The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index; 22 of the 52 sources its claims cite were checked for support.
Human-like systematic generalization through a meta-learning neural network
Open accessAbstract
The power of human language and thought arises from systematic compositionality—the algebraic ability to understand and produce novel combinations from known components. Fodor and Pylyshyn 1 famously argued that artificial neural networks lack this capacity and are therefore not viable models of the mind. Neural networks have advanced considerably in the years since, yet the systematicity challenge persists. Here we successfully address Fodor and Pylyshyn’s challenge by providing evidence that neural networks can achieve human-like systematicity when optimized for their compositional skills. To do so, we introduce the meta-learning for compositionality (MLC) approach for guiding training through a dynamic stream of compositional tasks. To compare humans and machines, we conducted human behavioural experiments using an instruction learning paradigm. After considering seven different models, we found that, in contrast to perfectly systematic but rigid probabilistic symbolic models, and perfectly flexible but unsystematic neural networks, only MLC achieves both the systematicity and flexibility needed for human-like generalization. MLC also advances the compositional skills of machine learning systems in several systematic generalization benchmarks. Our results show how a standard neural network architecture, optimized for its compositional skills, can mimic human systematic generalization in a head-to-head comparison.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces meta-learning for compositionality (MLC), an optimization procedure that trains a standard transformer over a dynamic stream of few-shot compositional tasks. In head-to-head human experiments and machine-learning benchmarks, MLC achieves human-like systematic generalization and reproduces human-like inductive biases and error patterns, though it still fails on some productivity and structural-generalization challenges.
Key findings
- On the few-shot instruction-learning task, the most systematic MLC run achieved 100% exact-match accuracy; when sampling, MLC produced systematic outputs at 82.4% versus 80.7% for humans, and handled longer output sequences at 77.8% versus 72.5% for humans.
- MLC reproduced human-like errors: one-to-one translation errors were 56.3% of MLC errors versus 24.4% for people, and iconic-concatenation errors involving function 3 were 13.8% versus 23.3% for people.
- In the open-ended task, MLC matched the modal human response in 65.0% of samples; across samples, 66.0% followed one-to-one (62.1% for people), 85.0% followed iconic concatenation (79.3% for people), and 99.0% chose a unique response for each unique command (93.1% for people).
- On benchmarks, MLC reached error rates of 0.22% on SCAN “add jump,” 0.04% on “around right,” 0.06% on “opposite right,” and 0.87% on COGS lexical generalization; however, it failed SCAN length and three COGS structural splits with 100% error.
Methods
- Human behavioural experiments were run on Amazon Mechanical Turk using a pseudolanguage instruction-learning paradigm: n = 25 participants completed a few-shot task with 14 study instructions and 10 query instructions, and n = 29 different participants completed an open-ended task with seven unknown instructions.
- MLC used a standard seq2seq transformer with 3 encoder/decoder layers, 8 attention heads per layer, embeddings of size 128, feedforward hidden size 512, and about 1.4 million parameters; it was meta-trained over episodes generated from randomly generated interpretation grammars, with frozen weights at test.
- MLC variants included algebraic-only, copy-only, behaviourally informed, and joint training; benchmark experiments used SCAN and COGS with surface-level word-type permutations and a modular procedure for long in-context sequences, scoring exact-match error rates.
Limitations
- The paper states that MLC does not automatically handle unpractised forms of generalization or concepts outside the meta-learning distribution, fails SCAN length and three COGS structural splits at 100% error, lacks a mechanism for emitting new symbols, and is untested on the full complexity of natural language and other modalities.
- The paper also states that optimizing over many related grammar-based tasks is not developmentally plausible, and that MLC does not solve every challenge raised by Fodor and Pylyshyn.
- Evident from the text: the human experiments use modest samples (n = 25 and n = 29) and a pseudolanguage with abstract coloured-circle outputs rather than natural language; some MLC variants are optimized directly on human behavioural data, so human-like error patterns may partly reflect those training targets.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
39 Own result · 26 Own method · 13 No basis given · 11 Own interpretation · 5 Supported · 4 Partly supported · 4 Sources not found · 4 No full text of the sources · 3 Not checked yet · 2 Supported so far · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Notably, modern neural networks still struggle on tests of systematicity11–18—tests that even a minimally algebraic mind should pass2.”
§ Human-like systematic generalization through a meta-learning neural network > Main
What the sources say
- Not matchedcites [11] Lake, B. M. & Baroni, M. Generalization without systematicity: on the compositional skills of sequence-to-sequence recurrent networks. In Proc. International Conference on Machine Learning (ICML) (eds. Dy, J. & Krause, A.) 2873–2882 (PMLR, 2018).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites [12] Ettinger, A., Elgohary, A., Phillips, C. & Resnik, P. Assessing composition in sentence vector representations. In Proc. 7th International Conference on Computational Linguistics, (COLING 2018) 1790–1801 (Association for Computational Linguistics, 2018).
- Supportedcites [13] Bahdanau, D. et al. CLOSURE: assessing systematic generalization of CLEVR models. In Proc. NAACL Workshop on Visually Grounded Interaction and Language (ViGIL) (2019). source
The Q1 (CLEVR) section says that the evaluated neural models often struggle on CLOSURE questions. Section 6 (Discussion) likewise reports that their performance deteriorates on unfamiliar combinations of known primitives, supporting the claim about tests of systematicity. (LLM, conf=0.93)
- Supportedcites [14] Keysers, D. et al. Measuring compositional generalization: a comprehensive method on realistic data. In Proc. International Conference on Learning Representations (ICLR) (2019). source
The Introduction says state-of-the-art machine-learning methods often fail to capture compositional structure and generalize compositionally. The additional SCAN results give a concrete test: both Transformer models scored 0% on the length split, while the LSTM scored around 14%. These passages support the attributed claim that modern neural networks struggle on tests of systematicity. (LLM, conf=0.92)
- Partly supportedcites [15] Yu, L. & Ettinger, A. Assessing phrasal representation and composition in transformers. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP) 4896–4907 (Association for Computational Linguistics, 2020). sourceExperimental — accuracy not yet measured
The Discussion (p. 8) reports little evidence of sophisticated phrase composition in the tested transformers once word overlap is controlled, supporting a narrower claim that these models struggle on controlled composition tests. It also reports strong performance on classic sense-selection items. The source therefore does not establish that modern neural networks generally struggle on tests of systematicity. (LLM, conf=0.84)
- Supportedcites [16] Kim, N. & Linzen, T. COGS: a compositional generalization challenge based on semantic interpretation. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP) 9087–9105 (2020). source
The Introduction (p. 1) identifies systematicity as a form of compositional generalization. The Results (p. 6) show that the tested Transformers and LSTMs performed well in distribution but poorly on the COGS generalization test, supporting the cited claim. (LLM, conf=0.93)
- Not checked — no full text of the sourcecites [17] Hupkes, D., Dankers, V., Mul, M. & Bruni, E. Compositionality decomposed: how do neural networks generalize? J. Artif. Int. Res. 67, 757–795 (2020). source
- Supportedcites [18] Press, O. et al. Measuring and narrowing the compositionality gap in language models. Preprint at https://arxiv.org/abs/2210.03350 (2022). source
The Introduction reports that language models fail to combine facts in a large fraction of compositional questions, even when they answer the constituent questions correctly. It also reports a compositionality gap of roughly 40% across the model sizes tested, supporting the attributed claim that modern neural networks still struggle on such tests. (LLM, conf=0.90)
- Not matchedcites [2] Marcus, G. F. The Algebraic Mind: Integrating Connectionism and Cognitive Science (MIT Press, 2003).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The tasks from the machine-learning literature that we experimented with, SCAN11,66 and COGS16, feature long sequences as (in-context) study examples.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
What the sources say
- Not matchedcites [11] Lake, B. M. & Baroni, M. Generalization without systematicity: on the compositional skills of sequence-to-sequence recurrent networks. In Proc. International Conference on Machine Learning (ICML) (eds. Dy, J. & Krause, A.) 2873–2882 (PMLR, 2018).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [66] Loula, J., Baroni, M. & Lake, B. M. Rearranging the familiar: testing compositional generalization in recurrent networks. Preprint at http://arxiv.org/abs/1807.07545 (2018). sourceExperimental — accuracy not yet measured
Section 2 describes SCAN as a sequence-to-sequence task that maps commands to action sequences, and notes a test split with longer action sequences than the training set. This supports the reference to sequences in SCAN, but the paper does not characterize them as long in-context study examples; it describes conventional training and test sets instead. COGS is attached to a separate citation in the citing sentence. (LLM, conf=0.82)
- Partly supportedcites [16] Kim, N. & Linzen, T. COGS: a compositional generalization challenge based on semantic interpretation. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP) 9087–9105 (2020). sourceExperimental — accuracy not yet measured
The COGS paper describes sentence-to-logical-form examples and reports long output sequences, including a training example of 153 tokens (Section 5.2.1, p. 7). It does not present COGS examples as in-context study examples; its models were trained on the dataset (Section 5.1, p. 6). (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MLC also advances the compositional skills of machine learning systems in several systematic generalization benchmarks.”
- Also stated in: § Machine learning benchmarks · plain
“Beyond predicting human behaviour, MLC can achieve error rates of less than 1% on machine learning benchmarks for systematic generalization.”
§ Human-like systematic generalization through a meta-learning neural network > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Machine learning benchmarks · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MLC outperforms all alternatives, including the same highly algebraic MLC model as described in the previous experiment (MLC (algebraic only)) and a probabilistic symbolic model that uses the three inductive biases to generate responses but, in contrast to MLC, is not capable of optimizing for other patterns in the human behaviour (Table 1; symbolic (oracle/biases)).”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all model samples, 66.0% followed one-to-one (62.1% for people), 85.0% followed iconic concatenation (79.3% for people) and the vast majority (99.0%) chose a unique response for each unique command (93.1% for people).”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
In the paper
“People’s responses also followed a third bias related to mutual exclusivity that encourages assigning unique meanings to unique words27.”
§ Human-like systematic generalization through a meta-learning neural network > Behavioural results
What the sources say
- Not checked — no full text of the sourcecites [27] Markman, E. M. & Wachtel, G. F. Children’s use of mutual exclusivity to constrain the meanings of words. Cogn. Psychol. 20, 121–157 (1988). source
plaincorrelationaln = 29- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human participants recruited via Amazon Mechanical Turk are representative of the broader human population for these compositional skills.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Although human few-shot learning behaviour can be well characterized by either MLC or a probabilistic symbolic model, a test of more open-ended behaviour emphasizes MLC’s relative strengths.”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Importantly, a single transformer can be optimized for both the few-shot learning and open-ended instruction tasks (MLC (joint)); in fact, this is the strongest overall model across experiments for predicting human behaviour (additional analysis is shown in Extended Data Fig. 5 and Supplementary Information 1).”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
In the paper
“Notably, participants also generalized correctly in 72.5% of cases to longer output sequences than seen during training (an example is shown as the last instruction in Fig. 2b (i)), which is a type of generalization that neural networks often struggle with11.”
§ Human-like systematic generalization through a meta-learning neural network > Behavioural results
What the sources say
- Not matchedcites [11] Lake, B. M. & Baroni, M. Generalization without systematicity: on the compositional skills of sequence-to-sequence recurrent networks. In Proc. International Conference on Machine Learning (ICML) (eds. Dy, J. & Krause, A.) 2873–2882 (PMLR, 2018).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainn = 25- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
- Rests on an assumption: Human participants recruited via Amazon Mechanical Turk are representative of the broader human population for these compositional skills.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The MLC transformer performs comparably to a probabilistic symbolic model that assumes that people infer the gold grammar but respond stochastically with lapses based on the human inductive biases (symbolic (oracle/biases)).”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all responses, 18 out of 29 participants followed one-to-one (62.1%), 23 out of 29 (79.3%) followed iconic concatenation and all but two followed mutual exclusivity in producing a unique response to each instruction (27 out of 29; 93.1%).”
§ Human-like systematic generalization through a meta-learning neural network > Behavioural results
plainn = 29- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human participants recruited via Amazon Mechanical Turk are representative of the broader human population for these compositional skills.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“An informal analysis further revealed that MLC captured more nuanced patterns of response that only partially use the inductive biases (Fig. 3b (right)).”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Reflecting the strong influence of the biases, the majority of participants (17 out of 29; 58.6%) responded with a pattern analogous to that in Fig. 3a,b (left), which is perfectly consistent with all three inductive biases.”
§ Human-like systematic generalization through a meta-learning neural network > Behavioural results
boostedn = 29- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human participants recruited via Amazon Mechanical Turk are representative of the broader human population for these compositional skills.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MLC is capable of optimizing models for highly systematic behaviour.”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plaindirectn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“An informal analysis of this run further shows that MLC is also capable of more subtle and bias-driven behaviours; when sampling from the distribution of model outputs (Fig. 2b), the transformer produced systematic outputs at an average rate (82.4%) close to human performance (80.7%), and appropriately handled longer output sequences at a rate (77.8%) near human levels (72.5%).”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, like people, the MLC transformer made errors reflecting one-to-one translations (56.3% of errors; 24.4% for people) and iconic concatenations (13.8% of errors involving function 3; 23.3% for people).”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“An informal analysis of this run further shows that MLC is also capable of more subtle and bias-driven behaviours; when sampling from the distribution of model outputs (Fig. 2b), the transformer produced systematic outputs at an average rate (82.4%) close to human performance (80.7%), and appropriately handled longer output sequences at a rate (77.8%) near human levels (72.5%).”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
In the paper
“In our experiments, we found that the most common human responses were algebraic and systematic in exactly the ways that Fodor and Pylyshyn1 discuss.”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
What the sources say
- Not checked — no full text of the sourcecites [1] Fodor, J. A. & Pylyshyn, Z. W. Connectionism and cognitive architecture: a critical analysis. Cognition 28, 3–71 (1988). source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
- Rests on an assumption: Human participants recruited via Amazon Mechanical Turk are representative of the broader human population for these compositional skills.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all model samples, 66.0% followed one-to-one (62.1% for people), 85.0% followed iconic concatenation (79.3% for people) and the vast majority (99.0%) chose a unique response for each unique command (93.1% for people).”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Preliminary experiments reported in Supplementary Information 3 suggest that systematicity is still a challenge, or at the very least an open question, even for recent large language models such as GPT-4.”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Other errors involved applying a function but mixing up its arguments, often in ways that suggest an ‘iconic concatenation’ bias for maintaining the order of the input words in the order of the output symbols (23.3% of all errors involving function 3 followed this pattern; marked with IC in Fig. 2b (i)).”
§ Human-like systematic generalization through a meta-learning neural network > Behavioural results
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
- Rests on an assumption: Human participants recruited via Amazon Mechanical Turk are representative of the broader human population for these compositional skills.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Without the benefit of meta-learning, basic seq2seq has error rates at least seven times as high across the benchmarks, despite using the same transformer architecture.”
§ Human-like systematic generalization through a meta-learning neural network > Machine learning benchmarks
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all model samples, 66.0% followed one-to-one (62.1% for people), 85.0% followed iconic concatenation (79.3% for people) and the vast majority (99.0%) chose a unique response for each unique command (93.1% for people).”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Importantly, a single transformer can be optimized for both the few-shot learning and open-ended instruction tasks (MLC (joint)); in fact, this is the strongest overall model across experiments for predicting human behaviour (additional analysis is shown in Extended Data Fig. 5 and Supplementary Information 1).”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, MLC is failing to generalize to nuances in inductive biases that it was not optimized for, as we explore further through an additional behavioural and modelling experiment in Supplementary Information 2.”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MLC can also predict which instructions are easier or harder for people on average (Pearson’s r = 0.788, P = 0.031, two-tailed permutation test, n = 10 items; item-level performance is shown in Extended Data Fig. 1).”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plaincorrelationaln = 10 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MLC does not automatically handle unpractised forms of generalization or concepts outside the meta-learning distribution, reducing the scope of entirely novel structures it can correctly process (compare the encouraging results on learning novel rules reported in Supplementary Information 1, with its failure on the SCAN and COGS productivity splits).”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Many errors involved ‘one-to-one’ translations that mapped each input word to exactly one output symbol, as if all words were primitives rather than functions (24.4% of all errors; marked with 1-to-1 in Fig. 2b (i)).”
§ Human-like systematic generalization through a meta-learning neural network > Behavioural results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
- Rests on an assumption: Human participants recruited via Amazon Mechanical Turk are representative of the broader human population for these compositional skills.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all responses, 18 out of 29 participants followed one-to-one (62.1%), 23 out of 29 (79.3%) followed iconic concatenation and all but two followed mutual exclusivity in producing a unique response to each instruction (27 out of 29; 93.1%).”
§ Human-like systematic generalization through a meta-learning neural network > Behavioural results
plainn = 29- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human participants recruited via Amazon Mechanical Turk are representative of the broader human population for these compositional skills.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, MLC did handle novel sentence structures in our few-shot instruction-learning task (77.8% correct on queries with both longer input and output sequences than seen during study; Fig. 2), suggesting that the right meta-training procedure can promote productivity—a challenge we leave to future work.”
§ Human-like systematic generalization through a meta-learning neural network > Machine learning benchmarks
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, like people, the MLC transformer made errors reflecting one-to-one translations (56.3% of errors; 24.4% for people) and iconic concatenations (13.8% of errors involving function 3; 23.3% for people).”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Beyond predicting human behaviour, MLC can achieve error rates of less than 1% on machine learning benchmarks for systematic generalization.”
§ Human-like systematic generalization through a meta-learning neural network > Machine learning benchmarks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all responses, 18 out of 29 participants followed one-to-one (62.1%), 23 out of 29 (79.3%) followed iconic concatenation and all but two followed mutual exclusivity in producing a unique response to each instruction (27 out of 29; 93.1%).”
§ Human-like systematic generalization through a meta-learning neural network > Behavioural results
plainn = 29- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human participants recruited via Amazon Mechanical Turk are representative of the broader human population for these compositional skills.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The participants were able to produce output sequences that exactly matched the algebraic standard in 80.7% of cases (indicated by an asterisk in Fig. 2b (i)).”
§ Human-like systematic generalization through a meta-learning neural network > Behavioural results
plainn = 25- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
- Rests on an assumption: Human participants recruited via Amazon Mechanical Turk are representative of the broader human population for these compositional skills.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On COGS, MLC achieves an error rate of 0.87% across the 18 types of lexical generalization.”
§ Human-like systematic generalization through a meta-learning neural network > Machine learning benchmarks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results show how a standard neural network architecture, optimized for its compositional skills, can mimic human systematic generalization in a head-to-head comparison.”
- Also stated in: § Discussion · boosted
“We showed how MLC enables a standard neural network optimized for its compositional skills to mimic or exceed human systematic generalization in a side-by-side comparison.”
§ Human-like systematic generalization through a meta-learning neural network > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Discussion · boosted
In the paper
“Here we successfully address Fodor and Pylyshyn’s challenge by providing evidence that neural networks can achieve human-like systematicity when optimized for their compositional skills.”
- Also stated in: § Main · plain
“In this Article, we provide evidence that neural networks can achieve human-like systematic generalization through MLC—an optimization procedure that we introduce for encouraging systematicity through a series of few-shot compositional tasks (Fig. 1).”
§ Human-like systematic generalization through a meta-learning neural network > Abstract
What the sources say
- Not checked — no full text of the sourcecites [1] Fodor, J. A. & Pylyshyn, Z. W. Connectionism and cognitive architecture: a critical analysis. Cognition 28, 3–71 (1988). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Main · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, meta-learning alone will not allow a standard network to generalize to episodes that are in turn out-of-distribution with respect to the ones presented during meta-learning.”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Such tasks require handling ‘productivity’ (page 33 of ref. 1), in ways that are largely distinct from systematicity.”
§ Human-like systematic generalization through a meta-learning neural network > Machine learning benchmarks
What the sources say
- Not checked — no full text of the sourcecites [1] Fodor, J. A. & Pylyshyn, Z. W. Connectionism and cognitive architecture: a critical analysis. Cognition 28, 3–71 (1988). source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we gave it an additional advantage not offered to any other model class: we tracked each step of the optimizer and selected the best parameter values on the basis of the test loss.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Alternative neural and symbolic models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When deviating from this algebraic standard, the responses were still highly non-random and suggestive of strong inductive biases.”
§ Human-like systematic generalization through a meta-learning neural network > Behavioural results
hedgedcorrelationaln = 25- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
- Rests on an assumption: Human participants recruited via Amazon Mechanical Turk are representative of the broader human population for these compositional skills.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the language of machine learning, we conclude that the meta-learning strategy succeeds when generalization makes a new episode in-distribution with respect to the training episodes, even when the specific test items are out-of-distribution with respect to the study examples in the episode.”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although human few-shot learning behaviour can be well characterized by either MLC or a probabilistic symbolic model, a test of more open-ended behaviour emphasizes MLC’s relative strengths.”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The bottlenecks are the encoder self-attention layers, which are O(S²).”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Specifically, it is intractable to process a single source sequence that consists of the concatenated query input sequence and multiple study example sequences, which could have a worst-case source sequence of length S ≈ 1,500 on COGS and potentially longer in other applications”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Given that only the architecture’s use has changed (not the architecture itself), the model has approximately the same number of learnable parameters as in MLC (except for the smaller input vocabulary).”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Alternative neural and symbolic models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 43
- It may state two separate claims as one. 38
- The quote only partly states this claim. 1
Every source the paper cites for this claim was checked and supports it.
In the paper
“For COGS, both models were optimized for 300 epochs (also without early stopping), which is slightly more training than the extended amount prescribed in ref. 67 for their strong seq2seq baseline.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
What the sources say
- Supportedcites [67] Csordás, R., Irie, K. & Schmidhuber, J. The devil is in the detail: simple tricks improve systematic generalization of transformers. In Proc. EMNLP 2021—2021 Conference on Empirical Methods in Natural Language Processing 619–634 (Association for Computational Linguistics, 2021). source
Section 3.2 reports training COGS models without early stopping for a fixed 50,000 steps. The dataset statistics in Appendix G give 24,155 training examples, and Table 4 gives a batch size of 128; together these amount to about 265 epochs, slightly fewer than the citing authors’ 300. The source supports the comparison, not the citing authors’ statement about how they trained their own two models. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, children are not born with an adult-like ability to compose functions; in fact, there seem to be important changes between infancy58 and pre-school59 that could be tied to learning.”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
What the sources say
- Partly supportedcites [58] Piantadosi, S. T., Palmeri, H. & Aslin, R. Limits on composition of conceptual operations in 9-month-olds. Infancy 23, 310–324 (2018). sourceExperimental — accuracy not yet measured
The Introduction and Conclusion report that 9-month-olds did not predictively compose simple feature-change functions, in contrast to preschoolers. The General discussion says the developmental change remains unexplained and suggests developing abilities as possible contributors. The paper does not test newborns or establish that learning causes the change. (LLM, conf=0.82)
- Partly supportedcites [59] Piantadosi, S. & Aslin, R. Compositional reasoning in early childhood. PLoS ONE 11, e0147734 (2016). sourceExperimental — accuracy not yet measured
The Abstract reports that children aged 3.5–4.5 can compose novel functions, supporting the preschool part of the sentence. The Discussion allows that prior learning may have contributed, but the paper does not establish a change from infancy to preschool or show that children lack this ability at birth. (LLM, conf=0.84)
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“On test episodes, the model weights are frozen and no task-specific parameters are provided32.”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
What the sources say
- Supportedcites [32] Chan, S. C. Y. et al. Data distributional properties drive emergent in-context learning in transformers. In Advances in Neural Information Processing Systems 35 (eds Koyejo, S. et al.) 18878–18891 (Curran Associates, 2022). source
Section 2.3 (p. 4) says trained models are evaluated on sequences whose labels are reassigned for each sequence, so the model must use the current context to predict the query label. Appendix C.4 (p. 16) states that the same model is used across training and evaluation. The described test setup uses the trained model and contextual examples, not task-specific parameters or weight updates. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“These response patterns can be compared to biases in language acquisition more generally; indeed, forms of one-to-one27 and iconic concatenation28,29 are widely attested in natural language.”
§ Human-like systematic generalization through a meta-learning neural network > Behavioural results
What the sources say
- Not checked — no full text of the sourcecites [27] Markman, E. M. & Wachtel, G. F. Children’s use of mutual exclusivity to constrain the meanings of words. Cogn. Psychol. 20, 121–157 (1988). source
- Not checked — no full text of the sourcecites [28] Haiman, J. The iconicity of grammar: isomorphism and motivation. Language 56, 515–540 (1980). source
- Supportedcites [29] de Ruiter, L., Theakston, A., Brandt, S. & Lieven, E. Iconicity affects children’s comprehension of complex sentences: the role of semantics, clause order, input and individual differences. Cognition 171, 202–224 (2018). source
This source is cited for iconic concatenation, not the separately cited one-to-one bias. The Introduction describes clause order mirroring event order as iconic in English and other languages, and Section 1.2.3 reports corpus evidence that iconic orders are frequent for some sentence types. That supports the broad claim that forms of this bias occur in natural language. (LLM, conf=0.87)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“MLC also allows neural networks to tackle other existing challenges, including making systematic use of isolated primitives11,16 and using mutual exclusivity to infer meanings44.”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
What the sources say
- Not matchedcites [11] Lake, B. M. & Baroni, M. Generalization without systematicity: on the compositional skills of sequence-to-sequence recurrent networks. In Proc. International Conference on Machine Learning (ICML) (eds. Dy, J. & Krause, A.) 2873–2882 (PMLR, 2018).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites [16] Kim, N. & Linzen, T. COGS: a compositional generalization challenge based on semantic interpretation. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP) 9087–9105 (2020). source
Section 3.1 (p. 4) describes a COGS challenge in which primitives appear only as isolated words in training but must be interpreted in sentences at generalization time. This citation is attached to the isolated-primitives challenge, not to the citing authors’ claim about MLC or to the separately cited mutual-exclusivity challenge. (LLM, conf=0.91)
- Not matchedcites [44] Gandhi, K. & Lake, B. M. Mutual exclusivity as a challenge for deep neural networks. In Proc. Advances in Neural Information Processing Systems (NeurIPS) 33 (eds Larochelle, H. et al.) 14182–14192 (Curran Associates, 2020).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Systematicity continues to challenge models11–18 and motivates new frameworks34–41.”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
What the sources say
- Not matchedcites [11] Lake, B. M. & Baroni, M. Generalization without systematicity: on the compositional skills of sequence-to-sequence recurrent networks. In Proc. International Conference on Machine Learning (ICML) (eds. Dy, J. & Krause, A.) 2873–2882 (PMLR, 2018).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites [12] Ettinger, A., Elgohary, A., Phillips, C. & Resnik, P. Assessing composition in sentence vector representations. In Proc. 7th International Conference on Computational Linguistics, (COLING 2018) 1790–1801 (Association for Computational Linguistics, 2018).
- Supportedcites [13] Bahdanau, D. et al. CLOSURE: assessing systematic generalization of CLEVR models. In Proc. NAACL Workshop on Visually Grounded Interaction and Language (ViGIL) (2019). source
The Abstract reports that state-of-the-art CLEVR models often lack systematicity on CLOSURE and describes a new Vector-NMN architecture developed to improve systematic generalization. The Discussion likewise reports persistent generalization gaps and proposes combining the strengths of different model types in future work. (LLM, conf=0.90)
- Supportedcites [14] Keysers, D. et al. Measuring compositional generalization: a comprehensive method on realistic data. In Proc. International Conference on Learning Representations (ICLR) (2019). source
The Abstract reports that three machine-learning architectures fail to generalize compositionally. It also says the lack of realistic benchmarks makes improvements difficult to evaluate and introduces a new benchmark-construction method in response, supporting the attributed motivation for a new framework. (LLM, conf=0.90)
- Supportedcites [15] Yu, L. & Ettinger, A. Assessing phrasal representation and composition in transformers. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP) 4896–4907 (Association for Computational Linguistics, 2020). source
The Discussion (p. 8) says that, after controlling for word overlap, the tested transformers show little evidence of sophisticated phrase composition and predicts related weaknesses on future tests. This supports the claim that systematic meaning composition remains a challenge for these models. The separate statement about new frameworks has its own citation range and is not attributed to this source. (LLM, conf=0.88)
- Supportedcites [16] Kim, N. & Linzen, T. COGS: a compositional generalization challenge based on semantic interpretation. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP) 9087–9105 (2020). source
The Conclusion (p. 9) reports that Transformers and LSTMs performed poorly on COGS generalization tests despite near-perfect in-domain performance. It also says the challenge can motivate development of new computational models, supporting the sentence’s broader point. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [17] Hupkes, D., Dankers, V., Mul, M. & Bruni, E. Compositionality decomposed: how do neural networks generalize? J. Artif. Int. Res. 67, 757–795 (2020). source
- Supportedcites [18] Press, O. et al. Measuring and narrowing the compositionality gap in language models. Preprint at https://arxiv.org/abs/2210.03350 (2022). source
The Abstract reports that larger GPT-3 models showed no corresponding improvement in the compositional reasoning tested, and introduces self-ask to address that gap. The Conclusion likewise describes self-ask as an improvement over chain-of-thought prompting. This supports the sentence's broad point about a continuing model challenge motivating a new approach, although the paper studies compositional reasoning in multi-hop questions rather than systematicity in every setting. (LLM, conf=0.87)
- Supportedcites [34] Collins, A. G. E. & Frank, M. J. Cognitive control over learning: creating, clustering, and generalizing task-set structure. Psychol. Rev. 120, 190–229 (2013). source
The citation is attached to the sentence’s “new frameworks” clause. The Introduction motivates computational models for creating and reusing task-set structure to generalize learned behavior, and the Discussion describes the authors’ new C-TS model for that purpose. The paper is therefore an example of the framework the citing sentence refers to. (LLM, conf=0.85)
- Not matchedcites [35] Chen, X., Liang, C., Yu, A. W., Song, D. & Zhou, D. Compositional generalization via neural-symbolic stack machines. In Proc. Advances in Neural Information Processing Systems 33 (eds Larochelle, H. et al.) 1690–1701 (Curran Associates, 2020).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“As shown in Fig. 4 and detailed in the ‘Architecture and optimizer’ section of the Methods, MLC uses the standard transformer architecture26 for memory-based meta-learning.”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
What the sources say
- Supportedcites [26] Vaswani, A. et al. Attention is all you need. In Proc. Advances in Neural Information Processing Systems 30 (eds Guyon, I. et al.) 5998–6008 (Curran Associates, 2017). source
The citation identifies the transformer architecture used by MLC, rather than attributing MLC's design to Vaswani et al. Section 3 (p. 2) describes the Transformer's encoder–decoder architecture and its stacked self-attention layers. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Second, children become better word learners over the course of development60, similar to a meta-learner improving with training.”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
What the sources say
- Supportedcites [60] Bergelson, E. The comprehension boost in early word learning: older infants are better learners. Child Dev. Perspect. 14, 142–149 (2020). source
The Abstract reports a robust improvement in infants’ word comprehension after their first birthday and proposes that older infants learn to make better use of relatively stable language input. The Language Input section likewise says children take greater advantage of that input. The meta-learner comparison is the citing authors’ analogy, not a claim attributed to this source. (LLM, conf=0.90)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“COGS is a multi-faceted benchmark that evaluates many forms of systematic generalization.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
What the sources say
- Supportedcites [16] Kim, N. & Linzen, T. COGS: a compositional generalization challenge based on semantic interpretation. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP) 9087–9105 (2020). source
The Overview of COGS (pp. 2–3) describes a challenge set targeting five categories of generalization, with a full list of cases in Table 1. The Introduction (p. 1) names examples including novel combinations of words and grammatical roles, deeper phrase nesting, and verb argument alternation. This supports describing COGS as a benchmark evaluating multiple forms of systematic generalization. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After considering seven different models, we found that, in contrast to perfectly systematic but rigid probabilistic symbolic models, and perfectly flexible but unsystematic neural networks, only MLC achieves both the systematicity and flexibility needed for human-like generalization.”
§ Human-like systematic generalization through a meta-learning neural network > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the few-shot instruction task, this improves the test loss marginally, but not accuracy.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Alternative neural and symbolic models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For successful optimization, it is also important to pass each study example (input sequence only) as an additional query when training on a particular episode.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Implementation of MLC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“The current architecture also lacks a mechanism for emitting new symbols2, although new symbols introduced through the study examples could be emitted through an additional pointer mechanism55.”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
What the sources say
- Not matchedcites [2] Marcus, G. F. The Algebraic Mind: Integrating Connectionism and Cognitive Science (MIT Press, 2003).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [55] Vinyals, O., Fortunato, M. & Jaitly, N. Pointer networks. In Proc. Advances in Neural Information Processing Systems (eds Cortes, C. et al.) (Curran Associates, 2015).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In our experiments, only MLC closely reproduced human behaviour with respect to both systematicity and biases, with the MLC (joint) model best navigating the trade-off between these two blueprints of human linguistic behaviour.”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MLC also outperforms a basic seq2seq transformer fit to the patterns in Fig. 2 without meta-learning and an MLC model optimized for copying rather than systematic generalization (MLC (copy only); during training, the query examples always match one of the study examples).”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The MLC transformer responded exactly like the modal human participant in 65.0% of samples (Fig. 3b (left)), perfectly instantiating the three key inductive biases.”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite the unconstrained nature of the test, people’s responses were highly structured and confirm the previous two inductive biases.”
§ Human-like systematic generalization through a meta-learning neural network > Behavioural results
plaincorrelationaln = 29- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Human participants recruited via Amazon Mechanical Turk are representative of the broader human population for these compositional skills.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MLC shows much stronger systematicity than neural networks trained in standard ways, and shows more nuanced behaviour than pristine symbolic models.”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However surface-level permutations were not enough for MLC to solve the structural generalization tasks in the benchmarks. MLC fails to handle longer output sequences (SCAN length split) as well as novel and more complex sentence structures (three types in COGS), with error rates at 100%.”
§ Human-like systematic generalization through a meta-learning neural network > Machine learning benchmarks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Chance performance is 2.8% for two-length output sequences if the length is known, and exponentially less for longer sequences.”
§ Human-like systematic generalization through a meta-learning neural network > Behavioural results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nevertheless, all 10 runs failed to generalize systematically on the few-shot instruction task (0% exact-match accuracy).”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Alternative neural and symbolic models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, we evaluated lower-capacity transformers but found that they did not perform better.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Alternative neural and symbolic models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On SCAN, MLC solves three systematic generalization splits with an error rate of 0.22% or lower (99.78% accuracy or above), including the already mentioned ‘add jump’ split and ‘around right’ and ‘opposite right’, which examine novel combinations of known words.”
§ Human-like systematic generalization through a meta-learning neural network > Machine learning benchmarks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“However, people also relied on inductive biases that sometimes support the algebraic solution and sometimes deviate from it; indeed, people are not purely algebraic machines3,6,7.”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
What the sources say
- Not checked — no full text of the sourcecites [3] Johnson, K. On the systematicity of language and thought. J. Philos. 101, 111–139 (2004). source
- Not checked — no full text of the sourcecites [6] O’Reilly, R. C. et al. in The Architecture of Cognition: Rethinking Fodor and Pylyshyn’s Systematicity Challenge (eds Calvo, P. & Symons, J.) 191–226 (MIT Press, 2014). source
- Not matchedcites [7] Nam, A. J. & McClelland, J. L. What underlies rapid learning and systematic generalization in humans? Preprint at http://arxiv.org/abs/2107.06994 (2021).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
- Rests on an assumption: Human participants recruited via Amazon Mechanical Turk are representative of the broader human population for these compositional skills.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Indeed, MLC was similarly optimized to (implicitly) infer systematic rules and respond with the same biased-based patterns, and it is therefore natural that the two models would perform similarly.”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“MLC was evaluated on this task in several ways; in each case, MLC responded to this novel task through learned memory-based strategies, as its weights were frozen and not updated further.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Implementation of MLC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Model predictions were also evaluated through fivefold cross-validation33: MLC and other models were optimized on responses for either 23 or 24 participants (depending on the cross-validation split) and then predicted responses for held-out participants.”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
What the sources say
- Not matchedcites [33] Myung, J. I. & Pitt, M. A. in Stevens’ Handbook of Experimental Psychology and Cognitive Neuroscience (ed. Wixted, J. T.) 85–118 (John Wiley & Sons, 2018).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainn = 24- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The study items remained on the screen for reference, so that performance would reflect generalization in the absence of memory limitations.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Behavioural methods: few-shot learning task
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, MLC did handle novel sentence structures in our few-shot instruction-learning task (77.8% correct on queries with both longer input and output sequences than seen during study; Fig. 2), suggesting that the right meta-training procedure can promote productivity—a challenge we leave to future work.”
§ Human-like systematic generalization through a meta-learning neural network > Machine learning benchmarks
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Beyond natural language, people require a years-long process of education to master other forms of systematic generalization and symbolic reasoning6,7, including mathematics, logic and computer programming.”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
What the sources say
- Not checked — no full text of the sourcecites [6] O’Reilly, R. C. et al. in The Architecture of Cognition: Rethinking Fodor and Pylyshyn’s Systematicity Challenge (eds Calvo, P. & Symons, J.) 191–226 (MIT Press, 2014). source
- Not matchedcites [7] Nam, A. J. & McClelland, J. L. What underlies rapid learning and systematic generalization in humans? Preprint at http://arxiv.org/abs/2107.06994 (2021).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The study and test items always differed from one another by more than one primitive substitution (except in the function 1 stage, where a single primitive was presented as a novel argument to function 1).”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Behavioural methods: few-shot learning task
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that here the examples used for optimization were generated by the benchmark designers through algebraic rules, and there is therefore no direct imitation of human behavioural data.”
§ Human-like systematic generalization through a meta-learning neural network > Machine learning benchmarks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The study and test set only used four output symbols, but the pool provided six possibilities (that is, there were two extra colours that were not associated to words), to discourage reasoning by exclusion.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Behavioural methods: few-shot learning task
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The COGS output expressions were converted to uppercase to remove any incidental overlap between input and output token indices (which MLC, but not basic seq2seq, could exploit).”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This more scalable MLC variant, the original MLC architecture (see the ‘Architecture and optimizer’ section) and basic seq2seq all have approximately the same number of learnable parameters (except for the fact that basic seq2seq has a smaller input vocabulary).”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“MLC shows how meta-learning can be used like hierarchical Bayesian models for reverse-engineering inductive biases (see ref. 47 for a formal connection), although with the aid of neural networks for greater expressive power.”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
What the sources say
- Not matchedcites ref. 47
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The decoder cross-attention layers are less expensive, O(ST), because the target sequence length T, which does not include any study examples, is typically much shorter (T ≪ S).”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Our experimental paradigm asks participants to process instructions in a pseudolanguage in order to generate abstract outputs (meanings), differing from artificial grammar learning22, statistical learning23 and program learning24 in that explicit or implicit judgments of grammaticality are not needed.”
§ Human-like systematic generalization through a meta-learning neural network > Behavioural results
What the sources say
- Not checked — no full text of the sourcecites [22] Reber, A. Implicit learning of artificial grammars. Verb. Learn. Verb. Behav. 5, 855–863 (1967). source
- Not checked — no full text of the sourcecites [23] Aslin, R. N., Saffran, J. R. & Newport, E. L. Computation of conditional probability statistics by 8-month-old infants. Psychol. Sci. 9, 321–324 (1998). source
- Not matchedcites [24] Stuhlmuller, A., Tenenbaum, J. B. & Goodman, N. D. Learning structured generative concepts. In Proc. Thirty-Second Annual Conference of the Cognitive Science Society, 2296–2301 (2010).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“using the test corpus for study examples would inadvertently leak test information”
- Also stated in: § Machine learning benchmarks · hedged
“picking study examples from the generalization corpus would inadvertently leak test information”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Also stated in: § Machine learning benchmarks · hedged
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, query responses were also excluded if the corresponding study phases were not completed correctly (for all items) within three attempts (13% of remaining data).”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Behavioural methods: few-shot learning task
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The validation episodes were defined by new grammars that differ from the training grammars. Grammars were only considered new if they did not match any of the meta-training grammars, even under permutations of how the rules are ordered.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Implementation of MLC
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“First, before the experiment, the participants practiced using the response interface and had to pass an instructions quiz; they cycled through the quiz until they passed it.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Behavioural methods: few-shot learning task
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We used several strategies to ensure that our participants were paying attention.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Behavioural methods: few-shot learning task
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For statistical analyses of the data from this experiment and elsewhere, we tested for data normalcy and applied alternative nonparametric or permutation tests when the assumptions were not met.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Behavioural methods: few-shot learning task
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“All of the studies were approved by the NYU IRB, protocol FY2018-1728, and obtained informed consent.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Behavioural methods: few-shot learning task
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Importantly, this approach seeks to model adult compositional skills but not the process by which adults acquire those skills, which is an issue that is considered further in the general discussion.”
§ Human-like systematic generalization through a meta-learning neural network > Modelling results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The vocabulary in COGS is much larger than in SCAN; thus, the study examples cannot be sampled arbitrarily with any reasonable hope that they would inform the query of interest.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our implementation of MLC uses only common neural networks without added symbolic machinery, and without hand-designed internal representations or inductive biases.”
§ Human-like systematic generalization through a meta-learning neural network > Main
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Without access to study examples, the model is poorly equipped for learning words with changing meanings; it has no in-context memory and, therefore, all of its knowledge must be stored in the learned weights.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Alternative neural and symbolic models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, MLC derives its abilities through meta-learning, where both systematic generalization and the human biases are not inherent properties of the neural network architecture but, instead, are induced from data.”
§ Human-like systematic generalization through a meta-learning neural network > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, catch trials were included during the test phases, probing the study items rather than new items, with the answers clearly presented on the screen above. There was one catch trial per stage (except the last stage had two); participants were excluded if they missed two or more catch trials (n = 5).”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Behavioural methods: few-shot learning task
plainn = 5- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“One participant was excluded because they reported using an external aid in a post-test survey.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Behavioural methods: open-ended task
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Occasionally skipping the permutations in this way helps to break symmetries that can slow optimization; that is, the association between the input and output primitives is no longer perfectly balanced.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The gold interpretation grammar that produced the few-shot instruction-learning task with humans and machines (Extended Data Fig. 2) was also reserved for testing in this way, with an additional structural requirement that grammars for producing the training and validation episodes should also not match the gold grammar through any permutation of the input and output symbol assignments.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Implementation of MLC
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“for the ‘add jump’ split, one study example is fixed to be ‘jump → JUMP’, ensuring that MLC has access to the basic meaning before attempting compositional uses of ‘jump’.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Performance was averaged over 200 passes through the dataset, each episode with different random query orderings as well as word and colour assignments.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Implementation of MLC
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For comparison with the gold grammar or with human behaviour via log-likelihood, performance was averaged over 100 random word/colour assignments.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Implementation of MLC
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The assignment of words to colours and functions was randomized for each participant (drawn from nine possible words and six colours), and the first three stages were presented in random order.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Behavioural methods: few-shot learning task
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To make sense of an episode, MLC must become adept at inferring, from just a few study examples, how words map to meanings.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To master the lexical generalization splits, the meta-training procedure targets several lexical classes that participate in particularly challenging compositional generalizations.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- The authors state this as an assumption: Broad lexical classes (common nouns, proper nouns, dative verbs, infinitive verbs) are assumed.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that several other input words (the mostly ‘functional’ words ‘turn’, ‘twice’, ‘thrice’, ‘around’, ‘opposite’, ‘and’, ‘after’) have stable meanings that can be stored in the model weights.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Importantly, although the broad classes are assumed and could plausibly arise through simple distributional learning68,69, the correspondence between input and output word types is unknown and not used.”
- Also stated in: § Machine learning benchmarks · plain
“As before, the correspondence between input and output tokens is unknown, both at the levels of a sentence and the whole dataset.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Also stated in: § Machine learning benchmarks · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“MLC must also become adept at composition: it must systematically compose the inferred word meanings to correctly answer the queries.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Other verbs, punctuation and logical symbols have stable meanings that can be stored in the model weights.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Machine learning benchmarks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that all of the models compared in Table 1 have the same opportunity to fit a lapse parameter.”
§ Human-like systematic generalization through a meta-learning neural network > Methods > Alternative neural and symbolic models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
“The study instructions were consistent with an underlying interpretation grammar, which derives outputs from inputs through a set of compositional rewrite rules (see the ‘Interpretation grammars’ section of the Methods).”
The paper relies on this without support.
Human participants recruited via Amazon Mechanical Turk are representative of the broader human population for these compositional skills.
“All experiments were run on Amazon Mechanical Turk, and detailed procedures are described in the ‘Behavioural methods: few-shot learning task’ and ‘Behavioural methods: open-ended task’ sections of the Methods.”
The authors state this as an assumption.
Broad lexical classes (common nouns, proper nouns, dative verbs, infinitive verbs) are assumed.
“Importantly, although the broad classes are assumed and could plausibly arise through simple distributional learning68,69, the correspondence between input and output word types is unknown and not used.”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Connectionism and cognitive architecture: A critical analysis
Fodor, J. A., & Pylyshyn, Z. W. (1988) · Cognition · doi:10.1016/0010-0277(88)90031-5 · Cited by 4,022
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511
Implicit learning of artificial grammars
Reber, A. S. (1967) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(67)80149-x · Cited by 2,026
Children's use of mutual exclusivity to constrain the meanings of words
Markman, E. M., & Wachtel, G. F. (1988) · Cognitive Psychology · doi:10.1016/0010-0285(88)90017-5 · Cited by 1,337
Computation of conditional probability statistics by 8-month-old infants
Aslin, R. N., Saffran, J. R., & Newport, E. L. (1998) · Psychological Science · doi:10.1111/1467-9280.00063 · Cited by 1,189
Smolensky, P. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90007-m · Cited by 959
Research works printed but not linked: 21
- Marcus, G. F. The Algebraic Mind: Integrating Connectionism and Cognitive Science (MIT Press, 2003).
- Hill, F. et al. Environmental drivers of systematicity and generalisation in a situated agent. In Proc. International Conference on Learning Representations (ICLR) (2020).
- Lake, B. M. & Baroni, M. Generalization without systematicity: on the compositional skills of sequence-to-sequence recurrent networks. In Proc. International Conference on Machine Learning (ICML) (eds. Dy, J. & Krause, A.) 2873–2882 (PMLR, 2018).
- Ettinger, A., Elgohary, A., Phillips, C. & Resnik, P. Assessing composition in sentence vector representations. In Proc. 7th International Conference on Computational Linguistics, (COLING 2018) 1790–1801 (Association for Computational Linguistics, 2018).
- Stuhlmuller, A., Tenenbaum, J. B. & Goodman, N. D. Learning structured generative concepts. In Proc. Thirty-Second Annual Conference of the Cognitive Science Society, 2296–2301 (2010).
- Sutskever, I., Vinyals, O. & Le, Q. V. Sequence to sequence learning with neural networks. In Proc. Advances in Neural Information Processing Systems (eds Ghahramani, Z. et al.) (Curran Associates, 2014).
- Lake, B. M. Compositional generalization through meta sequence-to-sequence learning. In Proc. Advances in Neural Information Processing Systems (NeurIPS) 32 (eds Wallach, H. et al.) 9791–9801 (Curran Associates, 2019).
- Myung, J. I. & Pitt, M. A. in Stevens’ Handbook of Experimental Psychology and Cognitive Neuroscience (ed. Wixted, J. T.) 85–118 (John Wiley & Sons, 2018).
- Chen, X., Liang, C., Yu, A. W., Song, D. & Zhou, D. Compositional generalization via neural-symbolic stack machines. In Proc. Advances in Neural Information Processing Systems 33 (eds Larochelle, H. et al.) 1690–1701 (Curran Associates, 2020).
- Russin, J., Jo, J., O’Reilly, R. C. & Bengio, Y. Systematicity in a recurrent neural network by factorizing syntax and semantics. In Proc. 42nd Annual Meeting of the Cognitive Science Society (eds Denison, S. et al.) (Cognitive Science Society. 2020).
- Nye, M. I., Solar-Lezama, A., Tenenbaum, J. B. & Lake, B. M. Learning compositional rules via neural program synthesis. In Proc. Advances in Neural Information Processing Systems (NeurIPS) 33 (eds Larochelle, H. et al.) (Curran Associates, 2020).
- Singh, G., Deng, F. & Ahn, S. Illiterate DALL-E learns to compose. In Proc. ICLR https://openreview.net/group?id=ICLR.cc/2022/Conference (2022).
- Gandhi, K. & Lake, B. M. Mutual exclusivity as a challenge for deep neural networks. In Proc. Advances in Neural Information Processing Systems (NeurIPS) 33 (eds Larochelle, H. et al.) 14182–14192 (Curran Associates, 2020).
- Grant, E., Finn, C., Levine, S., Darrell, T. & Griffiths, T. Recasting gradient-based meta-learning as hierarchical bayes. In Proc. International Conference on Learning Representations (ICLR) (2019).
- Grant, E., Peterson, J. C. & Griffiths, T. Learning deep taxonomic priors for concept learning from few positive examples. In Proc. Annual Meeting of the Cognitive Science Society (eds Goel, A. K. et al.) 1865–1870 (Cognitive Science Society, 2019).
- Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D. & Lillicrap, T. Meta-learning with memory-augmented neural networks. In Proc. International Conference on Machine Learning (ICML) 1842–1850 (PMLR, 2016).
- McCoy, R. T., Grant, E., Smolensky, P., Griffiths, T. L. & Linzen, T. Universal linguistic inductive biases via meta-learning. In Proc. 42nd Annual Conference of the Cognitive Science Society (eds Denison, S. et al.) (Cognitive Science Society, 2020).
- Vinyals, O., Fortunato, M. & Jaitly, N. Pointer networks. In Proc. Advances in Neural Information Processing Systems (eds Cortes, C. et al.) (Curran Associates, 2015).
- Radford, A., Narasimhan, K. R., Salimans, T. & Sutskever, I. Improving language understanding by generative pre-training. Preprint at https://openai.com/research/language-unsupervised (2018).
- Mitchell, E., Finn, C. & Manning, C. Challenges of acquiring compositional inductive biases via meta-learning. In Proc. AAAI Workshop on Meta-Learning and MetaDL Challenge 138–148 (2021).
- Lake, B. M., Linzen, T. & Baroni, M. Human few-shot learning of compositional instructions. In Proc. 41st Annual Conference of the Cognitive Science Society (eds Goel, A. K. et al.) 611–617 (Cognitive Science Society, 2019).
Cited by
Sorted by citations
Artificial intelligence and illusions of understanding in scientific research
Messeri, L., & Crockett, M. J. (2024) · Nature · doi:10.1038/s41586-024-07146-0 · Cited by 707
Towards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2024) · NEJM AI · doi:10.1056/aioa2300138 · Cited by 483
Language is primarily a tool for communication rather than thought
Fedorenko, E., Piantadosi, S. T., & Gibson, E. A. F. (2024) · Nature · doi:10.1038/s41586-024-07522-w · Cited by 190
Theory Is All You Need: AI, Human Cognition, and Causal Reasoning
(2024) · doi:10.1287/stsc.2024.0189 · Cited by 151
Nakaura, T., Ito, R., Ueda, D., Nozaki, T., Fushimi, Y., Matsui, Y., Yanagawa, M., Yamada, A., Tsuboyama, T., Fujima, N., et al. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-024-01552-0 · Cited by 131
Metadata
- Authors
- Lake, B. M., & Baroni, M.
- Affiliations
- New York University (US)
- Institució Catalana de Recerca i Estudis Avançats (ES)
- Corresponding author
- Lake, Brenden M.
- Venue
- Nature
- Published
- 2023-10-25
- Volume
- 623
- Issue
- 7985
- Pages
- 115-121
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- PMC
- PMC10620072
- OpenAlex
- W4387949708
- Semantic Scholar
- dd4dfee7ad7a2ed179f9b2e80b83685b37661dbf
- Cited by
- 230
- References
- 48
- Citations by year
- 2026: 66 · 2025: 87 · 2024: 71 · 2023: 6
- FWCI
- 16.38
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Repositori digital de la UPF (Universitat Pompeu Fabra) (submittedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- systematic generalization, compositional generalization, meta-learning, human-machine comparison
- MeSH terms
- Machine Learning, Humans, Language, Verbal Behavior, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
