Human-like systematic generalization through a meta-learning neural network

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

For COGS, both models were optimized for 300 epochs without early stopping, which is slightly more training than the extended amount prescribed in ref. 67 for their strong seq2seq baseline.

“For COGS, both models were optimized for 300 epochs (also without early stopping), which is slightly more training than the extended amount prescribed in ref. 67 for their strong seq2seq baseline.”

• cites [67] Csordás, R., Irie, K. & Schmidhuber, J. The devil is in the detail: simple tricks improve systematic generalization of transformers. In Proc. EMNLP 2021—2021 Conference on Empirical Methods in Natural Language Processing 619–634 (Association for Computational Linguistics, 2021). sourceSupported

It may state two separate claims as one.

Children are not born with an adult-like ability to compose functions; there seem to be important changes between infancy and pre-school that could be tied to learning.

“First, children are not born with an adult-like ability to compose functions; in fact, there seem to be important changes between infancy58 and pre-school59 that could be tied to learning.”

• cites [58] Piantadosi, S. T., Palmeri, H. & Aslin, R. Limits on composition of conceptual operations in 9-month-olds. Infancy 23, 310–324 (2018). sourcePartly supported

  The Introduction and Conclusion report that 9-month-olds did not predictively compose simple feature-change functions, in contrast to preschoolers. The General discussion says the developmental change remains unexplained and suggests developing abilities as possible contributors. The paper does not test newborns or establish that learning causes the change. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [59] Piantadosi, S. & Aslin, R. Compositional reasoning in early childhood. PLoS ONE 11, e0147734 (2016). sourcePartly supported

  The Abstract reports that children aged 3.5–4.5 can compose novel functions, supporting the preschool part of the sentence. The Discussion allows that prior learning may have contributed, but the paper does not establish a change from infancy to preschool or show that children lack this ability at birth. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Probably not a substantial claim. It may state two separate claims as one.

On test episodes, the model weights are frozen and no task-specific parameters are provided.

“On test episodes, the model weights are frozen and no task-specific parameters are provided32.”

• cites [32] Chan, S. C. Y. et al. Data distributional properties drive emergent in-context learning in transformers. In Advances in Neural Information Processing Systems 35 (eds Koyejo, S. et al.) 18878–18891 (Curran Associates, 2022). sourceSupported

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Forms of one-to-one and iconic concatenation biases are widely attested in natural language.

“These response patterns can be compared to biases in language acquisition more generally; indeed, forms of one-to-one27 and iconic concatenation28,29 are widely attested in natural language.”

• cites [27] Markman, E. M. & Wachtel, G. F. Children’s use of mutual exclusivity to constrain the meanings of words. Cogn. Psychol. 20, 121–157 (1988). sourceNot checked — no full text of the source
• cites [28] Haiman, J. The iconicity of grammar: isomorphism and motivation. Language 56, 515–540 (1980). sourceNot checked — no full text of the source
• cites [29] de Ruiter, L., Theakston, A., Brandt, S. & Lieven, E. Iconicity affects children’s comprehension of complex sentences: the role of semantics, clause order, input and individual differences. Cognition 171, 202–224 (2018). sourceSupported

It may state two separate claims as one.

MLC allows neural networks to tackle other existing challenges, including making systematic use of isolated primitives and using mutual exclusivity to infer meanings.

“MLC also allows neural networks to tackle other existing challenges, including making systematic use of isolated primitives11,16 and using mutual exclusivity to infer meanings44.”

• cites [11] Lake, B. M. & Baroni, M. Generalization without systematicity: on the compositional skills of sequence-to-sequence recurrent networks. In Proc. International Conference on Machine Learning (ICML) (eds. Dy, J. & Krause, A.) 2873–2882 (PMLR, 2018).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [16] Kim, N. & Linzen, T. COGS: a compositional generalization challenge based on semantic interpretation. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP) 9087–9105 (2020). sourceSupported
• cites [44] Gandhi, K. & Lake, B. M. Mutual exclusivity as a challenge for deep neural networks. In Proc. Advances in Neural Information Processing Systems (NeurIPS) 33 (eds Larochelle, H. et al.) 14182–14192 (Curran Associates, 2020).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Systematicity continues to challenge models and motivates new frameworks.

“Systematicity continues to challenge models11–18 and motivates new frameworks34–41.”

• cites [11] Lake, B. M. & Baroni, M. Generalization without systematicity: on the compositional skills of sequence-to-sequence recurrent networks. In Proc. International Conference on Machine Learning (ICML) (eds. Dy, J. & Krause, A.) 2873–2882 (PMLR, 2018).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [12] Ettinger, A., Elgohary, A., Phillips, C. & Resnik, P. Assessing composition in sentence vector representations. In Proc. 7th International Conference on Computational Linguistics, (COLING 2018) 1790–1801 (Association for Computational Linguistics, 2018).Not checked yet
• cites [13] Bahdanau, D. et al. CLOSURE: assessing systematic generalization of CLEVR models. In Proc. NAACL Workshop on Visually Grounded Interaction and Language (ViGIL) (2019). sourceSupported
• cites [14] Keysers, D. et al. Measuring compositional generalization: a comprehensive method on realistic data. In Proc. International Conference on Learning Representations (ICLR) (2019). sourceSupported
• cites [15] Yu, L. & Ettinger, A. Assessing phrasal representation and composition in transformers. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP) 4896–4907 (Association for Computational Linguistics, 2020). sourceSupported
• cites [16] Kim, N. & Linzen, T. COGS: a compositional generalization challenge based on semantic interpretation. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP) 9087–9105 (2020). sourceSupported
• cites [17] Hupkes, D., Dankers, V., Mul, M. & Bruni, E. Compositionality decomposed: how do neural networks generalize? J. Artif. Int. Res. 67, 757–795 (2020). sourceNot checked — no full text of the source
• cites [18] Press, O. et al. Measuring and narrowing the compositionality gap in language models. Preprint at https://arxiv.org/abs/2210.03350 (2022). sourceSupported
• cites [34] Collins, A. G. E. & Frank, M. J. Cognitive control over learning: creating, clustering, and generalizing task-set structure. Psychol. Rev. 120, 190–229 (2013). sourceSupported
• cites [35] Chen, X., Liang, C., Yu, A. W., Song, D. & Zhou, D. Compositional generalization via neural-symbolic stack machines. In Proc. Advances in Neural Information Processing Systems 33 (eds Larochelle, H. et al.) 1690–1701 (Curran Associates, 2020).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

MLC uses the standard transformer architecture for memory-based meta-learning.

“As shown in Fig. 4 and detailed in the ‘Architecture and optimizer’ section of the Methods, MLC uses the standard transformer architecture26 for memory-based meta-learning.”

• cites [26] Vaswani, A. et al. Attention is all you need. In Proc. Advances in Neural Information Processing Systems 30 (eds Guyon, I. et al.) 5998–6008 (Curran Associates, 2017). sourceSupported

It may state two separate claims as one.

Children become better word learners over the course of development, similar to a meta-learner improving with training.

“Second, children become better word learners over the course of development60, similar to a meta-learner improving with training.”

• cites [60] Bergelson, E. The comprehension boost in early word learning: older infants are better learners. Child Dev. Perspect. 14, 142–149 (2020). sourceSupported

Probably not a substantial claim. The paper’s conclusions may not depend on it.

COGS is a multi-faceted benchmark that evaluates many forms of systematic generalization.

“COGS is a multi-faceted benchmark that evaluates many forms of systematic generalization.”

• cites [16] Kim, N. & Linzen, T. COGS: a compositional generalization challenge based on semantic interpretation. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP) 9087–9105 (2020). sourceSupported

It may state two separate claims as one.

Only MLC achieves both the systematicity and flexibility needed for human-like generalization, unlike perfectly systematic but rigid probabilistic symbolic models and perfectly flexible but unsystematic neural networks.

“After considering seven different models, we found that, in contrast to perfectly systematic but rigid probabilistic symbolic models, and perfectly flexible but unsystematic neural networks, only MLC achieves both the systematicity and flexibility needed for human-like generalization.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Pretraining the basic seq2seq model on the entire meta-training set including study examples, then fine-tuning, improves test loss marginally but not accuracy on the few-shot instruction task.

“On the few-shot instruction task, this improves the test loss marginally, but not accuracy.”

Probably not a substantial claim. The paper may give no reason why it holds.

For successful optimization, it is also important to pass each study example (input sequence only) as an additional query when training on a particular episode.

“For successful optimization, it is also important to pass each study example (input sequence only) as an additional query when training on a particular episode.”

It may state two separate claims as one.

The current architecture lacks a mechanism for emitting new symbols, although new symbols introduced through the study examples could be emitted through an additional pointer mechanism.

“The current architecture also lacks a mechanism for emitting new symbols2, although new symbols introduced through the study examples could be emitted through an additional pointer mechanism55.”

• cites [2] Marcus, G. F. The Algebraic Mind: Integrating Connectionism and Cognitive Science (MIT Press, 2003).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [55] Vinyals, O., Fortunato, M. & Jaitly, N. Pointer networks. In Proc. Advances in Neural Information Processing Systems (eds Cortes, C. et al.) (Curran Associates, 2015).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

In our experiments, only MLC closely reproduced human behaviour with respect to both systematicity and biases, with the MLC (joint) model best navigating the trade-off between these two blueprints of human linguistic behaviour.

“In our experiments, only MLC closely reproduced human behaviour with respect to both systematicity and biases, with the MLC (joint) model best navigating the trade-off between these two blueprints of human linguistic behaviour.”

It may state two separate claims as one.

MLC outperforms a basic seq2seq transformer fit without meta-learning and an MLC model optimized for copying rather than systematic generalization (MLC (copy only)) at predicting human few-shot learning behaviour.

“MLC also outperforms a basic seq2seq transformer fit to the patterns in Fig. 2 without meta-learning and an MLC model optimized for copying rather than systematic generalization (MLC (copy only); during training, the query examples always match one of the study examples).”

• AssumptionRests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.

It may state two separate claims as one.

The MLC transformer responded exactly like the modal human participant in 65.0% of samples, perfectly instantiating the three key inductive biases.

“The MLC transformer responded exactly like the modal human participant in 65.0% of samples (Fig. 3b (left)), perfectly instantiating the three key inductive biases.”

• AssumptionRests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.

It may state two separate claims as one.

In the open-ended task, people's responses were highly structured and confirmed the previous two inductive biases.

“Despite the unconstrained nature of the test, people’s responses were highly structured and confirm the previous two inductive biases.”

• AssumptionRests on an assumption: Human participants recruited via Amazon Mechanical Turk are representative of the broader human population for these compositional skills.

It may state two separate claims as one.

MLC shows much stronger systematicity than neural networks trained in standard ways, and shows more nuanced behaviour than pristine symbolic models.

“MLC shows much stronger systematicity than neural networks trained in standard ways, and shows more nuanced behaviour than pristine symbolic models.”

It may state two separate claims as one.

MLC fails to handle longer output sequences (SCAN length split) and novel and more complex sentence structures (three types in COGS), with error rates at 100%.

“However surface-level permutations were not enough for MLC to solve the structural generalization tasks in the benchmarks. MLC fails to handle longer output sequences (SCAN length split) as well as novel and more complex sentence structures (three types in COGS), with error rates at 100%.”

It may state two separate claims as one.

Chance performance is 2.8% for two-length output sequences if the length is known, and exponentially less for longer sequences.

“Chance performance is 2.8% for two-length output sequences if the length is known, and exponentially less for longer sequences.”

It may state two separate claims as one.

All 10 runs of the basic seq2seq model failed to generalize systematically on the few-shot instruction task, with 0% exact-match accuracy.

“Nevertheless, all 10 runs failed to generalize systematically on the few-shot instruction task (0% exact-match accuracy).”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Lower-capacity transformers did not perform better than the basic seq2seq model.

“First, we evaluated lower-capacity transformers but found that they did not perform better.”

It may state two separate claims as one.

On SCAN, MLC solves three systematic generalization splits with an error rate of 0.22% or lower (99.78% accuracy or above), including 'add jump', 'around right', and 'opposite right'.

“On SCAN, MLC solves three systematic generalization splits with an error rate of 0.22% or lower (99.78% accuracy or above), including the already mentioned ‘add jump’ split and ‘around right’ and ‘opposite right’, which examine novel combinations of known words.”

It may state two separate claims as one.

People also relied on inductive biases that sometimes support the algebraic solution and sometimes deviate from it; people are not purely algebraic machines.

“However, people also relied on inductive biases that sometimes support the algebraic solution and sometimes deviate from it; indeed, people are not purely algebraic machines3,6,7.”

• cites [3] Johnson, K. On the systematicity of language and thought. J. Philos. 101, 111–139 (2004). sourceNot checked — no full text of the source
• cites [6] O’Reilly, R. C. et al. in The Architecture of Cognition: Rethinking Fodor and Pylyshyn’s Systematicity Challenge (eds Calvo, P. & Symons, J.) 191–226 (MIT Press, 2014). sourceNot checked — no full text of the source
• cites [7] Nam, A. J. & McClelland, J. L. What underlies rapid learning and systematic generalization in humans? Preprint at http://arxiv.org/abs/2107.06994 (2021).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: The algebraic interpretation grammar provides the correct normative standard for the few-shot instruction-learning task.
• AssumptionRests on an assumption: Human participants recruited via Amazon Mechanical Turk are representative of the broader human population for these compositional skills.

It may state two separate claims as one.

MLC was similarly optimized to infer systematic rules and respond with the same biased-based patterns as the symbolic (oracle/biases) model.

“Indeed, MLC was similarly optimized to (implicitly) infer systematic rules and respond with the same biased-based patterns, and it is therefore natural that the two models would perform similarly.”

Probably not a substantial claim. It may state two separate claims as one.

MLC was evaluated on this task in several ways; in each case, MLC responded to this novel task through learned memory-based strategies, as its weights were frozen and not updated further.

“MLC was evaluated on this task in several ways; in each case, MLC responded to this novel task through learned memory-based strategies, as its weights were frozen and not updated further.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Model predictions were evaluated through fivefold cross-validation.

“Model predictions were also evaluated through fivefold cross-validation33: MLC and other models were optimized on responses for either 23 or 24 participants (depending on the cross-validation split) and then predicted responses for held-out participants.”

• cites [33] Myung, J. I. & Pitt, M. A. in Stevens’ Handbook of Experimental Psychology and Cognitive Neuroscience (ed. Wixted, J. T.) 85–118 (John Wiley & Sons, 2018).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim.

The study items remained on the screen for reference, so that performance would reflect generalization in the absence of memory limitations.

“The study items remained on the screen for reference, so that performance would reflect generalization in the absence of memory limitations.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The right meta-training procedure can promote productivity.

“However, MLC did handle novel sentence structures in our few-shot instruction-learning task (77.8% correct on queries with both longer input and output sequences than seen during study; Fig. 2), suggesting that the right meta-training procedure can promote productivity—a challenge we leave to future work.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

People require a years-long process of education to master other forms of systematic generalization and symbolic reasoning, including mathematics, logic and computer programming.

“Beyond natural language, people require a years-long process of education to master other forms of systematic generalization and symbolic reasoning6,7, including mathematics, logic and computer programming.”

• cites [6] O’Reilly, R. C. et al. in The Architecture of Cognition: Rethinking Fodor and Pylyshyn’s Systematicity Challenge (eds Calvo, P. & Symons, J.) 191–226 (MIT Press, 2014). sourceNot checked — no full text of the source
• cites [7] Nam, A. J. & McClelland, J. L. What underlies rapid learning and systematic generalization in humans? Preprint at http://arxiv.org/abs/2107.06994 (2021).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim.

The study and test items always differed from one another by more than one primitive substitution (except in the function 1 stage, where a single primitive was presented as a novel argument to function 1).

“The study and test items always differed from one another by more than one primitive substitution (except in the function 1 stage, where a single primitive was presented as a novel argument to function 1).”

Probably not a substantial claim. It may state two separate claims as one.

The examples used for optimization on machine learning benchmarks were generated by the benchmark designers through algebraic rules, and there is no direct imitation of human behavioural data.

“Note that here the examples used for optimization were generated by the benchmark designers through algebraic rules, and there is therefore no direct imitation of human behavioural data.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The study and test set only used four output symbols, but the pool provided six possibilities (that is, there were two extra colours that were not associated to words), to discourage reasoning by exclusion.

“The study and test set only used four output symbols, but the pool provided six possibilities (that is, there were two extra colours that were not associated to words), to discourage reasoning by exclusion.”

Probably not a substantial claim. It may state two separate claims as one.

The COGS output expressions were converted to uppercase to remove incidental overlap between input and output token indices, which MLC but not basic seq2seq could exploit.

“The COGS output expressions were converted to uppercase to remove any incidental overlap between input and output token indices (which MLC, but not basic seq2seq, could exploit).”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The more scalable MLC variant, the original MLC architecture, and basic seq2seq all have approximately the same number of learnable parameters, except that basic seq2seq has a smaller input vocabulary.

“This more scalable MLC variant, the original MLC architecture (see the ‘Architecture and optimizer’ section) and basic seq2seq all have approximately the same number of learnable parameters (except for the fact that basic seq2seq has a smaller input vocabulary).”

It may state two separate claims as one.

MLC shows how meta-learning can be used like hierarchical Bayesian models for reverse-engineering inductive biases, with neural networks for greater expressive power.

“MLC shows how meta-learning can be used like hierarchical Bayesian models for reverse-engineering inductive biases (see ref. 47 for a formal connection), although with the aid of neural networks for greater expressive power.”

• cites ref. 47Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Decoder cross-attention layers are less expensive, O(ST), because the target sequence length T is typically much shorter than S.

“The decoder cross-attention layers are less expensive, O(ST), because the target sequence length T, which does not include any study examples, is typically much shorter (T ≪ S).”

The quote only partly states this claim.

The pseudolanguage instruction-learning task is a valid measure of human systematic generalization, differing from artificial grammar learning, statistical learning, and program learning in that explicit or implicit judgments of grammaticality are not needed.

“Our experimental paradigm asks participants to process instructions in a pseudolanguage in order to generate abstract outputs (meanings), differing from artificial grammar learning22, statistical learning23 and program learning24 in that explicit or implicit judgments of grammaticality are not needed.”

• cites [22] Reber, A. Implicit learning of artificial grammars. Verb. Learn. Verb. Behav. 5, 855–863 (1967). sourceNot checked — no full text of the source
• cites [23] Aslin, R. N., Saffran, J. R. & Newport, E. L. Computation of conditional probability statistics by 8-month-old infants. Psychol. Sci. 9, 321–324 (1998). sourceNot checked — no full text of the source
• cites [24] Stuhlmuller, A., Tenenbaum, J. B. & Goodman, N. D. Learning structured generative concepts. In Proc. Thirty-Second Annual Conference of the Cognitive Science Society, 2296–2301 (2010).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim.

Using the test corpus for study examples would inadvertently leak test information.

“using the test corpus for study examples would inadvertently leak test information”

• Also stated in: § Machine learning benchmarks · hedged

  “picking study examples from the generalization corpus would inadvertently leak test information”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Query responses were excluded if the corresponding study phases were not completed correctly (for all items) within three attempts (13% of remaining data).

“Finally, query responses were also excluded if the corresponding study phases were not completed correctly (for all items) within three attempts (13% of remaining data).”

Probably not a substantial claim. It may state two separate claims as one.

The validation episodes were defined by new grammars that differ from the training grammars. Grammars were only considered new if they did not match any of the meta-training grammars, even under permutations of how the rules are ordered.

“The validation episodes were defined by new grammars that differ from the training grammars. Grammars were only considered new if they did not match any of the meta-training grammars, even under permutations of how the rules are ordered.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Before the experiment, the participants practiced using the response interface and had to pass an instructions quiz; they cycled through the quiz until they passed it.

“First, before the experiment, the participants practiced using the response interface and had to pass an instructions quiz; they cycled through the quiz until they passed it.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

We used several strategies to ensure that our participants were paying attention.

“We used several strategies to ensure that our participants were paying attention.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

For statistical analyses of the data from this experiment and elsewhere, we tested for data normalcy and applied alternative nonparametric or permutation tests when the assumptions were not met.

“For statistical analyses of the data from this experiment and elsewhere, we tested for data normalcy and applied alternative nonparametric or permutation tests when the assumptions were not met.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

All of the studies were approved by the NYU IRB, protocol FY2018-1728, and obtained informed consent.

“All of the studies were approved by the NYU IRB, protocol FY2018-1728, and obtained informed consent.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The MLC approach seeks to model adult compositional skills but not the process by which adults acquire those skills.

“Importantly, this approach seeks to model adult compositional skills but not the process by which adults acquire those skills, which is an issue that is considered further in the general discussion.”

It may state two separate claims as one.

The vocabulary in COGS is much larger than in SCAN, so study examples cannot be sampled arbitrarily with any reasonable hope that they would inform the query of interest.

“The vocabulary in COGS is much larger than in SCAN; thus, the study examples cannot be sampled arbitrarily with any reasonable hope that they would inform the query of interest.”

It may state two separate claims as one.

MLC uses only common neural networks without added symbolic machinery, hand-designed internal representations, or inductive biases.

“Our implementation of MLC uses only common neural networks without added symbolic machinery, and without hand-designed internal representations or inductive biases.”

It may state two separate claims as one.

Without access to study examples, the basic seq2seq model is poorly equipped for learning words with changing meanings; it has no in-context memory and therefore all of its knowledge must be stored in the learned weights.

“Without access to study examples, the model is poorly equipped for learning words with changing meanings; it has no in-context memory and, therefore, all of its knowledge must be stored in the learned weights.”

It may state two separate claims as one.

MLC derives its abilities through meta-learning, where both systematic generalization and human biases are not inherent properties of the neural network architecture but induced from data.

“Furthermore, MLC derives its abilities through meta-learning, where both systematic generalization and the human biases are not inherent properties of the neural network architecture but, instead, are induced from data.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Catch trials were included during the test phases, probing the study items rather than new items, with the answers clearly presented on the screen above. There was one catch trial per stage (except the last stage had two); participants were excluded if they missed two or more catch trials (n = 5).

“Second, catch trials were included during the test phases, probing the study items rather than new items, with the answers clearly presented on the screen above. There was one catch trial per stage (except the last stage had two); participants were excluded if they missed two or more catch trials (n = 5).”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

One participant was excluded because they reported using an external aid in a post-test survey.

“One participant was excluded because they reported using an external aid in a post-test survey.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Occasionally skipping permutations helps break symmetries that can slow optimization; the association between input and output primitives is no longer perfectly balanced.

“Occasionally skipping the permutations in this way helps to break symmetries that can slow optimization; that is, the association between the input and output primitives is no longer perfectly balanced.”

Probably not a substantial claim. It may state two separate claims as one.

The gold interpretation grammar that produced the few-shot instruction-learning task with humans and machines was reserved for testing, with an additional structural requirement that grammars for producing the training and validation episodes should also not match the gold grammar through any permutation of the input and output symbol assignments.

“The gold interpretation grammar that produced the few-shot instruction-learning task with humans and machines (Extended Data Fig. 2) was also reserved for testing in this way, with an additional structural requirement that grammars for producing the training and validation episodes should also not match the gold grammar through any permutation of the input and output symbol assignments.”

Probably not a substantial claim.

For the 'add jump' split, fixing one study example to 'jump → JUMP' ensures MLC has access to the basic meaning before attempting compositional uses of 'jump'.

“for the ‘add jump’ split, one study example is fixed to be ‘jump → JUMP’, ensuring that MLC has access to the basic meaning before attempting compositional uses of ‘jump’.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Performance was averaged over 200 passes through the dataset, each episode with different random query orderings as well as word and colour assignments.

“Performance was averaged over 200 passes through the dataset, each episode with different random query orderings as well as word and colour assignments.”

Probably not a substantial claim.

For comparison with the gold grammar or with human behaviour via log-likelihood, performance was averaged over 100 random word/colour assignments.

“For comparison with the gold grammar or with human behaviour via log-likelihood, performance was averaged over 100 random word/colour assignments.”

Probably not a substantial claim. It may state two separate claims as one.

The assignment of words to colours and functions was randomized for each participant (drawn from nine possible words and six colours), and the first three stages were presented in random order.

“The assignment of words to colours and functions was randomized for each participant (drawn from nine possible words and six colours), and the first three stages were presented in random order.”

Probably not a substantial claim. The paper may give no reason why it holds.

To make sense of an episode, MLC must become adept at inferring, from just a few study examples, how words map to meanings.

“To make sense of an episode, MLC must become adept at inferring, from just a few study examples, how words map to meanings.”

Probably not a substantial claim.

To master the lexical generalization splits, the meta-training procedure targets several lexical classes that participate in particularly challenging compositional generalizations.

“To master the lexical generalization splits, the meta-training procedure targets several lexical classes that participate in particularly challenging compositional generalizations.”

• AssumptionThe authors state this as an assumption: Broad lexical classes (common nouns, proper nouns, dative verbs, infinitive verbs) are assumed.

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Several other input words (functional words such as 'turn', 'twice', 'thrice', 'around', 'opposite', 'and', 'after') have stable meanings that can be stored in the model weights.

“Note that several other input words (the mostly ‘functional’ words ‘turn’, ‘twice’, ‘thrice’, ‘around’, ‘opposite’, ‘and’, ‘after’) have stable meanings that can be stored in the model weights.”

Probably not a substantial claim. It may state two separate claims as one.

The correspondence between input and output word types is unknown and not used.

“Importantly, although the broad classes are assumed and could plausibly arise through simple distributional learning68,69, the correspondence between input and output word types is unknown and not used.”

• Also stated in: § Machine learning benchmarks · plain

  “As before, the correspondence between input and output tokens is unknown, both at the levels of a sentence and the whole dataset.”

Probably not a substantial claim. The paper may give no reason why it holds.

MLC must become adept at composition: it must systematically compose the inferred word meanings to correctly answer the queries.

“MLC must also become adept at composition: it must systematically compose the inferred word meanings to correctly answer the queries.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Other verbs, punctuation and logical symbols have stable meanings that can be stored in the model weights.

“Other verbs, punctuation and logical symbols have stable meanings that can be stored in the model weights.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

All models compared in Table 1 have the same opportunity to fit a lapse parameter.

“Note that all of the models compared in Table 1 have the same opportunity to fit a lapse parameter.”