The DOI matches its DataCite record; 4 of the 4 references it cites were found in a registry or an index.
Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
Open accessAbstract
State-of-the-art machine learning methods exhibit limited compositional generalization. At the same time, there is a lack of realistic benchmarks that comprehensively measure this ability, which makes it challenging to find and evaluate improvements. We introduce a novel method to systematically construct such benchmarks by maximizing compound divergence while guaranteeing a small atom divergence between train and test sets, and we quantitatively compare this method to other approaches for creating compositional generalization benchmarks. We present a large and realistic natural language question answering dataset that is constructed according to this method, and we use it to analyze the compositional generalization ability of three machine learning architectures. We find that they fail to generalize compositionally and that there is a surprisingly strong negative correlation between compound divergence and accuracy. We also demonstrate how our method can be used to create new compositionality benchmarks on top of the existing SCAN dataset, which confirms these findings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Modern neural networks still struggle on tests of systematicity.
“Notably, modern neural networks still struggle on tests of systematicity11–18—tests that even a minimally algebraic mind should pass2.”
From Human-like systematic generalization through a meta-learning neural network (2023) · cites this paper as [14]
Systematicity continues to challenge models and motivates new frameworks.
“Systematicity continues to challenge models11–18 and motivates new frameworks34–41.”
From Human-like systematic generalization through a meta-learning neural network (2023) · cites this paper as [14]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ASPECTS OF THE THEORY OF SYNTAX
Chomsky, N. (1964) · doi:10.21236/ad0616323 · Cited by 13,587
Connectionism and cognitive architecture: A critical analysis
Fodor, J. A., & Pylyshyn, Z. W. (1988) · Cognition · doi:10.1016/0010-0277(88)90031-5 · Cited by 4,022
The Web as a Knowledge-Base for Answering Complex Questions
Talmor, A., & Berant, J. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1059 · Cited by 520
Flagged first, then by citations
Research works printed but not linked: 38
- Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In CVPR, 2018. URL https: //arxiv.org/pdf/1712.00377.pdf.
- Jacob Andreas. Measuring compositionality in representation learning. In ICLR, 2019. URL https: //openreview.net/pdf?id=HJz05o0qK7.
- Franz Baader, Diego Calvanese, Deborah McGuinness, Peter Patel-Schneider, and Daniele Nardi. The description logic handbook: Theory, implementation and applications. Cambridge University Press, 2003.
- Franz Baader, Sebastian Brandt, and Carsten Lutz. Pushing the EL envelope. In IJCAI, 2005. URL http://dl.acm.org/citation.cfm?id=1642293.1642351.
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015. URL http://arxiv.org/abs/1409.0473. · arXiv
- Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can it be learned? In ICLR, 2019. URL http://arxiv.org/abs/1811.12889. · arXiv
- Jasmijn Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela. Jump to better conclusions: SCAN both left and right. In BlackboxNLP@EMNLP, 2018. URL https://www. aclweb.org/anthology/W18-5407.
- Peter Battaglia, Jessica Blake Chandler Hamrick, Victor Bapst, Alvaro Sanchez, Vinicius Zam- baldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andy Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Jayne Langston, Chris Dyer, Nicolas Heess, Daan Wier- stra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Rela- tional inductive biases, deep learning, and graph networks. CoRR, abs/1806.01261, 2018. URL https://arxiv.org/pdf/1806.01261.pdf. · arXiv
- Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In EMNLP, 2013. URL https://www.aclweb.org/anthology/D13-1160.
- Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collabo- ratively created graph database for structuring human knowledge. In ACM SIGMOD, 2008. URL https://doi.org/10.1145/1376616.1376746. · DOI · Paper page
- Michael Chang, Abhishek Gupta, Sergey Levine, and Thomas L. Griffiths. Automatically com- posing representation transformations as a means for generalization. In ICLR, 2019. URL https://openreview.net/pdf?id=B1ffQnRcKX.
- JK Chung, PL Kannappan, CT Ng, and PK Sahoo. Measures of distance between probability distri- butions. JMAA, 138(1):280-292, 1989. URL https://core.ac.uk/download/pdf/82205465.pdf.
- Michael A. Covington. Gulp 3.1: An extension of prolog for unification-based grammar. Research Report AI-1994-06, University of Georgia, 1994. URL http://hdl.handle.net/10724/30221.
- Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. CoRR, abs/1807.03819, 2018. URL https://arxiv.org/pdf/1807.03819.pdf. · arXiv
- Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasi- vam, Rui Zhang, and Dragomir Radev. Improving text-to-SQL evaluation methodology. In ACL, 2018. URL http://aclweb.org/anthology/P18-1033.
- Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matko Bosnjak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: Learning hierarchical compositional visual concepts. In ICLR, 2018. URL https://openreview.net/pdf?id= rkN2Il-RZ.
- Drew A. Hudson and Christopher D. Manning. Compositional attention networks for machine rea- soning. In ICLR, 2018. URL https://openreview.net/pdf?id=S1Euwz-Rb.
- Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019a. URL https://arxiv.org/pdf/1902.09506. pdf. · arXiv
- Drew A. Hudson and Christopher D. Manning. Learning by abstraction: The neural state machine. CoRR, abs/1907.03950, 2019b. URL http://arxiv.org/abs/1907.03950. · arXiv
- Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Fei-Fei Li, Lawrence C. Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. URL https://arxiv.org/pdf/1612.06890.pdf. · arXiv
- Lauri Karttunen. Features and values. In ACL, 1984. URL https://doi.org/10.3115/980491.980499. · DOI
- Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In ICML, 2018. URL https://arxiv.org/pdf/ 1711.00350.pdf.
- João Loula, Marco Baroni, and Brenden Lake. Rearranging the familiar: Testing compositional gen- eralization in recurrent networks. In BlackboxNLP@EMNLP, 2018. URL https://www.aclweb. org/anthology/W18-5413.
- Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In ICLR, 2019. URL https://arxiv.org/abs/1904.12584. · arXiv
- Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. Key-value memory networks for directly reading documents. EMNLP, 2016. doi: 10.18653/v1/d16-1147. URL http://dx.doi.org/10.18653/v1/D16-1147. · DOI
- Jake Russin, Jason Jo, Randall C. O'Reilly, and Yoshua Bengio. Compositional generalization in a deep seq2seq model by separating syntax and semantics. CoRR, abs/1904.09708, 2019. URL http://arxiv.org/abs/1904.09708. · arXiv
- David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In ICLR, 2019. URL https://openreview.net/pdf?id= H1gR5iR5FX.
- Yu Su, Huan Sun, Brian Sadler, Mudhakar Srivatsa, Izzeddin Gur, Zenghui Yan, and Xifeng Yan. On generating characteristic-rich question sets for QA evaluation. In EMNLP, 2016. URL https: //www.aclweb.org/anthology/D16-1054.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017. URL https: //arxiv.org/pdf/1706.03762.pdf.
- Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. Tensor2tensor for neural machine translation. CoRR, abs/1803.07416, 2018. URL http://arxiv.org/abs/1803.07416. · arXiv
- Yushi Wang, Jonathan Berant, and Percy Liang. Building a semantic parser overnight. In ACL, 2015. URL http://aclweb.org/anthology/P15-1129.
- Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards AI-complete question answering: A set of prerequisite toy tasks. In ICLR, 2016. URL https://arxiv.org/pdf/1502.05698.pdf. · arXiv
- Xuchen Yao and Benjamin Van Durme. Information extraction over structured data: Question an- swering with freebase. In ACL, 2014. URL https://www.aclweb.org/anthology/P14-1090.
- Scott Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang, and Jina Suh. The value of semantic parse labeling for knowledge base question answering. In ACL, 2016. URL https: //www.aclweb.org/anthology/P16-2033.
- Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J Smola, and Le Song. Variational rea- soning for question answering with knowledge graph. In AAAI, 2018. URL https://arxiv.org/pdf/ 1709.04071.pdf.
- Did [Emerald Cities]'s cinematographer, writer, and editor edit, executive produce, and direct [Blues for the Avatar] and [White Stork Is Coming]?
- What male employee of [Weta Workshop] was [Bad Taste]'s editor? 26. Were [Weta Digital] and
- Workshop] founded by a cinematographer and founded by a film editor? 27. What art director influenced [DreamWorks Animation]'s founder? 28. Did [Daisies] star [Fruit of Paradise]'s costume designer and writer, star [Jaromír Vomácka], and star [Jirina Myskova]?
Cited by
Sorted by citations
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.67 · Cited by 428
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2210.03350 · Cited by 294
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N., & Lewis, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.378 · Cited by 303
Human-like systematic generalization through a meta-learning neural network
Lake, B. M., & Baroni, M. (2023) · Nature · doi:10.1038/s41586-023-06668-3 · Cited by 230
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193
“One specific type of generalisation examined frequently in supervised learning which is relevant to RL is compositional generalisation (Hupkes et al., 2020; Keysers et al., 2020).”
BackgroundMcCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2021) · arXiv · doi:10.48550/arxiv.2111.09509 · Cited by 70
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00567 · Cited by 70
Diverse Demonstrations Improve In-context Compositional Generalization
Levy, I., Bogin, B., & Berant, J. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.78 · Cited by 66
On the Binding Problem in Artificial Neural Networks
Greff, K., van Steenkiste, S., & Schmidhuber, J. (2020) · arXiv · doi:10.48550/arxiv.2012.05208 · Cited by 102
Metadata
- Authors
- Keysers, D., Schärli, N., Scales, N., Buisman, H., Furrer, D., Kashubin, S., Momchev, N., Sinopalnikov, D., Stafiniak, L., Tihon, T., Tsarkov, D., Wang, X., van Zee, M., & Bousquet, O.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1912.09713
- OpenAlex
- W2996094825
- Cited by
- 264
- References
- 4
- Citations by year
- 2026: 21 · 2025: 22 · 2024: 18 · 2023: 72 · 2022: 53 · 2021: 50 · 2020: 27 · 2019: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- compositional generalization, benchmark design, natural language question answering, compositional reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
