The DOI matches its DataCite record; 8 of the 8 references it cites were found in a registry or an index.
CLOSURE: Assessing Systematic Generalization of CLEVR Models
Open accessAbstract
The CLEVR dataset of natural-looking questions about 3D-rendered scenes has recently received much attention from the research community. A number of models have been proposed for this task, many of which achieved very high accuracies of around 97-99%. In this work, we study how systematic the generalization of such models is, that is to which extent they are capable of handling novel combinations of known linguistic constructs. To this end, we test models' understanding of referring expressions based on matching object properties (such as e.g. "another cube that is the same size as the brown cube") in novel contexts. Our experiments on the thereby constructed CLOSURE benchmark show that state-of-the-art models often do not exhibit systematicity after being trained on CLEVR. Surprisingly, we find that an explicitly compositional Neural Module Network model also generalizes badly on CLOSURE, even when it has access to the ground-truth programs at test time. We improve the NMN's systematic generalization by developing a novel Vector-NMN module architecture with vector-valued inputs and outputs. Lastly, we investigate how much few-shot transfer learning can help models that are pretrained on CLEVR to adapt to CLOSURE. Our few-shot learning experiments contrast the adaptation behavior of the models with intermediate discrete programs with that of the end-to-end continuous models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Modern neural networks still struggle on tests of systematicity.
“Notably, modern neural networks still struggle on tests of systematicity11–18—tests that even a minimally algebraic mind should pass2.”
From Human-like systematic generalization through a meta-learning neural network (2023) · cites this paper as [13]
Systematicity continues to challenge models and motivates new frameworks.
“Systematicity continues to challenge models11–18 and motivates new frameworks34–41.”
From Human-like systematic generalization through a meta-learning neural network (2023) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Connectionism and cognitive architecture: A critical analysis
Fodor, J. A., & Pylyshyn, Z. W. (1988) · Cognition · doi:10.1016/0010-0277(88)90031-5 · Cited by 4,022
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317
Self-citationFilm: Visual reasoning with a general conditioning layer
Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11671 · Cited by 2,490
Self-citationAnalyzing the Behavior of Visual Question Answering Models
Agrawal, A., Batra, D., & Parikh, D. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1203 · Cited by 297
Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
Mascharka, D., Tran, P., Soklaski, R., & Majumdar, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00519 · Cited by 203
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2016) · arXiv · doi:10.48550/arxiv.1612.06890 · Cited by 40
Flagged first, then by citations
Research works printed but not linked: 21
- Agrawal, A., Kembhavi, A., Batra, D., and Parikh, D. C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset. arXiv:1704.08243 [cs], April 2017. URL http://arxiv.org/abs/1704.08243. arXiv: 1704.08243. · arXiv
- Andreas, J., Rohrbach, M., Darrell, T., and Klein, D. Neural Module Networks. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. URL http://arxiv. org/abs/1511.02799.
- Bahdanau, D., Cho, K., and Bengio, Y. Neural Ma- chine Translation by Jointly Learning to Align and Translate. In International Conference on Learning Representations, ICLR 2015, 2015.
- Bahdanau, D., Murty, S., Noukhovitch, M., Nguyen, T. H., Vries, H. d., and Courville, A. Systematic Generalization: What Is Required and Can It Be Learned? In International Conference on Learning Representations, ICLR 2019, 2019. URL https: //openreview.net/forum?id=HkezXnA9YX.
- Bastings, J., Baroni, M., Weston, J., Cho, K., and Kiela, D. Jump to better conclusions: SCAN both left and right. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for Computa- tional Linguistics, November 2018. URL https: //www.aclweb.org/anthology/W18-5407.
- Hu, R., Andreas, J., Rohrbach, M., Darrell, T., and Saenko, K. Learning to Reason: End-to-End Module Networks for Visual Question Answering. In Proceedings of 2017 IEEE International Con- ference on Computer Vision, April 2017. URL http://arxiv.org/abs/1704.05526. arXiv: 1704.05526. · arXiv
- Hu, R., Andreas, J., Darrell, T., and Saenko, K. Explainable Neural Computation via Stack Neu- ral Module Networks. In arXiv:1807.08556 [cs], July 2018. URL http://arxiv.org/abs/1807. 08556. arXiv: 1807.08556. · arXiv
- Hudson, D. A. and Manning, C. D. Composi- tional Attention Networks for Machine Reason- ing. In Proceedings of the 2018 International Conference on Learning Representations, February 2018. URL https://openreview.net/forum? id=S1Euwz-Rb.
- Hupkes, D., Dankers, V., Mul, M., and Bruni, E. The compositionality of neural networks: integrating symbolism and connectionism. arXiv:1908.08351 [cs, stat], August 2019. URL http://arxiv. org/abs/1908.08351. arXiv: 1908.08351. · arXiv
- Ioffe, S. and Szegedy, C. Batch Normalization: Ac- celerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learn- ing, ICML 2015, 2015. URL http://jmlr.org/ proceedings/papers/v37/ioffe15.html.
- Johnson, J., Hariharan, B., van der Maaten, L., Hoff- man, J., Fei-Fei, L., Zitnick, C. L., and Girshick, R. Inferring and Executing Programs for Visual Rea- soning. In Proceedings of 2017 IEEE International Conference on Computer Vision, ICCV 2017, 2017. URL http://arxiv.org/abs/1705.03633. · arXiv
- Keysers, D., Schärli, N., Scales, N., Buisman, H., Fur- rer, D., Kashubin, S., Momchev, N., Sinopalnikov, D., Stafiniak, L., Tihon, T., Tsarkov, D., Wang, X., van Zee, M., and Bousquet, O. Measuring Compositional Generalization: A Comprehensive Method on Realistic Data. In International Con- ference on Learning Representations, 2020. arXiv: 1912.09713. · arXiv
- Kingma, D. P. and Ba, J. Adam: A Method for Stochastic Optimization. In Proceedings of the 2015 International Conference on Learning Representa- tions, ICLR 2015, 2015. URL http://arxiv. org/abs/1412.6980. arXiv: 1412.6980. · arXiv
- Kuhnle, A. and Copestake, A. ShapeWorld -A new test methodology for multimodal language under- standing. arXiv:1704.04517 [cs], April 2017. URL http://arxiv.org/abs/1704.04517. arXiv: 1704.04517. · arXiv
- Lake, B. M. and Baroni, M. Generalization with- out systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Pro- ceedings of the 36th International Conference on Machine Learning, 2018. URL http://arxiv. org/abs/1711.00350. arXiv: 1711.00350. · arXiv
- Loula, J., Baroni, M., and Lake, B. M. Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks. In Proceedings of the 2018 BlackboxNLP EMNLP Workshop, July 2018. URL https://arxiv.org/abs/1807.07545. · arXiv
- Perez, E., de Vries, H., Strub, F., Dumoulin, V., and Courville, A. Learning Visual Reasoning With- out Strong Priors. arXiv:1707.03017 [cs, stat], July 2017a. URL http://arxiv.org/abs/1707. 03017. arXiv: 1707.03017. · arXiv
- Russin, J., Jo, J., and O'Reilly, R. C. Compositional generalization in a deep seq2seq model by separat- ing syntax and semantics. arXiv:1904.09708 [cs, stat], April 2019. URL http://arxiv.org/abs/ 1904.09708. arXiv: 1904.09708. · arXiv
- Santoro, A., Raposo, D., Barrett, D. G. T., Malinowski, M., Pascanu, R., Battaglia, P., and Lillicrap, T. A simple neural network module for relational reason- ing. In Advances in Neural Information Processing Systems 31, June 2017. URL http://arxiv.org/ abs/1706.01427. arXiv: 1706.01427. · arXiv
- Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to Sequence Learning with Neural Networks. In Ad- vances in Neural Information Processing Systems 27, pp. 3104-3112, 2014.
- Yi, K., Wu, J., Gan, C., Torralba, A., Kohli, P., and Tenenbaum, J. B. Neural-Symbolic VQA: Disentan- gling Reasoning from Vision and Language Under- standing. Advances in Neural Information Process- ing Systems 31: NeurIPS 2018, October 2018. URL http://arxiv.org/abs/1810.02338. arXiv: 1810.02338. · arXiv
Cited by
Sorted by citations
Inductive biases for deep learning of higher-level cognition
Goyal, A., & Bengio, Y. (2022) · Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rspa.2021.0068 · Cited by 300
Self-citationHuman-like systematic generalization through a meta-learning neural network
Lake, B. M., & Baroni, M. (2023) · Nature · doi:10.1038/s41586-023-06668-3 · Cited by 230
Metadata
- Authors
- Bahdanau, D., de Vries, H., O'Donnell, T. J., Murty, S., Beaudoin, P., Bengio, Y., & Courville, A.
- Affiliations
- Université de Montréal (CA)
- Canadian Institute for Advanced Research (CA)
- Mila - Quebec Artificial Intelligence Institute (CA)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1912.05783
- OpenAlex
- W2995479498
- Cited by
- 96
- References
- 8
- Citations by year
- 2026: 6 · 2025: 2 · 2024: 8 · 2023: 33 · 2022: 19 · 2021: 19 · 2020: 9
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- CLEVR, systematic generalization, referring expressions, few-shot transfer learning, 3D scene understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
