The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
A Corpus for Reasoning about Natural Language Grounded in Photographs
Open accessAbstract
We introduce a new dataset for joint reasoning about natural language and images, with a focus on semantic diversity, compositionality, and visual reasoning challenges.The data contains 107,292 examples of English sentences paired with web photographs.The task is to determine whether a natural language caption is true about a pair of photographs.We crowdsource the data using sets of visually rich images and a compare-and-contrast task to elicit linguistically diverse language.Qualitative analysis shows the data requires compositional joint reasoning, including about quantities, comparisons, and relations.Evaluation using state-of-the-art visual reasoning methods shows the data presents a strong challenge.* Contributed equally.† Work done as an undergraduate at Cornell University. 1 In parts of this paper, we use the term compositional differently than it is commonly used in linguistics to refer to reasoning that requires composition.This type of reasoning often manifests itself in highly compositional language.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., & Dollár, P. (2014) · arXiv · doi:10.48550/arxiv.1405.0312 · Cited by 2,579 · cited 1×
“We detect the objects in the pair of images using a Mask R-CNN model (He et al., 2017; Girshick et al., 2018) pretrained on the COCO detection task (Lin et al., 2014).”
MethodFilm: Visual reasoning with a general conditioning layer
Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11671 · Cited by 2,490
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614
A simple neural network module for relational reasoning
Santoro, A., Raposo, D., Barrett, D. G. T., Malinowski, M., Pascanu, R., Battaglia, P., & Lillicrap, T. (2017) · arXiv · doi:10.48550/arxiv.1706.01427 · Cited by 1,281 · cited 1×
“…2017b; Perez et al., 2018; Hu et al., 2017; Mascharka et al., 2018; Hu et al., 2018; Suarez et al., 2018) and attention-or memory-based meth-ods (Santoro et al., 2017; Zhu et al., 2017; Hud-son and Manning, 2018; Tan and Bansal, 2018; Malinowski et al., 2018; Pavez et al., 2018; Yao et al.,…”
BackgroundAnderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sunderhauf, N., Reid, I., Gould, S., & van den Hengel, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00387 · Cited by 1,187
Generation and Comprehension of Unambiguous Object Descriptions
Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A., & Murphy, K. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.9 · Cited by 1,174
ReferItGame: Referring to Objects in Photographs of Natural Scenes
Kazemzadeh, S., Ordonez, V., Matten, M., & Berg, T. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1086 · Cited by 1,048
Andreas, J., Rohrbach, M., Darrell, T., & Klein, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.12 · Cited by 908
From Recognition to Cognition: Visual Commonsense Reasoning
Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00688 · Cited by 769
Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
Agrawal, A., Batra, D., Parikh, D., & Kembhavi, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00522 · Cited by 566
TVQA: Localized, Compositional Video Question Answering
Lei, J., Yu, L., Bansal, M., & Berg, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1167 · Cited by 541 · cited 1×
“, 2018) Visual Question Answering TVQA (Lei et al., 2018) Video Question Answering 4 4”
BackgroundLearning to Reason: End-to-End Module Networks for Visual Question Answering
Hu, R., Andreas, J., Rohrbach, M., Darrell, T., & Saenko, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.93 · Cited by 517
Inferring and Executing Programs for Visual Reasoning
Johnson, J., Hariharan, B., Van Der Maaten, L., Hoffman, J., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.325 · Cited by 467
Das, A., Kottur, S., Gupta, K., Singh, A., Jain, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · Cited by 453
FigureQA: An Annotated Figure Dataset for Visual ReasoningPreprint
Kahou, S. E., Michalski, V., Atkinson, A., Kádár, Á., Trischler, A., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.07300 · Cited by 446
Learning to Compose Neural Networks for Question Answering
Andreas, J., Rohrbach, M., Darrell, T., & Klein, D. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1181 · Cited by 446
Learning to Interpret Natural Language Navigation Instructions from Observations
Chen, D., & Mooney, R. (2011) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v25i1.7974 · Cited by 426
Visual Entailment: A Novel Task for Fine-Grained Image UnderstandingPreprint
Xie, N., Lai, F., Doran, D., & Kadav, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.06706 · Cited by 381
TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments
Chen, H., Suhr, A., Misra, D., Snavely, N., & Artzi, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01282 · Cited by 343
Self-citationNeural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
Yi, K., Wu, J., Gan, C., Torralba, A. B., Kohli, P., & Tenenbaum, J. B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.02338 · Cited by 320
Yin and Yang: Balancing and Answering Binary Visual Questions
Zhang, P., Goyal, Y., Summers-Stay, D., Batra, D., & Parikh, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.542 · Cited by 312
Walk the talk: connecting language, knowledge, and action in route instructions
MacMahon, M., Stankiewicz, B. J., & Kuipers, B. J. (2006) · Cited by 306
A Joint Model of Language and Perception for Grounded Attribute Learning
Matuszek, C., FitzGerald, N., Zettlemoyer, L., Bo, L., & Fox, D. (2012) · Maryland Shared Open Access Repository · doi:10.13016/m2ehu6-1m90 · Cited by 248
An Analysis of Visual Question Answering Algorithms
Kafle, K., & Kanan, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.217 · Cited by 247
Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction
Misra, D. K., Bennett, A., Blukis, V., Niklasson, E., Shatkhin, M., & Artzi, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1287 · Cited by 221
Self-citationTransparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
Mascharka, D., Tran, P., Soklaski, R., & Majumdar, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00519 · Cited by 203 · cited 2×
“Several recent approaches to visual reasoning make use of modular networks (Hu et al., 2017; Johnson et al., 2017b; Mascharka et al., 2018; Perez et al., 2018).”
BackgroundBringing Semantics into Focus Using Visual Abstraction
Zitnick, C. L., & Parikh, D. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.387 · Cited by 193
A Corpus of Natural Language for Visual Reasoning
Suhr, A., Lewis, M., Yeh, J., & Artzi, Y. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p17-2034 · Cited by 175
Self-citationScalable multi-label annotation
Deng, J., Russakovsky, O., Krause, J., Bernstein, M. S., Berg, A., & Fei-Fei, L. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557011 · Cited by 166
Explainable Neural Computation via Stack Neural Module Networks
Hu, R., Andreas, J., Darrell, T., & Saenko, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_4 · Cited by 162
Talk the Walk: Navigating New York City through Grounded DialoguePreprint
de Vries, H., Shuster, K., Batra, D., Parikh, D., Weston, J., & Kiela, D. (2018) · arXiv (Cornell University) · doi:10.4230/lipics.cosit.2022.24 · Cited by 149
Structured Attentions for Visual Question Answering
Zhu, C., Zhao, Y., Huang, S., Tu, K., & Ma, Y. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.145 · Cited by 125
WordNet: a lexical database for English
Miller, G. A. (1992) · DARPA Workshop on Speech and Natural Language (HLT) · doi:10.3115/1075527.1075662 · Cited by 99
Learning Distributions over Logical Forms for Referring Expression Generation
FitzGerald, N., Artzi, Y., & Zettlemoyer, L. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1197 · Cited by 92
Self-citationShapeWorld - A new test methodology for multimodal language understandingPreprint
Kuhnle, A., & Copestake, A. A. (2017) · arXiv.org · doi:10.48550/arxiv.1704.04517 · Cited by 81
GQA: a new dataset for compositional question answering over real-world images.Preprint
Hudson, D. A., & Manning, C. D. (2019) · arXiv (Cornell University) · Cited by 81
C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 DatasetPreprint
Agrawal, A., Kembhavi, A., Batra, D., & Parikh, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.08243 · Cited by 75
TallyQA: Answering Complex Counting Questions
Acharya, M., Kafle, K., & Kanan, C. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018076 · Cited by 69
A Survey of Current Datasets for Vision and Language Research
Ferraro, F., Mostafazadeh, N., Huang, T.-H., Vanderwende, L., Devlin, J., Galley, M., & Mitchell, M. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1021 · Cited by 68
Weakly Supervised Semantic Parsing with Abstract Examples
Goldman, O., Latcinnik, V., Naveh, U., Globerson, A., & Berant, J. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1168 · Cited by 64
A Dataset and Architecture for Visual Reasoning with a Working Memory
Yang, G. R., Ganichev, I., Wang, X., Shlens, J., & Sussillo, D. (2018) · Conference on Cognitive Computational Neuroscience (CCN) · doi:10.1007/978-3-030-01249-6_44 · Cited by 48
Natural reference to objects in a visual domain
Mitchell, M., van Deemter, K., & Reiter, E. (2010) · International Conference on Natural Language Generation · Cited by 41
Using Syntax to Ground Referring Expressions in Natural ImagesPreprint
Cirik, V., Berg-Kirkpatrick, T., & Morency, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.10547 · Cited by 39
DDRprog: A CLEVR Differentiable Dynamic Reasoning ProgrammerPreprint
Suárez, J., Johnson, J., & Fei-Fei, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.11361 · Cited by 32
Learning to Disambiguate by Asking Discriminative Questions
Li, Y., Huang, C., Tang, X., & Loy, C. C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.370 · Cited by 26
“Caption” as a Coherence Relation: Evidence and Implications
Alikhani, M., & Stone, M. (2019) · doi:10.18653/v1/w19-1806 · Cited by 24
Towards a Dataset for Human Computer Communication via Grounded Language Acquisition
Bisk, Y., Marcu, D., & Wong, W. (2016) · National Conference on Artificial Intelligence · Cited by 23
Working Memory Networks: Augmenting Memory Networks with a Relational Reasoning Module
Pavez, J., Allende, H., & Allende‐Cid, H. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1092 · Cited by 22
Cascaded Mutual Modulation for Visual Reasoning
Yao, Y., Xu, J., Wang, F., & Xu, B. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1118 · Cited by 14
Learning Visual Question Answering by Bootstrapping Hard AttentionPreprint
Malinowski, M., Doersch, C., Santoro, A., & Battaglia, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.00300 · Cited by 13
Effectively Crowdsourcing Radiology Report Annotations
Cocos, A., Masino, A. J., Qian, T., Pavlick, E., & Callison-Burch, C. (2015) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/w15-2614 · Cited by 11
Object Ordering with Bidirectional Matchings for Visual Reasoning
Tan, H. X., & Bansal, M. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2071 · Cited by 11
Binary Image Selection (BISON): Interpretable Evaluation of Visual Grounding.Preprint
Hu, H., Misra, I., & van der Maaten, L. J. P. (2019) · arXiv (Cornell University) · Cited by 7
Cited by
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055
Li, J., Li, D., Xiong, C., & Hoi, S. (2022) · arXiv · doi:10.48550/arxiv.2201.12086 · Cited by 1,741
Metadata
- Authors
- Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H., & Artzi, Y.
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 6418-6428
- OpenAlex
- W2899335602
- Cited by
- 448
- References
- 68
- FWCI
- 18.55
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.