Visual7W: Grounded Question Answering in Images
Open accessAbstract
We have seen great progress in basic perceptual tasks such as object recognition and detection. However, AI models still fail to match humans in high-level vision tasks due to the lack of capacities for deeper reasoning. Recently the new task of visual question answering (QA) has been proposed to evaluate a model's capacity for deep image understanding. Previous works have established a loose, global association between QA sentences and images. However, many questions and answers, in practice, relate to local regions in the images. We establish a semantic link between textual descriptions and image regions by object-level grounding. It enables a new type of QA with visual answers, in addition to textual answers used in previous work. We study the visual QA tasks in a grounded setting with a large collection of 7W multiple-choice QA pairs. Furthermore, we evaluate human performance and several baseline models on the QA tasks. Finally, we propose a novel LSTM model with spatial attention to tackle the 7W QA tasks.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most currently available datasets for training question generation systems are crowd-sourced, which largely affects the quality of generated questions.
“Most datasets that are currently available for training question generation systems are crowd-sourced [42, 47, 84, 92, 114, 115, 119, 120, 122, 123], which largely impacts the quality of generated questions.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [120]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1× · see citing works
“The recent development of new technologies, especially deep learning, has achieved successes in many perceptual visual tasks such as object recognition, image classification and pose estimation [15, 18, 22, 35, 36, 39, 40].”
BackgroundHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1× · see citing works
“LSTM models [11] have achieved state-of-the-art results in several sequence processing tasks [5, 13, 41].”
BackgroundAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 1× · see citing works
“Yet the status quo of computer vision is still far from matching human capabilities, especially when it comes to understanding an image in all its details.”
BackgroundSelf-citationRich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1× · see citing works
“The questions are represented by 200dimensional averaged word embeddings from a pre-trained model [29].”
BackgroundSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1× · see citing works
“LSTM models [11] have achieved state-of-the-art results in several sequence processing tasks [5, 13, 41].”
BackgroundShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542 · cited 2× · see citing works
“The recent development of deep learning technologies has achieved successes in many perceptual visual tasks such as object recognition, image classification and pose estimation [15, 21, 25, 38, 39, 42, 43].”
BackgroundLarge-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · cited 2× · see citing works
“The recent development of deep learning technologies has achieved successes in many perceptual visual tasks such as object recognition, image classification and pose estimation [15, 21, 25, 38, 39, 42, 43].”
BackgroundSelf-citationShow and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 2× · see citing works
“Pioneer work in image captioning [4, 5, 13, 45, 47], sentence-based image retrieval [14, 40] and visual QA [1, 7, 36] shows promising results.”
BackgroundLong-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · cited 5× · see citing works
“We elaborate on the details of the data collection we conducted upon 47,300 images from COCO [25] (a subset of images from Visual Genome [20]).”
MethodInfluentialSelf-citationDeep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
Self-citationVQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
DeepPose: Human Pose Estimation via Deep Neural Networks
Toshev, A., & Szegedy, C. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.214 · Cited by 3,288 · cited 1× · see citing works
“The recent development of deep learning technologies has achieved successes in many perceptual visual tasks such as object recognition, image classification and pose estimation [15, 21, 25, 38, 39, 42, 43].”
Background(2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2,534 · see citing works
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · cited 2× · see citing works
“However, as Flickr30K [34, 48] and Visual Madlibs [49] Which paw is lifted? What is the dog doing? Why is there foam?”
BackgroundLearning to detect unseen object classes by between-class attribute transfer
Lampert, C. H., Nickisch, H., & Harmeling, S. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206594 · Cited by 1,849 · see citing works
Learning to detect unseen object classes by between-class attribute transfer
Lampert, C. H., Nickisch, H., & Stefan Harmeling, B. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206594 · Cited by 1,563 · see citing works
Building Watson: An Overview of the DeepQA Project
Ferrucci, D. A., Brown, E. W., Chu‐Carroll, J., Fan, J., Gondek, D. C., Kalyanpur, A. A., Lally, A., Murdock, J. W., Nyberg, E., Prager, J. M., et al. (2010) · AI Magazine · doi:10.1609/aimag.v31i3.2303 · Cited by 1,523 · cited 1× · see citing works
“Traditional question answering system relies on an elaborate pipeline of models involving natural language parsing, knowledge base querying, and answer generation [6].”
BackgroundFlickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444 · see citing works
DRAW: A Recurrent Neural Network For Image GenerationPreprint
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.04623 · Cited by 1,394 · cited 1× · see citing works
“We add spatial attention [10, 47] to the standard LSTM model for visual QA, illustrated in Fig.”
MethodLarge-scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G. D., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · Cited by 1,266 · see citing works
Self-citationReferItGame: Referring to Objects in Photographs of Natural Scenes
Kazemzadeh, S., Ordóñez, V., Matten, M., & Berg, T. L. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1086 · Cited by 1,082 · cited 1× · see citing works
“Inspired by Geman et al. [8], we establish the link by grounding objects in the images and perform experiments in the grounded QA setting.”
MethodLong-term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. (2014) · doi:10.21236/ada623249 · Cited by 1,081 · see citing works
VQA: Visual Question AnsweringPreprint
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Batra, D., & Parikh, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00468 · Cited by 968 · cited 9× · see citing works
“’s prototype of a visual Turing test based on image regions [8] and the comprehensive data collection of QA pairs on COCO images [25] such as VQA [1] and Baidu [7], we fuse visual QA and grounding in order to create a new QA dataset with dense annotations and a more flexible evaluation environment.”
BackgroundMethodInfluentialSUN attribute database: Discovering, annotating, and recognizing scene attributes
Patterson, G., & Hays, J. H. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247998 · Cited by 878 · cited 1× · see citing works
“The 7W questions roughly correspond to an array of standard vision tasks: what [9, 15, 39], where [24, 50], when [30, 32], who [35, 42], why [33], how [23, 31] and which [16, 17].”
BackgroundDeep Fragment Embeddings for Bidirectional Image Sentence MappingPreprint
Karpathy, A., Joulin, A., & Fei-Fei, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.5679 · Cited by 802 · cited 1× · see citing works
“Pioneer work in image captioning [4, 5, 13, 45, 47], sentence-based image retrieval [14, 40] and visual QA [1, 7, 36] shows promising results.”
BackgroundDRAW: A Recurrent Neural Network For Image Generation
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015) · Cited by 785 · see citing works
Grounded Compositional Semantics for Finding and Describing Images with Sentences
Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., & Ng, A. Y. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00177 · Cited by 779 · cited 2× · see citing works
“Pioneer work in image captioning [4, 5, 13, 45, 47], sentence-based image retrieval [14, 40] and visual QA [1, 7, 36] shows promising results.”
BackgroundTowards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015) · arXiv · doi:10.48550/arxiv.1502.05698 · Cited by 670 · cited 3× · see citing works
“Table 2 compares Visual7W with VQA [1] and Facebook bAbI [46], which have reported model and human performances.”
BackgroundAsk Your Neurons: A Neural-Based Approach to Answering Questions about Images
Malinowski, M., Rohrbach, M., & Fritz, M. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.9 · Cited by 633 · see citing works
Exploring Models and Data for Image Question AnsweringPreprint
Ren, M., Kiros, R., & Zemel, R. S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.02074 · Cited by 618 · cited 8× · see citing works
“Several QA datasets [1, 7, 27, 36, 49] have been released since last year.”
BackgroundMethodInfluentialVizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866080 · Cited by 601 · cited 1× · see citing works
“Visual question answering is a natural extension to the captioning tasks, but is more interactive and has a stronger connection to real-world applications [3].”
BackgroundBigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A. E., Miller, R. C., Tatarowicz, A., White, B., White, S., & YEH, T. P. (2010) · International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806020 · Cited by 536 · see citing works
Mind's eye: A recurrent visual representation for image caption generation
Chen, X., & Zitnick, C. L. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298856 · Cited by 496 · see citing works
Barnard, K., Duygulu, P., Forsyth, D. A., de Freitas, N., Blei, D. M., Jordan, M. I., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2003) · Cited by 459 · see citing works
Where to Buy It: Matching Street Clothing Photos in Online Shops
Kiapour, M. H., Han, X., Lazebnik, S., Berg, A. C., & Berg, T. L. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.382 · Cited by 447 · cited 1× · see citing works
“The 7W questions roughly correspond to an array of standard vision tasks: what [9, 15, 39], where [24, 50], when [30, 32], who [35, 42], why [33], how [23, 31] and which [16, 17].”
BackgroundAre You Talking to a Machine? Dataset and Methods for Multilingual Image Question AnsweringPreprint
Gao, H., Mao, J., Zhou, J., Huang, Z., Wang, L., & Xu, W. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.05612 · Cited by 437 · cited 11× · see citing works
“Inspired by Geman et al.’s prototype of a visual Turing test based on image regions [8] and the comprehensive data collection of QA pairs on COCO images [25] such as VQA [1] and Baidu [7], we fuse visual QA and grounding in order to create a new QA dataset with dense annotations and a more flexible evaluation environme…”
BackgroundMethodInfluentialA Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain InputPreprint
Malinowski, M., & Fritz, M. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0210 · Cited by 432 · cited 5× · see citing works
“The DAQUAR dataset is the first toy-sized QA benchmark built upon indoor scene RGB-D images.”
BackgroundInfluentialHarvey, D., & Moran, S. (2009) · Cited by 429 · see citing works
Learning deep representations for ground-to-aerial geolocalization
Lin, T.-Y., Cui, Y., Belongie, S., & Hays, J. H. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299135 · Cited by 409 · cited 1× · see citing works
“The 7W questions roughly correspond to an array of standard vision tasks: what [9, 15, 39], where [24, 50], when [30, 32], who [35, 42], why [33], how [23, 31] and which [16, 17].”
BackgroundTranslating Video Content to Natural Language Descriptions
Rohrbach, M., Qiu, W., Titov, I., Thater, S., Pinkal, M., & Schiele, B. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.61 · Cited by 384 · cited 1× · see citing works
“Image and video captioning has become a popular task in the past year [4, 5, 13, 37, 45, 47].”
BackgroundVisual Turing test for computer vision systems
Geman, D., Geman, S., Hallonquist, N., & Laurent Younes, B. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1422953112 · Cited by 323 · see citing works
A Neural Network for Factoid Question Answering over Paragraphs
Iyyer, M., Boyd-Graber, J. L., Claudino, L., Socher, R., & Daumé, H. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1070 · Cited by 308 · cited 1× · see citing works
“Recent neural network models attempt to learn end-to-end directly from questions and answers [12, 46].”
BackgroundTowards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2016) · International Conference on Learning Representations · Cited by 261 · see citing works
Learning to Answer Questions from Image Using Convolutional Neural Network
Ma, L., Lu, Z., & Li, H. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10442 · Cited by 260 · cited 1× · see citing works
“The proposed models range from probabilistic inference [27, 44, 51] and recurrent neural networks [1, 7, 28, 36] to convolutional networks [26].”
BackgroundVisual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsPreprint
Krishna, R., Zhu, Y., Groth, O., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.07332 · Cited by 248 · cited 5× · see citing works
“We elaborate on the details of the data collection we conducted upon 47,300 images from COCO [25] (a subset of images from Visual Genome [20]).”
MethodInfluentialSelf-citationReasoning about Object Affordances in a Knowledge Base Representation
Zhu, Y., Fathi, A., & Fei-Fei, L. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10605-2_27 · Cited by 210 · cited 1× · see citing works
“The proposed models range from probabilistic inference [27, 44, 51] and recurrent neural networks [1, 7, 28, 36] to convolutional networks [26].”
BackgroundSelf-citationWhat Are You Talking About? Text-to-Image Coreference
Kong, C., Lin, D., Bansal, M., Urtasun, R., & Fidler, S. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.455 · Cited by 191 · cited 3× · see citing works
“The object-level groundings enable examining the object distributions and resolve the coreference ambiguity [19, 35].”
BackgroundLearning the Visual Interpretation of Sentences
Zitnick, C. L., Parikh, D., & Vanderwende, L. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.211 · Cited by 156 · cited 1× · see citing works
“There have been years of effort in connecting the visual and textual information for joint learning [2, 19, 33, 35, 40, 52].”
BackgroundJoint Video and Text Parsing for Understanding Events and Answering Queries
Tu, K., Meng, M., Lee, M. W., Choe, T. E., & Zhu, S. C. (2014) · IEEE Multimedia · doi:10.1109/mmul.2014.29 · Cited by 113 · cited 1× · see citing works
“The proposed models range from probabilistic inference [27, 44, 51] and recurrent neural networks [1, 7, 28, 36] to convolutional networks [26].”
BackgroundPolitical Journalism: New Challenges, New Practices
Kuhn, R., & Neveu, É. (2003) · Cited by 106 · see citing works
Dating Historical Color Images
Palermo, F., Hays, J. H., & Efros, A. A. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33783-3_36 · Cited by 93 · cited 1× · see citing works
“The 7W questions roughly correspond to an array of standard vision tasks: what [9, 15, 39], where [24, 50], when [30, 32], who [35, 42], why [33], how [23, 31] and which [16, 17].”
BackgroundPickup, L., Zheng, P., Wei, D., Shih, Y., Zhang, C., Zisserman, A., Schölkopf, B., & Freeman, W. T. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.262 · Cited by 80 · cited 1× · see citing works
“The 7W questions roughly correspond to an array of standard vision tasks: what [9, 15, 39], where [24, 50], when [30, 32], who [35, 42], why [33], how [23, 31] and which [16, 17].”
BackgroundVisual Madlibs: Fill in the blank Image Generation and Question AnsweringPreprint
Yu, L., Park, E., Berg, A. C., & Berg, T. L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00278 · Cited by 75 · see citing works
Pirsiavash, H., Vondrick, C., & Torralba, A. B. (2014) · doi:10.21236/ada612444 · Cited by 31 · see citing works
Flagged first, then by citations
1 of the 61 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
“Typically, these models can be characterized as top-down approaches, with context provided by a representation of a partially-completed caption in the case of image captioning [33, 27, 47, 45], or a representation of the question in the case of VQA [11, 28, 44, 46, 49].”
BackgroundMethodWang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · see citing works
“Deep neural network representations further make the joint language and vision learning tasks more feasible to solve, in image captioning [47, 23, 32, 46, 22], visual question answering [2, 45, 49, 53] and knowledge-guided transfer learning [4, 33], and so on.”
BackgroundInfluentialLXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · see citing works
“, 2015), GQA balanced version (Hudson and Manning, 2019), and VG-QA (Zhu et al., 2016).”
BackgroundAnchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · see citing works
“We explain predictions from the Visual7W openended VQA system (Zhu et al. 2016) using Figure 3a as the reference image.”
MethodSCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning
Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., & Chua, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.667 · Cited by 2,073 · see citing works
“[43] adopted the “soft” attention to merge image region features.”
MethodCLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781 · see citing works
Self-citationGQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
Hudson, D. A., & Manning, C. D. (2019) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2019.00686 · Cited by 3,703 · see citing works
“popular across the board, with a host of datasets being constructed [3, 9, 13, 36, 17] and numerous models being proposed [4, 33, 5, 8, 10].”
BackgroundMultimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · see citing works
“We use the splits from (Zhu et al., 2016) to train, validate, and test our models.”
MethodInfluentialHierarchical Question-Image Co-Attention for Visual Question AnsweringPreprint
Lu, J., Yang, J., Batra, D., & Parikh, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00061 · Cited by 1,303 · see citing works
“[26] add spatial attention to the standard LSTM model for pointing and grounded QA.”
BackgroundVL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019) · arXiv · doi:10.48550/arxiv.1908.08530 · Cited by 950 · see citing works
“l commonsense reasoning 3) masked visual-feature classification 3) grounding referring expressions zLXMERT is pre-trained on COCO Caption (Chen et al., 2015), VG Caption (Krishna et al., 2017), VG QA (Zhu et al., 2016), VQA (Antol et al., 2015) and GQA (Hudson & Manning, 2019). Table 1: Comparison among our VL-BERT…”
ResultFrom Recognition to Cognition: Visual Commonsense Reasoning
Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00688 · Cited by 769 · see citing works
“This line of work also includes ‘pointing’ questions [45, 93] and templated questions with open ended answers [86].”
BackgroundAsk, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
Xu, H., & Saenko, K. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_28 · Cited by 753 · see citing works
“[25] proposes a new dataset and uses a similar attention model to that in image captioning [19], but does not give results on the more common VQA benchmark [2], and our own implementation of this model is less accurate on [2] than other baseline models.”
ResultTowards VQA Models That Can Read
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., & Rohrbach, M. (2019) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2019.00851 · Cited by 2,473 · see citing works
“VQA models typically use some variant of attention to get a representation of the image that is relevant for answering the question [2, 7, 30, 47, 48, 51, 17].”
BackgroundMUTAN: Multimodal Tucker Fusion for Visual Question Answering
Ben-younes, H., Cadène, R., Cord, M., & Thome, N. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.285 · Cited by 733 · see citing works
“A specific benchmark has been first proposed [19], and large scale datasets have been recently collected [21, 2, 31], enabeling the development of more powerful models.”
BackgroundOK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
Marino, K., Rastegari, M., Farhadi, A., & Mottaghi, R. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00331 · Cited by 748 · see citing works
“The idea of Visual 7W [47] is to provide object-level grounding for question-answer pairs as opposed to image-level associations between images and QA pairs.”
BackgroundInfluentialVizWiz Grand Challenge: Answering Visual Questions from Blind People
Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., & Bigham, J. P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00380 · Cited by 621 · see citing works
“0: Real [17] MSCOCO [25] Crowd workers (AMT) Typed Visual7W [45] MSCOCO [25] Crowd workers (AMT) Typed CLEVR [19] Synthetic Shapes Automatically generated (templates) ——– SHAPES [6] Synthetic Shapes Automatically generated (templates) ——– Visual Genome [23] MSCOCO [25] & YFCC100M [35] Crowd workers (AMT) Typed FVQA [3…”
BackgroundMethodInfluentialDon't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
Agrawal, A., Batra, D., Parikh, D., & Kembhavi, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00522 · Cited by 586 · see citing works
Picón, A., Álvarez-Gila, A., Seitz, M., Ortiz‐Barredo, A., Echazarra, J., & Johannes, A. (2018) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2018.04.002 · Cited by 601 · see citing works
“Visual Question Answering (Zhu et al. (2016)) or Dense Captioning (Johnson et al.”
BackgroundInfluentialWang, X., Peng, Y., Lü, L., Lu, Z., & Summers, R. M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00943 · Cited by 574 · see citing works
TVQA: Localized, Compositional Video Question Answering
Lei, J., Yu, L., Bansal, M., & Berg, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1167 · Cited by 541 · see citing works
“Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al., 2017), etc. Addit…”
BackgroundInfluentialMultimodal intelligence: Representation learning, information fusion, and applications
Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511 · see citing works
“However, it is often simplified as a classification task defined in different ways to focus on different core problems [46], [47], [150], [235], [236].”
BackgroundMaking the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., & Parikh, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1116-0 · Cited by 487 · see citing works
“For example, in another popular VQA dataset Visual7W [62], a question-only baseline achieves an accuracy of 46.”
BackgroundCAD-Net: A Context-Aware Detection Network for Objects in Remote Sensing Imagery
Zhang, G., Lu, S., & Zhang, W. (2019) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2930982 · Cited by 497 · see citing works
“For example, [39] uses visual attention to determine the most relevant regions in image captioning, [40] uses visual attention for feature fusion for visual question answering, [41] com-”
BackgroundA dataset of clinically generated visual questions and answers about radiology images
Lau, J. J., Gayen, S., Abacha, A. B., & Demner‐Fushman, D. (2018) · Scientific Data · doi:10.1038/sdata.2018.251 · Cited by 496 · see citing works
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
Jang, Y., Song, Y., Yu, Y., Kim, Y., & Kim, G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.149 · Cited by 496 · see citing works
“Most existing VQA datasets are image-centric, e.g ., DAQUAR [25], abstract scenes [24], VQA [2], Visual Madlibs [41], DAQUAR-Consensus [26], FM-IQA [11], COCO-QA [29], and Visual7W [43].”
BackgroundInfluentialFVQA: Fact-Based Visual Question Answering
Wang, P., Wu, Q., Shen, C., Dick, A., & van den Hengel, A. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2754246 · Cited by 490 · see citing works
The multi-modal fusion in visual question answering: a review of attention mechanisms
Lu, S., Liu, M., Yin, L., Yin, Z., Liu, X., & Zheng, W. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1400 · Cited by 478 · see citing works
“Given a picture and a natural language question including which, when, where, who, what, how and why ( Zhu et al., 2016 ) that is correlative with this picture, the system need to produce a natural language answer as an output.”
BackgroundInferring and Executing Programs for Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Hoffman, J., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.325 · Cited by 487 · see citing works
“Like the CLEVR dataset, benchmark datasets for VQA typically comprise a set of questions on images with associated answers [3, 31, 39, 24, 50]; both questions and answers are generally posed in natural language.”
BackgroundSelf-citationGraph-Structured Representations for Visual Question Answering
Teney, D., Liu, L., & van den Hengel, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.344 · Cited by 464 · see citing works
“A major improvement to the basic method is to use an attention mechanism [31, 27, 5, 12, 3, 28].”
BackgroundMethodSemantically Equivalent Adversarial Rules for Debugging NLP models
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1079 · Cited by 445 · see citing works
“, 2017), and visual question answering (Zhu et al., 2016), models are becoming increasingly challenging to debug, and to determine whether they are ready for deployment.”
BackgroundMethodInfluentialVisual question answering: A survey of methods and datasets
Wu, Q., Teney, D., Wang, P., Shen, C., Dick, A., & van den Hengel, A. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.05.001 · Cited by 442 · see citing works
“The Visual7w [100] dataset is a subset of the Visual Genome that contains additional annotations.”
BackgroundMethodResultInfluentialDas, A., Kottur, S., Gupta, K., Singh, A., Jain, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · Cited by 432 · see citing works
A Fast and Accurate One-Stage Approach to Visual Grounding
Yang, Z., Gong, B., Wang, L., Huang, W., Yu, D., & Luo, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00478 · Cited by 431 · see citing works
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence
Park, D. H., Hendricks, L. A., Akata, Z., Rohrbach, A., Schiele, B., Darrell, T., & Rohrbach, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00915 · Cited by 418 · see citing works
Attention mechanism in neural networks: where it comes and where it goes
Soydaner, D. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07366-3 · Cited by 441 · see citing works
Attention-Based Dropout Layer for Weakly Supervised Object Localization
Choe, J., & Shim, H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00232 · Cited by 412 · see citing works
“This process is called attention mechanism and is actively used in various fields such as machine translation [43], image captioning [55], image inpainting [56, 23], transfer learning [57], visual question answering [64], and generative model [28, 60].”
Background12-in-1: Multi-Task Vision and Language Representation Learning
Lu, J., Goswami, V., Rohrbach, M., Parikh, D., & Lee, S. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01045 · Cited by 410 · see citing works
Adversarial Attacks on Deep-learning Models in Natural Language Processing
Zhang, W. E., Sheng, Q. Z., Alhazmi, A., & Li, C. (2020) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3374217 · Cited by 617 · see citing works
“…[34] Enron Spam, Datasets from [156] Gender Identification [117] Twitter Gender Image Captioning [20, 148] MSCOCO, Visual Genome Visual Question Answering [148] Datasets from [6], Datasets from [158] Visual-Semantic Embedding [125] MSCOCO Speech Tecognition [18] Mozilla Common Voice and negative).”
MethodFloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding
Rahnemoonfar, M., Chowdhury, T., Sarkar, A., Varshney, D., Yari, M., & Murphy, R. R. (2021) · IEEE Access · doi:10.1109/access.2021.3090981 · Cited by 422 · see citing works
Image Captioning and Visual Question Answering Based on Attributes and External Knowledge
Wu, Q., Shen, C., Wang, P., Dick, A., & van den Hengel, A. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2708709 · Cited by 408 · see citing works
“Inspired by Xu et al. [39] who encode visual attention in the Image Captioning, [46], [47], [48], [49], [50], [51] propose to use the spatial attention to help answering visual questions.”
BackgroundTips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge
Teney, D., Anderson, P., He, X., & van den Hengel, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00444 · Cited by 398 · see citing works
Forecasting Stock Prices from the Limit Order Book Using Convolutional Neural Networks
Tsantekidis, A., Passalis, N., Tefas, A., Kanniainen, J., Gabbouj, M., & Iosifidis, A. (2017) · IEEE Conference on Business Informatics · doi:10.1109/cbi.2017.23 · Cited by 394 · see citing works
“Deep Learning has been shown to significantly improve upon previous machine learning methods in tasks such as speech recognition [5], image captioning [6], [7], and question answering [8].”
BackgroundModeling Relationships in Referential Expressions with Compositional Modular Networks
Hu, R., Rohrbach, M., Andreas, J., Darrell, T., & Saenko, K. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.470 · Cited by 386 · see citing works
““which” questions) in visual question answering on the Visual-7W dataset [34].”
BackgroundMethodResultInfluentialWang, X., Peng, Y., Lü, L., Lu, Z., Bagheri, M. H., & Summers, R. M. (2019) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-030-13969-8_18 · Cited by 377 · see citing works
Hierarchical Question-Image Co-Attention for Visual Question Answering
Lu, J. (2024) · TIB Data Manager · doi:10.57702/uhdmfu1r · Cited by 357 · see citing works
VirTex: Learning Visual Representations from Textual Annotations
Desai, K., & Johnson, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01101 · Cited by 350 · see citing works
“Vision-and-Language Pretraining attempts to learn joint representations of image-text paired data that can be transferred to multimodal downstream tasks such as visual question answering [13, 14, 59, 60], visual reasoning [61, 62], referring expressions [63], and language-based image retrieval [35].”
Background3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera
Armeni, I., He, Z.-Y., Zamir, A., Gwak, J., Malik, J., Fischer, M., & Savarese, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00576 · Cited by 350 · see citing works
“Examples are on scene graph generation [30, 46], image cap-tioning/description [26, 3, 23], image retrieval [24] and visual question-answering [17, 51].”
BackgroundTemporally Grounding Natural Sentence in Video
Chen, J., Chen, X., Ma, L., Jie, Z., & Chua, T. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1015 · Cited by 330 · see citing works
“Previous approaches on multimodal interactions were limited to concatenation (Zhu et al., 2016), element-wise product or sum (Gao et al., 2017), and bilinear pooling (Fukui et al., 2016).”
Visual Attention Model for Name Tagging in Multimodal Social Media
Lu, D., Neves, L., Carvalho, V., Zhang, N., & Ji, H. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1185 · Cited by 324 · see citing works
“…(Bahdanau et al., 2014), it has been widely adopted to language and vision re-lated tasks, such as Image Captioning and Visual Question Answering (VQA), by retrieving the visual features most related to text context (Zhu et al., 2016; Anderson et al., 2017; Xu and Saenko, 2016; Chen et al., 2015).”
BackgroundAttention, please! A survey of neural attention models in deep learning
Correia, A. D. S., & Colombini, E. L. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10148-x · Cited by 330 · see citing works
A Diagram is Worth a Dozen Images
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_15 · Cited by 290 · see citing works
“Recently, answering questions about real images has drawn much attention in both NLP and vision communities [22,23,24].”
BackgroundA-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge
Schwenk, D., Khandelwal, A., Clark, C., Marino, K., & Mottaghi, R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20074-8_9 · Cited by 295 · see citing works
Generating Natural Questions About an Image
Mostafazadeh, N., Misra, I., Devlin, J., Mitchell, M., He, X., & Vanderwende, L. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1170 · Cited by 270 · see citing works
“A recently published work on VQA, Visual7W (Zhu et al., 2016), establishes a grounding link on the object regions corresponding to the textual answer.”
BackgroundIdentity-Aware Textual-Visual Matching with Latent Co-attention
Li, S., Xiao, T., Li, H., Yang, W. P., & Wang, X. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.209 · Cited by 288 · see citing works
“It gains a lot of attention in recent years because of its wide applications in image captioning [20, 31, 11, 5], visual QA [3, 41, 24, 19], and textimage embedding [7, 26, 12, 35, 32].”
BackgroundTowards artificial general intelligence via a multimodal foundation model
Fei, N., Lu, Z., Gao, Y., Yang, G., Huo, Y., Wen, J., Lu, H., Song, R., Gao, X., Xiang, T., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-30761-2 · Cited by 296 · see citing works
Recurrent Multimodal Interaction for Referring Image Segmentation
Liu, C., Lin, Z., Shen, X., Yang, J., Lu, X., & Yuille, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.143 · Cited by 278 · see citing works
“also adopted by recent methods on referring object localization [38] and visual question answering tasks [1].”
MethodKimera: From SLAM to spatial perception with 3D dynamic scene graphs
Rosinol, A., Violette, A., Abate, M., Hughes, N., Chang, Y., Shi, J., Gupta, A., & Carlone, L. (2021) · Int. J. Robotics Res. · doi:10.1177/02783649211056674 · Cited by 418 · see citing works
“2018), visual questionanswering (Fukui et al. 2016; Zhu et al. 2016), and action detection (Lu et al.”
BackgroundKembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., & Hajishirzi, H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.571 · Cited by 244 · see citing works
“released the Visual7W dataset [30] which contained multiple choice visual answers in addition to textual answers.”
BackgroundMethodUnifying Vision-and-Language Tasks via Text GenerationPreprint
Cho, J., Lei, J., Tan, H. X., & Bansal, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.02779 · Cited by 268 · see citing works
“, 2019), GQA balanced version (Hudson & Manning, 2019), and Visual7W (Zhu et al., 2016)) as in Tan & Bansal (2019), but only used them for the visual question answering task.”
MethodMotion-Appearance Co-memory Networks for Video Question Answering
Gao, J., Ge, R., Chen, K., & Nevatia, R. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00688 · Cited by 252 · see citing works
Adversarial Attacks on Deep Learning Models in Natural Language Processing: A SurveyPreprint
Zhang, W. E., Sheng, Q. Z., Alhazmi, A., & Li, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.06796 · Cited by 253 · see citing works
An Analysis of Visual Question Answering Algorithms
Kafle, K., & Kanan, C. (2017) · IEEE International Conference on Computer Vision · doi:10.1109/iccv.2017.217 · Cited by 272 · see citing works
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2020) · International Conference on Learning Representations · Cited by 276 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and VQA.Preprint
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2017) · arXiv (Cornell University) · Cited by 252 · see citing works
Visual question answering: Datasets, algorithms, and future challenges
Kafle, K., & Kanan, C. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.06.005 · Cited by 245 · see citing works
“Free form QA: What does the sky look like? Region based QA: What color is the horse? (b) Example of the pointing QA task in Visual7W [34].”
BackgroundMethodCLEVRER: CoLlision Events for Video REpresentation and ReasoningPreprint
Yi, K., Gan, C., Li, Y., Kohli, P., Wu, J., Torralba, A. B., & Tenenbaum, J. B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.01442 · Cited by 283 · see citing works
“…visual reasoning has been widely studied in artificial intelligence and computer vision, driven by the introduction of various datasets on both static images (Antol et al., 2015; Zhu et al., 2016; Hudson & Manning, 2019) and videos (Jang et al., 2017; Tapaswi et al., 2016; Zadeh et al., 2019).”
MethodWide-Slice Residual Networks for Food Recognition
Martinel, N., Foresti, G. L., & Micheloni, C. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00068 · Cited by 233 · see citing works
ViP-CNN: Visual Phrase Guided Convolutional Neural Network
Li, Y., Ouyang, W., Wang, X., & Tang, X. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.766 · Cited by 233 · see citing works
“With the research on object detection gradually becoming mature [37, 36, 32, 24, 22, 23], increasingly more researchers put their attention on higher-level understanding of the scene [21, 51, 2, 48, 49, 46, 9, 6, 7, 47].”
BackgroundMulti-level Attention Networks for Visual Question Answering
Yu, D., Fu, J., Mei, T., & Rui, Y. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.446 · Cited by 233 · see citing works
“Visual7W is a more recent VQA dataset built by [37], which is a subset of Visual Genome [3] (the largest visual QA dataset to date with 1.”
BackgroundMethodInfluentialWinoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., & Ross, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00517 · Cited by 240 · see citing works
“Unlike Winoground, these stress tests tend to come from existing datasets that have images and text from typical training domains, such as Conceptual Captions [63], COCO [48], Visual7W [93] and VQA [3, 27].”
BackgroundRevisiting Visual Question Answering Baselines
Jabri, A., Joulin, A., & van der Maaten, L. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46484-8_44 · Cited by 235 · see citing works
HA-CCN: Hierarchical Attention-Based Crowd Counting Network
Sindagi, V. A., & Patel, V. M. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2928634 · Cited by 216 · see citing works
“[70] employed soft attention to combine image region features for the task of VQA.”
MethodStimulus-driven and concept-driven analysis for image caption generation
Ding, S., Qu, S., Xi, Y., & Wan, S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.04.095 · Cited by 214 · see citing works
Toward the third generation artificial intelligence
Zhang, B., Zhu, J., & Su, H. (2023) · Science China Information Sciences · doi:10.1007/s11432-021-3449-x · Cited by 217 · see citing works
“More work can be referred to in [92,93].”
BackgroundLXMERT: Learning Cross-Modality Encoder Representations from TransformersPreprint
Tan, H. X., & Bansal, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.07490 · Cited by 223 · see citing works
“, 2015), GQA balanced version (Hudson and Manning, 2019), and VG-QA (Zhu et al., 2016).”
Background3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans
Rosinol, A., Gupta, A., Abate, M., Shi, J., & Carlone, L. (2020) · Robotics: Science and Systems · doi:10.15607/rss.2020.xvi.079 · Cited by 214 · see citing works
Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
Mascharka, D., Tran, P., Soklaski, R., & Majumdar, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00519 · Cited by 203 · see citing works
“Many authors [1, 4, 6, 12, 17, 18, 26, 27, 23, 28, 33] use a spatial softmax to compute attention weights.”
How Much Can CLIP Benefit Vision-and-Language Tasks?Preprint
Shen, S., Li, L. H., Tan, H. X., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., & Keutzer, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06383 · Cited by 272 · see citing works
“Following LXMERT, we use the same corpora aggregated from MS COCO Cap-tions (Chen et al., 2015), Visual Genome Cap-tions (Krishna et al., 2017), VQA (Antol et al., 2015), GQA (Hudson and Manning, 2019), and VG-QA (Zhu et al., 2016) for pre-training.”
MethodAttention Correctness in Neural Image Captioning
Liu, C., Mao, J., Sha, F., & Yuille, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11197 · Cited by 210 · see citing works
Dynamic Multimodal Instance Segmentation Guided by Natural Language Queries
Margffoy-Tuay, E. A., Pérez, J. C., Botero, E., & Arbeláez, P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_39 · Cited by 192 · see citing works
Improving the Reliability of Deep Neural Networks in NLP: A Review
Alshemali, B., & Kalita, J. K. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.105210 · Cited by 192 · see citing works
“They also tested the model of Zhu et al.’s [74] using their dataset.”
TVQA+: Spatio-Temporal Grounding for Video Question Answering
Lei, J., Yu, L., Berg, T. L., & Bansal, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.730 · Cited by 187 · see citing works
RAVEN: A Dataset for Relational and Analogical Visual REasoNing
Zhang, C., Gao, F., Jia, B., Zhu, Y., & Zhu, S. C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00546 · Cited by 189 · see citing works
“Unlike VQA, RPM lies directly at the center of human intelligence [7], is diagnostic of abstract and structural reasoning ability [9], and characterizes the defining feature of high-level intelligence, i.e ., fluid intelligence [21].”
BackgroundInfluentialSelf-citationOut of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question AnsweringPreprint
Narasimhan, M., Lazebnik, S., & Schwing, A. G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00538 · Cited by 184 · see citing works
“Therefore, numerous approaches address VQA, VQG and dialog tasks by extracting visual cues using deep network architectures [1,2,11,16,18,19,24,32,33,35,41,44,53,56,57,59,60,63,65, 66,68], while general knowledge remains unavailable.”
BackgroundUncovering the Temporal Context for Video Question Answering
Zhu, L., Xu, Z., Yang, Y., & Hauptmann, A. G. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1033-7 · Cited by 210 · see citing works
“Biases in answers Jabri et al. (2016) found that there is a strong bias in Visual7W dataset (Zhu et al. 2016). λ = 1 in Fig.”
BackgroundCross-Modal and Hierarchical Modeling of Video and Text
Zhang, B., Hu, H., & Sha, F. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01261-8_23 · Cited by 172 · see citing works
“Examples are embedding images and the visual categories [8], embedding images and texts for VSE [19], and embedding images, questions, and answers for VQA [13].”
BackgroundConnecting Vision and Language with Localized Narratives
Pont-Tuset, J., Uijlings, J., Changpinyo, S., Soricut, R., & Ferrari, V. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58558-7_38 · Cited by 176 · see citing works
Fang, B., Li, Y., Zhang, H., & Chan, J. C.-W. (2019) · Remote Sensing · doi:10.3390/rs11020159 · Cited by 171 · see citing works
Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and ApproachesPreprint
Storks, S., Gao, Q., & Chai, J. Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01172 · Cited by 234 · see citing works
Audio Visual Scene-Aware Dialog
Alamri, H., Cartillier, V., Das, A., Wang, J., Cherian, A., Essa, I. A., Batra, D., Marks, T. K., Hori, C., Anderson, P., et al. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00774 · Cited by 172 · see citing works
“Video-based Question Answering: Inspired by the success of image-based question answering [1, 9, 32, 34], recent work has addressed the task of video-based question answering [11,20,30].”
BackgroundKVQA: Knowledge-Aware Visual Question Answering
Shah, S. B., Mishra, A., Yadati, N., & Talukdar, P. P. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018876 · Cited by 171 · see citing works
“…image, also popularly known as Visual Question Answering (VQA), has gained huge interest in recent years (Goyal et al. 2017; Antol et al. 2015; Zhu et al. 2016; Kembhavi et al. 2017 Recognizing the need of external knowledge in answering questions about a given image, commonsense-enabled VQA…”
BackgroundOtter: A Multi-Modal Model With In-Context Instruction Tuning
Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Cahyono, J. A., Yang, J., Li, C., & Liu, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3571946 · Cited by 253 · see citing works
“Multi-Instruct covers most multi-modal tasks that require visual understanding and multi-modal reasoning, such as Visual Question Answering [11, 41, 30], Image Captioning [16], Image Generation [7], and Visual Relationship Understanding [14], among others.”
Spatial Memory for Context Reasoning in Object Detection
Chen, X., & Gupta, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.440 · Cited by 172 · see citing works
“Memory has been successfully used in the recognition community recently for tasks such as captioning [13, 17, 59, 84, 85, 92] or visual question answering [2, 3, 24, 44, 55, 57, 72, 90, 91, 93, 97].”
MethodDid the Model Understand the Question?
Mudrakarta, P. K., Taly, A., Sundararajan, M., & Dhamdhere, K. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1176 · Cited by 166 · see citing works
“The Visual Question Answering Task (Agrawal et al., 2015; Teney et al., 2017; Kazemi and Elqursh, 2017; Ben-younes et al., 2017; Zhu et al., 2016) requires a system to answer questions about images (fig.”
BackgroundUsing deep learning to detect price change indications in financial markets
Tsantekidis, A., Passalis, N., Tefas, A., Kanniainen, J., Gabbouj, M., & Iosifidis, A. (2017) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2017.8081663 · Cited by 157 · see citing works
“Recent Deep Learning methods has been shown to significantly improve upon previous machine learning techniques in tasks such as speech recognition [5], image captioning [6], [7], and question answering [8].”
MethodVision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155 · see citing works
See-Through-Text Grouping for Referring Image Segmentation
Chen, D.-J., Jia, S., Lo, Y.-C., Chen, H.-T., & Liu, T.-L. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00755 · Cited by 148 · see citing works
“Methods for VQA usually use a Recurrent Neural Network (RNN) to encode or generate sequences.”
BackgroundInfluentialVisual Entailment: A Novel Task for Fine-Grained Image UnderstandingPreprint
Xie, N., Lai, F., Doran, D., & Kadav, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.06706 · Cited by 156 · see citing works
“0 [23], CLEVR [32], Visual7w [81], Visual Genome [41], COCO-QA [57], and models [33, 60, 29, 31, 1, 67, 17, 37] have been used to measure the progress in understanding the interaction between vision and language modalities.”
MethodOvercoming Language Priors in Visual Question Answering with Adversarial RegularizationPreprint
Ramakrishnan, S., Agrawal, A., & Lee, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.03649 · Cited by 141 · see citing works
“In response to these challenges, there has been extensive work on VQA in recent years both in terms of dataset curation [6, 12, 2, 17, 13, 32, 3] and modeling [2, 5, 28, 14, 16, 4, 22, 20].”
BackgroundDeep Neural Network Compression by In-Parallel Pruning-Quantization
Tung, F., & Mori, G. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2886192 · Cited by 140 · see citing works
“The task integrates computer vision, natural language processing, and logical reasoning, and is sometimes considered to be “AI-complete” or a proxy to evaluating a system for deep image understanding [68], [69].”
Background
- Authors
- Zhu, Y., Groth, O., Bernstein, M., & Fei-Fei, L.
- Affiliations
- Stanford University (US)
- Technische Universität Dresden (DE)
- Venue
- 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2016-06
- Pages
- 4995-5004
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2962749469
- Cited by
- 794
- References
- 61
- Citations by year
- 2026: 22 · 2025: 48 · 2024: 59 · 2023: 65 · 2022: 81 · 2021: 100 · 2020: 103 · 2019: 122 · 2018: 104 · 2017: 61 · 2016: 29
- FWCI
- 36.05
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Technische Universität Dresden
- Multidisciplinary University Research Initiative
- Office of Naval Research
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- visual question answering, spatial attention, LSTM, image understanding, multiple-choice questions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
