Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years.However, tasks such as visual question answering require combining these vector representations with each other.Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations.We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors.As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features.We extensively evaluate MCB on the visual question answering and grounding tasks.We consistently show the benefit of MCB over ablations without MCB.For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation.This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, the PJ-X backbone MCB [16] (which is crucial for the answering performance) preceded the publication of the CLEVR dataset.”
From CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We replace the concatenation of the visual representation and the encoded phrase in Grounder with MCB to combine both modalities.”
MethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“, 2015) that is pretrained on ImageNet data (Deng et al., 2009).”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Also, for this model, we concatenated the learned word embedding with pretrained GloVe vectors (Pennington et al., 2014).”
MethodGirshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Research works printed but not linked: 3
- Ross Girshick. 2015. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
- Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014. Multimodal neural language mod- els. In Proceedings of the International Conference on Machine Learning (ICML), pages 595-603.
- Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems (NIPS).
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00636 · Cited by 4,948
“Typically, these models can be characterized as top-down approaches, with context provided by a representation of a partially-completed caption in the case of image captioning [33, 27, 47, 45], or a representation of the question in the case of VQA [11, 28, 44, 46, 49].”
BackgroundMethodMultimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507
“This system builds upon the winner of the 2016 VQA challenge [73], with several simplification and additions.”
BackgroundMaking the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055
“y on joint multimodel embeddings to bridge the semantic gap between visual and textual clues in images and text, although such representations are usually tailored for specic tasks. For example, MCB [9], BAN [16] and DFAF [10] proposed advanced multimodal fusion methods for Visual Question Answering (VQA) [3]. SCAN [2…”
MethodGQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
Hudson, D. A., & Manning, C. D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00686 · Cited by 1,451
“popular across the board, with a host of datasets being constructed [3, 9, 13, 36, 17] and numerous models being proposed [4, 33, 5, 8, 10].”
BackgroundCLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Bidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445
“Several approaches to constructing the attention matrix have been used including element-wise product, element-wise sum, concatenation and Multimodal Compact Bilinear Pooling (Fukui et al., 2016).”
MethodFlickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444
Hierarchical Question-Image Co-Attention for Visual Question Answering
Lu, J., Yang, J., Batra, D., & Parikh, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00061 · Cited by 1,129
“Many recent works [2, 7, 13, 16, 17, 27, 12, 6] have proposed models for VQA.”
BackgroundA simple neural network module for relational reasoning
Santoro, A., Raposo, D., Barrett, D. G. T., Malinowski, M., Pascanu, R., Battaglia, P., & Lillicrap, T. (2017) · arXiv · doi:10.48550/arxiv.1706.01427 · Cited by 1,281
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors
Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1209 · Cited by 1,170
“, 2017), (Fukui et al., 2016) performs contextdependent fusion of multimodal data.”
BackgroundMethodResultInfluentialAttention on Attention for Image Captioning
Huang, L., Wang, W., Chen, J., & Wei, X.-Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00473 · Cited by 1,092
“AoA generates an attention gate and an information vector via two linear transformations and applies the gate to the vector to add a second attention, where the techniques are similar to some other work: GLU [10], which replaces RNN and CNN to capture long-range dependencies for language modeling; multi-modal fusion […”
MethodMISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Hazarika, D., Zimmermann, R., & Poria, S. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413678 · Cited by 1,073
“g cross-modal dynamics using sophisticated fusion mechanisms. These works include variety of methods, such as, multiple kernel learning [42], and tensor-based fusion (including its low-rank variants) [14, 21, 26, 29, 31, 58]. While these works perform fusion over representations of utterances, another line of work tak…”
Deep Modular Co-Attention Networks for Visual Question Answering
Yu, Z., Yu, J., Cui, Y., Tao, D., & Tian, Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00644 · Cited by 964
“We set a confidence threshold to the probabilities of detected objects and obtain a dynamic number of objects m ∈ [10, 100].”
BackgroundMethodBlind Image Quality Assessment Using a Deep Bilinear Convolutional Neural Network
Zhang, W., Ma, K., Yan, J., Deng, D., & Wang, Z. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2886771 · Cited by 949
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., & Sivic, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00272 · Cited by 986
“These applications include automatic image and video captioning [20, 36, 60, 61], visual question answering [9, 29, 51, 63], visual content retrieval based on textual queries [32, 56, 63], temporal localization of events in videos using natural language [15, 26] or video summarization with natural language [38].”
BackgroundTowards VQA Models That Can Read
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., & Rohrbach, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00851 · Cited by 711
“VQA models typically use some variant of attention to get a representation of the image that is relevant for answering the question [2, 7, 30, 47, 48, 51, 17].”
MethodInfluentialSelf-citationOK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
Marino, K., Rastegari, M., Farhadi, A., & Mottaghi, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00331 · Cited by 703
“Attention-based models [11, 25, 39, 40, 41, 47] better guide the model in answering the questions by highlighting image regions that are relevant to the question.”
BackgroundMulti-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering
Yu, Z., Yu, J., Fan, J., & Tao, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.202 · Cited by 758
Kim, J.-H., Jun, J., & Zhang, B.-T. (2018) · arXiv · doi:10.48550/arxiv.1805.07932 · Cited by 761
“networks (MRN) to efficiently utilize the multiple bilinear attention maps of the BAN, unlike the previous works [6, 15] where multiple attention maps are used by concatenating the attended features.”
BackgroundMethodDual Attention Networks for Multimodal Reasoning and Matching
Nam, H., Ha, J.-W., & Kim, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.232 · Cited by 726
“For fair evaluation, we compare single-model accuracies obtained without data augmentation, even though [5] reported better performance using model ensembles and additional training data.”
BackgroundMethodMUTAN: Multimodal Tucker Fusion for Visual Question Answering
Ben-younes, H., Cadene, R., Cord, M., & Thome, N. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.285 · Cited by 716
“Therefore, current bilinear approaches must be simplified or approximated by reducing the model complexity: in [5], the authors sacrifice trainability by using a handcrafted multi-modal projection, while a global tensor rank constraint is applied in [8], reducing correlations to a simple element-wise product.”
BackgroundMethodInfluentialX-Linear Attention Networks for Image Captioning
Pan, Y., Yao, T., Li, Y., & Mei, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01098 · Cited by 720
Hadamard Product for Low-rank Bilinear Pooling
Kim, J.-H., On, K.-W., Lim, W., Kim, J., Ha, J.-W., & Zhang, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.04325 · Cited by 154
Adversarial Attacks on Deep-learning Models in Natural Language Processing: A Survey
Zhang, W. E., Sheng, Q. Z., Alhazmi, A., & Li, C. (2020) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3374217 · Cited by 407
“Visual Question Answering [148] Y white-box MCB [33], N2NMN [51] L2”
BackgroundDon't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
Agrawal, A., Batra, D., Parikh, D., & Kembhavi, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00522 · Cited by 566
Person Search with Natural Language Description
Li, S., Xiao, T., Li, H., Zhou, B., Yue, D., & Wang, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.551 · Cited by 550
“Visual QA methods were proposed to answer questions about given images [31, 29, 40, 33, 26, 7].”
MethodWhat Makes Training Multi-modal Classification Networks Hard?
Wang, W., Tran, D., & Feiszli, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01271 · Cited by 588
“Our work is related to previous research on multi-modal networks [7] for classifications [41, 49, 19, 21, 12, 6, 10, 31], which primarily uses pre-training in contrast to our joint training.”
ResultLearned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572
“For VQA, Fukui et al. [2016] propose to combine text representations with visual representations extracted by a pretrained residual network [He et al., 2016].”
MethodA dataset of clinically generated visual questions and answers about radiology images
Lau, J. J., Gayen, S., Ben Abacha, A., & Demner-Fushman, D. (2018) · Scientific Data · doi:10.1038/sdata.2018.251 · Cited by 483
Learning Two-Branch Neural Networks for Image-Text Matching Tasks
Wang, L., Li, Y., Huang, J., & Lazebnik, S. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2797921 · Cited by 535
Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering
Yu, Z., Yu, J., Xiang, C., Fan, J., & Tao, D. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2817340 · Cited by 532
“From the previous work such as [8] and [14], we have witnessed that the bilinear pooling models have superior”
BackgroundMethodInfluentialLearning to Reason: End-to-End Module Networks for Visual Question Answering
Hu, R., Andreas, J., Rohrbach, M., Darrell, T., & Saenko, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.93 · Cited by 517
“Recent years have seen a proliferation of datasets [19, 4] and approaches, including models based on differentiable memory [25, 24], dynamic prediction of question-specific computations [20, 2], and core improvements to the implementation of the multi-modal representation and attention mechanism [9, 18].”
BackgroundMethodInfluentialSelf-citationDeep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535
“Following this strategy, promising results have been shown in many multimodal classification or clustering tasks, such as video classification [6], [21], event detection [7], [8], sentiment analysis [9], [10], and visual question answering [23].”
BackgroundMethodTGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
Jang, Y., Song, Y., Yu, Y., Kim, Y., & Kim, G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.149 · Cited by 488
“VQA-MCB, on the other hand, uses multimodal compact bilinear pooling to handle visual-textual fusion and spatial attention [10].”
BackgroundMethodResultInfluentialMultimodal intelligence: Representation learning, information fusion, and applications
Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511
“This setup can be implemented as a classification problem based on the features of images, questions, and answer candidates [172].”
BackgroundMethodInfluentialpeng (2017) · doi:10.1145/3072959.3073602 · Cited by 502
“To help the network better distinguish between di erent phases of a motion, we take inspiration from bilinear pooling models for vision tasks [Fukui et al. 2016].”
MethodInferring and Executing Programs for Visual Reasoning
Johnson, J., Hariharan, B., Van Der Maaten, L., Hoffman, J., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.325 · Cited by 467
“Many systems for VQA employ a very similar architecture [3, 8, 9, 30, 32, 33, 44]: they combine an RNN-based embedding of the question with a convolutional network-based embedding of an image in a classification model over possible answers.”
BackgroundMulti-Modality Cross Attention Network for Image and Sentence Matching
Wei, X., Zhang, T., Li, Y., Zhang, Y., & Wu, F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01095 · Cited by 451
“The goal of such a cross-modal matching task is how to accurately measure the visual-semantic similarity between an image and a sentence, and is related to many visionlanguage tasks including image-sentence cross-modal retrieval [21, 48, 55], visual captioning [1, 5], visual grounding [8, 53] and visual question answe…”
BackgroundHuang, S.-C., Shen, L., Lungren, M. P., & Yeung, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00391 · Cited by 489
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., & Parikh, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1116-0 · Cited by 487
“Table 2: Accuracy breakdown over different answer types achieved by MCB [9] and HieCoAtt [25] models when trained/tested on unbalanced/balanced VQA datasets.”
BackgroundMethodInfluentialMultimodal Transformer With Multi-View Visual Representation for Image Captioning
Yu, J., Li, J., Yu, Z., & Huang, Q. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2947482 · Cited by 468
“visual question answering [2]–[4], visual grounding [5] and”
BackgroundFloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding
Rahnemoonfar, M., Chowdhury, T., Sarkar, A., Varshney, D., Yari, M., & Murphy, R. R. (2021) · IEEE Access · doi:10.1109/access.2021.3090981 · Cited by 394
Graph-Structured Representations for Visual Question Answering
Teney, D., Liu, L., & Van Den Hengel, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.344 · Cited by 447
Counterfactual VQA: A Cause-Effect Look at Language Bias
Niu, Y., Tang, K., Zhang, H., Lu, Z., Hua, X.-S., & Wen, J.-R. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01251 · Cited by 418
“Over the recent years, we have witnessed the fast evolution of VQA, especially multi-modal fusion [8, 9, 10] and attention mechanism [11, 12].”
BackgroundVisual question answering: A survey of methods and datasets
Wu, Q., Teney, D., Wang, P., Shen, C., Dick, A., & van den Hengel, A. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.05.001 · Cited by 438
“[21] combine the attention mechanism into their “Multimodal Compact Bilinear pooling” (MCB) already mentioned in Section 2.”
BackgroundMethodScaling Open-Vocabulary Image Segmentation with Image-Level Labels
Ghiasi, G., Gu, X., Cui, Y., & Lin, T.-Y. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20059-5_31 · Cited by 339
“Visual grounding: Image captioning and image text datasets [37,9,27] enable research on the interplay of captions and grounded visual concepts [14,41,15,19,25].”
BackgroundFoundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Liang, P. P., Zadeh, A., & Morency, L.-P. (2024) · ACM Computing Surveys · doi:10.1145/3656580 · Cited by 195
“For example, Socher et al. [304] map image embeddings into word embedding spaces for zero-shot image classification.”
Mixed High-Order Attention Network for Person Re-Identification
Chen, B., Deng, W., & Hu, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00046 · Cited by 415
“Moreover, the bilinear pooling and highorder pooling methods are also applied in Visual-QuestionAnswering task, such as [14, 22, 56, 57].”
MethodComposing Text and Image for Image Retrieval - an Empirical Odyssey
Vo, N., Jiang, L., Sun, C., Murphy, K., Li, L.-J., Fei-Fei, L., & Hays, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00660 · Cited by 399
“Many techniques have been proposed to combine the text and image inputs effectively [7, 34, 22, 35, 37, 43, 27, 48, 25].”
BackgroundFlickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0965-7 · Cited by 298
Multiscale Feature Extraction and Fusion of Image and Text in VQA
Lu, S., Ding, Y., Liu, M., Yin, Z., Yin, L., & Zheng, W. (2023) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-023-00233-6 · Cited by 422
Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering
Gao, P., Jiang, Z., You, H., Lu, P., Hoi, S. C. H., Wang, X., & Li, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00680 · Cited by 381
“Thirdly, better multi-modality fusion approaches, such Bilinear Fusion [9], MCB [7] and MUTAN [4], have been proposed for better capturing the high-level interactions between language and visual features.”
MethodInfluentialFigureQA: An Annotated Figure Dataset for Visual Reasoning
Kahou, S. E., Michalski, V., Atkinson, A., Kádár, Á., Trischler, A., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.07300 · Cited by 122
“Global or local image representations are typically combined with the language encodings through attention [28, 29, 30] and pooling [31] mechanisms, then fed to a decoder that outputs a final answer in language.”
BackgroundMultimodal Explanations: Justifying Decisions and Pointing to the Evidence
Park, D. H., Hendricks, L. A., Akata, Z., Rohrbach, A., Schiele, B., Darrell, T., & Rohrbach, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00915 · Cited by 418
Self-citationTips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge
Teney, D., Anderson, P., He, X., & Hengel, A. V. D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00444 · Cited by 396
Relation-Aware Graph Attention Network for Visual Question Answering
Li, L., Gan, Z., Cheng, Y., & Liu, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01041 · Cited by 379
“[15, 25, 60, 4, 24] explored higherorder fusion methods to better combine textual information with visual information (e.”
BackgroundEntangled Transformer for Image Captioning
Li, G., Zhu, L., Liu, P., & Yang, Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00902 · Cited by 375
“The widely used co-attention mechanism [26, 45, 13, 21] in visual question answering (VQA) can explore the visual and semantic information jointly.”
BackgroundDas, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., & Batra, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00008 · Cited by 410
Bilinear Convolutional Neural Networks for Fine-Grained Visual Recognition
Lin, T.-Y., RoyChowdhury, A., & Maji, S. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2723400 · Cited by 328
“The idea of combining two features using the outer product has also been shown to be effective for other tasks such as visual question-answering [52] where text and visual features are combined, action recognition [53] where optical flow and image features are combined.”
BackgroundMultimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images
Chen, R. J., Lu, M. Y., Weng, W.-H., Chen, T. Y., Williamson, D. F., Manz, T., Shady, M., & Mahmood, F. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00398 · Cited by 398
“…sum, element-wise multiplication (Hadamard Product), bilinear pooling (Kronecker Product), and co-attention mechanisms are often used in many multimodal learning tasks such as VQA [19, 29, 28, 20], sentiment analysis [65], survival analysis [42, 9, 10], and other tasks in medicine [8, 41].”
MethodModeling Relationships in Referential Expressions with Compositional Modular Networks
Hu, R., Rohrbach, M., Andreas, J., Darrell, T., & Saenko, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.470 · Cited by 375
“The problem of grounding referential expressions can be naturally formulated as a retrieval problem over image regions [19, 10, 24, 7, 32, 20].”
BackgroundMethodSelf-citationModulating early visual processing by language
de Vries, H., Strub, F., Mary, J., Larochelle, H., Pietquin, O., & Courville, A. C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.00683 · Cited by 369
“Attention mechanisms [26] are often used to let the question attend to particular parts of the extracted higher-level feature maps.”
BackgroundMethodInfluentialIn Defense of Grid Features for Visual Question Answering
Jiang, H., Misra, I., Rohrbach, M., Learned-Miller, E., & Chen, X. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01028 · Cited by 358
Self-citationHow Much Can CLIP Benefit Vision-and-Language Tasks?
Shen, S., Li, L. H., Tan, H. X., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., & Keutzer, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06383 · Cited by 147
“Various methods have been introduced (Fukui et al., 2016; Yang et al., 2016; Anderson et al., 2018a; Jiang et al., 2018; Gao et al., 2019; Jiang et al., 2020).”
Self-citationCAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval
Wang, Z., Liu, X., Li, H., Sheng, L., Yan, J., Wang, X., & Shao, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00586 · Cited by 359
“Despite the success of feature fusion in other problems such as visual question answering [7, 8, 13, 32, 23], cross-modal feature fusion for text-image retrieval is nontrivial and has not been investigated before.”
BackgroundVisual attention methods in deep learning: An in-depth survey
Hassanin, M., Anwar, S., Radwan, I., Khan, F. S., & Mian, A. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102417 · Cited by 317
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Chen, L., Yan, X., Xiao, J., Zhang, H., Pu, S., & Zhuang, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01081 · Cited by 341
Kernel Pooling for Convolutional Neural Networks
Cui, Y., Zhou, F., Wang, J., Liu, X., Lin, Y., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.325 · Cited by 339
“Recently, fueled by compact 2nd order polynomial kernel approximation with Tensor Sketch [6, 31], same visual recognition performances can be preserved with much lower feature dimension [11] and new application on visual question answering is enabled [10].”
BackgroundMethodInfluentialLarge-Scale Adversarial Training for Vision-and-Language Representation Learning
Gan, Z., Chen, Y.-C., Li, L., Zhu, C., Cheng, Y., & Liu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06195 · Cited by 260
“Prominent work includes: (i) advanced attention mechanisms [40, 79, 51]; (ii) better multimodal fusion methods [14, 80, 27, 26]; (iii) multi-step reasoning [77, 22, 15]; (iv) incorporation of object relations [55, 48, 6, 32]; and (v) neural module networks for compositional reasoning [3, 25, 20, 11].”
BackgroundDurand, T., Mordan, T., Thome, N., & Cord, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.631 · Cited by 342
“Visual Question Answering [64, 4] or Visual Grounding [17].”
Nguyen, D.-K., & Okatani, T. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00637 · Cited by 347
3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera
Armeni, I., He, Z.-Y., Zamir, A., Gwak, J., Malik, J., Fischer, M., & Savarese, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00576 · Cited by 350
“Examples are on scene graph generation [30, 46], image captioning/description [26, 3, 23], image retrieval [24] and visual question-answering [17, 51].”
BackgroundSpatially Adaptive Computation Time for Residual Networks
Figurnov, M., Collins, M. D., Zhu, Y., Zhang, L., Huang, J., Vetrov, D., & Salakhutdinov, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.194 · Cited by 329
“In recent years deep convolutional networks have become an integral part of state-of-the-art systems for a diverse set of computer vision problems such as object detection [34], image segmentation [32], imageto-text [22, 42], visual question answering [11] and image generation [9].”
BackgroundTemporally Grounding Natural Sentence in Video
Chen, J., Chen, X., Ma, L., Jie, Z., & Chua, T.-S. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1015 · Cited by 328
“Previous approaches on multimodal interactions were limited to concatenation (Zhu et al., 2016), element-wise product or sum (Gao et al., 2017), and bilinear pooling (Fukui et al., 2016).”
MUREL: Multimodal Relational Reasoning for Visual Question Answering
Cadene, R., Ben-younes, H., Cord, M., & Thome, N. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00209 · Cited by 330
“Instead, state-of-the-art methods often rely on the much simpler attentional framework [11, 8, 20, 10].”
BackgroundMethodInfluentialVisual Entailment: A Novel Task for Fine-Grained Image Understanding
Xie, N., Lai, F., Doran, D., & Kadav, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.06706 · Cited by 149
“This includes simple linear models using ranking loss [16, 36], bi-linear pooling meth-ods [45, 20, 55, 17, 37], attention-based methods [1, 52, 64] and reasoning based approaches [54, 27, 33, 38, 29] on CLEVER and VQA-v1.”
MethodDas, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., & Batra, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00279 · Cited by 288
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs
Rosinol, A., Violette, A., Abate, M., Hughes, N., Chang, Y., Shi, J., Gupta, A., & Carlone, L. (2021) · The International Journal of Robotics Research · doi:10.1177/02783649211056674 · Cited by 271
“, 2018; Zhao and Zhu, 2013a), visual question-answering (Fukui et al., 2016; Zhu et al., 2016), and action detection (Liang et al.”
BackgroundRSVQA: Visual Question Answering for Remote Sensing Data
Lobry, S., Marcos, D., Murray, J., & Tuia, D. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.2988782 · Cited by 305
“Instead, [25] proposed a fusion method that first selects relevant visual features based on the textual feature (attention step) and then combines them with the textual feature.”
MethodImVoteNet: Boosting 3D Object Detection in Point Clouds With Image Votes
Qi, C. R., Chen, X., Litany, O., & Guibas, L. J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00446 · Cited by 294
DVQA: Understanding Data Visualizations via Question Answering
Kafle, K., Price, B., Cohen, S., & Kanan, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00592 · Cited by 254
Analyzing the Behavior of Visual Question Answering Models
Agrawal, A., Batra, D., & Parikh, D. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1203 · Cited by 297
Explicit Knowledge-based Reasoning for Visual Question Answering
Wang, P., Wu, Q., Shen, C., Dick, A., & van den Hengel, A. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/179 · Cited by 233
Anwer, R. M., Khan, F. S., van de Weijer, J., Molinier, M., & Laaksonen, J. (2018) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2018.01.023 · Cited by 287
“It is being investigated in the context of action recognition [34], [35], [36], RGB-D [37], and multi-modal networks [38], [39].”
BackgroundIdentity-Aware Textual-Visual Matching with Latent Co-attention
Li, S., Xiao, T., Li, H., Yang, W., & Wang, X. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.209 · Cited by 277
REMIND Your Neural Network to Prevent Catastrophic Forgetting
Hayes, T. L., Kafle, K., Shrestha, R., Acharya, M., & Kanan, C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58598-3_28 · Cited by 248
Recurrent Multimodal Interaction for Referring Image Segmentation
Liu, C., Lin, Z., Shen, X., Yang, J., Lu, X., & Yuille, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.143 · Cited by 272
“Multimodal Interaction Representation Our work is also related to multimodal feature fusion in visual question answering [16, 9, 25] and image captioning [6].”
BackgroundExploring Hate Speech Detection in Multimodal Publications
Gomez, R., Gibert, J., Gomez, L., & Karatzas, D. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093414 · Cited by 250
“Later methods, such as Multimodal Compact Bilinear pooling [6], utilize bilinear pooling to learn multimodal features.”
MethodActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., & Tao, D. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019127 · Cited by 281
Domain Adaptation: Challenges, Methods, Datasets, and Applications
Singhal, P., Walambe, R., Ramanna, S., & Kotecha, K. (2023) · IEEE Access · doi:10.1109/access.2023.3237025 · Cited by 234
“The bilinear model approach ([90], [91]) was used for fusion, and then tucker decomposition was used”
MethodAn Analysis of Visual Question Answering Algorithms
Kafle, K., & Kanan, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.217 · Cited by 247
“However, the annotations can be both non-exhaustive and duplicated, which makes using them to automatically make QA pairs difficult.”
BackgroundMethodInfluentialREMIND Your Neural Network to Prevent Catastrophic Forgetting
Hayes, T. L., Kafle, K., Shrestha, R., Acharya, M., & Kanan, C. (2019) · arXiv · doi:10.48550/arxiv.1910.02509 · Cited by 268
“, attention-based [6,46,78], compositional [7,28], bi-modal fusion [10,19,71]) and it can be applied to similar tasks like image captioning [12] and referring expression recognition [43,63,67].”
BackgroundDeep Spatial-Semantic Attention for Fine-Grained Sketch-Based Image Retrieval
Song, J., Yu, Q., Song, Y.-Z., Xiang, T., & Hospedales, T. M. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.592 · Cited by 261
“Attention Modelling Visual attention models have been studied extensively in a wide range of vision problems including image caption generation [44, 20], VQA [5, 26], image classification [25, 34, 42] and particularly finegrained image recognition [34, 42].”
Background3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans
Rosinol, A., Gupta, A., Abate, M., Shi, J., & Carlone, L. (2020) · Robotics: Science and Systems XVI · doi:10.15607/rss.2020.xvi.079 · Cited by 201
Stochastic Neural Networks for Hierarchical Reinforcement Learning
Florensa, C., Duan, Y., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03012 · Cited by 180
“Richer forms of integrations, such as multiplicative integrations and bilinear pooling, have been shown to have greater representation power and improve the optimization landscape, achieving better results when complex interactions are needed (Fukui et al., 2016; Wu et al., 2016).”
BackgroundVisual question answering: Datasets, algorithms, and future challenges
Kafle, K., & Kanan, C. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.06.005 · Cited by 245
“This is shown by experiments in [46] and [54], where the models were evaluated with and without attention, and the attentive version performed better in both cases.”
BackgroundMethodInfluential
Metadata
- Authors
- Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M.
- Affiliations
- Sony (Japan) (JP)
- University of California, Berkeley (US)
- Max Planck Institute for Informatics (DE)
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 457-468
- OpenAlex
- W2963383024
- Cited by
- 1,582
- References
- 60
- Citations by year
- 2026: 42 · 2025: 105 · 2024: 126 · 2023: 165 · 2022: 189 · 2021: 222 · 2020: 225 · 2019: 201 · 2018: 179 · 2017: 105 · 2016: 20
- FWCI
- 58.39
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Max Planck Digital Library (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1427425, 1212798, N000141110688)
- Defense Advanced Research Projects Agency
- Multidisciplinary University Research Initiative (N000141110688)
- Air Force Research Laboratory
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- visual question answering, visual grounding, multimodal fusion, outer product, attention mechanism, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.