Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years.However, tasks such as visual question answering require combining these vector representations with each other.Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations.We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors.As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features.We extensively evaluate MCB on the visual question answering and grounding tasks.We consistently show the benefit of MCB over ablations without MCB.For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation.This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, the PJ-X backbone MCB [16] (which is crucial for the answering performance) preceded the publication of the CLEVR dataset.”
From CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We replace the concatenation of the visual representation and the encoded phrase in Grounder with MCB to combine both modalities.”
MethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“, 2015) that is pretrained on ImageNet data (Deng et al., 2009).”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Also, for this model, we concatenated the learned word embedding with pretrained GloVe vectors (Pennington et al., 2014).”
MethodGirshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Research works printed but not linked: 3
- Ross Girshick. 2015. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
- Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014. Multimodal neural language mod- els. In Proceedings of the International Conference on Machine Learning (ICML), pages 595-603.
- Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems (NIPS).
Cited by
Items 501–600 of 1,002
Passage Retrieval for Outside-Knowledge Visual Question Answering
Qu, C., Zamani, H., Yang, L., Croft, W. B., & Miller, E. G. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462987 · Cited by 49
Dense Modality Interaction Network for Audio-Visual Event Localization
Liu, S., Quan, W., Wang, C., Liu, Y., Liu, B., & Yan, D.-M. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3150469 · Cited by 46
Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning
Li, Z., Wong, X., Stengel-Eskin, E., Kortylewski, A., Ma, W., Durme, B. V., & Yuille, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01437 · Cited by 149
“Two-stream methods extract features for image and questions using CNN and LSTM respectively, then enable interaction between the two modalities with different feature fusing methods [4, 13, 24, 32, 37, 50, 60].”
MethodFigure Captioning with Reasoning and Sequence-Level TrainingPreprint
Chen, C. C., Zhang, R., Koh, E., Kim, S., Cohen, S. D., Yu, T., Bunescu, R. C., & Bunescu, R. (2019) · arXiv.org · doi:10.48550/arxiv.1906.02850 · Cited by 44
“Bilinear pooling or related techniques are further proposed to efficiently and expressively combine the image and question features (Fukui et al., 2016; Kim et al., 2016).”
Revisiting Visual Question Answering BaselinesPreprint
Jabri, A., Joulin, A., & van der Maaten, L. J. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.08390 · Cited by 47
Scene Graph Reasoning with Prior Visual Relationship for Visual Question AnsweringPreprint
Yang, Z., Qin, Z., Yu, J., & Hu, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.09681 · Cited by 44
“Besides straightforward solutions, several works apply bilinear pooling [28, 8, 14] or more complex fusion methods [22].”
MethodHuman-Adversarial Visual Question AnsweringPreprint
Sheng, S., Singh, A., Goswami, V., Magana, J. A. L., Galuba, W., Parikh, D., & Kiela, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02280 · Cited by 77
“From simple LSTM [21] and VGGNet [49] fused models to more advanced fusion techniques (MCB [15], Pythia [25]) to better object detectors (MCAN [56], MoViE+MCAN [42], BUTD [2]) to transformers [54, 51] and self-supervised pretraining (MCAN [56], UNIMO [34]), the community has brought the performance…”
MethodDesta, M. T., Chen, L., & Kornuta, T. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00201 · Cited by 41
Polarimetric Thermal to Visible Face Verification via Attribute Preserved Synthesis
Di, X., Zhang, H., & Patel, V. M. (2018) · IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS) · doi:10.1109/btas.2018.8698554 · Cited by 42
“yer (stride 2, kernelsize 4, and padding-size 1), L stands for Leaky Relu layer (negative slope=0.02), B stands for the batch-normalization layer, MCB indicates the Multimodal Compact Bilinear module [4, 5], D stands for the deconvolutional layer (stride 2, kernel-size 4 and padding-size 1), R is the RuLU layer, and T…”
BackgroundMethodInfluentialSelf-supervised Pre-training and Contrastive Representation Learning for Multiple-choice Video QA
Kim, S., Jeong, S., Kim, E., Kang, I.-H., & Kwak, N. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17556 · Cited by 48
Disentangled Motif-aware Graph Learning for Phrase Grounding
Mu, Z., Tang, S., Tan, J., Yu, Q., & Zhuang, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17602 · Cited by 36
Learning to Answer Visual Questions From Web Videos
Yang, A., Miech, A., Šivic, J., Laptev, I., & Schmid, C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3173208 · Cited by 53
“Various approaches to combine spatial image representations and sequential question representations have been proposed [6], [11], [30], [66], [99], [101], [106].”
Modality-specific Learning Rates for Effective Multimodal Additive Late-fusion
Yao, Y., & Mihalcea, R. F. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.143 · Cited by 75
“…first approach is late-fusion, in which the features from different models are first encoded separately by unimodal encoders, and the single-vector representation is then combined into a joint representation and fed into the final classifier (Kim et al., 2016; Fukui et al., 2016; Yu et al., 2017b).”
BackgroundMethodCreativity: Generating Diverse Questions using Variational Autoencoders
Jain, U., Zhang, Z., & Schwing, A. G. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03493 · Cited by 29
“When considering generation of text from images, caption and paragraph generation [32, 4, 11, 18, 19, 20, 13, 33, 37, 40, 48, 60, 64, 69], as well as visual question answering [2, 22, 47, 52, 57, 67, 68, 71, 21, 34, 74, 1, 15, 31, 72, 73, 66, 46, 45, 31] come to mind immediately.”
Guo, Y., Ge, H., & Jin-hong, L. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1159063 · Cited by 40
“Therefore, The MCBP module merges the text feature T m and the image feature V m into a multimodal feature.”
BackgroundInfluentialThe Promise of Premise: Harnessing Question Premises in Visual Question Answering
Mahendru, A., Prabhu, V., Mohapatra, A., Batra, D., & Lee, S. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1097 · Cited by 38
“Many top-performing models integrate attention mechanisms (over the image, the question, or both) to focus on important structures (Fukui et al., 2016; Lu et al., 2016, 2017), and some have been designed with compositionality in mind (Andreas et al.”
Embodied Multimodal Multitask Learning
Chaplot, D. S., Lee, L., Salakhutdinov, R., Parikh, D., & Batra, D. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/338 · Cited by 41
Visual question answering via Attention-based syntactic structure tree-LSTM
Liu, Y., Zhang, X., Huang, F., Tang, X., & Li, Z. (2019) · Applied Soft Computing · doi:10.1016/j.asoc.2019.105584 · Cited by 36
Figure Captioning with Relation Maps for Reasoning
Chen, C., Zhang, R., Koh, E., Kim, S., Cohen, S., & Rossi, R. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093592 · Cited by 36
“Bilinear pooling or related techniques are further proposed to efficiently and expressively combine the image and question features [13, 23].”
BackgroundImage captioning improved visual question answering
Sharma, H., & Jalal, A. S. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11276-2 · Cited by 43
“These models are improved by applying various approaches like merging of better visual and linguistic features [14, 27] and improved visual feature representation based on attention mechanism [14].”
MethodBilateral Cross-Modality Graph Matching Attention for Feature Fusion in Visual Question Answering
Cao, J., Qin, X., Zhao, S., & Shen, J. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3135655 · Cited by 46
“Some sophisticated schemes, including MCB [11], MLB [12], and MUTAN [13], which designed different linear fusion strategies, demonstrated better performances and less computational burden than bilinear fusion.”
See and Learn More: Dense Caption-Aware Representation for Visual Question Answering
Bi, Y., Jiang, H., Hu, Y., Sun, Y., & Yin, B. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3291379 · Cited by 37
“For multi-modal information fusion, previous approaches use global image and question features for cross-modal information fusion [2], [3], which are difficult to answer questions about details, resulting in inefficient models.”
MethodComprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulation
Yan, X., Yuan, Z., Du, Y., Liao, Y., Guo, Y., Cui, S. R., & Li, Z. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3340679 · Cited by 52
“To answer questions by reference images, early models [23], [24], [25], [26], [27], [28] design various of features fusion method between vision and language.”
MethodCurlingNet: Compositional Learning between Images and Text for Fashion IQ DataPreprint
Yu, Y., Lee, S., 최윤철, & Kim, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.12299 · Cited by 35
“For final ensemble, we use several variants of fusion functions such as MUTAN [2] and MCB [4].”
MethodLi, J., Liu, H., Liu, W., Zong, P., Huang, K., Li, Z., Li, H., Xiong, T., Tian, G., Li, C., et al. (2023) · Briefings in Functional Genomics · doi:10.1093/bfgp/elad032 · Cited by 37
Zero-Shot Learning Across Heterogeneous Overlapping Domains
Kumar, A., Muddireddy, P. R., Dreyer, M., & Hoffmeister, B. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-516 · Cited by 34
Answer Again: Improving VQA with Cascaded-Answering Model
Peng, L., Yang, Y., Zhang, X., Ji, Y., Lu, H., & Shen, H. T. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.2998805 · Cited by 45
Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA
Kim, H., Tang, Z., & Bansal, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.435 · Cited by 39
Multimodal deep fusion for image question answering
Zhang, W., Yu, J., Wang, Y., & Wang, W. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106639 · Cited by 37
Contrast and Classify: Training Robust VQA Models
Kant, Y., Moudgil, A., Batra, D., Parikh, D., & Agrawal, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00163 · Cited by 36
A Case Study of the Shortcut Effects in Visual Commonsense Reasoning
Ye, K., & Kovashka, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16428 · Cited by 50
“To solve visual question answering tasks, studies used feature fusion (Antol et al. 2015; Zhou et al. 2015), attention (Lu et al. 2016; Fukui et al. 2016; Anderson et al. 2018), or question-related modular structures (Andreas et al. 2016b,a).”
MethodSim VQA: Exploring Simulated Environments for Visual Question Answering
Cascante-Bonilla, P., Wu, H., Wang, L., Feris, R., & Ordóñez, V. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00500 · Cited by 55
Video Captioning and Retrieval Models with Semantic Attention.Preprint
Yu, Y., Ko, H., Choi, J., & Kim, G. (2016) · arXiv (Cornell University) · Cited by 34
Han, W., Dong, X., Zhang, Y., Crandall, D., Xu, C., & Shen, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3400873 · Cited by 34
“For example, cross-modality tasks, such as visual grounding [17], [24], [53] and visual question answering [2], [36] depend on fused features extracted from multi-modal inputs [3], [48], [57].”
BackgroundA Multimodal Benchmark Dataset and Model for Crop Disease Diagnosis
Liu, X., Liu, Z., Hu, H., Chen, Z., Wang, K., Wang, K., & Lian, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73016-0_10 · Cited by 65
“The imperative for a multimodal visual question answering (VQA) system [5, 13], proficient in diagnosing crop diseases while possessing extensive agricultural knowledge, becomes evident.”
Improving visual question answering using dropout and enhanced question encoder
Fang, Z., Liu, J., Li, Y., Qiao, Y., & Lu, H. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.01.038 · Cited by 33
Approximated Bilinear Modules for Temporal Modeling
Zhu, X., Xu, C., Hui, L., Lu, C., & Tao, D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00359 · Cited by 35
“Bilinear pooling has been promising in many computer vision tasks [29, 11, 10, 21, 53, 28, 52, 12] Lin et al . utilized two streams of CNNs for fine-grained classification by extracting two branches of features and fusing them with outer product [29].”
InfluentialVisual Question Answering on 360° Images
Chou, S.-H., Chao, W., Lai, W., Sun, M., & Yang, M. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093452 · Cited by 54
Multimodal Neural Graph Memory Networks for Visual Question Answering
Khademi, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.643 · Cited by 44
“Fukui et al. (2016) proposed a VQA model based on multimodal compact bilinear (MCB) pooling to get a joint representation for image and question.”
MethodWhat Deep CNNs Benefit From Global Covariance Pooling: An Optimization Perspective
Wang, Q., Zhang, L., Wu, B., Ren, D., Li, P., Zuo, W., & Hu, Q. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01078 · Cited by 34
Propagating Over Phrase Relations for One-Stage Visual Grounding
Yang, S., Li, G., & Yu, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58529-7_35 · Cited by 33
“They focus on learning the feature fusion in language and vision modalities [4,19,29], reconstructing the phrases from phrase-region features [7, 18] or matching the phrase embedding with the encoded phrase-related/phrase-unrelated region features [12,23] to address the first challenge mentioned…”
BackgroundALSA: Adversarial Learning of Supervised Attentions for Visual Question Answering
Liu, Y., Zhang, X., Zhao, Z., Zhang, B., Cheng, L., & Li, Z. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3029423 · Cited by 35
“MCB [33] and MLB [34] both adopt bilinear models with multimodal pooling scheme to learn the multimodal embedding for answer prediction.”
MethodBiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialogues
Le, H., Sahoo, D., Chen, N. F., & Hoi, S. C. H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.145 · Cited by 42
AiR: Attention with Reasoning Capability
Chen, S., Jiang, M. X., Yang, J., & Zhao, Q. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58452-8_6 · Cited by 47
Towards Fully Automated Manga Translation
Hinami, R., Ishiwatari, S., Yasuda, K., & Matsui, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17537 · Cited by 54
Zia, K., Chiumento, A., & Havinga, P. (2022) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2022.3215731 · Cited by 39
“Multi-modal learning has proved very beneficial in different multi-modal classification tasks like video classification [73], sentiment analysis [74] and visual question answering [75].”
BackgroundMultimodal Fine-Grained Transformer Model for Pest Recognition
Zhang, Y., Chen, L., & Yuan, Y. (2023) · Electronics · doi:10.3390/electronics12122620 · Cited by 36
Visual question answering: from early developments to recent advances -- a surveyPreprint
Huynh, N. D., Bouadjenek, M. R., Aryal, S., Razzak, I., & Hacid, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.03939 · Cited by 50
An Improved Attention and Hybrid Optimization Technique for Visual Question Answering
Sharma, H., & Jalal, A. S. (2021) · Neural Processing Letters · doi:10.1007/s11063-021-10655-y · Cited by 35
HRVQA: A Visual Question Answering benchmark for high-resolution aerial images
Li, K., Vosselman, G., & Yang, M. Y. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.06.002 · Cited by 50
Zhan, C., Peng, P., Wang, H., Wang, G., Lin, Y., Chen, T., & Wang, H. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103464 · Cited by 35
Adversarial Multimodal Network for Movie Story Question Answering
Yuan, Z., Sun, S., Duan, L., Li, C., Wu, X. Y., & Xu, C. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3002667 · Cited by 36
Stroke constrained attention network for online handwritten mathematical expression recognition
Wang, J., Du, J., Zhang, J., Wang, B., & Ren, B. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108047 · Cited by 32
Single-branch Network for Multimodal Training
Saeed, M. S., Nawaz, S., Khan, M. H., Zaheer, M. Z., Nandakumar, K., Yousaf, M. H., & Mahmood, A. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097207 · Cited by 50
“extensively studied to solve several multimodal tasks including cross-modal retrieval [1], [2], cross-modal verification [3], [4], multimodal named entity recognition [5], [6], visual question answering [7], [8], image captioning [9], [10], and multimodal classification [11]–[13].”
BackgroundConvolutional Self-Attention NetworksPreprint
Yang, B., Wang, L., Wong, D. F., Chao, L. S., & Tu, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03107 · Cited by 45
“The vanilla SAN models linearly aggregate features from different heads, and this procedure limits the extent of abstraction (Fukui et al., 2016; Li et al., 2019).”
BackgroundMultimodal Differential Network for Visual Question Generation
Patro, B. N., Kumar, S., Kurmi, V. K., & Namboodiri, V. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1434 · Cited by 29
“…2014; Lin et al., 2014; Antol et al., 2015; Ren et al., 2015; Ma et al., 2016; Noh et al., 2016) for solving VQA task including attention-based methods (Zhu et al., 2016; Fukui et al., 2016; Gao et al., 2015; Xu and Saenko, 2016; Lu et al., 2016; Shih et al., 2016; Patro and Namboodiri, 2018).”
MethodInverse Visual Question Answering: A New Benchmark and VQA Diagnosis Tool
Liu, F., Xiang, T., Hospedales, T. M., Yang, W., & Sun, C. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2880185 · Cited by 36
Where to Look?: Mining Complementary Image Regions for Weakly Supervised Object Localization
Babar, S., & Das, S. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00105 · Cited by 35
“Since then, attention has been used in a wide variety of applications including image captioning [37], visual grounding [12], visual question answering [2], sound source localization [3].”
BackgroundDiscover Micro-Influencers for Brands via Better Understanding
Wang, S., Gan, T., Liu, Y.-A., Zhang, L., Wu, J., & Nie, L. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3087038 · Cited by 41
Improving Deep Learning Based Second-Order Side-Channel Analysis With Bilinear CNN
Cao, P., Zhang, C., Lu, X., Gu, D., & Xu, S. (2022) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3216959 · Cited by 31
“Our work is also related to papers focusing on bilinear pooling, which is widely used for fine-grained visual recognition [24] and multimodal fusion [25].”
BackgroundMethodWord-to-region attention network for visual question answering
Peng, L., Yang, Y., Bin, Y., Xie, N., Shen, F., Ji, Y., & Xu, X. (2018) · Multimedia Tools and Applications · doi:10.1007/s11042-018-6389-3 · Cited by 31
“Recently, attention mechanism is employed widely in computer vision and natural language processing [3, 6, 16, 28, 37, 44, 45, 48].”
BackgroundMethodInfluentialA Novel Framework for Robustness Analysis of Visual QA Models
Huang, J.-H., Dao, C. D., Alfadly, M., & Ghanem, B. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018449 · Cited by 43
MSRC: Multimodal Spatial Regression with Semantic Context for Phrase Grounding
Chen, K., Kovvuri, R., Gao, J., & Nevatia, R. (2017) · International Conference on Multimedia Retrieval · doi:10.1145/3078971.3078976 · Cited by 31
Language Features Matter: Effective Language Representations for Vision-Language Tasks
Burns, A., Tan, R., Saenko, K., Sclaroff, S., & Plummer, B. A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00757 · Cited by 34
“In recent years many methods have been proposed for vision-language tasks such as image and video captioning [12, 27, 47, 48, 52], multimodal retrieval [16, 24, 20, 49, 37, 46, 51], phrase grounding [42, 19, 41, 43], and visual question answering [14, 2, 56, 44, 54].”
BackgroundMethodMultichannel Attention Refinement for Video Question Answering
Zhuang, Y., Xu, D., Yan, X., Cheng, W., Zhao, Z., Pu, S., & Xiao, J. (2020) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3366710 · Cited by 34
“VIS+LSTM [33] and VQA-MCB [8] are recent state-of-the-art works in image-based question answering.”
BackgroundFact-based visual question answering via dual-process system
Liu, L., Wang, M., He, X., Qing, L., & Chen, H. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107650 · Cited by 33
Learning from Inside: Self-driven Siamese Sampling and Reasoning for Video Question Answering
Yu, W., Zheng, H., Li, M., Ji, L., Wu, L., Xiao, N., & Duan, N. (2021) · neural information processing systems · Cited by 30
Medical visual question answering based on question-type reasoning and semantic space constraint
Wang, M., He, X., Liu, L., Qing, L., Chen, H., Liu, Y., & Ren, C. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102346 · Cited by 32
Deep Residual Weight-Sharing Attention Network With Low-Rank Attention for Visual Question Answering
Qin, B., Hu, H. R., & Zhuang, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3173131 · Cited by 32
“In [15], [53]–[55], the authors used the bilinear attention maps to connect vision and language modalities, which allowed multiplicative interaction between all features to get joint representations.”
MethodShetty, S., Ananthanarayana, V. S., & Mahale, A. R. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14940-x · Cited by 37
“In applications such as visual question answering [22], the concept of integrating visual and textual features by outer product has produced competent outcomes.”
BackgroundK-PathVQA: Knowledge-Aware Multimodal Representation for Pathology Visual Question Answering
Naseem, U., Khushi, M., Dunn, A. G., & Kim, J. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3294249 · Cited by 40
“…VQA models such as Bilinear Attention Networks (BAN) [15], Stacked Attention Networks (SAN) [16], and Multi-modal Compact Bilinear (MCB) [17], where attention mechanism and bilinear pooling schemes are applied to capture cross-modal feature fusion which captures the textual and visual…”
MethodVision transformer-based visual language understanding of the construction process
Yang, B., Zhang, B., Han, Y., Liu, B., Hu, J., & Jin, Y. (2024) · Alexandria Engineering Journal · doi:10.1016/j.aej.2024.05.015 · Cited by 30
Multi-source Multi-level Attention Networks for Visual Question Answering
Yu, D., Fu, J., Tian, X., & Mei, T. (2019) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3316767 · Cited by 31
“First, some methods introduce a visual attention mechanism to localize the target regions queried by a question, while masking the irrelevant regions by a learned attention map [11, 21, 31, 34, 41, 50, 51].”
BackgroundMethodTextual-Visual Reference-Aware Attention Network for Visual Dialog
Guo, D., Wang, H., Wang, S., & Wang, M. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2992888 · Cited by 29
Adversarial Learning With Multi-Modal Attention for Visual Question Answering
Liu, Y., Zhang, X., Huang, F., Cheng, L., & Li, Z. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3016083 · Cited by 32
Relational Graph Learning for Grounded Video Description Generation
Zhang, W., Wang, X. E., Tang, S., Shi, H., Shi, H., Xiao, J., Zhuang, Y., & Wang, W. Y. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413746 · Cited by 35
“Visual Grounding : Visual grounding models encourage captioning generators to link phrases with specific spatial regions of images or videos, thereby presenting a potential way to improve the ex-plainability of models [7, 24, 35, 46, 49, 53].”
BackgroundLi, M., & Moens, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21346 · Cited by 36
Cross-Modal Multistep Fusion Network With Co-Attention for Visual Question Answering
Lao, M., Guo, Y., Wang, H., & Zhang, X. (2018) · IEEE Access · doi:10.1109/access.2018.2844789 · Cited by 30
A Study on Multimodal and Interactive Explanations for Visual Question AnsweringPreprint
Alipour, K., Schulze, J. P., Yao, Y., Ziskind, A. J., & Burachas, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.00431 · Cited by 35
“A common approach is to use attentional masks that highlight specific regions of the image, conditioned on the question (Kazemi and Elqursh 2017; Lu et al. 2016b; Teney et al. 2017; Xu and Saenko 2015; Jiang et al. 2018b; Fukui et al. 2016; Xu and Saenko 2016; Teney et al. 2018).”
BackgroundA Survey of Medical Vision-and-Language Applications and Their Techniques
Chen, Q., Zhao, R., Wang, S., Phan, V. M. H., van den Hengel, A., Verjans, J. W. H., Liao, Z., To, M., Xia, Y., Chen, J., et al. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-026-02973-2 · Cited by 37
Recurrent convolutional video captioning with global and local attention
Jin, T., Li, Y., & Zhang, Z. M. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.08.042 · Cited by 28
Explanation vs Attention: A Two-Player Game to Obtain Attention for VQA
Patro, B. N., Anupriy, & Namboodiri, V. P. (2020) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v34i07.6858 · Cited by 28
Multiple answers to a question: a new approach for visual question answering
Hosseinabad, S. H., Safayani, M., & Mirzaei, A. (2020) · The Visual Computer · doi:10.1007/s00371-019-01786-4 · Cited by 28
“Later on, the models became very complex [5,15], for instance, some of them retrieving an attention mask for the image based on the question [14,22,30].”
BackgroundAn effective strategy for multi-modal fake news detection
Xu, P., & Bao, X. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-12290-8 · Cited by 34
Achieving Human Parity on Visual Question Answering
Yan, M., Xu, H., Li, C., Tian, J., Bi, B., Wang, W., Xu, X., Zhang, J., Huang, S., Huang, F., et al. (2023) · ACM Transactions on Information Systems · doi:10.1145/3572833 · Cited by 32
“Traditional cross-modal fusionmethods gradually improve from a simple linear model [1] to bilinear pooling ones [20, 94], and attention-based fusion ones [52, 89], where questions and image features are fully interactedwith each other.”
BackgroundMemexQA: Visual Memex Question AnsweringPreprint
Jiang, L., Liang, J., Cao, L., Kalantidis, Y., Farfade, S., & Hauptmann, A. G. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.01336 · Cited by 30
“Visual QA VQA was first proposed by Antol et al. in [2] and has received a large amount of attention in recent years [31, 28, 14, 15, 8, 30, 27, 23].”
Can you fool AI with adversarial examples on a visual Turing testPreprint
Xu, X., Chen, X., Liu, C., Rohrbach, A., Darrell, T., & Song, D. X. (2017) · arXiv (Cornell University) · Cited by 29
Self-citationVQA and Visual Reasoning: An Overview of Recent Datasets, Methods and ChallengesPreprint
Zakari, R. Y., Owusu, J. W., Wang, H., Qin, K., Lawal, Z. K., & Dong, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.13296 · Cited by 35
“To integrate the two modalities, it uses Multimodal Compact Bilinear pooling (MCB) [125].”
MethodInfluentialBai, L., Wang, G., Islam, M., Seenivasan, L., Wang, A., & Ren, H. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102602 · Cited by 45
“For instance, Fukui et al. [31] proposed a compact bilinear pooling model for VQA and visual grounding.”
MethodVQA and Visual Reasoning: An overview of approaches, datasets, and future direction
Zakari, R. Y., Owusu, J. W., Qin, K., Wang, H., Lawal, Z. K., & He, T. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129345 · Cited by 31
Learning to Specialize with Knowledge Distillation for Visual Question Answering
Mun, J., Lee, K., Shin, J., & Han, B. (2018) · Seoul National University Open Repository (Seoul National University) · Cited by 45
Examine before You Answer: Multi-task Learning with Adaptive-attentions for Multiple-choice VQA
Gao, L., Zeng, P., Song, J., Liu, X., & Shen, H. T. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240687 · Cited by 35
“[8] and MLAN [37].”
MethodInfluentialMHSAN: Multi-Head Self-Attention Network for Visual Semantic Embedding
Park, G., Han, C., Kim, D., & Yoon, W. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093548 · Cited by 27
“MCB [5] achieved a core improvement for visual question answering by their own fusion method and attention mechanism that computes one attention weight from multiple image features and the text feature.”
BackgroundCross-modality co-attention networks for visual question answering
Han, D., Zhou, S., Li, K., & de Mello, R. F. (2021) · Soft Computing · doi:10.1007/s00500-020-05539-7 · Cited by 27
“Nevertheless, the feature dimensions of the MCB method still tend to be in higher dimensions.”
MethodVisual question answering based on local-scene-aware referring expression generation
Kim, J.-J., Lee, D.-G., Wu, J., Jung, H.-G., & Lee, S. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.02.001 · Cited by 31
Yu, W., Huang, H., & Shen, G. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3222480 · Cited by 27
“The MCBP [32] has been proven to be an efficient feature fusion method.”
MethodREX: Reasoning-aware and Grounded Explanation
Chen, S., & Zhao, Q. G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01514 · Cited by 37
“There is a large body of research on constructing VQA datasets [3, 4, 9, 12, 13, 29, 32, 43] and developing VQA models [2, 8, 10, 11, 16, 17, 21, 26,33,35,42].”
BackgroundMethodZhi-hua, W., Yu, L., Ding, X., Liao, X., & Wang, L. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3287256 · Cited by 34
“…multi-modal learning methods usually adopt the late-stage fusion methods ( e.g. , average pooling [14], concatenation [1], [12], bilinear pooling [15], [16]), which makes it difficult for the model to effectively learn the relationship between different modal features and the flexibility of the…”
BackgroundMethodInfluentialVisual Grounding Via Accumulated Attention
Deng, C., Wu, Q., Wu, Q., Hu, F., Lyu, F., & Tan, M. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3023438 · Cited by 30
“Recently, the attention mechanism has been widely used in many applications such as Natural Language Processing [2, 23], Image Captioning [29, 30] and VQA [4, 14, 16, 24] due to its effectiveness in dealing with information redun-dancies.”
BackgroundMethodVisual Sentiment Analysis With Social Relations-Guided Multiattention Networks
Xu, J. J., Li, Z., Huang, F., Li, C., & Yu, P. S. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3027766 · Cited by 27
Metadata
- Authors
- Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M.
- Affiliations
- Sony (Japan) (JP)
- University of California, Berkeley (US)
- Max Planck Institute for Informatics (DE)
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 457-468
- OpenAlex
- W2963383024
- Cited by
- 1,582
- References
- 60
- Citations by year
- 2026: 42 · 2025: 105 · 2024: 126 · 2023: 165 · 2022: 189 · 2021: 222 · 2020: 225 · 2019: 201 · 2018: 179 · 2017: 105 · 2016: 20
- FWCI
- 58.39
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Max Planck Digital Library (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1427425, 1212798, N000141110688)
- Defense Advanced Research Projects Agency
- Multidisciplinary University Research Initiative (N000141110688)
- Air Force Research Laboratory
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- visual question answering, visual grounding, multimodal fusion, outer product, attention mechanism, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.