Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years.However, tasks such as visual question answering require combining these vector representations with each other.Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations.We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors.As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features.We extensively evaluate MCB on the visual question answering and grounding tasks.We consistently show the benefit of MCB over ablations without MCB.For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation.This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, the PJ-X backbone MCB [16] (which is crucial for the answering performance) preceded the publication of the CLEVR dataset.”
From CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We replace the concatenation of the visual representation and the encoded phrase in Grounder with MCB to combine both modalities.”
MethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“, 2015) that is pretrained on ImageNet data (Deng et al., 2009).”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Also, for this model, we concatenated the learned word embedding with pretrained GloVe vectors (Pennington et al., 2014).”
MethodGirshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Research works printed but not linked: 3
- Ross Girshick. 2015. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
- Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014. Multimodal neural language mod- els. In Proceedings of the International Conference on Machine Learning (ICML), pages 595-603.
- Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems (NIPS).
Cited by
Items 901–1,000 of 1,002
VL-Meta: Vision-Language Models for Multimodal Meta-Learning
Ma, H., Fan, B., Ng, B. K., & Lam, C. (2024) · Mathematics · doi:10.3390/math12020286 · Cited by 11
Revisiting Counterfactual Problems in Referring Expression Comprehension
Yu, Z., & Li, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01276 · Cited by 11
“REC connects image regions with natural language, which contributes to downstream vision-language tasks such as image captioning [3, 16, 49] and visual question answering [11, 41, 45].”
BackgroundTri-VQA: Triangular Reasoning Medical Visual Question Answering for Multi-Attribute Analysis
Fan, L., Gong, X., Zheng, C., & Ou, Y. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822825 · Cited by 11
“The predominant approach in Med-VQA is the employment of a joint embedding framework, which integrates extracted image features with question features to predict or generate an answer [2, 3].”
Elliptical Higher-Order-Spectra Periocular Code
Algashaam, F. M. A., Nguyen, K., Chandran, V., & Banks, J. (2017) · IEEE Access · doi:10.1109/access.2017.2697898 · Cited by 14
“Multimodal Compact Bilinear Pooling [51]”
MethodInfluentialAttention-Based Natural Language Person Retrieval
Zhou, T., Chen, M., Yu, J., & Terzopoulos, D. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.10 · Cited by 14
“Sophisticated approaches have been developed [9, 20], which substantially improve the results in performance of visual question answering on the VQA datasets [9].”
MethodCQ-VQA: Visual Question Answering on Categorized Questions
Mishra, A., Anand, A., & Guha, P. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206913 · Cited by 11
“The attention mechanism is used with outer product based fusion of image and question embeddings [26].”
BackgroundMethodMulti-View Attention Network for Visual Dialog
Park, S., Whang, T., Yoon, Y. C., & Lim, H. (2021) · Applied Sciences · doi:10.3390/app11073009 · Cited by 28
Towards More Explainability: Concept Knowledge Mining Network for Event Recognition
Qi, Z., Wang, S., Su, C., Su, L., Huang, Q., & Tian, Q. C. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413954 · Cited by 11
“Bilinear pooling [22] is first proposed to solve fined-grained classification problem and is widely used to combine features from different modalities [12, 45].”
BackgroundGuessing State Tracking for Visual Dialogue
Pang, W., & Wang, X. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58517-4_40 · Cited by 13
Commands for Autonomous Vehicles by Progressively Stacking Visual-Linguistic Representations
Dai, H., Luo, S., Ding, Y. S., & Shao, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-66096-3_2 · Cited by 14
Improving Zero-Shot Phrase Grounding via Reasoning on External Knowledge and Spatial Relations
Shi, Z., Shen, Y., Jin, H., & Zhu, X. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i2.20123 · Cited by 14
Flexible Fashion Product Retrieval Using Multimodality-Based Deep Learning
Jo, Y., Wi, J., Kim, M., & Lee, J. Y. (2020) · Applied Sciences · doi:10.3390/app10051569 · Cited by 11
“The MCB pooling showed better performance than the ptwise product for the VQA problem, but it showed lower performance in the proposed approach [27].”
BackgroundMethodHu, J., Qian, S., Fang, Q., & Xu, C. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3009491 · Cited by 11
Improving users' mental model with attention‐directed counterfactual edits
Alipour, K., Ray, A., Lin, X., Cogswell, M., Schulze, J. P., Yao, Y., & Burachas, G. (2021) · Applied AI Letters · doi:10.1002/ail2.47 · Cited by 17
Deep Multimodal Feature Encoding for Video Ordering
Sharma, V., Tapaswi, M., & Stiefelhagen, R. (2019) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.48550/arxiv.2004.02205 · Cited by 12
“Inspired by this, multimodal feature representation learning [36, 14, 64, 65, 20, 22, 41, 66, 28] has attracted quite a lot of attention.”
BackgroundCross-Dataset Adaptation for Visual Question AnsweringPreprint
Chao, W., Hu, H., & Sha, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.03726 · Cited by 14
“Alternatively, the other setting is to select one from the top frequent answers and compare it to multiple human-annotated ones [3, 4, 5, 9, 16, 22, 29, 44, 45, 47, 48], avoiding constructing decoys that are too easy such that the performance is artificially boosted [4, 16].”
InfluentialImage-Question-Answer Synergistic Network for Visual DialogPreprint
Guo, D., Xu, C., & Tao, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.09774 · Cited by 11
“However, MCB needs to sam- ple features, which is computationally intractable and has very large projected dimension, so low-rank bilinear pooling (MLB) [15] and multi-factor bilinear pooling (MFB) [32] have been proposed to project the two features into a common low-rank space, and the bilinear attention network (BAN…”
MethodGiving Commands to a Self-driving Car: A Multimodal Reasoner for Visual Grounding
Deruyttere, T., Collell, G., & Moens, M. (2020) · Lirias · doi:10.48550/arxiv.2003.08717 · Cited by 12
“The first one consists in fusing the extracted features of the image together with the encoded textual question [6,14,29].”
BackgroundSur, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.14264 · Cited by 10
“Different multimodal bilinear pooling methods are proposed like in [9], [5].”
BackgroundMethodCan 3D Vision-Language Models Truly Understand Natural Language?Preprint
Deng, W., Yang, J., Ding, R., Liu, J., Li, Y., Qi, X., & Ngai, E. C. ‐. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.14760 · Cited by 13
“Significant progress has been made on various tasks, such as Visual Grounding and Visual Question Answering (VQA), in the context of both 2D Vision Language (2D-VL) [3,4,17,18,46,53,63] and 3D vision-language (3D-VL) understanding [14–16, 25,55–58].”
MethodMAFN: multi-level attention fusion network for multimodal named entity recognition
Zhou, X., Zhang, Y., Wang, Z., Lu, M., & Liu, X. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17376-5 · Cited by 10
Object Attribute Matters in Visual Question Answering
Li, P., Si, Q., Fu, P., Lin, Z., & Wang, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29816 · Cited by 11
Multi-SEA: Multi-stage Semantic Enhancement and Aggregation for image–text retrieval
Tian, Z., Ou, Z., Zhu, Y., Lyu, S., Zhang, H., Xiao, J., & Song, M. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104165 · Cited by 11
Pezzelle, S., Sorodoc, I.-T., & Bernardi, R. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1039 · Cited by 13
“, are very frequent and have been shown to be particularly challenging for any model (Antol et al., 2015; Malinowski et al., 2015; Ren et al., 2015; Fukui et al., 2016).”
Object Detection Boosting using Object Attributes in Detect and Describe Framework
Khan, M. J., Zafar, A., Tumanian, V., Yue, D., & Li, G. (2019) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai.2019.00126 · Cited by 10
“4) Fusion Strategy: Multimodal Compact Bilinear Pooling: We also experiment with Multimodal Compact Bilinear Pooling (MCB) [24] for merging of two convolutional layers.”
MethodDouble Attention-based Multimodal Neural Machine Translation with Semantic Image Regions
Zhao, Y., Komachi, M., Kajiwara, T., & Chu, C. (2020) · Cited by 10
Leveraging Bilinear Attention to Improve Spoken Language Understanding
Chen, D., Huang, Z., & Zou, Y. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747553 · Cited by 13
“By taking all pair-wise interactions between query and key into account, it triggers the second order feature interactions, and thus provides more discriminative representations, which has been widely explored in Computer Vision research [11–14].”
BackgroundGrouping Bilinear Pooling for Fine-Grained Image Classification
Zeng, R., & He, J. (2022) · Applied Sciences · doi:10.3390/app12105063 · Cited by 10
Label graph learning for multi-label image recognition with cross-modal fusion
Xie, Y., Wang, Y., Liu, Y., & Zhou, K. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-12397-y · Cited by 11
“Besides, Multi-modal Compact Bilinear (MCB) [5] pooling and Multi-modal Low-rank Bilinear (MLB) [18] pooling have been developed to reduce the computational complexity of the original bilinear pooling model and make it practicable for VQA.”
MethodMultimodal Features Alignment for Vision–Language Object Tracking
Ye, P., Xiao, G., & Liu, J. (2024) · Remote Sensing · doi:10.3390/rs16071168 · Cited by 13
Geometry-Constrained Car Recognition Using a 3D Perspective Network
Zeng, R., Zongyuan, G., Simon, D., Sridharan, S., & Fookes, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5468 · Cited by 11
Learning User Representations for Open Vocabulary Image Hashtag Prediction
Durand, T. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00979 · Cited by 14
“In our experiments, we use the MUTAN model [5] to approximate the bilinear product (Equation 6) but other models [19, 25, 49, 14] can be used.”
BackgroundMethodBelainine, B., Sadat, F., & Boukadoum, M. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3151347 · Cited by 12
“The responses generated can be based on inputs of many forms, for example, text requests [5], [6] visual captioning [7], [8], visual grounding [9], [10], and visual question answering (VQA) [4], [11], [12].”
BackgroundDual Self-Guided Attention with Sparse Question Networks for Visual Question Answering
Xiang, S. H. E. N., Han, D., Chang, C., & Zong, L. (2022) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2021edp7189 · Cited by 10
Multimodal high-order relational network for vision-and-language tasks
Pan, H., & Huang, J. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.03.071 · Cited by 11
LRF-WiVi: A WiFi and Visual Indoor Localization Method Based on Low-Rank Fusion
Liu, W., Qin, C., Deng, Z., & Jiang, H. (2022) · Sensors · doi:10.3390/s22228821 · Cited by 10
“Fingerprint localization can be implemented using only one of these two signals, which means that the location information of each signal has an independent positioning action, also called signal-specific action; In the field of multimodal learning, modeling the interaction between different modal data has been shown…”
BackgroundNeural Textual Features Composition for CBIR
Aboali, M. A., El-Maddah, I. A. M., & Munim, H. E. A. E. (2023) · IEEE Access · doi:10.1109/access.2023.3259737 · Cited by 11
“A Multimodal Compact Bilinear Pooling as a feature fusion mechanism to combine image and text was proposed in [19].”
MethodCo-attention graph convolutional network for visual question answering
Liu, C., Tan, Y.-Y., Xia, T.-T., Zhang, J., & Zhu, M. B. (2023) · Multimedia Systems · doi:10.1007/s00530-023-01125-7 · Cited by 10
“In order to obtain finer grained integration of image-question features, a large number of methods based on visual attention [4, 5, 7, 9, 11–13, 15] or co-attention [6, 8, 10, 14, 16] as well as bilinear pooling such as MCB [4], MLB [7], MFB [6], are proposed.”
MethodMultimodal Local Perception Bilinear Pooling for Visual Question Answering
Lao, M., Guo, Y., Wang, H., & Zhang, X. (2018) · IEEE Access · doi:10.1109/access.2018.2873570 · Cited by 10
“In order to compare our MLPB method with other stateof-the-art works, we adapted MLPB into the co-attention framework proposed in [5].”
MethodPivot Correlational Neural Network for Multimodal Video Categorization
Kang, S., Kim, J., Choi, H., Kim, S.-J., & Yoo, C. D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_24 · Cited by 10
“Outer product is also used to fuse input modalities [6, 13, 20].”
MethodEnsembling Visual Explanations
Rajani, N. F., & Mooney, R. J. (2018) · The Springer series on challenges in machine learning · doi:10.1007/978-3-319-98131-4_7 · Cited by 10
Exploiting hierarchical visual features for visual question answering
Hong, J. K., Fu, J., Uh, Y., Mei, T., & Byun, H. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.03.035 · Cited by 10
Qiao, S., Li, J., Zhao, C., & Zhang, T. (2021) · Nuclear Instruments and Methods in Physics Research Section A Accelerators Spectrometers Detectors and Associated Equipment · doi:10.1016/j.nima.2021.165406 · Cited by 10
Adaptive attention fusion network for visual question answering
Gu, G., Kim, S. T., & Ro, Y. M. (2017) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2017.8019540 · Cited by 10
“For given a pair of image and question, VQA model automatically generates answer phrases [2] or classifies an answer based on a set of pre-specified answer candidates [1, 3-7, 11, 15].”
MethodEssay-Anchor Attentive Multi-Modal Bilinear Pooling for Textbook Question Answering
Li, J., Su, H., Zhu, J., & Zhang, B. (2018) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2018.8486468 · Cited by 10
“Bilinear models [2, 3] are effective at learning high-level associations between questions and images, but are inefficient to handle the long essays.”
BackgroundMethodInfluentialSurvey of Recent Advances in Visual Question AnsweringPreprint
Pandhre, S., & Sodhani, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.08203 · Cited by 10
“To resolve this problem, Noh et al [16] proposed DPPnet architecture that learns to dynamically change the network parameter based on the question Algorithm VQA Open-ended test-dev VQA multiple-choice COCO-QA DAQUAR all-single answer DAQUAR reduced-single answer MCB[7] 66.7 70.2 - - - Show,ask,attend,ans[10] 64.5 - -…”
BackgroundMethodInfluentialMultimodal Machine Translation through Visuals and Speech
Sulubacak, U., Çağlayan, O., Grönroos, S.-A., Rouhe, A., Elliott, D., Specia, L., & Tiedemann, J. (2019) · Spiral (Imperial College London) · doi:10.5281/zenodo.3690790 · Cited by 12
Higher-Order Count Sketch: Dimensionality Reduction that Retains Efficient Tensor Operations
Shi, Y., & Anandkumar, A. (2020) · DCC · doi:10.1109/dcc47342.2020.00045 · Cited by 10
“In Visual Question Answering task, people use compact bilinear pooling to compute joint features from language and image (Fukui et al., 2016).”
Background3D-Aware Visual Question Answering about Parts, Poses and Occlusions
Wang, X., Ma, W., Li, Z., Kortylewski, A., & Yuille, A. (2023) · neural information processing systems · doi:10.52202/075280-2560 · Cited by 27
Semantically Guided Visual Question Answering
Zhao, H., Fan, Q., Gutfreund, D., & Fu, Y. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00205 · Cited by 9
“We also point out that two recent papers [8] and [15], are not listed in Table 2 as they measure performance on the 3000 and 2000 most frequent answers in the dataset respectively.”
ResultIntra- and Inter-modal Multilinear Pooling with Multitask Learning for Video Grounding
Zhou, Y., Song, Y., Yu, J., Wang, M., & Huang, Q. (2020) · Neural Processing Letters · doi:10.1007/s11063-020-10205-y · Cited by 9
“Therefore, many approximated bilinear pooling models were investigated [3,11,33].”
BackgroundMethodInfluentialLearning to Recognize Visual Concepts for Visual Question Answering With Structural Label Space
Gao, D., Wang, R., Shan, S., & Chen, X. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2989701 · Cited by 9
“The first branch [2], [9], [16], [21], [24], [32], [34], [38] proposes powerful attention mechanisms to compute the matching score between the question and the image regions.”
BackgroundMethodBarlow constrained optimization for Visual Question Answering
Jha, A., Patro, B. N., Gool, L. V., & Tuytelaars, T. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00114 · Cited by 9
“In later works [11,24,43,44], it was observed that attending to specific parts in the images and questions helps to better reason and answer.”
Multi-source Heterogeneous Data Fusion Algorithm Based on Federated Learning
Zhou, J., & Lei, Y. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-0405-1_4 · Cited by 11
Chouhayebi, H., Mahraz, M. A., Riffi, J., & Tairi, H. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16779-8 · Cited by 10
“The multimodal compact bilinear pooling strategy has been extended in dealing with multimodal data by Fukui et al. [48]and Delbrouck and Dupont [49].”
FedVQA: Personalized Federated Visual Question Answering over Heterogeneous Scenes
Lao, M., Pu, N., Zhong, Z., Sebe, N., & Lew, M. S. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611958 · Cited by 12
“Most of earlier VQA works seek to establish efficient model architectures to achieve fine-grained vision-language interactions for answer prediction, such as multimodal fusion [13, 33, 57], attention [4, 11, 21, 51, 56], and large-scale pre-training models [28, 36, 61].”
BackgroundLatent Attention Network With Position Perception for Visual Question Answering
Zhang, J., Liu, X., & Wang, Z. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3377636 · Cited by 11
“The previous models [7], [8], [9] commit to effectively fusing visual and textual features for obtaining more accurate answers by classification.”
ARDN: Attention Re-distribution Network for Visual Question Answering
Yi, J., Han, D., Chen, C., Shen, X., & Zong, L. (2024) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-024-09067-6 · Cited by 10
“Regarding multimodal feature fusion, Fukui et al. [36] found that joint vector representation struggled to capture complex multi-modal interaction information.”
Jang, Y., Jung, J., Hong, Y., Lee, J., Jeong, H., Shim, H., & Chang, H. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3431028 · Cited by 9
“For example, Fukui et al. [25] provided multimodal compact bilinear (MCB) pooling for visual question answering.”
HoMap: End-to-End Vectorized HD Map Construction With High-Order Modeling
Cai, Y., Dong, W., Liu, Z., Wang, H., & Chen, L. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3445374 · Cited by 12
“Further research demonstrates the beneficial effects of GCP in image classification [22], [31], [32], [33] and visual question answering [34], [35].”
Dai, R., Wang, Z., Wang, W., Jie, J., Chen, J., & Ye, Q. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112251 · Cited by 9
Hierarchical Multimodality Graph Reasoning for Remote Sensing Visual Question Answering
Zhang, H., Wang, K., Zhang, L., Wang, B., & Li, X. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3502800 · Cited by 11
Using Explanations to Improve Ensembling of Visual Question Answering Systems
Mooney, N. F. R. A. R. J. (2017) · Cited by 9
Multi-stage Attention based Visual Question Answering
Mishra, A., Anand, A., & Guha, P. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9413173 · Cited by 9
“In [3], authors have proposed compact bilinear pooling based fusion to obtain a richer representation.”
BackgroundMethodVirtual Teaching Assistants: Technologies, Applications and Challenges
Liu, J., Zhang, L., Wei, B., & Zheng, Q. (2021) · Springer eBooks · doi:10.1007/978-3-030-72188-6_13 · Cited by 10
MGMP: Multimodal Graph Message Propagation Network for Event Detection
Li, J., Wang, Y., & Li, W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-98358-1_12 · Cited by 9
Traffic flow prediction based on depthwise separable convolution fusion network
Yu, Y., Sun, W., Liu, J., & Zhang, C. (2022) · Journal Of Big Data · doi:10.1186/s40537-022-00637-9 · Cited by 9
“[30] proposed to fuse the lingual and image modals using the multi-modal compact bi-linearity.”
MethodM2FNet: Multi-granularity Feature Fusion Network for Medical Visual Question Answering
Wang, H., Pan, H., Zhang, K., He, S., & Chen, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20865-2_11 · Cited by 9
AttentionRNN: A Structured Spatial Attention Mechanism
Khandelwal, S., & Sigal, L. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00352 · Cited by 10
“1Reported performance for the baselines are from our PyTorch [23] re-implementation of [5], and are directly comparable to MCB+ARNN.”
BackgroundMethodResultInfluentialJoint Wasserstein Autoencoders for Aligning Multimodal Embeddings
Mahajan, S., Botschen, T., Gurevych, I., & Roth, S. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00557 · Cited by 12
“Image-text matching has also been studied in a classification setting, employing logistic regression or a softmax with cross-entropy [11, 43].”
MethodVC-VQA: Visual Calibration Mechanism For Visual Question Answering
Qiao, Y., Yu, Z., & Liu, J. (2020) · Proceedings - International Conference on Image Processing · doi:10.1109/icip40778.2020.9190828 · Cited by 9
“Existing models do not perform well on the balanced dataset [5][10], which shows that these models tend to be influenced by dataset bias, without focusing on visual information.”
BackgroundCross-modal Relational Reasoning Network for Visual Question Answering
Chen, H., Liu, R., & Peng, B. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00441 · Cited by 9
“A popular framework for VQA first represents images and questions as global features, then fuses them into a common space by bilinear fusion methods [9, 49, 5, 10, 23].”
BackgroundIlievski, I., & Feng, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.00584 · Cited by 9
“Recently, several representation fusion methods were developed [2, 11, 1] and some novel attention mechanisms were introduced [15, 14].”
BackgroundiParaphrasing: Extracting Visually Grounded Paraphrases via an ImagePreprint
Chu, C., Otani, M., & Nakashima, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.04284 · Cited by 10
“Wang et al. (2016a) proposed joint embeddings using a two-branch NN. Fukui et al. (2016) used a multimodal compact bilinear pooling method to combine textual and visual embeddings.”
Zero-Shot Transfer VQA DatasetPreprint
Li, Y., Yang, Y., Wang, J., & Xu, W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00692 · Cited by 10
“Non-attention methods include multimodal compact bilinear networks [5], relational networks [25], and Deeper LSTM Question+Image [18].”
Semantically-Aware Attentive Neural Embeddings for Image-based Visual LocalizationPreprint
Seymour, Z., Sikka, K., Chiu, H.-P., Samarasekera, S., & Kumar, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.03402 · Cited by 11
“Although more sophisticated fusion techniques e.g ., bilinear pooling have been used recently [17], we opted for projected sum pooling since it is easy to use, uses few trainable parameters, and also maintains the spatial configuration of the feature maps.”
MethodExploring Uncertainty in Conditional Multi-Modal Retrieval SystemsPreprint
Taha, A., Chen, Y.-T., Yang, X., Misu, T., & Davis, L. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.07702 · Cited by 9
“Other fusion methods, including gated [1], max [5] and bilinear [12] fusion, are employed in integrating natural language and visual modalities.”
MethodBiomedical Question Answering: A Comprehensive Review.Preprint
Jin, Q., Yuan, Z. P., Xiong, G., Yu, Q., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2021) · arXiv (Cornell University) · Cited by 11
Cai, L., Fang, H., & Li, Z. (2023) · The Journal of Supercomputing · doi:10.1007/s11227-023-05195-2 · Cited by 8
Xu, X., Tu, W., & Yang, Y. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102749 · Cited by 21
Yusuf, A. A., Feng, C., Mao, X., Haruna, Y., Li, X., & Duma, R. A. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128850 · Cited by 10
Multimodal Fusion Framework Based on Low-Rank Interaction for Tumor Prognostic Prediction
An, Y., Lan, R., Lin, H., Wang, Y., Deng, H., Liu, Z., Wang, L., Liu, Z., Lu, C. Y., Yang, H., et al. (2025) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbbio.2025.3578334 · Cited by 9
“Previous studies have shown that TFNs are more effective in capturing multimodal interactions than simple concatenation or pooling methods [37], [38].”
SCBM-Net: a multimodal feature fusion-based dual-channel method for bearing fault diagnosis
liu, Q., Tong, W.-Y., Bai, H.-W., & Dong, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-21665-4 · Cited by 8
“Multimodal Compact Bilinear Pooling (MCB) 27 enables efficient approximation of outer product interactions between multiple modalities by projecting the input features into a higher-dimensional space using randomized mappings and convolution in the frequency domain.”
Robust Explanations for Visual Question Answering
Patro, B. N., Patel, S., & Namboodiri, V. P. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093295 · Cited by 33
“Recent work from [60, 14, 15, 52, 31, 45, 27, 38, 41] considers region-based image attention.”
BackgroundLiu, D., Chen, L., Wang, L., & Wang, Z. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11260-w · Cited by 8
“The authors of [5] propose an improved MCB method.”
MethodMulti‐modal object detection via transformer network
Liu, W., Wang, H., Gao, Q., & Zhu, Z. (2023) · IET Image Processing · doi:10.1049/ipr2.12884 · Cited by 8
Tabata, M., Kurata, K., & Tamamatsu, J. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00363 · Cited by 10
“Various fusion methods have been explored, including concatenation [22], bilinear pooling [8, 40], and co-attention [39, 40].”
BackgroundMethodDivide and Conquer: Answering Questions with Object Factorization and Compositional Reasoning
Chen, S., & Zhao, Q. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00651 · Cited by 14
“These data efforts lead to the development of methods that advance VQA models from different perspectives, including multi-modal fusion [13,29,57], attention mechanism [2,28], structured inference [21–23, 48], and vision-and-language pretraining [33, 34, 38, 49].”
Fen, L., Chen, J., Li, K., Bai, J., Tan, W., Cai, C., & Ayub, M. S. (2023) · Digital Signal Processing · doi:10.1016/j.dsp.2023.104265 · Cited by 9
Asri, H. S., & Safabakhsh, R. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18871-z · Cited by 10
“Authors in [34, 35] devised a more complex combination with this assumption that point-wise addition and multiplication may not greatly denote the relationship among different features.”
Hu, Z., Wei, J., Huang, Q., Liang, H., Zhang, X., & Liu, Q.-G. (2020) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc50466.2020.00040 · Cited by 8
Cascading Top-Down Attention for Visual Question Answering
Tian, W., Zhou, R., & Zhao, Z. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207390 · Cited by 8
How LSTM Encodes Syntax: Exploring Context Vectors and Semi-Quantization on Natural Text
Shibata, C., Uchiumi, K., & Mochihashi, D. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.356 · Cited by 10
Robust Visual Reasoning via Language Guided Neural Module Networks
Akula, A., Jampani, V., Changpinyo, S., & Zhu, S. C. (2021) · neural information processing systems · Cited by 8
A document analysis deep learning regression model for initial coin offerings success prediction
Wang, J., Chen, R., Xu, W. X., Tang, Y., & Qin, Y. W. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118367 · Cited by 10
Transformer Gate Attention Model: An Improved Attention Model for Visual Question Answering
Zhang, H., & Wu, W. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9891887 · Cited by 9
“Fukui et al. [29] proposed “Multi -modal Compact Bilinear pooling” (MCB) that randomly projects the image and text features to a higher-dimensional space and then convolves both vectors with multiplications to predict the attention.”
BackgroundManmadhan, S., & Kovoor, B. C. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14981-2 · Cited by 8
“Joint Comprehension Concatenation, Element-wise addition, Element-wise multiplication, Canonical Correlation Analysis [11], Multimodal Compact Bilinear Pooling (MCB) [8], Attention [13, 50] • Med-VQA research is still in its early stages, and thus difficult to conclude the best way of comprehension.”
InfluentialLogical Implications for Visual Question Answering Consistency
Tascon-Morales, S., Márquez-Neila, P., & Sznitman, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00650 · Cited by 15
Zhang, W., Zeng, K., Xinming, Y., Shi, T., & Wang, P. (2023) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3584371.3613008 · Cited by 12
“The goal of QA is to automatically answer natural language questions about various data sources, such as databases [49], knowledge graphs [7], narrative texts [34], images [3], and multi-modal data types [11, 20].”
Background
Metadata
- Authors
- Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M.
- Affiliations
- Sony (Japan) (JP)
- University of California, Berkeley (US)
- Max Planck Institute for Informatics (DE)
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 457-468
- OpenAlex
- W2963383024
- Cited by
- 1,582
- References
- 60
- Citations by year
- 2026: 42 · 2025: 105 · 2024: 126 · 2023: 165 · 2022: 189 · 2021: 222 · 2020: 225 · 2019: 201 · 2018: 179 · 2017: 105 · 2016: 20
- FWCI
- 58.39
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Max Planck Digital Library (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1427425, 1212798, N000141110688)
- Defense Advanced Research Projects Agency
- Multidisciplinary University Research Initiative (N000141110688)
- Air Force Research Laboratory
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- visual question answering, visual grounding, multimodal fusion, outer product, attention mechanism, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.