Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years.However, tasks such as visual question answering require combining these vector representations with each other.Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations.We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors.As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features.We extensively evaluate MCB on the visual question answering and grounding tasks.We consistently show the benefit of MCB over ablations without MCB.For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation.This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, the PJ-X backbone MCB [16] (which is crucial for the answering performance) preceded the publication of the CLEVR dataset.”
From CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We replace the concatenation of the visual representation and the encoded phrase in Grounder with MCB to combine both modalities.”
MethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“, 2015) that is pretrained on ImageNet data (Deng et al., 2009).”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Also, for this model, we concatenated the learned word embedding with pretrained GloVe vectors (Pennington et al., 2014).”
MethodGirshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Research works printed but not linked: 3
- Ross Girshick. 2015. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
- Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014. Multimodal neural language mod- els. In Proceedings of the International Conference on Machine Learning (ICML), pages 595-603.
- Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems (NIPS).
Cited by
Items 701–800 of 1,002
V-Doc : Visual questions answers with Documents
Ding, Y., Huang, Z., Wang, R., Zhang, Y., Chen, X., Ma, Y., Chung, H., & Han, S. C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02083 · Cited by 20
“different multi-modal models, such as bilinear models [2], image-text fusing based transformers [11,16], self-attention [3], etc.”
BackgroundEmoMV: Affective music-video correspondence learning datasets for classification and retrieval
Thao, H. T. P., Roig, G., & Herremans, D. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.10.002 · Cited by 26
How to find a good image-text embedding for remote sensing visual question answering?Preprint
Chappuis, C., Lobry, S., Kellenberger, B., Saux, B. L., & Tuia, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.11848 · Cited by 27
“The first approach, multimodal compact bilinear pooling (MCB [9]), uses a random projection on the image and question features before combining them.”
BackgroundDai, W., Sun, X., Zhang, W., Wang, D., Tang, D., & Peng, Y. (2025) · Journal of Manufacturing Processes · doi:10.1016/j.jmapro.2025.01.033 · Cited by 19
Visual Grounding in 2D and 3D: A unified perspective and survey
Guo, K., Huang, Y., Jia, T., Song, X., Sun, S., Wei, H., Han, X.-F., Huang, S., Strisciuglio, N., & Li, S. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103625 · Cited by 38
Plenty is Plague: Fine-Grained Learning for Visual Question Answering
Zhou, Y., Ji, R., Sun, X., Su, J., Meng, D., Gao, Y., & Shen, C. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2956699 · Cited by 19
“At present, VQA is typically regarded as a multi-modal classification problem [1] [2] [7] [13] [6].”
BackgroundMethodInfluentialAccuracy vs. complexity: A trade-off in visual question answering models
Farazi, M. R., Khan, S. H., & Barnes, N. M. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108106 · Cited by 20
Visual question answering by pattern matching and reasoning
Zhan, H., Xiong, P., Wang, X., Wang, X., & Yang, L. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.10.016 · Cited by 21
Quality-Aware Multimodal Biometric Recognition
Soleymani, S., Dabouei, A., Taherkhani, F., Iranmanesh, S. M., Dawson, J. M., & Nasrabadi, N. M. (2021) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2021.3131664 · Cited by 25
“Recently, to overcome this shortcoming, compact bilinear pooling has been proposed [13], [14], [15], [21].”
BackgroundVAQA: Visual Arabic Question Answering
kamel, S. M., Hassan, S. I., & Elrefaei, L. A. (2023) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-023-07687-y · Cited by 19
“Therefore, multiple attempts were proposed to compress the bilinear pooling operation, including the Multi-modal Compact Bilinear pooling (MCB) method [23] that won the VQA workshop challenge 2016, Multi-modal lowrank Bilinear pooling (MLB) [34], Multi-modal Factorized Bilinear pooling (MFB) [24], Multi-modal Tucker F…”
Image Captioning with X-Gelu Activated Xgl Transformer
Sharma, D., Dhiman, C., & Kumar, D. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15291-3 · Cited by 21
“The very nature of it makes it suitable for applications such as image retrieval [12, 20], human-robot interaction [6, 27], aid to the blind [44], and visual question answering [11] and the like.”
Gupta, S., Saini, N., Kundu, S., & Das, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56060-6_2 · Cited by 19
A Multimodal Transfer Learning Approach Using PubMedCLIP for Medical Image Classification
Dao, H. N., Nguyen, T. D., Mugisha, C., & Paik, I. (2024) · IEEE Access · doi:10.1109/access.2024.3401777 · Cited by 24
Divert More Attention to Vision-Language Object Tracking
Guo, M., Zhang, Z., Jing, L., Ling, H., & Fan, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3409078 · Cited by 28
“Natural language contains high-level semantics and has been leveraged to foster vision-related tasks [45], [46], [47] including tracking [14], [15], [16].”
BackgroundAligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks
Gupta, T., Shih, K. J., Singh, S., & Hoiem, D. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.452 · Cited by 24
Multispectral Periocular Classification With Multimodal Compact Multi-Linear Pooling
Algashaam, F. M. A., Nguyen, K., Alkanhal, M. I., Chandran, V., Boles, W., & Banks, J. (2017) · IEEE Access · doi:10.1109/access.2017.2731118 · Cited by 19
“Hence the outer product can be efficiently calculated, allowing multiplicative interaction between all elements of component vectors with a small number of parameters, activations and computation [17].”
BackgroundMethodInfluentialCompositing-Aware Image Search
Zhao, H., Shen, X., Lin, Z., Sunkavalli, K., Price, B. L., & Jia, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01219-9_31 · Cited by 20
“The image and word features are then fed into a multimodal compact bilinear pooling (MCB) module [8].”
MethodInfluentialAttentive Interactive Convolutional Matching for Community Question Answering in Social Multimedia
Hu, J., Qian, S., Fang, Q., & Xu, C. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240626 · Cited by 19
“Fukui et al. [7] first introduce the bilinear model to solve the problem of multi-modal feature fusion in QA.”
AnswerNet: Learning to Answer Questions
Wan, Z., & He, H. (2018) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2018.2884486 · Cited by 20
“The results for MCB are reported in [43].”
BackgroundMethodInfluentialDensely Connected Attention Flow for Visual Question Answering
Liu, F., Liu, J., Fang, Z., Hong, R., & Lu, H. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/122 · Cited by 19
“To capture high-level interactions between the two modalities, some bilinear pooling methods, including MCB [Fukui et al., 2016], MLB [Kim et al.”
BackgroundMethodInfluentialNeural Reasoning, Fast and Slow, for Video Question Answering
Le, T. M., Le, V., Venkatesh, S., & Tran, T. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207580 · Cited by 21
“We also list here some other methods provided by [14] including those proposed in [6] and [31].”
MethodVideo question answering via grounded cross-attention network learning
Ye, Y., Zhang, S., Li, Y., Qian, X., Tang, S., Pu, S., & Xiao, J. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102265 · Cited by 19
Combination of deep speaker embeddings for diarisation
Sun, G., Zhang, C., & Woodland, P. C. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.04.020 · Cited by 25
“Commonly used bilinear pooling decomposition methods include count sketches and convolutions (Gao et al., 2016), low-rank approximations (Kim et al., 2017), and various other tensor decomposition methods (Fukui et al., 2016; Ben-younes et al., 2017, 2019).”
MethodRegularizing Attention Networks for Anomaly Detection in Visual Question Answering
Lee, D., Cheon, Y., & Han, W.-S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16279 · Cited by 22
Deep Cross-Attention Network for Crowdfunding Success Prediction
Tang, Z., Yang, Y., Li, W., Lian, D., & Duan, L. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3141256 · Cited by 23
“We also experiment with various other feature fusion methods that are element-wise product (EWP), element-wise sum (EWS), and multimodal compact bilinear pooling [48] (MCB).”
MethodInner Knowledge-based Img2Doc Scheme for Visual Question Answering
Li, Q., Xiao, F., Bhanu, B., Sheng, B., & Hong, R. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3489142 · Cited by 19
“The multi-modal fusion methods include element-wise product [46], Multi-modal Compact Bilinear ( MCB ) [14], Multi-modal Factorized Bilinear pooling ( MFB ) [57], attention mechanisms [58], knowledge base-enhanced approaches [63], and so on.”
MethodAnomaly Matters: An Anomaly-Oriented Model for Medical Visual Question Answering
Cong, F., Xu, S., Guo, L., & Tian, Y. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3185113 · Cited by 22
“Multimodal compact bilinear pooling (MCB) [18] is designed to map the input feature to a lower-dimensional space by a linear transformation to reduce the computational cost of the outer product.”
BackgroundGuo, D., Zhang, J., Yang, B., & Lin, Y. (2022) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3572792 · Cited by 21
Large-Scale Traffic Congestion Prediction based on Multimodal Fusion and Representation Mapping
Zhou, B., Liu, J., Cui, S., & Zhao, Y. (2022) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa54385.2022.10032443 · Cited by 23
“Based on joint representation, many classification or clustering tasks have been applied, such as event detection [32] [14], sentiment analysis [22] [37], and visual question answering [13].”
MethodYORO - Lightweight End to End Visual Grounding
Ho, C.-H., Appalaraju, S., Jasani, B. A., Manmatha, R., & Vasconcelos, N. M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25085-9_1 · Cited by 26
Stacking with Auxiliary Features for Visual Question Answering
Rajani, N. F., & Mooney, R. J. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1201 · Cited by 19
“The MCB model combines the vision and language vector representations using an outer product instead of the traditional approach of using concatenation or element-wise product or sum of the two vectors (Fukui et al., 2016).”
BackgroundMethodInfluentialHaurilet, M., Roitberg, A., & Stiefelhagen, R. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00203 · Cited by 21
“In general, the ways of addressing this problem can be divided into four categories: 1) Global embedding methods [31, 1, 28, 34, 35, 30] that use a joint embedding of the global image representation and the question to produce an answer; 2) Models that attend to parts of the image are able to improve performance [50,…”
MethodSelective residual learning for Visual Question Answering
Hong, J. K., Park, S., & Byun, H. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.03.098 · Cited by 21
Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking
Akhtar, M., Cocarascu, O., & Simperl, E. (2023) · Findings of the Association for Computational Linguistics: EACL 2023 · doi:10.18653/v1/2023.findings-eacl.30 · Cited by 18
VGRSS: Datasets and Models for Visual Grounding in Remote Sensing Ship Images
Chen, Y., Zhan, L., Zhao, Y., Xiong, S., & Lu, X. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3562717 · Cited by 23
“Early works utilized CNNs combined with LSTM or Bi-LSTM to extract and fuse visual and textual features, with multimodal compact bilinear (MCB) pooling being introduced to enhance multimodal feature integration [23], [24], [25].”
Biswas, D., & Tešić, J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127857 · Cited by 18
Understanding question-answering systems: Evolution, applications, trends, and challenges
Farea, A., & Emmert‐Streib, F. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110997 · Cited by 21
Overcoming language priors in VQA via adding visual module
Zhao, J., Zhang, X., Wang, X., Yang, Y., & Sun, G. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-06923-0 · Cited by 19
“At present, the methods of VQA can be divided into base modal fusion methods [1–5], attention model methods [6–9], based on external knowledge model methods [10–12], relational reasoning model methods [13–15], etc.”
MethodPang, H., Wei, S., Zhang, G., Zhang, S., Qiu, S., & Zhao, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3208742 · Cited by 19
“In visual question answering, some works [47], [48], [49] utilize the method of bi-linear pooling to interact and fuse the information of language and vision.”
MethodMulti-modal adaptive gated mechanism for visual question answering
Xu, Y., Zhang, L., & Shen, X. (2023) · PLoS ONE · doi:10.1371/journal.pone.0287557 · Cited by 22
“Based on Deep Modular Co-Attention [25] and inspired by related work [29– 33], this paper proposes a novel and efficient Multi-modal Adaptive Gated Mechanism (MAGM) model for visual question answering noise information filtering problems.”
Ramanarayanan, S., Palla, A., Ram, K., & Sivaprakasam, M. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110633 · Cited by 17
“The emerging directions in multimodal learning have shown that the training data acquired from one modality can benefit from the shared knowledge across different modalities [29, 30].”
BackgroundBose, P., Rana, P., & Ghosh, P. (2023) · IEEE Access · doi:10.1109/access.2023.3299877 · Cited by 23
“Attention mechanisms have been applied on image-text based multimodal learning tasks starting in 2015-16 [16], [20], [21], [22], [23].”
MethodRelational reasoning and adaptive fusion for visual question answering
Shen, X., Han, D., Zong, L., Guo, Z., & Hua, J. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05437-7 · Cited by 18
“Traditional modality fusion methods [16, 17] mainly focus on fusing shallow features.”
InfluentialLiu, G., Li, P., Zhao, Z., He, J., He, G., & Zhong, S. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104748 · Cited by 22
Choi, H., Zhang, L., & Watkins, C. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129415 · Cited by 18
PIRC Net: Using Proposal Indexing, Relationships and Context for Phrase Grounding
Kovvuri, R., & Nevatia, R. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20870-7_28 · Cited by 21
Kim, E.-S., On, K.-W., Kim, J.-S., Heo, Y., Choi, S.-H., Lee, H.-D., & Zhang, B.-T. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11018-5_28 · Cited by 18
“Na et al[14] tried to learn multimodal joint representation using multimodal compact bilinear pooling[8].”
BackgroundRevisiting Image-Language Networks for Open-Ended Phrase Detection
Plummer, B. A., Shih, K. J., Li, Y., Xu, K., Lazebnik, S., Sclaroff, S., & Saenko, K. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3029008 · Cited by 25
“In the most common scenario, the text query to be localized is assumed to be present in the test image [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14].”
BackgroundToward Faster and Simpler Matrix Normalization via Rank-1 Update
Yu, T., Cai, Y., & Li, P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58529-7_13 · Cited by 18
“Bilinear pooling has achieved excellent performance in many computer vision tasks, such as fine-grained recognition [22, 30, 20, 29, 19], generic image recognition [18], visual question answering [7, 36] and action classification [14, 31, 3, 28].”
BackgroundDual-level interactive multimodal-mixup encoder for multi-modal neural machine translation
Ye, J., & Guo, J. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03331-8 · Cited by 18
“[19] adopt multimodal compact bilinear pooling to fuse the grid features and textual features; [15] employ grid visual features with multi-modal attention mechanism on the decoder; [2] adopt two independent attention mechanisms to process source language words and grid visual features separately; [11] concatenate text…”
MethodAn Effective Dense Co-Attention Networks for Visual Question Answering
He, S., & Han, D. (2020) · Sensors · doi:10.3390/s20174897 · Cited by 18
“[29] used multiple attention layers or generated multiple attention maps to realize multi-step reasoning or multiple attention on images.”
MethodTIGEr: Text-to-Image Grounding for Image Caption EvaluationPreprint
Jiang, M., Huang, Q., Zhang, L., Wang, X. E., Zhang, P., Gan, Z., Diesner, J., & Gao, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02050 · Cited by 29
“, softmax function) (Jabri et al., 2016; Fukui et al., 2016).”
BackgroundExplainable High-order Visual Question Reasoning: A New Benchmark and Knowledge-routed NetworkPreprint
Cao, Q., Li, B., Liang, X., & Lin, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.10128 · Cited by 18
“Moreover, the human-annotated questions-answers suffer from high correlations, which enables the end-to-end networks [1], [3], [4], [9]–[11] to achieve high performance by naively exploiting the dataset bias rather than”
BackgroundDetailed Annotations of Chest X-Rays via CT Projection for Report UnderstandingPreprint
Seibold, C., Reiß, S., Sarfraz, M. S., Fink, M. A., Mayer, V. L., Sellner, J., Kim, M. S., Maier-Hein, K. H., Kleesiek, J., & Stiefelhagen, R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.03416 · Cited by 31
“We extract the top 200 scoring boxes for each labeled image following most phrase grounding methods [15, 18, 68].”
BackgroundMethodDas, A., Agarwal, V., & Shetty, N. P. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1608837 · Cited by 19
Instance-sequence reasoning for video question answering
Liu, R., & Han, Y. (2022) · Frontiers of Computer Science · doi:10.1007/s11704-021-1248-1 · Cited by 17
Depth-Aware and Semantic Guided Relational Attention Network for Visual Question Answering
Liu, Y., Wei, W., Peng, D., Mao, X.-L., Zhi-yong, H., & Zhou, P. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3190686 · Cited by 23
“Indeed, there already exist several early works in VQA domain to research on fusion strategies [4]–[6], [28] and attention mechanisms [1], [29] to preserve fine-gained feature of images for joint visual-language representation learning, e.g., small objects [29] or question-relevant regions [1].”
BackgroundMiao, Y., He, S., Cheng, W., Li, G., & Tong, M. (2022) · Scientific Reports · doi:10.1038/s41598-022-21149-9 · Cited by 19
Answer Mining from a Pool of Images: Towards Retrieval-Based Visual Question Answering
Penamakuri, A. S., Gupta, M., Gupta, M. D., & Mishra, A. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/146 · Cited by 27
Li, Y., Yang, Q., Wang, F. L., Lee, L., Qu, Y., & Hao, T. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102667 · Cited by 22
Layerwised multimodal knowledge distillation for vision-language pretrained model
Wang, J., Liao, D., Zhang, Y., Xu, D., & Zhang, X. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106272 · Cited by 22
Generative Attention Model with Adversarial Self-learning for Visual Question Answering
Ilievski, I., & Feng, J. (2017) · Thematic Workshops of ACM Multimedia · doi:10.1145/3126686.3126695 · Cited by 20
“[16] which showed that simple element-wise product results in a multimodal representations superior to representations obtained with more complex methods such as multimodal compact bilinear pooling [6].”
BackgroundMethodInfluentialExplicit ensemble attention learning for improving visual question answering
Lioutas, V., Passalis, N., & Tefas, A. (2018) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.04.031 · Cited by 19
Zhejiang University at ImageCLEF 2019 Visual Question Answering in the Medical Domain.
Yan, X., Li, L., Xie, C., Xiao, J., & Gu, L. (2019) · CLEF (Working Notes) · Cited by 17
Identifying Illicit Drug Dealers on Instagram with Large-scale Multimodal Data Fusion
Hu, C., Yin, M., Liu, B., Li, X., & Ye, Y. (2021) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3472713 · Cited by 21
“The third is compact bilinear pooling [15].”
BackgroundMethodInfluentialQuestion-Guided Erasing-Based Spatiotemporal Attention Learning for Video Question Answering
Liu, F., Liu, J., Hong, R., & Lu, H. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3105280 · Cited by 22
“Fukui et al. [22], Kim et al. [23], Ben-Younes et al. [24], and Yu et al. [25] introduced bilinear fusion in attention mechanisms to capture fine image-question interactions, and thus, generate more accurate attention weights.”
BackgroundEnd-to-end deep learning for reverse driving trajectory of autonomous bulldozer
You, K., Ding, L., Jiang, Y., Wu, Z., & Zhou, C. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109402 · Cited by 18
VGNMN: Video-grounded Neural Module Networks for Video-Grounded Dialogue Systems
Le, H., Chen, N. F., & Hoi, S. C. H. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.247 · Cited by 28
A Multi-level Mesh Mutual Attention Model for Visual Question Answering
Lei, Z., Zhang, G., Wu, L., Zhang, K., & Liang, R. (2022) · Data Science and Engineering · doi:10.1007/s41019-022-00200-9 · Cited by 17
“[8] built a multi-modal compact bilinear pooling model, which simultaneously uses the outer product and the Kronecker product to complete the multi-modal fusion operation.”
BackgroundMethodInfluentialLearning Convolutional Text Representations for Visual Question Answering
Wang, Z., & Ji, S. (2018) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611975321.67 · Cited by 26
“Meanwhile, unlike in [11], we do not use additional data sources like the pre-trained word embedding (Word2Vec, GloVe) and other dataset (Visual Genome [21]) to augment training.”
BackgroundMethodInfluentialLópez-Sánchez, D., Arrieta, A. G., & Corchado, J. M. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.05.104 · Cited by 17
“However, an interesting line for future work would be the possibility of extending CBP-KRP to the case where bilinear descriptors are formed as the outer product of the descriptors extracted by two different CNNs, as this could have applications in multi-modal problems [45,46] .”
BackgroundThe Color of the Cat is Gray: 1 Million Full-Sentences Visual Question Answering (FSVQA)Preprint
Shin, A., Ushiku, Y., & Harada, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.06657 · Cited by 17
“Words in the question were input to LSTM (Hochreiter and Schmidhuber 1997) one at a time as one-hot vector, where the dictionary contains only the words appearing more than once.”
BackgroundWays of Conditioning Generative Adversarial NetworksPreprint
Kwak, H., & Zhang, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01455 · Cited by 19
“E ven though this forms a joint representation, it is hard to fully capture the complex associations between the two different modalities[12].”
BackgroundMethodVideoNavQA: Bridging the Gap between Visual and Embodied Question AnsweringPreprint
Cangea, C., Belilovsky, E., Lió, P., & Courville, A. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04950 · Cited by 19
“Widely used VQA models include stacked attention networks [38], which use the question embedding as a query for attending over the visual input, multimodal compact bilinear pooling [8], which fuses the text and visual embeddings via multiplying them in the Fourier space, feature-wise linear modulation [26], which sele…”
MethodImproving VQA and its Explanations \\ by Comparing Competing Explanations
Wu, J., Chen, L., & Mooney, R. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15631 · Cited by 8
“A large number of previous VQA systems (Fukui et al., 2016; Ben-Younes et al., 2017; Ramakrishnan et al., 2018) utilize a trainable top-down at-”
Cong, Y., & Mo, H. (2025) · International Journal of Intelligent Systems · doi:10.1155/int/5124400 · Cited by 27
“Te most common strategies for fusing visual and text features involve methods like afne transformation [86], vector outer product [87], matrix decomposition [88], etc. Researches have shown that traditional VQA methods possess elements that are not pertinent to providing answers during feature…”
Wang, J., Ji, Y., Sun, J., Yang, Y., & Sakai, T. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.196 · Cited by 16
Improving visual question answering by combining scene-text information
Sharma, H., & Jalal, A. S. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-12317-0 · Cited by 18
Fen, L., Chen, J., Li, K., Tan, W., Cai, C., & Ayub, M. S. (2022) · Entropy · doi:10.3390/e24121836 · Cited by 18
Multi-label movie genre classification based on multimodal fusion
Cai, Z.-H., Ding, H., Wu, J., Xi, Y., Wu, X., & Cui, X. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16121-2 · Cited by 18
“…these fusion methods, some are relatively simple, such as concat-enation, element-wise sum and element-wise product, which tend to be easier to implement and less computing costs, but in performance they may be slightly inferior to some complex fusion methods, such as MCB [12, 40] and MCF [38].”
Gesture-to-Text: A Real-Time Indian Sign Language Translator with Pose Estimation and LSTMs
Shetty, S., Hirani, E., Singh, A., & Koshy, R. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.04.253 · Cited by 16
Deep Fuzzy Multiteacher Distillation Network for Medical Visual Question Answering
Liu, Y., Chen, B., Wang, S., Lu, G., & Zhang, Z. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3402086 · Cited by 19
“Numerous existing approaches have adopted cross-modal reasoning strategies directly from general VQA models, such as multimodal compact bilinear pooling (MCB) [41], stacked attention network (SAN) [42], and bilinear attention network (BAN) [43], to facilitate answer prediction.”
BackgroundChameleon: A Multimodal Learning Framework Robust to Missing Modalities
Liaqat, M. I., Nawaz, S., Zaheer, M. Z., Saeed, M. S., Sajjad, H., Schepper, T. D., Nandakumar, K., Khan, M. H., Gallo, I., & Schedl, M. (2025) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-025-00370-y · Cited by 23
“…tasks such as multimodal classification [1, 2], cross-modal retrieval [3], cross-modal verification [4], multimodal named entity recognition [5, 6], visual question answering [7, 8], image captioning [9], semantic relatedness [10], segmentation [11], and multi-modal machine translation [12, 13].”
Object Referring in Visual Scene with Spoken Language
Vasudevan, A. B., Dai, D., & Gool, L. V. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00206 · Cited by 20
Hinami, R., & Satoh, S. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1281 · Cited by 18
“81 Multimodal compact bilinear (Fukui et al., 2016) - - - - - - - - 48.”
Adversarial Learning of Answer-Related Representation for Visual Question Answering
Liu, Y., Zhang, X., Huang, F., & Li, Z. (2018) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3269206.3271765 · Cited by 17
“The last type of methods [2, 7, 10] focus on multi-modal feature embedding and resort to a bilinear pooling for VQA.”
MethodVQA With No Questions-Answers Training
Vatashsky, B.-Z., & Ullman, S. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01039 · Cited by 18
“As end-to-end training dominates current answering schemes, many works focused on improving the image-question fused features [10, 8, 45], various attention mechanisms for selecting important features [43, 17, 31, 29, 4] and incorporating outputs of other visual tasks [12, 3, 9, 36].”
BackgroundHierarchical Visual-Textual Knowledge Distillation for Life-Long Correlation Learning
Peng, Y., Qi, J., Ye, Z., & Zhuo, Y. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-020-01392-1 · Cited by 23
“Fukui et al. (2016) propose multimodal compact bilinear pooling to combine visual and textual representations.”
BackgroundLinguistically Routing Capsule Network for Out-of-distribution Visual Question Answering
Cao, Q., Wan, W., Wang, K., Liang, X., & Lin, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00164 · Cited by 16
“Later, lots of works [15, 28, 48, 6] focused on the joint embedding of image and question.”
BackgroundMHKD-MVQA: Multimodal Hierarchical Knowledge Distillation for Medical Visual Question Answering
Wang, J., Huang, S., Du, H., Qin, Y. X., Wang, H., & Zhang, W. (2022) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9995473 · Cited by 22
“In our experiments, we have compared our model with representative models, including VGG16+BERT+MFB [41], CGMVQA [8], MMBERT [11] on VQA-MED 2019 dataset and MCB [42], MEVF [5], CR [43], MMBERT on VQA-RAD dataset.”
MethodJose, J., Yılmaz, E., Magalhães, J., Castells, P., Ferro, N., Silva, M. J., & Martins, F. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-45439-5 · Cited by 21
Towers of Babel: Combining Images, Language, and 3D Geometry for Learning Multimodal Vision
Wu, X., Averbuch‐Elor, H., Sun, J. H., & Snavely, N. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00048 · Cited by 29
“Popular tasks include instruction following [5, 32, 8], visual question answering [6, 15, 22, 4], and phrase localization [29, 49, 45].”
BackgroundAMC: Attention guided Multi-modal Correlation Learning for Image SearchPreprint
Chen, K., Bui, T., Chen, F., Wang, Z., & Nevatia, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.00763 · Cited by 18
“The dictionary sizes for the query modality in Clickture and ASD are 85636 and Approach Img Key VAN LAN LF MTN MB [31] DSSM-Key [11] DSSM-Img [11] RCCA [39] Img ATT Key ATT Img ATT -Key ATT -LF AMC Full 17388.”
BackgroundMethodNeuron Interaction Based Representation Composition for Neural Machine Translation
Li, J., Wang, X., Yang, B., Shi, S., Lyu, M. R.-T., & Tu, Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6334 · Cited by 17
Hou, G., Jia, L., Zhang, Y., Wu, W. D., Zhao, L., Zhao, J., Wang, L., & Qiang, Y. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106140 · Cited by 18
Unified Transformer with Cross-Modal Mixture Experts for Remote-Sensing Visual Question Answering
Liu, G., He, J., Li, P., Zhong, S., Li, H., & He, G. (2023) · Remote Sensing · doi:10.3390/rs15194682 · Cited by 17
Localized Questions in Medical Visual Question Answering
Tascon-Morales, S., Márquez-Neila, P., & Sznitman, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43895-0_34 · Cited by 15
RSMoDM: Multimodal Momentum Distillation Model for Remote Sensing Visual Question Answering
Li, P., Liu, G., He, J., Meng, X., Zhong, S., & Chen, X. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3419035 · Cited by 15
“Techniques such as multimodal compact bilinear (MCB) pooling [36], multi-modal low-rank bilinear (MLB) pooling [37], and multimodal factorized bilinear (MFB) pooling [38] emerged, enhancing the interaction and integration between images and textual features more effectively.”
BackgroundPERS: Parameter-Efficient Multimodal Transfer Learning for Remote Sensing Visual Question Answering
He, J., Liu, G., Li, P., Su, X., Jiang, W., Zhang, D., & Zhong, S. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3447086 · Cited by 15
“However, Chappuis et al. [20] confirmed in their study that the elementwise fusion strategy in RSVQA was the least effective among the three existing feature fusion methods [14], [21], [22].”
BackgroundResultGeneralized Hadamard-Product Fusion Operators for Visual Question Answering
Duke, B., & Taylor, G. W. (2018) · Canadian Conference on Computer and Robot Vision · doi:10.1109/crv.2018.00016 · Cited by 15
Metadata
- Authors
- Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M.
- Affiliations
- Sony (Japan) (JP)
- University of California, Berkeley (US)
- Max Planck Institute for Informatics (DE)
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 457-468
- OpenAlex
- W2963383024
- Cited by
- 1,582
- References
- 60
- Citations by year
- 2026: 42 · 2025: 105 · 2024: 126 · 2023: 165 · 2022: 189 · 2021: 222 · 2020: 225 · 2019: 201 · 2018: 179 · 2017: 105 · 2016: 20
- FWCI
- 58.39
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Max Planck Digital Library (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1427425, 1212798, N000141110688)
- Defense Advanced Research Projects Agency
- Multidisciplinary University Research Initiative (N000141110688)
- Air Force Research Laboratory
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- visual question answering, visual grounding, multimodal fusion, outer product, attention mechanism, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.