Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years.However, tasks such as visual question answering require combining these vector representations with each other.Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations.We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors.As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features.We extensively evaluate MCB on the visual question answering and grounding tasks.We consistently show the benefit of MCB over ablations without MCB.For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation.This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, the PJ-X backbone MCB [16] (which is crucial for the answering performance) preceded the publication of the CLEVR dataset.”
From CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We replace the concatenation of the visual representation and the encoded phrase in Grounder with MCB to combine both modalities.”
MethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“, 2015) that is pretrained on ImageNet data (Deng et al., 2009).”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Also, for this model, we concatenated the learned word embedding with pretrained GloVe vectors (Pennington et al., 2014).”
MethodGirshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Research works printed but not linked: 3
- Ross Girshick. 2015. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
- Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014. Multimodal neural language mod- els. In Proceedings of the International Conference on Machine Learning (ICML), pages 595-603.
- Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems (NIPS).
Cited by
Items 401–500 of 1,002
Progressive Graph Attention Network for Video Question Answering
Peng, L., Yang, S., Bin, Y., & Wang, G. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475193 · Cited by 59
“Other approaches [7, 18, 38] proposed several models based on bilinear pooling to overcome the problem of multi-modal fusion.”
BackgroundMethodAn analysis of graph convolutional networks and recent datasets for visual question answering
Yusuf, A. A., Feng, C., & Xianling, M. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10151-2 · Cited by 59
“In joint embedding approaches, text features can be extracted using bag-of-words (BOW) or long short term memory (LSTM) (Ben-Younes et al. 2017; Fukui et al. 2016; Shih et al. 2016).”
BackgroundMethodInfluentialFine-Grained Video Categorization with Redundancy Reduction Attention
Zhu, C., Tan, X., Zhou, F., Liu, X., Yue, K., Ding, E., & Ma, Y. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01228-1_9 · Cited by 49
“The soft attention mechanism [5,33] is able to select the most discriminative regional features.”
BackgroundMethodMultimodal Document Image Classification
Jain, R., & Wigington, C. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00021 · Cited by 58
Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend
Chai, W., & Wang, G. (2022) · Applied Sciences · doi:10.3390/app12136588 · Cited by 56
Visual Question Answering as Reading Comprehension
Li, H., Wang, P., Shen, C., & van den Hengel, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00648 · Cited by 54
“[8] argue that these simple manners are not expressive enough and propose MCB which allows a multiplicative interaction between all elements of image and text vectors.”
BackgroundMethodVisual Question Answering With Dense Inter- and Intra-Modality Interactions
Liu, F., Liu, J., Fang, Z., Hong, R., & Lu, H. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3026892 · Cited by 48
Consensus Graph Representation Learning for Better Grounded Image Captioning
Zhang, W., Shi, H., Tang, S., Xiao, J., Yu, Q., & Zhuang, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16452 · Cited by 65
An Empirical Study on Leveraging Scene Graphs for Visual Question AnsweringPreprint
Zhang, C., Chao, W., & Xuan, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.12133 · Cited by 57
“ires comprehending and reasoning with visual and textual information. Existing algorithms mostly adopt the pipeline that first extracts image and question features [30], followed by multi-modal fusion [6,18] and attention [2,47,71] to obtain multi-modal features to infer the answer. While achieving promising results, i…”
Multimodal Unified Attention Networks for Vision-and-Language InteractionsPreprint
Zhou, Y., Cui, Y., Yu, J., Tao, D., & Tian, Q. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04107 · Cited by 56
“, X → Y ) [16][30], or the co-attention methods, which model bidirectional inter-modal interactions (i.”
BackgroundMethodA Review of Multimodal Explainable Artificial Intelligence: Past, Present and FuturePreprint
Sun, S., An, W., Tian, F., Fang, N., Liu, Q., Liu, J., Shah, N., & Chen, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.14056 · Cited by 64
“Systems for tasks like visual question answering [164] and fine-grained image classification [165] integrate explanation generation, providing multimodal outputs such as attention maps and textual interpretations [166].”
Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly
Whitehead, S., Petryk, S., Shakib, V., Gonzalez, J. E., Darrell, T., Rohrbach, A., & Rohrbach, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_9 · Cited by 90
Self-citationAl-Tameemi, I. K. S., Feizi‐Derakhshi, M., Pashazadeh, S., & Asadpour, M. (2023) · IEEE Access · doi:10.1109/access.2023.3307716 · Cited by 54
“Motivated by the attention mechanism’s notable achievements in various vision-based tasks [49], [50], our goal is to highlight the significant emotional segments of an image using the perspective of regions and scenes, respectively.”
BackgroundA Multimodal Memes Classification: A Survey and Open Research Issues
Afridi, T. H., Alam, A., Khan, M. N., Khan, J., & Lee, Y.-K. (2020) · Innovations in Smart Cities Applications Volume 4 · doi:10.1007/978-3-030-66840-2_109 · Cited by 45
“Later methods learned multimodal features using bilinear pooling [14].”
MethodRUArt: A Novel Text-Centered Solution for Text-Based Visual Question Answering
Jin, Z., Wu, H., Yang, C. H., Zhou, F., Qin, J., Xiao, L., & Yin, X.-C. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3120194 · Cited by 53
“[18], Kim et al.”
Are we pretraining it right? Digging deeper into visio-linguistic pretraining
Singh, A., Goswami, V., & Parikh, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.08744 · Cited by 27
“Visio-Linguistic Pretraining (V&L) Contrary to earlier works in vision and language [15, 19, 24, 32, 33, 56, 57] which designed specialized architectures for different tasks, many recently introduced models for V&L [2,12,25,26,30,31,47,48,59] provide a common architecture that can be pretrained…”
MethodFu, Z., Liu, F., Wang, H., Qi, J., Fu, X., Zhou, A., & Li, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.02172 · Cited by 66
“[21] Early 71.04 26.87M MCBP [22] Early 71.32 51.03M MMTM [24] Model 73.12 31.97M MSAF [25] Model 74.86 25.94M ERANNs [26] Model 74.80 - CFN-SR (Ours) Model 75.76 26.30M 5.”
InfluentialVideo Question Answering with Spatio-Temporal Reasoning
Jang, Y., Song, Y., Kim, C. D., Yu, Y., Kim, Y., & Kim, G. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01189-x · Cited by 46
“VQA-MCB, on the other hand, uses multimodal compact bilinear pooling to handle visual-textual fusion and spatial attention (Fukui et al. 2016).”
BackgroundMethodResultInfluentialZhang, Y., Han, J., Chen, H., Hu, F., Huang, Y., Tian, G., Zhong, D., & Yang, J. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf580 · Cited by 47
“Next, the patient’s image features and nuclear segmentation features were combined using the MCB [40] model, and the resulting fused features were utilized to train the MSI and TMB pr ediction models (Fig.”
Knowledge Acquisition for Visual Question Answering via Iterative Querying
Zhu, Y., Lim, J. J., & Fei-Fei, L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.651 · Cited by 50
“Our simple MLP model augmented by the memory bank show competitive results compared to existing models [9, 23] that have a much complex design.”
BackgroundMethodResultInfluentialMultimodal Learning and Reasoning for Visual Question Answering
Ilievski, I., & Feng, J. (2017) · neural information processing systems · Cited by 73
Visual Question Answering: A Tutorial
Teney, D., Wu, Q., & van den Hengel, A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2739826 · Cited by 48
“I [42] 80.6 36.5 43.7 58.2 63.1 Neural modules networks [4] 81.2 37.7 44 58.7 — Stacked attention networks [87] — — — 58.9 — Dynamic memory networks (DMNs+) [83] — — — 60.4 — DualNet [60] 81.9 37.8 49.7 61.7 66.7 Hierarchical coattention (HieCoAtt) [43] — — — 62.1 66.1 VQA-machine [74] 81.4 38.2 53.2 63.3 67.8 MLB [34…”
InfluentialStacked Latent Attention for Multimodal Reasoning
Fan, H., & Zhou, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00118 · Cited by 46
“It has been particularly successful for tasks requiring combined reasoning on multiple modalities such as visual question answering [15], image captioning [22], image-text matching [16], visual grounding [4] and others.”
BackgroundMethodResultInfluentialRevisiting Multi-Task Learning with ROCK: a Deep Residual Auxiliary Block for Visual Detection
Mordan, T., Thome, N., Hénaff, G., & Cord, M. (2018) · HAL · Cited by 63
AdaVQA: Overcoming Language Priors with Adapted Margin Cosine Loss
Guo, Y., Nie, L., Cheng, Z., Ji, F., Zhang, J., & Bimbo, A. D. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/98 · Cited by 60
New Ideas and Trends in Deep Multimodal Content Understanding: A Review
Chen, W., Wang, W., Liu, L., & Lew, M. S. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.10.042 · Cited by 50
Chandu, K. R., Bisk, Y., & Black, A. W. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.375 · Cited by 84
CRA-Net: Composed Relation Attention Network for Visual Question Answering
Peng, L., Yang, Y., Wang, Z., Wu, X. Y., & Huang, Z. H. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350925 · Cited by 45
Liang, T., Lin, G., Wan, M., Li, T., Ma, G., & Lv, F. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01505 · Cited by 65
“Our proposed approach is compared to the original uni-modal networks (i.e., DenseNet and language BERT), as well as the existing state-of-the-art image-text multimodal classification methods, including [1,7,12,14–16, 18,20,26].”
BackgroundInfluentialQSFVQA: A Time Efficient, Scalable and Optimized VQA Framework
Chowdhury, S., & Soni, B. (2023) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-023-07661-8 · Cited by 46
“The initial approach was VQA models to have a CNN module to generate image features and LSTM/CNN module to generate text features from question [6,9,20].”
MethodTransformer-based contrastive prototypical clustering for multimodal remote sensing data
Cai, Y., Zhang, Z., Ghamisi, P., Rasti, B., Liu, X., & Cai, Z. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119655 · Cited by 48
Kim, D., Kim, S., & Kwak, N. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1347 · Cited by 54
“Then, various attention algorithms have been mainly developed in this field and meth-ods of fusing textual and visual information such as bilinear pooling (Fukui et al., 2016; Yu et al.) have also been widely studied.”
MethodMulti-modal neural machine translation with deep semantic interactions
Su, J., Chen, J., Jiang, H. Q., Zhou, C., Lin, H., Ge, Y., Wu, Q., & Lai, Y. (2020) · Information Sciences · doi:10.1016/j.ins.2020.11.024 · Cited by 54
“In this work, we mainly explore image-text semantic interactions for multi-modal NMT, which have been proven effective in many tasks, such as VQA [32,44,22,42,31,41,52], NER [51], QA [43] etc.”
BackgroundMethodDriving behavior explanation with multi-level fusion
Ben-younes, H., Zablocki, É., Pérez, P., & Cord, M. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108421 · Cited by 52
“Some work perform this multi-modal fusion using bilinear models, which are made tractable by sampling methods like MCB [12] or through tensor factorization such as MUTAN [4], MLB [17], MFB [39] or BLOCK [5].”
MethodKnowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding
Cao, Q., Li, B., Liang, X., Wang, K., & Lin, L. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3045034 · Cited by 58
“Subsequently, a lot of efforts have been devoted to the joint embedding of image and question representation [5], [19]–[21].”
BackgroundGraph-Based Multi-Interaction Network for Video Question Answering
Gu, M., Zhao, Z., Jin, W., Hong, R., & Wu, F. Q. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3051756 · Cited by 45
“• VQA-MCB [78] is also an image-based method which utilizes multimodal compact bilinear pooling and spatial attention.”
Method“Subverting the Jewtocracy”: Online Antisemitism Detection Using Multimodal Deep Learning
Chandra, M., Pailla, D. R., Bhatia, H., Sanchawala, A., Gupta, M., Shrivastava, M., & Kumaraguru, P. (2021) · ACM Web Science Conference (WebSci) · doi:10.1145/3447535.3462502 · Cited by 73
“To combine the features obtained from the Text + OCR and the image encoder modules, T and I , we experiment with three different techniques of fusion – (1) Concatenation (2) Gated MCB [8] and (3) MFAS [22].”
MethodInfluentialIncorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory NetworksPreprint
Li, G., Su, H., & Zhu, W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.00733 · Cited by 48
“Most of the present methods solve VQA by jointly learning interactions and performing inference over the question and image contents based on the recent success of deep learning [19, 2, 22, 9, 8], which can be further improved by introducing the attention mechanisms [34, 32, 31, 17, 33, 1].”
MethodHadamard Product in Deep Learning: Introduction, Advances and Challenges
Chrysos, G. G., Wu, Y., Pascanu, R., Torr, P. H. S., & Cevher, V. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3560423 · Cited by 59
“For instance, a property of the Hadamard product connecting it with the Fast Fourier Transform (FFT) is exhibited in [12] for VQA.”
Language Person Search with Mutually Connected Classification Loss
Wang, Y., Bo, C., Wang, D., Wang, S., Qi, Y., & Lu, H. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682456 · Cited by 50
“For VQA, answering questions about given images [8, 9, 10, 11, 12, 13] learn a dynamic parameter layer with hashing techniques, which adaptively adjusts image features based on different questions for accurate answer classification.”
BackgroundCross-Attentional Audio-Visual Fusion for Weakly-Supervised Action Localization
Lee, J.-T., Jain, M., Park, H. W., & Yun, S. (2021) · International Conference on Learning Representations · Cited by 71
Sending or not? A multimodal framework for Danmaku comment prediction
Xi, D., Xu, W. X., Chen, R., Zhou, Y., & Yang, Z. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102687 · Cited by 48
Visual Entailment Task for Visually-Grounded Language LearningPreprint
Xie, N., Lai, F., Doran, D., & Kadav, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.10582 · Cited by 57
“…most relevant task to VE is VQA [Antol et al., 2015, Goyal et al., 2017, Zhu et al., 2016, Ren et al., 2015, Johnson et al., 2017b, Hudson and Manning, 2018, Anderson et al., 2018, Fukui et al., 2016], which is a representative multimodal task in machine learning that involves both images and text.”
BackgroundRTIC: Residual Learning for Text and Image Composition using Graph Convolutional NetworkPreprint
Shin, M., Cho, Y., Ko, B., & Gu, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.03015 · Cited by 45
“Image-text composition [2, 33, 29, 7, 6, 18] is a subset problem of feature fusion [4, 3, 21, 37, 38, 11], Figure 1.”
Efficient Counterfactual Debiasing for Visual Question Answering
Kolling, C., Móre, M. D., Gavenski, N. S., Pooch, E., Parraga, O., & Barros, R. C. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00263 · Cited by 58
“The performance of many state-of-the-art VQA models [32, 13] drops significantly on VQA-CP compared to other datasets, confirming the existing biases.”
BackgroundMethodTACFN: Transformer-Based Adaptive Cross-Modal Fusion Network for Multimodal Emotion Recognition
Liu, F., Fu, Z., Wang, Y., & Zheng, Q. (2023) · CAAI Artificial Intelligence Research · doi:10.26599/air.2023.9150019 · Cited by 51
BenchLMM: Benchmarking Cross-Style Visual Capability of Large Multimodal Models
Cai, R., Song, Z., Guan, D., Chen, Z., Li, Y., Luo, X., Yi, C., & Kot, A. C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72973-7_20 · Cited by 58
He, B., Wang, L., Zhou, W., Liu, H., Wang, Y., Lv, K., & He, K. (2025) · Scientific Reports · doi:10.1038/s41598-025-91420-2 · Cited by 46
“Then, fused features are obtained by Fast Fourier Transform (FFT) and Inverse Fast Fourier Transform (IFFT) 41 .”
Chain of Reasoning for Visual Question Answering
Wu, C., Liu, J., Wang, X., & Xuan, D. (2018) · neural information processing systems · Cited by 42
Neural Sequential Phrase Grounding (SeqGROUND)
Doğan, P., Sigal, L., & Groß, M. H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00430 · Cited by 48
iVQA: Inverse Visual Question Answering
Liu, F., Xiang, T., Hospedales, T. M., Yang, W., & Sun, C. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00898 · Cited by 57
Video Highlight Prediction Using Audience Chat Reactions
Fu, C.-Y., Lee, J., Bansal, M., & Berg, A. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1102 · Cited by 48
“, Bilinear Pooling (Fukui et al., 2016), Memory Networks (Xiong et al.”
BackgroundMethodMulti-Scale Thermal to Visible Face Verification via Attribute Guided Synthesis
Di, X., Riggan, B. S., Hu, S., Short, N. J., & Patel, V. M. (2021) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2021.3060641 · Cited by 46
“pooling module [13], [14] to generate high-quality visible images.”
BackgroundMethodInfluentialMulti-Modal Reasoning Graph for Scene-Text Based Fine-Grained Image Classification and Retrieval
Mafla, A., Dey, S., Biten, A. F., Gómez, L., & Karatzas, D. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00407 · Cited by 52
“Several fusion-based techniques such as Multimodal Compact Bilinear Pooling (MCB) [14, 12], Low-rank Bi-linear Attention Network (MLB) [27] and Block [5] have been explored to model relationships between language and vision.”
MethodMultitask Learning for Visual Question Answering
Ma, J., Liu, J., Lin, Q., Wu, B., Wang, Y., & You, Y. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3105284 · Cited by 43
“In the last few years, VQA has become one of the research hotspots–a number of VQA datasets have been built [1], [4]–[7] and a variety of deep learning models have been proposed [8]–[15].”
BackgroundMethodInfluentialMultimodal Compact Bilinear Pooling for Multimodal Neural Machine TranslationPreprint
Delbrouck, J.-B., & Dupont, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.08084 · Cited by 42
“We believe a step further could be to investigate different experimental settings or layer architectures as we felt MCB could perform much better as seen in similar previous work (Fukui et al., 2016).”
MethodResultBednarski, B. P., Singh, A. D., Zhang, W., Jones, W. M., Naeim, A., & Ramezani, R. (2022) · Scientific Reports · doi:10.1038/s41598-022-25472-z · Cited by 41
“However, Fukuia et al. 57 and Deng et al. 21 point out that these methods are likely suboptimal as they do not leverage the interaction between weights and features at each network layer.”
BackgroundCross-Dataset Adaptation for Visual Question Answering
Sha, F., Hu, H., & Chao, W.-L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00599 · Cited by 37
Visual Question Reasoning on General Dependency Tree
Cao, Q., Liang, X., Li, B., Li, G., & Lin, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00757 · Cited by 41
Revisiting Bilinear Pooling: A Coding Perspective
Gao, Z. F., Wu, Y., Zhang, X., Dai, J., Jia, Y., & Harandi, M. T. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5811 · Cited by 48
“Despite the success of BiP in many computer vision tasks including image classification (Lin, RoyChowdhury, and Maji 2015) and heterogeneous multi-modal tasks (Fukui et al. 2016), two shortcomings, i.e. , the redundancy and the burstiness, hinder the wide-application of BiP.”
BackgroundMethodInfluentialAdaptive Spatio-Temporal Graph Enhanced Vision-Language Representation for Video QA
Jin, W., Zhao, Z., Cao, X., Zhu, J., He, X., & Zhuang, Y. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3076556 · Cited by 46
“Fukui et al. [34] employed multimodal compact bilinear pooling (MCB) to efficiently and effectively combine visual and textual information.”
BackgroundSpatially Adaptive Computation Time for Residual NetworksPreprint
Figurnov, M., Collins, M. D., Zhu, Y., Zhang, L., Huang, J., Vetrov, D., & Salakhutdinov, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.02297 · Cited by 51
“In recent years deep convolutional networks have become an integral part of state-of-the-art systems for a diverse set of computer vision problems such as object detection [34], image segmentation [32], imageto-text [22, 42], visual question answering [11] and image generation [9].”
BackgroundCLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2016) · arXiv · doi:10.48550/arxiv.1612.06890 · Cited by 40
Same-different conceptualization: a machine vision perspective
Ricci, M. R., Cadène, R., & Serre, T. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.08.008 · Cited by 43
Singh, H. J., Bathla, G., Mehta, M., Chhabra, G., & Singh, P. (2023) · International Conference on Sustainable Computing and Data Communication Systems (ICSCDS) · doi:10.1109/icscds56580.2023.10104870 · Cited by 44
“[30] 53.74 Comp_QA (Andreas et al. 2015) [16] 54.8 MCB (Fukui et al. 2016) [34] 66.7 SAN (Yang al.”
InfluentialPooling acoustic and lexical features for the prediction of valence
Aldeneh, Z., Khorram, S., Dimitriadis, D., & Provost, E. M. (2017) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3136755.3136760 · Cited by 46
“Such methods showed success in computer vision applications [9, 15], but their use has not been investigated in linguistic/paralinguistic tasks.”
BackgroundMethodInfluentialLearning Answer Embeddings for Visual Question Answering
Sha, F., Chao, W., & Hu, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00569 · Cited by 45
Chao, W., Hu, H., & Sha, F. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1040 · Cited by 46
“accuracies before and after fine-tuning, together with the best results learned solely on those two datasets. As shown in Table6, fine-tuning largely improves the performance, justifying the finding by Fukui et al. (2016). 5.4 Qualitative Results In Fig.2, we present examples of image-questiontarget triplets from V7W, VQ…”
BackgroundInfluentialAnswer Questions with Right Image Regions: A Visual Attention Regularization Approach
Liu, Y., Guo, Y., Yin, J., Song, X., Liu, W., Nie, L., & Zhang, M. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3498340 · Cited by 50
“Canonical methods often cast VQA as a classification task [2, 3, 13, 16–18, 24, 29, 30, 46], where the image and question are processed via Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), respectively.”
MethodSpeaker-Guided Encoder-Decoder Framework for Emotion Recognition in Conversation
Bao, Y., Ma, Q., Wei, L., Zhou, W. G., & Hu, S. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/562 · Cited by 47
Beyond OCR + VQA: Involving OCR into the Flow for Robust and Accurate TextVQA
Zeng, G., Zhang, Y., Zhou, Y., & Yang, X. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475606 · Cited by 50
“For the visual entities, most related works rely on grid and region based features, which are usually used in VQA approaches [2, 14, 17, 26].”
Yi, Y., You, Y., Li, C., & Zhou, W.-L. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3265669 · Cited by 39
“Bilinear pooling is used in [41] to fuse the multimodal features of images and texts.”
MethodMultimodal Attention Network for Continuous-Time Emotion Recognition Using Video and EEG Signals
Choi, D. Y., Kim, D.-H., & Song, B. C. (2020) · IEEE Access · doi:10.1109/access.2020.3036877 · Cited by 39
“Fukui et al. achieved the compacter operation by applying fast Fourier transform (FFT) to the outer product process of the features of each modality [41].”
MethodA Review of Transformer-Based Approaches for Image Captioning
Ondeng, O., Absaloms, H. O., & Akuon, P. O. (2023) · Applied Sciences · doi:10.3390/app131911103 · Cited by 44
Li, Z., Pardos, Z. A., & Ren, C. (2024) · Computers & Education · doi:10.1016/j.compedu.2024.105027 · Cited by 46
Stock movement prediction with multimodal stable fusion via gated cross-attention mechanism
Zong, C. F., Wan, J., Cascone, L., & Zhou, H. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-02023-3 · Cited by 52
“However, existing research on stock movement prediction has been limited to unimodal [2] [3] [5] or bimodal [12] [16] [17] analysis, resulting in predictions that often diverge significantly from actual market trends.”
DeepCD: Learning Deep Complementary Descriptors for Patch Representations
Yang, T.-Y., Hsu, J.-H., Lin, Y., & Chuang, Y. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.359 · Cited by 43
“The results are displayed in the column named “2-stream PNNet+CBP.” It is obvious that the combination with CBP is not effective.”
MethodLearning visual and textual representations for multimodal matching and classification
Liu, Y., Liu, L., Guo, Y., & Lew, M. S. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.07.001 · Cited by 44
“Different from [35, 36] in which they simply perform the CBP module with the input visual or textual features, we build the CBP module based on the latent embeddings to generate a multimodal feature vector (Fig.”
MethodDifferential Networks for Visual Question Answering
Wu, C., Liu, J., Wang, X., & Li, R. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018997 · Cited by 40
“(Fukui et al. 2016) used the outer product to fuse image and question feature elements but resulting in the problem of dimension explosion.”
MethodConvolutional Reconstruction-to-Sequence for Video Captioning
Wu, A., Han, Y., Yang, Y., Hu, Q., & Wu, F. Q. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2956593 · Cited by 40
“To demonstrate the effectiveness of MCF, we respectively use MCB [45], MFB [46], MLB [47], Mutan [48], concatenation, element-wise addition, and element-wise product to replace MCF of the decoder and keep other components of our framework and their parameter settings invariant.”
MethodOn Modality Bias in the TVQA Dataset
Winterbottom, T., Xiao, S., McLean, A., & Moubayed, N. A. (2020) · British Machine Vision Conference (BMVC) · doi:10.5244/c.34.122 · Cited by 53
Su, H., Wang, P., Liu, L., Li, H., Li, Z., & Zhang, Y. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3034981 · Cited by 39
Entity Slot Filling for Visual Captioning
Bin, Y., Ding, Y., Peng, B., Peng, L., Yang, Y., & Chua, T. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3063297 · Cited by 41
Hou, J., Zhang, R., Xie, Y., Li, C., & Qin, W. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02257-y · Cited by 38
“J. Yang et al. employed ResNet50 to extract features from pathological images and fused selected clinical characteristics with image features using multimodal compact bilinear pooling (MCB) 42 , an outer product-based method that achieves improved prediction accuracy for breast cancer recurrence and metastasis with re…”
Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers
Geng, S., Gao, P., Chatterjee, M., Hori, C., Le Roux, J., Zhang, Y., Li, H., & Cherian, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16231 · Cited by 38
“Bilinear fusion methods (Ben-Younes et al. 2019; Fukui et al. 2016) have been explored towards inter-modality semantic alignment, however they often result in high-dimensional interaction tensors that are computationally expensive during inference.”
MethodEfficient Driver Anomaly Detection via Conditional Temporal Proposal and Classification Network
Su, L., Sun, C., Cao, D., & Khajepour, A. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3158480 · Cited by 38
Visual Question Answering: A Survey of Methods, Datasets, Evaluation, and Challenges
Kim, B. S., Kim, J., Lee, D., & Jang, B. (2025) · ACM Computing Surveys · doi:10.1145/3728635 · Cited by 79
“Experiments on the PASCAL VOC 2012 [90] and VQA v1/v2 datasets showed that WMA improved performance by 1.09% and 1.12% over previous methods like MCB [40] and MLB [78], achieving 67.79% and 67.89% on VQA v2.”
InfluentialAMC: Attention Guided Multi-modal Correlation Learning for Image Search
Chen, K., Bui, T., Chen, F., Wang, Z., & Nevatia, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.657 · Cited by 42
“The dictionary sizes for the query modality in Clickture and ASD are 85636 and Approach Img Key VAN LAN LF MTN MB [31] DSSM-Key [11] DSSM-Img [11] RCCA [39] Img ATT Key ATT Img ATT -Key ATT -LF AMC Full 17388.”
BackgroundMethodBeyond visual semantics: Exploring the role of scene text in image understanding
Dey, A. U., Ghosh, S. K., Valveny, E., & Harit, G. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.06.011 · Cited by 43
“While the community has witnessed lots of interplay between text and image in general [6, 11] they are mainly to model parallel text[34] present in the form of metadata, tags or annotation and not scene text contained in the image itself.”
BackgroundDeep ancient Roman Republican coin classification via feature fusion and attention
Anwar, H., Anwar, S., Zambanini, S., & Porikli, F. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107871 · Cited by 38
Region-attentive multimodal neural machine translation
Zhao, Y., Komachi, M., Kajiwara, T., & Chu, C. (2022) · Neurocomputing · doi:10.1016/j.neucom.2021.12.076 · Cited by 38
Generalized Bilinear Deep Convolutional Neural Networks for Multimodal Biometric Identification
Soleymani, S., Torfi, A., Dawson, J. M., & Nasrabadi, N. M. (2018) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2018.8451532 · Cited by 38
Multistep Question-Driven Visual Question Answering for Remote Sensing
Zhang, M., Chen, F., & Li, B. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3312479 · Cited by 52
Unsupervised Textual Grounding: Linking Words to Image Concepts
Yeh, R. A., Do, M. N., & Schwing, A. G. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00641 · Cited by 42
Revisiting EmbodiedQA: A Simple Baseline and Beyond
Wu, Y., Jiang, L., & Yang, Y. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2967584 · Cited by 39
“Recent studies show the attention-based model [4, 10, 26, 40] is a promising direction for this purpose.”
BackgroundRahnemoonfar, M., Dobbs, D., Yari, M., & Starek, M. J. (2019) · Remote Sensing · doi:10.3390/rs11091128 · Cited by 39
“Despite very promising results in “yes/no” and “what/where/who/when” questions, counting questions (how many) are the most difficult questions for the system, which have the lowest performance [10,11].”
BackgroundMoNet: Moments Embedding Network
Gou, M., Xiong, F., Camps, O., & Sznaier, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00335 · Cited by 38
Tensor Product Generation Networks for Deep NLP Modeling
Huang, Q., Smolensky, P., He, X., Deng, L., & Wu, D. O. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1114 · Cited by 41
“(The idea of using the outer product to construct internal representations was also explored in (Fukui et al., 2016).)”
BackgroundChallenges and Prospects in Vision and Language Research
Kafle, K., Shrestha, R., & Kanan, C. (2019) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2019.00028 · Cited by 52
“, bilinear fusion (Fukui et al., 2016), which are argued to better capture interactions between visual and linguistic representations.”
BackgroundKM 4 : Visual reasoning via Knowledge Embedding Memory Model with Mutual Modulation
Zheng, W., Yan, L., Gou, C., & Wang, F. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.10.007 · Cited by 41
Metadata
- Authors
- Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M.
- Affiliations
- Sony (Japan) (JP)
- University of California, Berkeley (US)
- Max Planck Institute for Informatics (DE)
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 457-468
- OpenAlex
- W2963383024
- Cited by
- 1,582
- References
- 60
- Citations by year
- 2026: 42 · 2025: 105 · 2024: 126 · 2023: 165 · 2022: 189 · 2021: 222 · 2020: 225 · 2019: 201 · 2018: 179 · 2017: 105 · 2016: 20
- FWCI
- 58.39
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Max Planck Digital Library (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1427425, 1212798, N000141110688)
- Defense Advanced Research Projects Agency
- Multidisciplinary University Research Initiative (N000141110688)
- Air Force Research Laboratory
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- visual question answering, visual grounding, multimodal fusion, outer product, attention mechanism, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.