Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years.However, tasks such as visual question answering require combining these vector representations with each other.Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations.We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors.As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features.We extensively evaluate MCB on the visual question answering and grounding tasks.We consistently show the benefit of MCB over ablations without MCB.For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation.This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, the PJ-X backbone MCB [16] (which is crucial for the answering performance) preceded the publication of the CLEVR dataset.”
From CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We replace the concatenation of the visual representation and the encoded phrase in Grounder with MCB to combine both modalities.”
MethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“, 2015) that is pretrained on ImageNet data (Deng et al., 2009).”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Also, for this model, we concatenated the learned word embedding with pretrained GloVe vectors (Pennington et al., 2014).”
MethodGirshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Research works printed but not linked: 3
- Ross Girshick. 2015. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
- Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014. Multimodal neural language mod- els. In Proceedings of the International Conference on Machine Learning (ICML), pages 595-603.
- Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems (NIPS).
Cited by
Items 801–900 of 1,002
Feature Enhancement in Attention for Visual Question Answering
Lin, Y., Pang, Z., Wang, D., & Zhuang, Y. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/586 · Cited by 15
Farazi, M. R., Khan, S. H., & Barnes, N. M. (2020) · Image and Vision Computing · doi:10.1016/j.imavis.2020.103985 · Cited by 15
Linguistically-aware attention for reducing the semantic gap in vision-language tasks
Gouthaman, K. V., Nambiar, A., Srinivas, K. S., & Mittal, A. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2020.107812 · Cited by 15
“decoder further fuses them using any standard fusion strategy, such as elementwise multiplication/sum, concatenation, or bi-linear operations [17, 18] and then performs a classification to one of the answers in a pre-defined set of answers.”
Multimodal sentiment analysis with asymmetric window multi-attentions
Lai, H., & Yan, X. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11234-y · Cited by 15
“Besides, Fukui et al. [8] employed Compact Bilinear Pooling (CBP) over the outer product of visual and linguistic representations to capture the interactions between vision and language for visual question answering.”
MethodPairwise VLAD Interaction Network for Video Question Answering
Wang, H., Guo, D., Hua, X., & Wang, M. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475620 · Cited by 15
“FrameQA Count RNN/Memory-based models VQA-MCB (R) [8] 58.”
Narayanan, A., Rao, A., Prasad, A., & Natarajan, S. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2021.104328 · Cited by 19
Visual context learning based on textual knowledge for image–text retrieval
Qin, Y., Gu, X., & Tan, Z. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.05.008 · Cited by 15
Semantic-Aware Modular Capsule Routing for Visual Question Answering
Han, Y., Yin, J., Wu, J., Wei, Y., & Nie, L. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3318949 · Cited by 21
“For conventional visual questions, the pioneer studies embed the visual and question features into a common space by various multimodal embedding techniques [8], [9].”
BackgroundFrom Vision to Content: Construction of Domain-Specific Multi-Modal Knowledge Graph
Zhang, X., Sun, X., Xie, C., & Lun, B. (2019) · IEEE Access · doi:10.1109/access.2019.2933370 · Cited by 18
Uncertainty Class Activation Map (U-CAM) Using Gradient Certainty Method
Patro, B. N., Lunayach, M., & Namboodiri, V. P. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3046916 · Cited by 16
“[6], [24], [35] advocate multimodal pooling and obtain close to state of the art results in VQA.”
BackgroundMethodInfluentialExplainable neural computation via stack neural module networks
Hu, R., Andreas, J., Darrell, T., & Saenko, K. (2021) · Applied AI Letters · doi:10.1002/ail2.39 · Cited by 18
Self-citationChartNet: Visual Reasoning over Statistical Charts using MAC-Networks
Sharma, M., Gupta, S., Chowdhury, A., & Vig, L. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852427 · Cited by 16
Robustness Analysis of Visual QA Models by Basic QuestionsPreprint
Huang, J.-H., Dao, C. D., Alfadly, M., Yang, C. H., & Ghanem, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.04625 · Cited by 15
“Currently, most of the state-of-the-art VQA models (Antol et al. 2015; Chen et al. 2016; Lu et al. 2016; Ben-younes et al. 2017; Fukui et al. 2016; Kim et al. 2017) only focus on how to improve accuracy.”
BackgroundMethodInfluentialJoint Image Captioning and Question AnsweringPreprint
Wu, J., Hu, Z., & Mooney, R. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.08389 · Cited by 15
Deep Bayesian Active Learning for Multiple Correct OutputsPreprint
Jedoui, K., Krishna, R., Bernstein, M. S., & Fei-Fei, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.01119 · Cited by 16
“VQA has received a considerable amount of attention in recent years with the development of several datasets, proposed as benchmarks [3, 50, 27, 20, 39, 57, 78], and of various models[3, 17, 45, 77, 75, 78, 26, 49, 71].”
BackgroundLarge-scale Bilingual Language-Image Contrastive LearningPreprint
Ko, B., & Gu, G. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.14463 · Cited by 24
“There have been demands of studies to combine more than two different modalities in feature analogy (Shin et al., 2021; Ben-Younes et al., 2017; Fukui et al., 2016; Yu et al., 2017; Ethayarajh et al., 2018).”
BackgroundA Comprehensive Survey on Visual Question Answering Datasets and AlgorithmsPreprint
Kabir, R., Haque, N., Islam, M. S., & Marium-E-Jannat (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.11150 · Cited by 24
“MCB [37] utilizes the above approach but replaces the convolution with a more efficient element-wise product in FFT(Fast Fourier Transform) space.”
InfluentialMultimodal Deception Detection: A Survey
Zhang, J., Lin, X., Huang, J., Ye, S., Guo, X., Zhu, D., Hu, R., Guo, D., Liang, Y., Yu, Z., et al. (2026) · Machine Intelligence Research · doi:10.1007/s11633-025-1625-x · Cited by 20
Overcoming language priors with self-contrastive learning for visual question answering
Yan, H., Liu, L., Feng, X., & Huang, Q. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-14167-2 · Cited by 21
“Existing methods [3, 11, 15, 26] design various fusion methods and attention mechanisms to understand the image and the question, but recent studies [1, 13, 18] have found that these works are heavily driven by the surface linguistic correlation in the training QA pairs (i.”
BackgroundMethodInfluentialMultimodal Graph Transformer for Multimodal Question Answering
He, X., & Wang, X. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.15 · Cited by 16
A Critical Analysis of Benchmarks, Techniques, and Models in Medical Visual Question Answering
Al-Hadhrami, S. A., Menaï, M. E. B., Al-Ahmadi, S. A., & Alnafessah, A. (2023) · IEEE Access · doi:10.1109/access.2023.3335216 · Cited by 15
Tong, S., Huang, K., Xing, W., Chu, Y., Nie, C., Ji, L., Wang, W., Tian, G., Wang, B., & Yang, J. (2024) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2024.108274 · Cited by 16
PTH-Net: Dynamic Facial Expression Recognition Without Face Detection and Alignment
Li, M., Zhang, X., Liao, T., Lin, S., & Xiao, G. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3504298 · Cited by 16
AerialVG: A Challenging Benchmark for Aerial Visual Grounding by Exploring Positional Relations
Liu, J., Chen, Q., Wang, Z., Tang, Y., Zhang, Y., Yan, C., Wang, D., Li, X., & Zhao, B. L. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00492 · Cited by 22
“Existing visual grounding methods can be broadly divided into two categories, i.e., (1) two-stage methods[13, 28, 44] and (2) one-stage methods[27, 37, 38].”
Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions
Wagner, M., Basevi, H. R. A., Shetty, R., Li, W., Malinowski, M., Fritz, M., & Leonardis, A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11009-3_32 · Cited by 14
Hierarchical Memory Decoder for Visual Narrating
Wu, A., Han, Y., Zhao, Z., & Yang, Y. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3020877 · Cited by 14
“To demonstrate the effectiveness of CCMF, we compare our method with two effective fusion method, i.e., bilinear pooling [41] and circulant fusion [60].”
MethodZhao, B., Tang, W., Pan, Y. R., Han, Y., & Wang, W. (2021) · Electronics · doi:10.3390/electronics10172046 · Cited by 15
“Therefore, in order to solve the problem of inefficient bilinear pooling calculation, we adopt the idea of Multimodal Compact Bilinear pooling [55], as shown in Figure 1, randomly project the features obtained by the MRELM-AE to a higher-dimensional space (using Count Sketch [77]), and then efficiently convolve the tw…”
MethodLowande, R. D. S., & Sevil, H. E. (2023) · Applied Sciences · doi:10.3390/app13085079 · Cited by 14
Liu, F., Peng, Y., & Rosen, M. P. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28577-7_20 · Cited by 14
Reciprocal Attention Fusion for Visual Question AnsweringPreprint
Farazi, M. R., & Khan, S. H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04247 · Cited by 14
“Language features: To represent the questions embedding in an end-to-end framework, GRUs [7] are used in a manner similar to [10, 5].”
BackgroundMethodInfluentialMuVAM: A Multi-View Attention-based Model for Medical Visual Question AnsweringPreprint
Pan, H., He, S., Zhang, K., Qu, B., Chen, C., & Shi, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03216 · Cited by 17
“Therefore, M ultimodal C ompact B ilinear pooling (MCB) [30] was proposed to perform the outer product calculation in a low-dimensional space to effectively combine multi-modal features.”
Karamcheti, S., Krishna, R., Fei-Fei, L., & Manning, C. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.02331 · Cited by 26
“…2015a; Johnson et al., 2017; Goyal et al., 2017; Krishna et al., 2017; Suhr et al., 2019; Hudson and Manning, 2019) and models (Zhou et al., 2015; Fukui et al., 2016; Lu et al., 2016; Yang et al., 2016; Zhu et al., 2016; Wu et al., 2016; Anderson et al., 2018a; Tan and Bansal, 2019; Chen et al.,…”
BackgroundXu, L. H., Liu, B., Khan, A. H., Fan, L., & Wu, X.-M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.06494 · Cited by 20
“We conduct experiments on VQA-RAD and SLAKE and compare VLT with general VQA models ( i.e ., SAN (Yang et al., 2016), MFH (Yu et al., 2018), MCB (Fukui et al., 2016), MUTAN (Ben-Younes et al., 2017), BAN (Kim et al., 2018)) as well as Med-VQA models ( i.e ., MEVF (Nguyen et al., 2019), CPRD (Liu et al., 2021a), CMSA-M…”
MethodMulti-modal multi-head self-attention for medical VQA
Joshi, V., Mitra, P., & Bose, S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17162-3 · Cited by 14
Guiding GPT models for specific one-for-all tasks in ground penetrating radar
Zheng, T., Zhang, Y., & Tao, M. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.105979 · Cited by 15
Combining Multiple Cues for Visual Madlibs Question Answering
Tommasi, T., Mallya, A., Plummer, B. A., Lazebnik, S., Berg, A. C., & Berg, T. L. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1096-0 · Cited by 13
“…answers is not straightforward and in most of the cases, it reduces to evaluating the predicted probability distribution on a fixed output space made by the 1000 most common answers of the used dataset (Fukui et al. 2016; Andreas et al. 2016b;Yang et al. 2016; Saito et al. 2017;Wang et al. 2017b).”
Image Captioning by Asking Questions
Yang, X., & Xu, C. (2019) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3313873 · Cited by 13
“Reference [13] proposes a multimodal compact bilinear pooling (MCB) method that combines the image and text features through Fourier space.”
MethodZheng, T., Ge, H., Li, J., & Wang, L. (2020) · Applied Intelligence · doi:10.1007/s10489-020-01821-1 · Cited by 14
“[54] proposed a multimodal compact bilinear pooling (MCBP) model that systematically investigated multiple feature fusion strategies.”
BackgroundMulti-modal Attentive Graph Pooling Model for Community Question Answer Matching
Hu, J., Fang, Q., Qian, S., & Xu, C. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413711 · Cited by 16
“TextGCN [37] trains a GCN model to convolve word and document embeddings in a large corpus graph for document classification, where the corpus graph consists of all the words and documents of a corpus as vertices and the relationships between them as edges.”
MethodDetecting adversarial examples by additional evidence from noise domain
Gao, S., Yu, S., Wu, L., Yao, S., & Zhou, X. (2021) · IET Image Processing · doi:10.1049/ipr2.12354 · Cited by 13
Attention in Reasoning: Dataset, Analysis, and Modeling
Chen, S., Jiang, M. X., Yang, J., & Zhao, Q. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3114582 · Cited by 13
“We compare the stacked attention from SAN [74], compositional attention from MAC [94] and the multi-head attention [69], [77], which all adopt object-based attention.”
BackgroundMethodCalibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images
Li, Z., Stengel-Eskin, E., Zhang, Y., Xie, C., Tran, Q., Durme, B. V., & Yuille, A. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01464 · Cited by 19
“[10] used multimodal compact bilinear pooling to combine visual and language features into a joint representation.”
MethodComprehensive-perception dynamic reasoning for visual question answering
Shuang, K., Guo, J., & Wang, Z. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108878 · Cited by 13
Yang, Q., Li, Y., Hu, B., Ma, L., Ding, Y., & Zhang, M. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548284 · Cited by 17
“Pointing and Justification (PJ-X) [23] uses a simplified MCB model [9] as the vision-language encoder and an LSTM-based language model as the decoder.”
BackgroundMethodANetQA: A Large-scale Benchmark for Fine-grained Compositional Reasoning over Untrimmed Videos
Zhou, Y., Zheng, L., Zhao, Z., Wu, F. Q., Jianping Fan, B., Ren, K., & Yu, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02221 · Cited by 13
“A number of image question answering (ImageQA) methods have been developed to push state-of-the-art performance on public benchmarks successively [6, 14, 46, 47].”
BackgroundLee, H., Lee, H., Park, M., & Kim, J. (2017) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2253978 · Cited by 13
What's in a Question: Using Visual Questions as a Form of Supervision
Ganju, S., Russakovsky, O., & Gupta, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.680 · Cited by 17
“Standard VQA systems [6, 47, 45, 22, 18, 24, 31, 2, 38, 16, 33, 20, 43, 44, 48, 37] take the image and its target question as input, with the expectation of producing an accurate answer for the question.”
BackgroundResultAn Attention-based Regression Model for Grounding Textual Phrases in Images
Ko, E., Aono, M., Nichols, E., & Funakoshi, K. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/558 · Cited by 16
“Our approach differs from [Hu et al., 2016b; Rohrbach et al., 2016; Fukui et al., 2016; Luo and Shakhnarovich, 2017] by using regression to ground textual phrases, eliminating the need for external candidate regions and avoiding adverse influence from candidate region prediction quality.”
MethodEfficient Fine-Grained Classification and Part Localization Using One Compact Network
Dai, X., Southall, B., Trinh, N. H., & Matei, B. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.122 · Cited by 13
“We use compact bilinear pooling [6, 5], a projection method that further reduces the number of dimensions in our feature vector while preserving desirable properties of the feature.”
MethodTransfer Learning via Unsupervised Task Discovery for Visual Question Answering
Noh, H., Kim, T., Mun, J., & Han, B. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00858 · Cited by 13
“Pretrained language models such as word embeddings [33] or sentence embeddings [21] are frequently used to initialize parameters of question encoders [12, 30, 35].”
BackgroundMethodWhat is Learned in Visually Grounded Neural Syntax Acquisition
Kojima, N., Averbuch‐Elor, H., Rush, A. M., & Artzi, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.234 · Cited by 20
Video and Text Matching with Conditioned Embeddings
Ali, A., Schwartz, I., Hazan, T., & Wolf, L. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00055 · Cited by 18
Lu, J., Ye, X., Ren, Y. C., & Yang, Y. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00539 · Cited by 23
“Traditionally, convolutional neural networks [14] and recurrent neural networks [17] are popular choices for image encoders and textual encoders, while the answer decoder ranges from a softmax classifier [7], an RNN decoder [29] to a dot product layer [19].”
BackgroundMethodInfluentialLocal relation network with multilevel attention for visual question answering
Sun, B., Zeng, Y., Zhang, Y., & Yu, L. (2020) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2020.102762 · Cited by 13
Multi-modal Learning with Prior Visual Relation Reasoning.Preprint
Yang, Z., Yu, J., Yang, C., Qin, Z., & Hu, Y. (2018) · arXiv (Cornell University) · Cited by 13
Scientific document processing: challenges for modern learning methods
Kashyap, A. R., Yang, Y., & Kan, M. (2023) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00352-7 · Cited by 15
“To address this, multimodal low-rank bilinear pooling [57, 96] targets to solve this dimensionality problem by means of a low-dimensional approximation.”
MethodEmoffMeme: identifying offensive memes by leveraging underlying emotions
Kumari, G., Bandyopadhyay, D., & Ekbal, A. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14807-1 · Cited by 23
“MSCLIP with [50]: Authors in this paper used multimodal compact bilinear pooling(MCB) [24] for the fusion of textual and visual models.”
MethodKwak, C., Jung, P., & Lee, S. (2023) · Applied Sciences · doi:10.3390/app13169456 · Cited by 16
To Boost Zero-Shot Generalization for Embodied Reasoning With Vision-Language Pre-Training
Su, K., Zhang, X., Zhang, S., Zhu, J., & Zhang, B. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3459800 · Cited by 16
“Another important one is bilinear pooling which was first used for visual reasoning tasks by MCB [47].”
MethodHanan, A., Khan, M. N., Fernandez-Añez, N., & Arghandeh, R. (2025) · Ecological Informatics · doi:10.1016/j.ecoinf.2025.103277 · Cited by 12
MSRC: multimodal spatial regression with semantic context for phrase grounding
Chen, K., Kovvuri, R., Gao, J., & Nevatia, R. (2017) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-017-0139-6 · Cited by 13
“Experiment shows MCB+Reg has 2.32% increase compared to MCB [7] model.”
BackgroundMethodInfluentialMulti-View Visual Question Answering with Active Viewpoint Selection
Qiu, Y., Satoh, Y., Suzuki, R., Iwata, K., & Kataoka, H. (2020) · Sensors · doi:10.3390/s20082281 · Cited by 12
Cross-modal learning with prior visual relation knowledge
Yu, J., Zhang, W., Yang, Z., Qin, Z., & Hu, Y. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106150 · Cited by 12
“MCB [22] is the winning entry of VQA Challenge 2016 which uses a multimodal compact bilinear pooling mechanism, while Adelaide is the champion of VQA Challenge 2017 [40], which is the first approach using object-level features as image representation.”
Cross-Modal Omni Interaction Modeling for Phrase Grounding
Yu, T., Hui, T., Yu, Z., Liao, Y., Yu, S., Zhang, F., & Liu, S. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413846 · Cited by 14
“As a relatively new topic, phrase grounding is far from being solved and yet has a breadth of applications that span image captioning [23, 24, 50], image retrieval[15], visual question answering [1, 12, 46] and referential expression generation [19].”
BackgroundFair-VQA: Fairness-Aware Visual Question Answering Through Sensitive Attribute Prediction
Park, S., Hwang, S., Hong, J. K., & Byun, H. (2020) · IEEE Access · doi:10.1109/access.2020.3041503 · Cited by 12
“Therefore, [39]–[41] proposemultimodal fusionmethods that sufficiently reflect relation between the two features with an efficient computational cost.”
BackgroundInfluentialKhan, H. R., Gupta, D., & Ekbal, A. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.151 · Cited by 23
Comprehensive Framework of Early and Late Fusion for Image–Sentence Retrieval
Wang, Y., Xu, X., Yu, W., Xu, R., Cao, Z., & Shen, H. T. (2022) · IEEE Multimedia · doi:10.1109/mmul.2022.3144972 · Cited by 13
Multi-modal co-attention relation networks for visual question answering
Guo, Z., & Han, D. (2022) · The Visual Computer · doi:10.1007/s00371-022-02695-9 · Cited by 14
“Some recent studies have proposed more complex multi-modal feature fusion meth-ods, such as bilinear-pooling-based methods [50,51], which can reflect the relationship between features of two different modalities at an effective computational cost.”
BackgroundMedical visual question answering with symmetric interaction attention and cross-modal gating
Chen, Z., Zou, B., Dai, Y., Zhu, C., Kong, G., & Zhang, W. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105049 · Cited by 12
Ma, Z., Yu, Z., Li, J., & Li, G. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26569 · Cited by 18
“2017), MCB (Fukui et al. 2016), MUTAN (Ben-Younes et al.”
MethodRethinking Multi-Modal Alignment in Multi-Choice VideoQA from Feature and Sample Perspectives
Xiao, S., Chen, L., Gao, K., Wang, Z., Yang, Y., Zhang, Z., & Xiao, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.561 · Cited by 12
“Then, we calculate a score for each candidate question-answer pair using multi-modal compact bilinear fusion (MCB) (Fukui et al., 2016), where ∗ denotes tq , vq and vt .”
MethodSimple Baselines for Interactive Video Retrieval with Questions and Answers
Liang, K., & Albanie, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01018 · Cited by 21
“We observe a similar trend to the recall@1 results reported in the main paper.”
ResultIncorporating texture and silhouette for video-based person re-identification
Bai, S., Chang, H., & Ma, B. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110759 · Cited by 16
More Than An Answer: Neural Pivot Network for Visual Qestion Answering
Zhou, Y., Ji, R., Su, J., Wu, Y., & Wu, Y. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123335 · Cited by 13
“Meanwhile, the usage of bilinear pooling methods also brings a significant improvement to VQA [7][12].”
BackgroundMethodInfluentialUMass at ImageCLEF Medical Visual Question Answering(Med-VQA) 2018 Task.
Peng, Y., & Liu, F. (2018) · CLEF (Working Notes) · Cited by 12
Question-Agnostic Attention for Visual Question Answering
Farazi, M., Khan, S., & Barnes, N. M. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9413330 · Cited by 12
“In order to further validate the complementary nature of our proposed QAA model, we include a simple spatial attention mechanism, commonly used in most VQA models [9, 4, 5] to refine the visual features according to the question.”
BackgroundMethodInfluentialVu, M. H., Sznitman, R., Nyholm, T., & Löfstedt, T. (2019) · Bern Open Repository and Information System (University of Bern) · doi:10.7892/boris.135251 · Cited by 12
Rankvqa: Answer Re-Ranking For Visual Question Answering
Qiao, Y., Yu, Z., & Liu, J. (2020) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme46284.2020.9102814 · Cited by 12
Multi-modal Contextual Graph Neural Network for Text Visual Question Answering
Liang, Y., Wang, X. E., Duan, X., & Zhu, W. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412891 · Cited by 13
“The communities’ interests include the image and question feature processing [14], [18], answer encoding [7], [21].”
BackgroundGeneralized pyramid co-attention with learnable aggregation net for video question answering
Gao, L., Chen, T., Li, X., Zeng, P., Zhao, L., & Li, Y.-F. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108145 · Cited by 12
Are VQA Systems RAD? Measuring Robustness to Augmented Data with Focused Interventions
Rosenberg, D., Gat, I., Feder, A., & Reichart, R. (2021) · doi:10.18653/v1/2021.acl-short.10 · Cited by 16
“While the accuracy of these systems appears to be constantly improving (Fukui et al., 2016; Yang et al., 2016; Lu et al., 2016), they are sensitive to small perturbations in their input and seem overfitted to their training data (Kafle et al., 2019).”
BackgroundRanek, J. S., Stanley, N., & Purvis, J. E. (2022) · Genome biology · doi:10.1186/s13059-022-02749-0 · Cited by 14
CS-VQA: Visual Question Answering with Compressively Sensed Images
Huang, L.-C., Kulkarni, K., Jha, A., Lohit, S., Jayasuriya, S., & Turaga, P. K. (2018) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2018.8451445 · Cited by 12
Deep-Learning-Based Financial Message Sentiment Classification in Business Management
Shao, C., & Chen, X. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/3888675 · Cited by 13
“In [24], it is proposed to further apply multimodal classification to the field of fake information detection. two different feature vectors are merged through the multimodal compact bilinear pooling (MCBP) method, which can effectively fuse the two kinds of information for classification.”
MethodAnswering Visual What-If Questions: From Actions to Predicted Scene DescriptionsPreprint
Wagner, M., Basevi, H. R. A., Shetty, R., Li, W., Malinowski, M., Fritz, M., & Leonardis, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03707 · Cited by 12
“Since then, the community has seen a proliferation of various datasets [17,18,19,20,21,22,23], including the most popular VQA [24], as well as numerous methods [15,18,25,26,27,28,29].”
BackgroundLearning Visual Question Answering by Bootstrapping Hard AttentionPreprint
Malinowski, M., Doersch, C., Santoro, A., & Battaglia, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.00300 · Cited by 29
“he answering module. work [7,9,22,23,24,25]. After a few layers of combined processing, we apply attention over spatial locations, following previous works which often apply soft attention mechanisms [7,8,9,10] at this point in the architecture. Finally, we aggregate features, using either sum-pooling, or relational […”
MethodAn Empirical Evaluation of Visual Question Answering for Novel ObjectsPreprint
Ramakrishnan, S. K., Pal, A., Sharma, G., & Mittal, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.02516 · Cited by 12
“[8] propose a multimodal bilinear pooling, using Fourier space computations for efficiency.”
MethodCompact bilinear pooling and multi-loss network for social media multimodal classification
Li, Y., Zheng, X., Zhu, M. L., Mei, J., Chen, Z., & Tao, Y. (2024) · Signal Image and Video Processing · doi:10.1007/s11760-024-03482-w · Cited by 15
“FuKui et al. [23] utilized multimodal compact bilinear pooling to efficiently combine expressive multimodal features.”
BVQA: Connecting Language and Vision Through Multimodal Attention for Open-Ended Question Answering
Bhuyan, M. S. M., Hossain, E., Sathi, K. A., Hossain, M. A., & Dewan, M. A. A. (2025) · IEEE Access · doi:10.1109/access.2025.3540388 · Cited by 13
A comprehensive survey on deep learning approaches for image captioning: a systematic review
Al-Malla, M. A., Hamdoun, O., & Ghneim, N. (2026) · Journal Of Big Data · doi:10.1186/s40537-026-01377-w · Cited by 12
Commonsense Justification for Action Explanation
Yang, S., Gao, Q., Saba-Sadiya, S., & Chai, J. Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1283 · Cited by 16
“An increasing amount of work on the Visual Question Answering (VQA) task (An-tol et al., 2015; Lu et al., 2016) has also looked into more interpretable approaches, for example, by utilizing attention-based models (Fukui et al., 2016) or reasoning based on explicit evidence (Wang et al., 2017).”
BackgroundThe Impact of Explanations on AI Competency Prediction in VQA
Alipour, K., Ray, A., Lin, X., Schulze, J. P., Yao, Y., & Burachas, G. (2020) · doi:10.1109/hccai49649.2020.00010 · Cited by 14
An Overview of Utilizing Knowledge Bases in Neural Networks for Question Answering
Kafle, S., de Silva, N., & Dou, D. (2020) · Information Systems Frontiers · doi:10.1007/s10796-020-10035-2 · Cited by 13
“Multimodal bilinear compact pooling (Fukui et al. 2016) proposes an efficient but highly optimized bilinear pooling over two data sources, enabling a robust embedding for visual question answering.”
BackgroundInterpretable Visual Reasoning via Probabilistic Formulation Under Natural Supervision
Han, X., Wang, S., Su, C., Zhang, W., Huang, Q., & Tian, Q. C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58545-7_32 · Cited by 12
“…developments in implicit reasoning can be steadily explained by our probabilistic framework, from one-step State Inference (one-stage fusion [1,16,13,32]) to multi-step State Inference (stacked attention [39,31]) and then to multi-step State Inference with dependent transitions (relation-based…”
BackgroundMethodDiverse Features Fusion Network for video-based action recognition
Deng, H., Kong, J., Jiang, M., & Liu, T. (2021) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2021.103121 · Cited by 11
Who’s Waldo? Linking People Across Text and Images
Cui, C. Y., Khandelwal, A., Artzi, Y., Snavely, N., & Averbuch‐Elor, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00141 · Cited by 16
“capability in visual semantic tasks with applications including image captioning [66, 41, 3], visual question answering [19, 20, 26] and instruction following [4, 43, 7].”
BackgroundLLQA - Lifelog Question Answering Dataset
Tran, L.-D., Ho, T. C., Pham, L. A., Nguyen, B. T., Gurrin, C. G., & Zhou, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-98358-1_18 · Cited by 22
Neural module networks: A review
Fashandi, H. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126518 · Cited by 13
Fine-grained bird image classification based on counterfactual method of vision transformer model
Chen, T., Li, Y., & Qiao, Q. (2023) · The Journal of Supercomputing · doi:10.1007/s11227-023-05701-6 · Cited by 11
Hierarchical reasoning based on perception action cycle for visual question answering
Mohamud, S. A. M., Jalali, A., & Lee, M. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122698 · Cited by 11
Metadata
- Authors
- Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M.
- Affiliations
- Sony (Japan) (JP)
- University of California, Berkeley (US)
- Max Planck Institute for Informatics (DE)
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 457-468
- OpenAlex
- W2963383024
- Cited by
- 1,582
- References
- 60
- Citations by year
- 2026: 42 · 2025: 105 · 2024: 126 · 2023: 165 · 2022: 189 · 2021: 222 · 2020: 225 · 2019: 201 · 2018: 179 · 2017: 105 · 2016: 20
- FWCI
- 58.39
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Max Planck Digital Library (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1427425, 1212798, N000141110688)
- Defense Advanced Research Projects Agency
- Multidisciplinary University Research Initiative (N000141110688)
- Air Force Research Laboratory
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- visual question answering, visual grounding, multimodal fusion, outer product, attention mechanism, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.