Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years.However, tasks such as visual question answering require combining these vector representations with each other.Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations.We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors.As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features.We extensively evaluate MCB on the visual question answering and grounding tasks.We consistently show the benefit of MCB over ablations without MCB.For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation.This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, the PJ-X backbone MCB [16] (which is crucial for the answering performance) preceded the publication of the CLEVR dataset.”
From CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We replace the concatenation of the visual representation and the encoded phrase in Grounder with MCB to combine both modalities.”
MethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“, 2015) that is pretrained on ImageNet data (Deng et al., 2009).”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Also, for this model, we concatenated the learned word embedding with pretrained GloVe vectors (Pennington et al., 2014).”
MethodGirshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Research works printed but not linked: 3
- Ross Girshick. 2015. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
- Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014. Multimodal neural language mod- els. In Proceedings of the International Conference on Machine Learning (ICML), pages 595-603.
- Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems (NIPS).
Cited by
Items 601–700 of 1,002
Multimodal integration strategies for clinical application in oncology
Zhang, B., Wan, Z., Luo, Y., Zhao, X., Samayoa, J. A., Zhao, W., & Wu, S. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1609079 · Cited by 36
Effective 3-D Shape Retrieval by Integrating Traditional Descriptors and Pointwise Convolution
Kuang, Z., Yu, J., Zhu, S., Li, Z., & Jianping Fan, B. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2918729 · Cited by 26
“ysis [51], [52], we generalize it as an amplifier to enhance the fitting power of the FC layer for more discriminative feature learning by using limited number of layers and parameters.”
BackgroundIQMA Network: Image Quality Multi-scale Assessment Network
Guo, H., Bin, Y., Hou, Y., Zhang, Q., & Luo, H. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00055 · Cited by 37
“For the pooling of the fused feature, bilinear pooling is widely used when features coming from different modalities [9].”
BackgroundMethodDual Attention and Question Categorization-Based Visual Question Answering
Mishra, A., Anand, A., & Guha, P. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3160418 · Cited by 31
“The state-of-the-art VQA models like MCB [19], MLB [21], MFH [36], MUTAN [6], and BLOCK [23] are chosen as baseline as they primarily contribute toward the fusion of two modalities.”
BackgroundMethodInfluentialLan, Y., Guo, Y., Chen, Q., Lin, S., Chen, Y., & Deng, X. (2023) · Frontiers in Plant Science · doi:10.3389/fpls.2022.1064399 · Cited by 30
“The image feature extraction models mainly include Convolutional Neural Network (CNN), VGGNet (Simonyan and Zisserman, 2014), GoogLeNet (Szegedy et al., 2015), and ResNet (Fukui et al., 2016; Kim et al., 2016; Ben-Younes et al., 2017) on the ImageNet dataset (Krizhevsky et al., 2017).”
BackgroundMethodInfluentialFC-KAN: Function combinations in Kolmogorov-Arnold networks
Ta, H. T., Quy, T. D., Rahman, A. B. S., Sidorov, G., & Gelbukh, A. F. (2026) · Information Sciences · doi:10.1016/j.ins.2026.123103 · Cited by 34
Local Temporal Bilinear Pooling for Fine-Grained Action Parsing
Zhang, Y., Tang, S., Muandet, K., Jarvers, C., & Neumann, H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01228 · Cited by 29
“Bilinear pooling (or second-order pooling) is widely used in fine-grained visual classification [2, 30, 22, 26, 27, 24, 25, 9, 42, 16, 21, 47, 48, 49, 45, 13, 38], visual questioning answering [8, 15, 50], feature fusion and disentangling [26, 27, 43, 4, 7, 14], action recognition [51, 46, 11, 14, 7] and other tasks.”
BackgroundModularized Textual Grounding for Counterfactual Resilience
Fang, Z., Kong, S., Fowlkes, C. C., & Yang, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00654 · Cited by 36
“For bilinear pooling, we adopt the Multimodal Compact Bilinear (MCB) pooling introduced in [16] that effectively pools over visual and textual features.”
MethodWhat Makes Training Multi-Modal Networks Hard?Preprint
Wang, W., Tran, D., & Feiszli, M. (2019) · arXiv (Cornell University) · Cited by 28
Learning visual language models for video understanding
Yang, A. (2023) · theses.fr (ABES) · doi:10.70675/802872a7z8092z4556za1fdz2c7af45f0e72 · Cited by 26
Visual Question Generation: The State of the Art
Patil, C., & Patwardhan, M. (2021) · ACM Computing Surveys · doi:10.1145/3383465 · Cited by 25
“Multi-modal compact bilinear (MCB) pooling for deeper interaction between image and question features has been proposed [8].”
BackgroundVisual Reasoning with Multi-hop Feature Modulation
Strub, F., Seurin, M., Perez, E., de Vries, H., Mary, J., Preux, P., Courville, A. C., & Pietquin, O. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01228-1_48 · Cited by 36
Modality-Balanced Models for Visual Dialogue
Kim, H., Tan, H. X., & Bansal, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6320 · Cited by 30
SCNET: A Novel UGI Cancer Screening Framework Based on Semantic-Level Multimodal Data Fusion
Ding, S., Huang, H. M., Li, Z., Liu, X., & Yang, S. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.2983126 · Cited by 30
“In the previous works on solving classification problems involving dual modalities such as text and images, a typical approach is to use the fused information of text and image features to weight the feature maps in CNNs using their spatial information [44].”
MethodDual Low-Rank Multimodal Fusion
Jin, T., Huang, S., Li, Y., & Zhang, Z. M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.35 · Cited by 32
“Recently, several methods have been proposed to learn joint embed-dings of multiple modalities (Fukui et al., 2016; Nojavanasghari et al., 2016; Zadeh et al., 2017).”
BackgroundLiu, D., Qu, X., Dong, J., & Zhou, P. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.167 · Cited by 25
“We use the same split as (Gao et al., 2017), which includes 10146, 4589, 4083 query-segment pairs for training, validation and testing.”
BackgroundMethodInfluentialA Bi-level representation learning model for medical visual question answering
Li, Y., Long, S., Yang, Z., Weng, H., Zeng, K., Huang, Z., Wang, F. L., & Hao, T. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104183 · Cited by 27
Egocentric video description based on temporally-linked sequences
Bolaños, M., Peris, Á., Casacuberta, F., Soler, S., & Radeva, P. I. (2017) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2017.11.022 · Cited by 31
“More precisely, in order to learn a representation of the input sequences and its relationships, we use a LSTM network (Hochreiter and Schmidhuber, 1997; Gers et al., 2000).”
MethodTextbook Question Answering Under Instructor Guidance with Memory Networks
Li, J., Su, H., Zhu, J., Wang, S., & Zhang, B. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00385 · Cited by 31
“Thanks to the availability of large-scale QA datasets with additional visual supporting information [23, 34, 2], Visual Question Answering (VQA) has attracted a substantial interest from the community of computer vision [10, 30, 8].”
BackgroundMulti-grained Attention with Object-level Grounding for Visual Question Answering
Huang, P., Huang, J., Guo, Y., Qiao, M., & Zhu, Y. T. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1349 · Cited by 25
“Then the multi-modal features are fused into one unified representation for which the answer is predicted (Malinowski et al., 2015; Fukui et al., 2016; Anderson et al., 2018).”
BackgroundImproving Visual Question Answering by Referring to Generated Paragraph Captions
Kim, H., & Bansal, M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1351 · Cited by 27
“In visual question answering (VQA) (Antol et al., 2015; Goyal et al., 2017; Lu et al., 2016; Fukui et al., 2016; Xu and Saenko, 2016; Yang et al., 2016; Zhu et al., 2016; Anderson et al., 2018), models are trained to choose the correct answer given a question about an image.”
BackgroundInfluentialVideo Question Answering: a Survey of Models and Datasets
Sun, G., Liang, L., Li, T., Yu, B., Wu, M., & Zhang, B. (2021) · Mobile Networks and Applications · doi:10.1007/s11036-020-01730-0 · Cited by 35
“The performance of the multi-modal compacts bilinear pooling (MCB) [70] is better than the linear pooling methods.”
BackgroundMethodInfluentialVQA-Aid: Visual Question Answering for Post-Disaster Damage Assessment and Analysis
Sarkar, A., & Rahnemoonfar, M. (2021) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss47720.2021.9553578 · Cited by 25
Zhou, S., Jia, J., Wu, Z., Yang, Z., Wang, Y., Chen, W., Meng, F., Huang, S., Shen, J., & Wang, X. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i7.16753 · Cited by 25
“The MCB method not only inherits the advantage of bi-linear models, allowing all elements of both acoustic, textual and social features to interact with each other by matrix mul-6041 tiplying, but also overcomes its disadvantage of high memory consumption and an excessive amount of parameters.”
MethodInfluentialAssessing the Robustness of Visual Question Answering ModelsPreprint
Huang, J.-H., Alfadly, M., Ghanem, B., & Worring, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.01452 · Cited by 29
“In Fukui et al (2016), the authors propose a method, Multimodal Compact Bilinear (MCB) pooling, to compute an outer product between visual and textual features.”
BackgroundMethodContrastive Visual-Linguistic PretrainingPreprint
Shi, L., Shuang, K., Geng, S., Su, P., Jiang, Z., Gao, P., Fu, Z., de Melo, G., & Su, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.13135 · Cited by 29
“Besides attention, bilinear models such as MCB [32], MLB [33] and MUTAN [34] have explored the benefit of channel interactions for multimodal understanding.”
Gai, X., Liu, J., Li, Y., Meng, Z., Wu, J., & Liu, Z. (2025) · neural information processing systems · doi:10.52202/085713-5656 · Cited by 38
Bridging Languages through Images with Deep Partial Canonical Correlation Analysis
Rotman, G., Vulić, I., & Reichart, R. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1084 · Cited by 25
“…and language has also enabled new tasks which require both visual and language understanding, such as visual question answering (Antol et al., 2015; Fukui et al., 2016; Xu and Saenko, 2016), image-to-text retrieval and text-to-image retrieval (Kiros et al., 2014; Mao et al., 2014), image caption…”
BackgroundRe-Attention for Visual Question Answering
Guo, W., Zhang, Y., Wu, X., Yang, J., Cai, X., & Yuan, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5338 · Cited by 24
“We compare our method with fusion-based methods, such as, MCB (Fukui et al. 2016) and ResNet-3000 (Ma et al. 2018).”
MethodA Unified Framework for Multilingual and Code-Mixed Visual Question Answering
Gupta, D. K., Lenka, P., Ekbal, A., & Bhattacharyya, P. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.90 · Cited by 43
“As the existing research on VQA are mainly focused on natural language questions written in English (Antol et al., 2015; Hu et al., 2017; Fukui et al., 2016; Anderson et al., 2018; Li et al., 2018; Xu and Saenko, 2016; Shih et al., 2016), their applications are often limited.”
BackgroundDynamic Convolution-based Encoder-Decoder Framework for Image Captioning in Hindi
Mishra, S. K., Sinha, S., Saha, S., & Bhattacharyya, P. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3573891 · Cited by 25
“There are just a few publications in the literature for Hindi in References [7, 14, 33, 35–39].”
MethodInfluentialA Regionally Indicated Visual Grounding Network for Remote Sensing Images
Hang, R., Xu, S., & Liu, Q. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3490847 · Cited by 28
“For example, Fukui et al. [51] proposed a multimodal compact bilinear pooling module to denote and learn the correlationships between visual and language features.”
Semantic Parsing to Probabilistic Programs for Situated Question Answering
Krishnamurthy, J., Tafjord, O., & Kembhavi, A. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1016 · Cited by 31
“, 2015), where many architectures have been proposed (Malinowski et al., 2015; Yang et al., 2015; Fukui et al., 2016).”
BackgroundU-CAM: Visual Explanation Using Uncertainty Based Class Activation Maps
Patro, B. N., Lunayach, M., Patel, S., & Namboodiri, V. P. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00754 · Cited by 106
“The figure shows the activation maps for baseline (MCB [13]) and our models (A-GCA and P-GCA).”
BackgroundMethodInfluentialBastida, J. O., Gallego, A., Rico-Juan, J. R., & Albarrán, P. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107693 · Cited by 27
TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning BaselinesPreprint
Lin, J., Jain, U., & Schwing, A. G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.14671 · Cited by 26
“…for VQA [9, 32], selecting sub-regions of an image [87], single attention [13, 98, 6, 19, 29, 82, 97, 39, 102], multimodal attention [59, 79, 70], memory nets and knowledge bases [96, 94, 91, 62], improvements in neural architecture [66, 63, 7, 8] and bilinear pooling representations [29, 46, 12].”
BackgroundMethodDomain Generalization for Face Anti-Spoofing via Content-Aware Composite Prompt Engineering
Guo, J., Liu, A., Diao, Y., Zhang, J., Ma, H., Zhao, B., Hong, R., & Wang, M. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3618575 · Cited by 23
“Study [62], [63] has emphasized that comprehensively harnessing the interactions among elements of multi-modal features can significantly bolster performance outcomes.”
Multi-Modal Explicit Sparse Attention Networks for Visual Question Answering
Guo, Z., & Han, D. (2020) · Sensors · doi:10.3390/s20236758 · Cited by 24
Multi-task learning for simultaneous script identification and keyword spotting in document images
Cheikhrouhou, A., Kessentini, Y., & Kanoun, S. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107832 · Cited by 23
Zhai, B., Guan, Y., Catt, M., & Plötz, T. (2021) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3494961 · Cited by 25
“It’s a method often used to fuse visual feature vectors with textual feature vectors to create a joint representation space, even though their distribution may vary dramatically[22, 45].”
MethodA Federated Attention-Based Multimodal Biometric Recognition Approach in IoT
Lin, L., Zhao, Y., Meng, J., & Zhao, Q. (2023) · Sensors · doi:10.3390/s23136006 · Cited by 24
Knowledge-Augmented Visual Question Answering With Natural Language Explanation
Xie, J., Cai, Y., Chen, J., Xu, R., Wang, J., & Li, Q. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3379900 · Cited by 32
“Park et al. [13] utilize the MCB [31] model for answer prediction and then use an LSTM-based [32] model for explanation generation.”
MethodAn Empirical Evaluation of Visual Question Answering for Novel Objects
Ramakrishnan, S. K., Pal, A., Sharma, G., & Mittal, A. (2017) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2017.773 · Cited by 33
“[8] propose a multimodal bilinear pooling, using Fourier space computations for efficiency.”
MethodLearning Unsupervised Visual Grounding Through Semantic Self-Supervision
Javed, S. A., Saxena, S., & Gandhi, V. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/112 · Cited by 26
A Genre-Aware Attention Model to Improve the Likability Prediction of Books
Maharjan, S., Montes, M., Gonzalez, F. A., & Solorio, T. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1375 · Cited by 27
“Bilinear models are used in the visual question answering community to fuse visual and textual information (Fukui et al., 2016).”
MethodRich Features Embedding for Cross-Modal Retrieval: A Simple Baseline
Fu, X., Zhao, Y., Wei, Y., Zhao, Y., & Wei, S. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2957948 · Cited by 24
“[37] systematically investigate multiple feature fusion strategies and find element-wise product to be among the most effective.”
BackgroundAre These Birds Similar: Learning Branched Networks for Fine-grained Representations
Nawaz, S., Calefati, A., Caraffini, M., Landro, N., & Gallo, I. (2019) · International Conference on Image and Vision Computing New Zealand (IVCNZ) · doi:10.1109/ivcnz48456.2019.8960960 · Cited by 28
“Recent years have seen a surge in tasks based on multimodal data including classification [18]–[20], Visual Question Answering [21], [22], multimodal named entity recognition [23], [24], cross-modal retrieval [25], [26] and verification [27], [28].”
BackgroundVisual question answering with attention transfer and a cross-modal gating mechanism
Li, W., Sun, J., Liu, G., Zhao, L., & Fang, X. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.02.031 · Cited by 23
Explaining VQA predictions using visual grounding and a knowledge base
Riquelme, F., Goyeneche, A. D., Zhang, Y., Niebles, J. C., & Soto, Á. (2020) · Image and Vision Computing · doi:10.1016/j.imavis.2020.103968 · Cited by 27
Vision Skills Needed to Answer Visual Questions
Zeng, X., Wang, Y., Chiu, T.-Y., Bhattacharya, N., & Gurari, D. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415220 · Cited by 31
“The AI community’s efforts to develop attention-based VQA algorithms [22, 42, 60]—finding the object(s) of interest in an image as an intermediate step towards answering a visual question—is a promising step in this direction, since object recognition is nearly always necessary for answering visual questions.”
BackgroundK-armed Bandit based Multi-Modal Network Architecture Search for Visual Question Answering
Zhou, Y., Ji, R., Sun, X., Luo, G., Hong, X., Su, J., Ding, X., & Shao, L. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413998 · Cited by 27
“Early developments resorted to bilinear-based approaches to improve the one-layer fusion of two modalities, e.g. , compact bilinear pooling [10], low-rank bilinear pooling[23] and factorized bilinear pooling [61].”
BackgroundMethodInfluentialMultimodal Multitask Representation Learning for Pathology Biobank Metadata PredictionPreprint
Weng, W., Cai, Y., Lin, A. S., Tan, F. E., & Chen, P.-H. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.07846 · Cited by 29
“For CBP, we followed the parameter setting in [10] with 16000-”
BackgroundMethodInfluentialMoVie: Revisiting Modulated Convolutions for Visual Counting and BeyondPreprint
Nguyen, D. K., Goswami, V., & Chen, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.11883 · Cited by 30
“Because the coverage of VQA is broad and existing benchmarks also include questions that require counting ability [1,43], it is important to note that general VQA models [44,23] without explicit counter also fall within the scope of ‘implicit’ ones when it comes to counting.”
BackgroundMethodDimitri, G. M. (2022) · Computers · doi:10.3390/computers11110163 · Cited by 22
One-Stage Object Referring with Gaze Estimation
Chen, J., Zhang, X., Wu, Y., Ghosh, S., Natarajan, P., Chang, S., & Allebach, J. P. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00550 · Cited by 22
“Methods w/o Gaze w/ Gaze S + D SimModel [25] 35.6 - MCB [8] 33.4 - NLOR [11] 36.9 - ORG [31] 38”
MethodInfluentialActively Seeking and Learning From Live Data
Teney, D., & van den Hengel, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00204 · Cited by 26
“Beyond apparent success on VQA benchmarks [14, 33], the approach was revealed to have severe limitations.”
BackgroundSertolli, B., Ren, Z. X., Schuller, B. W., & Cummins, N. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101204 · Cited by 24
Hu, C., Yin, M., Liu, B., Li, X., & Ye, Y. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3481908 · Cited by 30
“Bilinear pooling [9] based method was proposed to better capture the interactions between features in different modalities.”
Deep Tensor Evidence Fusion Network for Sentiment Classification
Wang, Z., Xu, G., Zhou, X., Kim, J., Zhu, H., & Deng, L. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3197994 · Cited by 23
“Recent techniques [33] sought to solve these shortcomings by simulating modal interactions using tensor representations.”
BackgroundEncoder–decoder cycle for visual question answering based on perception-action cycle
Mohamud, S. A. M., Jalali, A., & Lee, M. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109848 · Cited by 24
MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition
Xiang, P., Lin, C., Wu, K., & Bai, O. (2024) · International Conference on Pattern Recognition Systems (ICPRS) · doi:10.1109/icprs62101.2024.10677820 · Cited by 22
Stacked Self-Attention Networks for Visual Question Answering
Sun, Q., & Fu, Y. (2019) · International Conference on Multimedia Retrieval · doi:10.1145/3323873.3325044 · Cited by 22
“Furthermore, the feature fusion is explored from the simple element operation [23] to MLB [8], MCB [5] and MUTAN [3].”
MethodVisual-Textual Sentiment Analysis in Product Reviews
Ye, J., Peng, X., Qiao, Y., Xing, H., Li, J., & Ji, R. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8802992 · Cited by 22
“Bilinear pooling [20] or second-order pooling [21] is first introduced in image representations, and is further extent to Visual Question Answering (VQA) for jointly visual-textual representation learning [18, 19].”
BackgroundFecalNet: Automated detection of visible components in human feces using deep learning
Li, Q., Li, S., Liu, X., He, Z., Wang, T., Xu, Y., Guan, H., Chen, R., Qi, S., & Wang, F. (2020) · Medical Physics · doi:10.1002/mp.14352 · Cited by 22
A Human-Like Traffic Scene Understanding System: A Survey
Xia, Z.-X., Lai, W.-C., Tsao, L.-W., Hsu, L.-F., Yu, C.-C. H., Shuai, H.-H., & Cheng, W.-H. (2020) · IEEE Industrial Electronics Magazine · doi:10.1109/mie.2020.2970790 · Cited by 25
“, such as [4], [48]; 2) attention models that perform attention mechanisms to locate the relevant visual region for better performance [17], [68]; 3) composition models [3], [30], which use a modular representation of the neural networks; and 4) graph-based VQA models [24], [25], where a graph representation of the im…”
BackgroundModality adaptation in multimodal data
Razzaghi, P., Abbasi, K., Shirazi, M., & Shabani, N. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115126 · Cited by 22
A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanations
Rao, V. N., Zhen, X., Hovsepian, K., & Shen, M. (2021) · doi:10.18653/v1/2021.maiworkshop-1.4 · Cited by 26
Research on Visual Question Answering Based on GAT Relational Reasoning
Miao, Y., Cheng, W., He, S., & Jiang, H. (2022) · Neural Processing Letters · doi:10.1007/s11063-021-10689-2 · Cited by 25
Leveraging Medical Visual Question Answering with Supporting FactsPreprint
Kornuta, T., Rajan, D., Shivade, C., Asseman, A., & Özcan, A. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.12008 · Cited by 25
“These fusion techniques varied from concatenation of image and question representations, to more complex pooling mechanisms such as Multi-modal Compact Bilinear pooling (MCB) [7] and Multi-modal Low-rank Bilinear pooling (MLB) [12].”
Visual Robustness Benchmark for Visual Question Answering (VQA)
Ishmam, M. F., Tashdeed, I., Saadat, T. A., Ashmafee, M. H., Kamal, A. R. M., & Hossain, M. A. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00645 · Cited by 33
“Over the years, several works [4, 8, 20, 68] constantly improved the accuracy of VQA models.”
Lessons Learned in Multilingual Grounded Language Learning
Kádár, Á., Elliott, D., Côté, M.-A., Chrupała, G., & Alishahi, A. (2018) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k18-1039 · Cited by 26
“, 2015), visual question answering (Antol et al., 2015; Fukui et al., 2016; Jabri et al., 2016), text-toimage synthesis (Reed et al.”
BackgroundMulti-stage hybrid embedding fusion network for visual question answering
Lao, M., Guo, Y., Pu, N., Chen, W., Liu, Y., & Lew, M. S. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.10.071 · Cited by 22
Ren, B., Wu, Y., Huang, L., Zhang, Z., Huang, B., Zhang, H., Ma, J., Li, B., Liu, X., Wu, G., et al. (2021) · Human Brain Mapping · doi:10.1002/hbm.25748 · Cited by 23
“Multimodal bilinear fusion strategies have been recently proposed (Fukui et al., 2016; Yu, Yu, Xiang, Fan, & Tao, 2018) to concatenate the visual and textual representations.”
BackgroundSCATE: shared cross attention transformer encoders for multimodal fake news detection
Sachan, T., Pinnaparaju, N., Gupta, M., & Varma, V. (2021) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1145/3487351.3490965 · Cited by 27
“The vectors E′ t and E′ i are then combined together using compact bilinear pooling which has been shown to work well with multimodal systems [30].”
BackgroundWu, X., Mao, J., Xie, H., & Li, G. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.102977 · Cited by 28
Learning From Human Attention for Attribute-Assisted Visual Recognition
Bai, X. P., Zhang, P., Yu, X., Zheng, J. G., Hancock, E. R., Zhou, J. W., & Gu, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3458921 · Cited by 24
Face alignment recurrent network
Hou, Q., Wang, J., Bai, R., Zhou, S., & Gong, Y. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2017.09.028 · Cited by 21
Convolutional neural networks for computer vision-based detection and recognition of dumpsters
Ramírez, I., Cuesta‐Infante, A., Pantrigo, J. J., Montemayor, A. S., Moreno, J. L., Alonso, V., Anguita, G., & Palombarani, L. (2018) · Neural Computing and Applications · doi:10.1007/s00521-018-3390-8 · Cited by 24
“…object detection and tracking [18, 23], segmentation [13], human pose estimation [4] or visual attention and saliency [2], but it has also made possible and success of others which were unreachable a decade ago, namely image and video captioning and description [7, 15], or question answering [8].”
BackgroundVideo Question Answering via Knowledge-based Progressive Spatial-Temporal Attention Network
Jin, W., Zhao, Z., Li, Y., Li, J., Xiao, J., & Zhuang, Y. (2019) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3321505 · Cited by 21
“Fukui et al. [11] employ a multimodal pooling method to perform the joint embedding visual and text features.”
MethodToward Invisible Adversarial Examples Against DNN-Based Privacy Leakage for Internet of Things
Ding, X., Zhang, S., Song, M., Ding, X., & Li, F. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3008232 · Cited by 22
“…outstanding performance in many computer vision tasks, such as image classification [1], object detection [2], [3], and visual question answer [4], [5] because of the improvement of the model structure of DNNs, the optimization methods, the computing resources such as GPUs and the appearance of…”
MethodCho, J. W., Kim, D.-J., Choi, J., Jung, Y., & Kweon, I. S. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00175 · Cited by 28
“Visual Question Answering Recently, by virtue of the advancements in deep learning, there has been significant improvements in VQA models [28, 32, 35], with recent VQA models mostly focusing on designing better visual/question bilinear fusion [5, 10, 11, 26, 37] or better attention mechanisms [2, 7, 25, 28, 35, 36].”
MethodMultimodal Fusion Network with Latent Topic Memory for Rumor Detection
Chen, J., Wu, Z., Yang, Z., Xie, H., Wang, F. L., & Liu, W. (2021) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme51207.2021.9428404 · Cited by 21
“Fukui et al. [15] introduce the bilinear model to solve the problem of multimodal feature fusion in visual question answering (VQA).”
BackgroundSharma, H., & Jalal, A. S. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.116159 · Cited by 22
Speech-Based Visual Question AnsweringPreprint
Zhang, T., Dai, D., Tuytelaars, T., Moens, M., & Gool, L. V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.00464 · Cited by 22
“Some use an element-wise multiplication [29, 30] to pool these modalities, while [17] and [9] have shown much success in using more complex methods.”
MethodCLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations
Salewski, L., Koepke, A. S., Lensch, H. P. A., & Akata, Z. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_5 · Cited by 20
“It combines a modified MCB [16] framework, pre-trained on the VQA v2 dataset, with a visual pointing and textual justification module.”
MethodResultLRMM: Low rank multi-scale multi-modal fusion for person re-identification based on RGB-NI-TI
Wu, D., Liu, Z., Chen, Z., Gan, S., Tan, K., Wan, Q., & Wang, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125716 · Cited by 34
TACMT: Text-aware cross-modal transformer for visual grounding on high-resolution SAR images
Li, T., Wang, C., Tian, S., Zhang, B., Wu, F., Tang, Y., & Zhang, H. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.02.022 · Cited by 26
XTQA: Span-Level Explanations for Textbook Question Answering
Ma, J., Chai, Q., Liu, J., Yin, Q., Wang, P., & Zheng, Q. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3294991 · Cited by 21
“Fukui et al. [25] proposed the multimodal compact bilinear pooling method to learn joint input representations and project them into the answer space to predict answers.”
MethodEAML: ensemble self-attention-based mutual learning network for document image classification
Bakkali, S., Ming, Z., Coustaty, M., & Rusiñol, M. (2021) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-021-00378-0 · Cited by 23
“[22] proposed a multi-modal compact bilinear pooling to efficiently and expressively combine multi-modal features.”
BackgroundQian, Y., Wang, X., Liu, X., Ling, H., Jiang, Y., Chai, Y., & Liu, Y. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.102984 · Cited by 24
Yan, F., Silamu, W., Li, Y., & Chai, Y. (2022) · The Visual Computer · doi:10.1007/s00371-022-02524-z · Cited by 22
“A bilinear pooling-based feature fusionmethod [8] was applied to VQA to obtain high-level interaction functions between two modals.”
BackgroundMethodInfluentialDecoupled Cross-Modal Phrase-Attention Network for Image-Sentence Matching
Shi, Z., Zhang, T., Wei, X., Wu, F. H., & Zhang, Y. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3197972 · Cited by 22
“It is one of the fundamental problems for multimedia content understanding, and can provide inspiration for tasks such as cross-modal retrieval [4], [5], [6], [7], visual captioning [8], [9], visual grounding [10], [11] and visual question answering [12], [13], [14].”
BackgroundHuman Emotion Recognition Based on Spatio-Temporal Facial Features Using HOG-HOF and VGG-LSTM
Chouhayebi, H., Mahraz, M. A., Riffi, J., Tairi, H., & Alioua, N. (2024) · Computers · doi:10.3390/computers13040101 · Cited by 21
Text2Video: An End-to-end Learning Framework for Expressing Text With Videos
Yang, X., Zhang, T., & Xu, C. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2807588 · Cited by 21
“CT-SAN is the Concept-Tracing Semantic Attention Network [64] for video-to-language modeling by leveraging the concept word detector and the compact bilinear pooling [65].”
MethodWhat Action Causes This? Towards Naive Physical Action-Effect Prediction
Gao, Q., Yang, S., Chai, J. Y., & Vanderwende, L. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1086 · Cited by 29
“Recent years have seen an increasing amount of work integrating language and vision, for example, visual question answering (Antol et al., 2015; Fukui et al., 2016; Lu et al., 2016), image description generation (Xu et al., 2015; Vinyals et al., 2015), and grounding language to perception (Yang et…”
BackgroundStructured Triplet Learning with POS-Tag Guided Attention for Visual Question Answering
Wang, Z., Liu, X., Wang, L., Qiao, Y., Xie, X. S., & Fowlkes, C. C. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00209 · Cited by 20
Visual Query Answering by Entity-Attribute Graph Matching and Reasoning
Xiong, P., Zhan, H., Wang, X. Y., Sinha, B., & Wu, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00855 · Cited by 22
“[7] utilized Multimodal Compact Bilinear (MCB) pooling to efficiently and expressively combine multimodal features.”
MethodVisual Question Answering via Combining Inferential Attention and Semantic Space Mapping
Liu, Y., Zhang, X., Huang, F., Zhou, Z., Zhao, Z., & Li, Z. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106339 · Cited by 20
Beyond VQA: Generating Multi-word Answers and Rationales to Visual Questions
Dua, R., Kancheti, S. S., & Balasubramanian, V. N. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00178 · Cited by 20
“Other recent work has focused on better multimodal fusion methods [19, 20, 12, 43], the incorporation of relations [31, 23, 35], the use of multi-step reasoning [4], and neural module networks for compositional reasoning [18, 6, 16].”
Deep Modular Bilinear Attention Network for Visual Question Answering
Yan, F., Silamu, W., & Li, Y. (2022) · Sensors · doi:10.3390/s22031045 · Cited by 20
“Many approximated fusion methods, including MCB [2], MLB [26] and MUTAN [4], were proposed, which have shown better performance with fewer parameters.”
Metadata
- Authors
- Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M.
- Affiliations
- Sony (Japan) (JP)
- University of California, Berkeley (US)
- Max Planck Institute for Informatics (DE)
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 457-468
- OpenAlex
- W2963383024
- Cited by
- 1,582
- References
- 60
- Citations by year
- 2026: 42 · 2025: 105 · 2024: 126 · 2023: 165 · 2022: 189 · 2021: 222 · 2020: 225 · 2019: 201 · 2018: 179 · 2017: 105 · 2016: 20
- FWCI
- 58.39
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Max Planck Digital Library (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1427425, 1212798, N000141110688)
- Defense Advanced Research Projects Agency
- Multidisciplinary University Research Initiative (N000141110688)
- Air Force Research Laboratory
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- visual question answering, visual grounding, multimodal fusion, outer product, attention mechanism, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.