Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years.However, tasks such as visual question answering require combining these vector representations with each other.Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations.We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors.As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features.We extensively evaluate MCB on the visual question answering and grounding tasks.We consistently show the benefit of MCB over ablations without MCB.For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation.This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, the PJ-X backbone MCB [16] (which is crucial for the answering performance) preceded the publication of the CLEVR dataset.”
From CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We replace the concatenation of the visual representation and the encoded phrase in Grounder with MCB to combine both modalities.”
MethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“, 2015) that is pretrained on ImageNet data (Deng et al., 2009).”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Also, for this model, we concatenated the learned word embedding with pretrained GloVe vectors (Pennington et al., 2014).”
MethodGirshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Research works printed but not linked: 3
- Ross Girshick. 2015. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
- Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014. Multimodal neural language mod- els. In Proceedings of the International Conference on Machine Learning (ICML), pages 595-603.
- Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems (NIPS).
Cited by
Items 301–400 of 1,002
A Simple Baseline for Audio-Visual Scene-Aware Dialog
Schwartz, I., Schwing, A. G., & Hazan, T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01283 · Cited by 96
“Multimodal pooling methods were also explored [18, 33].”
BackgroundVisual-textual sentiment classification with bi-directional multi-level attention networks
Xu, J. J., Huang, F., Zhang, X., Wang, S., Li, C., Li, Z., & He, Y. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.04.018 · Cited by 75
Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
Li, Q., Fu, J., Yu, D., Mei, T., & Luo, J. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1164 · Cited by 77
“[8] incorporate a powerful feature fusion method into visual attention and obtain impressive results.”
BackgroundMethodDeep Multimodal Learning for the Diagnosis of Autism Spectrum Disorder
Tang, M., Kumar, P., Chen, H., & Shrivastava, A. (2020) · Journal of Imaging · doi:10.3390/jimaging6060047 · Cited by 76
“(2016) [14] propose a new method for multimodal compact bilinear (MCB) pooling by computing the outer product of the different feature vectors from multiple modalities.”
MethodZhang, W., Yu, J., Zhao, W., & Ran, C. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.02.006 · Cited by 75
Design of an explainable machine learning challenge for video interviews
Escalante, H. J., Guyon, I., Escalera, S., Jacques, J., Madadi, M., Baro, X., Ayache, S., Viegas, E., Gucluturk, Y., Guclu, U., et al. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966320 · Cited by 63
“the winning solutions for some recent challenges, [18], [19], [20], [21], [22], [23], [24], [25], [26]).”
Interpretable Visual Question Answering by Visual Grounding From Attention Supervision Mining
Zhang, Y., Niebles, J. C., & Soto, Á. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00043 · Cited by 71
Focal Visual-Text Attention for Memex Question Answering
Liang, J., Jiang, L., Cao, L., Kalantidis, Y., Li, L.-J., & Hauptmann, A. G. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2890628 · Cited by 76
“A variety of methods have been proposed to address the challenges from different aspects [5, 27, 14, 6, 20, 3, 16, 13], with remarkable progress on answering about a single image.”
BackgroundMethodResultInfluentialMultimodal feature fusion by relational reasoning and attention for visual question answering
Zhang, W., Yu, J., Hu, H., Hu, H., & Qin, Z. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.08.009 · Cited by 72
TallyQA: Answering Complex Counting Questions
Acharya, M., Kafle, K., & Kanan, C. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018076 · Cited by 69
Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognition
Arshad, O., Gallo, I., Nawaz, S., & Calefati, A. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00061 · Cited by 80
“337 2379-2140/19/$31.00 ©2019 IEEE DOI 10.1109/ICDAR.2019.00061 been extensively employed to vision and text related tasks, such as Image Captioning [15], VQA [10], Cross-Modal Retrieval [16] NER [11], [12], [13].”
BackgroundInfluentialLearning Visual Knowledge Memory Networks for Visual Question Answering
Zhou, S., Zhu, C., Dong, Y., Cai, D., Chen, Y., & Li, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00807 · Cited by 82
Deep Local Trajectory Replanning and Control for Robot Navigation
Pokle, A., Martín-Martín, R., Goebel, P., Chow, V., Ewald, H. M., Yang, J., Wang, Z., Sadeghian, A., Sadigh, D., Savarese, S., et al. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794062 · Cited by 80
“We also found that, as in [53], the outer product led to very high-dimensional features that made training prohibitively expensive or prone to over-fitting.”
Violin: A Large-Scale Dataset for Video-and-Language Inference
Liu, J., Chen, W., Cheng, Y., Gan, Z., Yu, L., Yang, Y., & Liu, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01091 · Cited by 91
Effective white-box testing of deep neural networks with adaptive neuron-selection strategy
Lee, S., Cha, S., Lee, D., & Oh, H. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3397346 · Cited by 88
“In particular, VGG-19 and ResNet-50 are widely used in practice when, for example, extracting the embedding of the images in various tasks such as style transfer [10, 14], image cap-tioning [35], and visual question answering [9].”
BackgroundScene Graph Refinement Network for Visual Question Answering
Qian, T., Chen, J., Chen, S., Wu, B., & Jiang, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3169065 · Cited by 85
“tention mechanisms [4]–[8] and can not address this task well since they lack the ability to perform explicit compositional rea-”
BackgroundUAV’s Status Is Worth Considering: A Fusion Representations Matching Method for Geo-Localization
Zhu, R., Yang, M., Yin, L., Wu, F. Q., & Yang, Y. (2023) · Sensors · doi:10.3390/s23020720 · Cited by 75
“[40] relied on CBP [39], and proposed multimodal compact bilinear pooling (MCB) to combine multimodal features and applied MCB on the visual question answering.”
MethodZero-Shot Visual Question AnsweringPreprint
Teney, D., & van den Hengel, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05546 · Cited by 69
“This distinction reflects the fact that CNNs pre-trained on ImageNet [4] are commonly used in existing VQA methods, and the fact that VQA is the task that we are actually interested in.”
Methode-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks
Kayser, M., Camburu, O.-M., Salewski, L., Emde, C., Do, V., Akata, Z., & Lukasiewicz, T. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00128 · Cited by 68
“Model MT model MT only Joint MT only Joint MT only Joint PJ-X MCB [18] N.”
MethodVQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering
Wang, Y., Yasunaga, M., Ren, H., Wada, S., & Leskovec, J. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01973 · Cited by 68
“It is crucial in many real-world tasks including scene understanding, autonomous vehicles, search engines, and recommendation systems [1,2,9,15].”
BackgroundDeep Attention Neural Tensor Network for Visual Question Answering
Bai, Y., Fu, J., Zhao, T., & Mei, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01258-8_2 · Cited by 73
“In this paper, we apply the attention mechanisms used in [9, 6] to learn the attention on the relevant visual regions and discard the useless information regarding the question.”
MethodMultimodal feature-wise co-attention method for visual question answering
Zhang, S., Chen, M., Chen, J., Zou, F., Li, Y.-F., & Lü, P. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.02.022 · Cited by 73
“The attention methods proposed in [5,9, 11,12,22] focus on solving the problem of visual spatial attention.”
InfluentialDual self-attention with co-attention networks for visual question answering
Liu, Y., Zhang, X., Zhang, Q., Li, C., Huang, F., Tang, X., & Li, Z. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107956 · Cited by 76
Vision-Language Transformer for Interpretable Pathology Visual Question Answering
Naseem, U., Khushi, M., & Kim, J. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3163751 · Cited by 63
“…challenges [11], [20], [21] on MedVQA tend to adapt the advance methods used in general-domain VQA such as Multi-modal Compact Bilinear (MCB) [22], Stacked Attention Networks (SAN) [23], Bilin-ear Attention Networks (BAN) [24], Multi-modal factorized bilinear (MFB) [25] and Multimodal…”
MethodAbouelaziz, I., Chetouani, A., Hassouni, M. E., Latecki, L. J., & Cherifi, H. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.107174 · Cited by 73
“Once the feature vectors are extracted from the above-described models, we combine them using the concept of Compact Bi-linear Pooling (CBP) [45] that computes the outer product of two feature vectors u and v .”
MethodC-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 DatasetPreprint
Agrawal, A., Kembhavi, A., Batra, D., & Parikh, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.08243 · Cited by 75
“To demonstrate the difficulty of our C-VQA splits, we report the performance of several existing VQA models [17, 20, 23, 27, 34] on our C-VQA splits.”
MethodDeepVecFont: synthesizing high-quality vector fonts via dual-modality learning
Wang, Y., & Lian, Z. (2021) · ACM Transactions on Graphics · doi:10.1145/3478513.3480488 · Cited by 112
“Representative methods for multimodal feature fusion include Concat+MLP, MCB [Fukui et al. 2016], MLB [Kim et al.”
MethodTowards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks
Luo, G., Zhou, Y., Sun, X., Wang, Y., Cao, L., Wu, Y., Huang, F., & Ji, R. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3139234 · Cited by 80
“For example, the recent X-Linear Transformer [49] integrates bilinear pooling [54] and squeeze-excitation module [55] into self-attention to selectively capitalize on visual information and to perform multimodal reasoning.”
BackgroundResultObject Referring in Videos with Language and Human Gaze
Vasudevan, A. B., Dai, D., & Gool, L. V. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00434 · Cited by 93
Training Lightweight Deep Convolutional Neural Networks Using Bag-of-Features Pooling
Passalis, N., & Tefas, A. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2872995 · Cited by 72
A Unified Framework for Multimodal Domain Adaptation
Qi, F., Yang, X., & Xu, C. (2018) · Proceedings of the 26th ACM international conference on Multimedia · doi:10.1145/3240508.3240633 · Cited by 64
“Bilinear models [14, 31] are recent popular and powerful solution to the fusion problem.”
BackgroundMethodAMAM: An Attention-based Multimodal Alignment Model for Medical Visual Question Answering
Pan, H., He, S., Zhang, K., Qu, B., Chen, C., & Shi, K. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109763 · Cited by 76
NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Tasks
Sammani, F., Mukherjee, T., & Deligiannis, N. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00814 · Cited by 110
“The authors used the MCB VQA model [16] as the task model and an LSTM as the explanation model.”
MethodDualNet: Domain-invariant network for visual question answering
Saito, K., Shin, A., Ushiku, Y., & Harada, T. (2017) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2017.8019436 · Cited by 64
“Compared with MCB (w/o attention) [8], the performance of our model is comparable.”
BackgroundAudio-Visual Deep Neural Network for Robust Person Verification
Qian, Y., Chen, Z., & Wang, S. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3057230 · Cited by 63
“The work in [46] introduced a method, called multi-modal compact bilinear pooling (MCB), to approximate the outer product result and reduce the dimension simultaneously.”
MethodRemoving Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies
Gat, I., Schwartz, I., Schwing, A., & Hazan, T. (2020) · arXiv · doi:10.48550/arxiv.2010.10802 · Cited by 63
“Subsequently, multiple works developed strong models to address these datasets [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26].”
BackgroundLong-Term Video Question Answering via Multimodal Hierarchical Memory Attentive Networks
Yu, T., Yu, J., Zhou, Y., Huang, Q., & Tian, Q. C. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.2995959 · Cited by 71
Medical Visual Question Answering via Conditional Reasoning and Contrastive Learning
Liu, B., Zhan, L.-M., Xu, L., & Wu, X.-M. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3232411 · Cited by 77
“…fusion, [71] adopted a simple concatenation operation, [1] exploited stacked attention networks (SAN) [64] and compact bilinear pooling (MCB) [15], [63] and [53] employed multimodal factorized bilinear pooling (MFB) [67], and [61] proposed to query an image by means of a written question…”
MethodA Spatial Hierarchical Reasoning Network for Remote Sensing Visual Question Answering
Zhang, Z., Jiao, L., Li, L., Liu, X., Chen, P., Liu, F., Li, Y., & Guo, Z. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3237606 · Cited by 86
“The multimodal compact bilinear (MCB) pooling [28], multimodal low-rank bilinear (MLB) attention network [31], and multimodal factorized bilinear (MFB) pooling [32] are the typical models, which exploit bilinear techniques to learn the high-level multimodal joint representation.”
BackgroundRobust Visual Question Answering: Datasets, Methods, and Future Challenges
Ma, J., Wang, P., Kong, D., Wang, Z., Liu, J., Pei, H., & Zhao, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3366154 · Cited by 87
“Current non-debiasing (generic) methods [16], [19], [20], [38], [39], [40] f typically leverage an image encoder Ev : V → Rnv×dv to learn nv regionlevel (patch-level) representations with dv dimensions, a question encoder Eq : Q → Rnq×dq to output nq word-level vectors with dq dimensions, a multi-modality encoder Em :…”
BackgroundCounterfactual Causal-Effect Intervention for Interpretable Medical Visual Question Answering
Cai, L., Fang, H., Xu, N., & Ren, B. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3425533 · Cited by 65
“Existing VQA-Med models tend to employ some advanced methods used in generic VQA, such as multi-modal compact bilinear [20], stacked attention networks [21], bilinear attention networks [22], focal visual-text attention [23], and multi-modal factorized high-order [24].”
ENVQA: Improving Visual Question Answering model by enriching the visual feature
Chowdhury, S., & Soni, B. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109948 · Cited by 65
Multi-interaction Network with Object Relation for Video Question Answering
Jin, W., Zhao, Z., Gu, M., Yu, J., Xiao, J., & Zhuang, Y. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3351065 · Cited by 67
“VQA-MCB [4] is also an image-based method which utilizes multimodal compact bilinear pooling and spatial attention.”
MethodKaramcheti, S., Krishna, R., Fei-Fei, L., & Manning, C. D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.564 · Cited by 84
“…2015a; Johnson et al., 2017; Goyal et al., 2017; Krishna et al., 2017; Suhr et al., 2019; Hudson and Manning, 2019) and models (Zhou et al., 2015; Fukui et al., 2016; Lu et al., 2016; Yang et al., 2016; Zhu et al., 2016; Wu et al., 2016; Anderson et al., 2018a; Tan and Bansal, 2019; Chen et al.,…”
BackgroundLearning to Learn Better Unimodal Representations via Adaptive Multimodal Meta-Learning
Ya, S., Mai, S., & Hu, H. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3178231 · Cited by 69
“These solutions tend to develop sophisticated fusion mechanisms for joint sentiment analysis, including tensor-based algorithms [14], [15], [16], [17], [18], [19], attention-based models [20], [21], [22], [23], [24], graph-based fusion [25], [26], [27] and so on.”
BackgroundObfuscation-Resilient Android Malware Analysis Based on Complementary Features
Gao, C., Cai, M., Yin, S., Huang, G., Li, H., Yuan, W., & Luo, X. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3302509 · Cited by 69
Multi-Modal Deep Analysis for Multimedia
Zhu, W., Wang, X. Y., & Li, H. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2940647 · Cited by 68
“Fukui et al. [116] propose a Multi-modal Compact Bilinear pooling (MCB) algorithm which adopts a sampling-based computation and projection method to reduce dimensionality while preserving the performance of full bilinear pooling.”
BackgroundMethodTIGEr: Text-to-Image Grounding for Image Caption Evaluation
Jiang, M., Huang, Q., Zhang, L., Wang, X., Zhang, P., Gan, Z., Diesner, J., & Gao, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1220 · Cited by 59
“, softmax function) (Jabri et al., 2016; Fukui et al., 2016).”
BackgroundDeep Generative Probabilistic Graph Neural Networks for Scene Graph Generation
Khademi, M., & Schulte, O. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6783 · Cited by 67
“MCB leverages the Visual Genome QA pairs as additional training data and the 152 -layer ResNet as a pretrained model.”
MethodInfluentialWill People Like Your Image? Learning the Aesthetic Space
Schwarz, K., Wieschollek, P., & Lensch, H. P. A. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00226 · Cited by 72
Object-Difference Attention: A Simple Relational Attention for Visual Question Answering
Wu, C., Liu, J., Wang, X., & Dong, X. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240513 · Cited by 68
“Recent studies use bilinear fusion to further increase the accuracy of the attention distribution on objects [5, 8, 14, 36].”
BackgroundMultimodal Encoder-Decoder Attention Networks for Visual Question Answering
Chen, C., Han, D., & Wang, J. (2020) · IEEE Access · doi:10.1109/access.2020.2975093 · Cited by 57
“[26], and it has become an integral part of the VQA model that requires fine-grained visual understanding [20], [29].”
BackgroundQuery-controllable Video Summarization
Huang, J.-H., & Worring, M. (2020) · International Conference on Multimedia Retrieval · doi:10.1145/3372278.3390695 · Cited by 76
“It has been shown that, in similar multi-modal contexts, the performance of models can decrease if feature fusion methods are improper; how to solve this issue, in general, remains an open question [2, 5].”
BackgroundInterpretable Visual Question Answering by Reasoning on Dependency Trees
Cao, Q., Liang, X., Li, B., & Lin, L. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2943456 · Cited by 66
“Among these works, the attention mechanism [2], [11], [12], [13], [14], [15] and the joint embedding of image and question representation [3], [16], [17] have been widely”
BackgroundExplore Multi-Step Reasoning in Video Question Answering
Song, X., Shi, Y., Chen, X., & Han, Y. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240563 · Cited by 70
HsgNet: A Road Extraction Network Based on Global Perception of High-Order Spatial Information
Xie, Y., Miao, F., Zhou, K., & Peng, J. (2019) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi8120571 · Cited by 58
“These have been continuously optimized in ways such as dimensionality reduction [27], multimodal feature fusion [28], low-rank reconstruction [29–31], etc.”
MethodMultimodal Entity Linking for Tweets
Adjali, O., Besançon, R., Ferret, O., Borgne, H. L., & Grau, B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-45439-5_31 · Cited by 95
“Joint multimodal representations are used in several multimodal applications such as visual question answering [1, 26] , text-image retrieval [4, 7, 48] and image captioning [28, 30] and exploits different approaches e.g., Canonical Correlation Analysis [46] , linear ranking-based models and non-linear deep learning m…”
BackgroundShin, A., Ishii, M., & Narihira, T. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-021-01547-8 · Cited by 65
“…scheme may be simple concatenation or element-wise addition and multiplications, more sophisticated schemes have also been proposed, such as compact bilinear pooling (Fukui et al. (2016)), low-rank bilinear pooling (Kim et al. (2017)), or cross-modal tucker fusion (Ben-younes et al. (2017)).”
BackgroundTemporal Reasoning via Audio Question Answering
Fayek, H. M., & Johnson, J. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3010650 · Cited by 90
Look Before you Speak: Visually Contextualized Utterances
Seo, P. H., Nagrani, A., & Schmid, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01660 · Cited by 89
“Popular vision and language tasks include visual question answering [4, 5, 22, 28, 58, 61], visual dialog [15, 16, 36, 40, 67], visual captioning [56, 59, 68, 73, 86], visual grounding [18, 22, 57, 83] and video-text retrieval [7, 25, 49, 53, 63].”
BackgroundCounterfactual Samples Synthesizing and Training for Robust Visual Question Answering
Chen, L., Zheng, Y., Niu, Y., Zhang, H., & Xiao, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3290012 · Cited by 78
Using Syntax to Ground Referring Expressions in Natural Images
Cirik, V., Berg-Kirkpatrick, T., & Morency, L. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12343 · Cited by 74
“In earlier stud- ies (Mao et al. 2016; Hu et al. 2016; Rohrbach et al. 2016; Fukui et al. 2016), the bounding boxes are scored based on their CNN and spatial features along with features for the whole image.”
BackgroundG3raphGround: Graph-Based Language Grounding
Bajaj, M., Wang, L., & Sigal, L. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00438 · Cited by 63
“The caption encoder and the Method Accuracy SMPL [30] 42.08 NonlinearSP [29] 43.89 GroundeR [26] 47.81 MCB [10] 48.69 RtP [24] 50.89 Similarity Network [28] 51.05 IGOP [38] 53.97 SPC+PPC [23] 55.49 SS+QRN (VGG det ) [6] 55.99 CITE [22] 59.27 SeqGROUND [9] 61.60 CITE [22] (finetuned) 61.89 QRC Net…”
MethodInfluentialInfrared and Visible Image Fusion Based on Deep Decomposition Network and Saliency Analysis
Jian, L., Rayhana, R., Ma, L., Wu, S., Liu, Z., & Jiang, H. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3096088 · Cited by 62
Fine-Grained Instance-Level Sketch-Based Image Retrieval
Yu, Q., Song, J., Song, Y.-Z., Xiang, T., & Hospedales, T. M. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-020-01382-3 · Cited by 60
“Attention modelling Visual attention models have been studied extensively in a wide range of vision problems including image caption generation (Xu et al. 2015; Lu et al. 2016), VQA (Fukui et al. 2016; Nam et al. 2016), image classification (Mnih et al. 2014; Sermanet et al. 2014; Xiao et al. 2015) and particularly fine…”
BackgroundMedical knowledge-based network for Patient-oriented Visual Question Answering
Huang, J., Chen, Y., Li, Y., Yang, Z., Gong, X., Wang, F. L., Xu, X., & Liu, W. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103241 · Cited by 58
Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering
Tang, R., Ma, C., Zhang, W. E., Wu, Q., & Yang, X. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58529-7_26 · Cited by 56
Visual question answering model based on graph neural network and contextual attention
Sharma, H., & Jalal, A. S. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2021.104165 · Cited by 60
Interpretable Counting for Visual Question Answering
Trott, A., Xiong, C., & Socher, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.08697 · Cited by 36
“In contrast, the predominant approach to VQA involves representing the visual input with the final feature map of a convolutional neural network (CNN), attending to regions based on an encoding of the question, and classifying the answer from the attention-weighted image features (Xu & Saenko, 2015; Yang et al., 2015;…”
BackgroundMethodInfluentialSpatial Dual-Modality Graph Reasoning for Key Information ExtractionPreprint
Sun, H., Kuang, Z., Yue, X., Lin, C., & Zhang, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.14470 · Cited by 77
“Our work is also related to multi-modality fusion methods [30], [31], [32], [33].”
BackgroundVision + X: A Survey on Multimodal Learning in the Light of Data
Zhu, Y., Wu, Y., Sebe, N., & Yan, Y. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3420239 · Cited by 80
“As a popular discriminative vision and language task, visual grounding aims to localize the object within an image given a text description as input [61], [63], [78], [106], [112], [140], [148], [153], [214], [276], [291], [292], [298].”
BackgroundDRAU: Dual Recurrent Attention Units for Visual Question Answering
Osman, A., & Samek, W. (2019) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2019.05.001 · Cited by 58
Question Type Guided Attention in Visual Question Answering
Shi, Y., Furlanello, T., Zha, S., & Anandkumar, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01225-0_10 · Cited by 65
Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54
“The unimodal sub-branches are usually fused by concatenation (Kiela & Bottou, 2014; Regneri, Rohrbach, Wetzel, Thater, Schiele, & Pinkal, 2013; Shekhar, Takmaz, Fernández, & Bernardi, 2019) or element-wise vector multiplication (Fukui, Park, Yang, Rohrbach, Darrell, & Rohrbach, 2016; Wang, Li, Huang, & Lazebnik, 2018)…”
Trajectory Prediction Network of Autonomous Vehicles With Fusion of Historical Interactive Features
Zuo, Z., Wang, X., Guo, S., Liu, Z., Li, Z., & Wang, Y. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3319024 · Cited by 68
“While the bilinear pooling method uses a matrix outer product to fuse features, which makes it possible to extract more complex feature interactions [39].”
MethodMulti-Task Learning of Hierarchical Vision-Language Representation
Nguyen, D.-K., & Okatani, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01074 · Cited by 55
“MCB [11] reported in [13]”
Generating Question Relevant Captions to Aid Visual Question Answering
Wu, J., Hu, Z., & Mooney, R. J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1348 · Cited by 56
“Recently, a large amount of attention-based deep-learning methods have been proposed for VQA, including top-down (Ren et al., 2015a; Fukui et al., 2016; Wu et al., 2016; Goyal et al., 2017; Li et al., 2018a) and bottom-up attention methods (Anderson et al., 2018; Li et al., 2018b; Wu and Mooney,…”
MethodInfluentialTwo-Stage Synthesis Networks for Transfer Learning in Machine Comprehension
Golub, D., Huang, P.-S., He, X., & Deng, L. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1087 · Cited by 65
“…have proven to be excellent feature extractors for diverse tasks such as image captioning (i.e., Lu et al. (2016); Fang et al. (2015); Karpathy and Fei-Fei (2015)) and visual question answering (i.e., Zhou et al. (2015); Xu and Saenko (2016); Fukui et al. (2016); Yang et al. (2016)), among others.”
BackgroundFrom Strings to Things: Knowledge-Enabled VQA Model That Can Read and Reason
Singh, A. K., Mishra, A., Shekhar, S., & Chakraborty, A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00470 · Cited by 66
“This has been leveraged in some of the earlier works in VQA such as [6, 11, 14, 15, 25, 29, 35, 41, 51, 52, 56, 60].”
ResultChen, X., Lin, X., Shen, Q., & Qian, X. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3035789 · Cited by 53
“The bilinear pooling method [38] can also be effectively used to fuse information from different modalities.”
MethodInformation Aggregation for Multi-Head Attention with Routing-by-Agreement
Li, J., Yang, B., Dou, Z.-Y., Wang, X., Lyu, M. R.-T., & Tu, Z. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1359 · Cited by 57
“For example, Fukui et al. (2016) and Ben-Younes et al. (2017) succeed on fusing multi-modal features (e.g., visual features and textual features) more effectively via employing the higher-order bilinear pooling instead of vector concatenation or element-wise operations.”
BackgroundDeepCU: Integrating both Common and Unique Latent Information for Multimodal Sentiment Analysis
Verma, S., Wang, C., Zhu, L., & Liu, W. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/503 · Cited by 56
Adaptive Modality Distillation for Separable Multimodal Sentiment Analysis
Peng, W., Hong, X., & Zhao, G. (2021) · IEEE Intelligent Systems · doi:10.1109/mis.2021.3057757 · Cited by 56
“BC-LSTM [8] and MV-LSTM [18] are based on LSTM.”
BackgroundMethodInfluentialProceedings of the 10th International Conference on Natural Language Generation
Krahmer, E., Gatt, A., & Goudbeek, M. B. (2017) · doi:10.18653/v1/w17-35 · Cited by 53
Parameter-Efficient Transfer Learning for Medical Visual Question Answering
Liu, J., Hu, T., Zhang, Y., Yang, F., Hao, J., Lv, J., & Liu, Z. (2023) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2023.3311333 · Cited by 72
“Abacha et al. [26] utilized stacked attention networks (SAN) [27] and compact bilinear pools (MCBS) [28].”
BackgroundA co-attention based multi-modal fusion network for review helpfulness prediction
Ren, G., Diao, L., Guo, F., & Hong, T. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103573 · Cited by 52
Co-Attention Network With Question Type for Visual Question Answering
Yang, C., Jiang, M., Jiang, B., Zhou, W., & Li, K. (2019) · IEEE Access · doi:10.1109/access.2019.2908035 · Cited by 55
Hierarchical Attention and Bilinear Fusion for Remote Sensing Image Scene Classification
Yu, D., Guo, H., Xu, Q., Lu, J., Zhao, C., & Lin, Y. (2020) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2020.3030257 · Cited by 52
“[46] proposed multimodal compact bilinear pooling and applied it to visual question answering.”
BackgroundMMKGR: Multi-hop Multi-modal Knowledge Graph Reasoning
Zheng, S., Wang, W., Qu, J., Yin, H., Chen, W., & Zhao, L. (2023) · 2023 IEEE 39th International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00015 · Cited by 45
“The limitation of this method is that the multi-modal noise affects the extraction of key features [14].”
MethodCompositional Attention Networks With Two-Stream Fusion for Video Question Answering
Yu, T., Yu, J., Zhou, Y., & Tao, D. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2940677 · Cited by 55
“To effectively integrate multimodal features of the images and questions, different bilinear pooling variants were introduced in VQA which were the state-of-the-art for common benchmark datasets at the time of each publication [4], [7], [11].”
BackgroundMethodA survey of methods, datasets and evaluation metrics for visual question answering
Sharma, H., & Jalal, A. S. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2021.104327 · Cited by 61
Factorized Multimodal Transformer for Multimodal Sequential LearningPreprint
Zadeh, A., Mao, C., Shi, K., Zhang, Y., Liang, P. P., Poria, S., & Morency, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09826 · Cited by 59
“Studies have also proposed generic fusion techniques that can be used in various models including Tensor Fusion (Zadeh et al., 2017) and its approximate variants (Liang et al.; Liu et al., 2018), as well as Compact Bilinear Pooling (Gao et al., 2015; Fukui et al., 2016; Kim et al., 2016).”
MethodDialogueTRM: Exploring Multi-Modal Emotional Dynamics in a Conversation
Mao, Y., Liu, G., Wang, X., Gao, W., & Li, X. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.229 · Cited by 50
“Bilnear (Fukui et al., 2016), GMU (Arevalo et al., 2020), and MulT (Tsai et al., 2019a) are three advanced single-grained fusion methods, in which the first two approaches only capture the prototype dependency, and the last one only captures the representation dependency.”
BackgroundMethodImproving Event Extraction via Multimodal Integration
Zhang, T., Whitehead, S., Zhang, H., Li, H., Ellis, J. G., Huang, L., Liu, W., Ji, H., & Chang, S. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123294 · Cited by 50
“Visul question answering is tackled in [8, 32].”
Vision-Dialog Navigation by Exploring Cross-Modal Memory
Zhu, Y., Zhu, F., Zhan, Z., Lin, B., Jiao, J., Chang, X., & Liang, X. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01074 · Cited by 49
On Modality Bias Recognition and Reduction
Guo, Y., Nie, L., Cheng, H. H., Cheng, Z., Kankanhalli, M., & Bimbo, A. D. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3565266 · Cited by 64
FNR: a similarity and transformer-based approach to detect multi-modal fake news in social media
Ghorbanpour, F., Ramezani, M., Fazli, M. A., & Rabiee, H. R. (2023) · Social Network Analysis and Mining · doi:10.1007/s13278-023-01065-0 · Cited by 55
“The researchers in AMFB (Kumari and Ekbal 2021) suggest an attention-based multi-modal factorized bilinear pooling that uses attention-based bidirectional LSTM to capture textual features and attention-based CNN-RNN blocks for capturing visual features (Fukui et al. 2016).”
Fast Graph Convolution Network Based Multi-label Image Recognition via Cross-modal Fusion
Wang, Y., Xie, Y., Liu, Y., Zhou, K., & Li, X. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3411880 · Cited by 60
“[7] first propose the Multi-modal Compact Bilinear pooling (MCB) which introduces the bilinear model to fuse multi-modal features by using the outer product of two vectors in different modalities to produce a very high-dimensional feature for quadratic expansion.”
BackgroundCross-Modal Hybrid Feature Fusion for Image-Sentence Matching
Xu, X., Wang, Y., He, Y., Yang, Y., Hanjalić, A., & Shen, H. T. (2021) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3458281 · Cited by 58
“As a result, multimodal compact bilinear pooling [13] in virtue of count sketch projection function [15] could reduce dimensions and parameters effectively.”
BackgroundGupta, V. K., Li, Z., Kortylewski, A., Zhang, C., Li, Y., & Yuille, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00502 · Cited by 81
“Multiple works have shown the effectiveness of attention in VQA [6, 10, 14, 23, 48, 50, 51].”
Background
Metadata
- Authors
- Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M.
- Affiliations
- Sony (Japan) (JP)
- University of California, Berkeley (US)
- Max Planck Institute for Informatics (DE)
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 457-468
- OpenAlex
- W2963383024
- Cited by
- 1,582
- References
- 60
- Citations by year
- 2026: 42 · 2025: 105 · 2024: 126 · 2023: 165 · 2022: 189 · 2021: 222 · 2020: 225 · 2019: 201 · 2018: 179 · 2017: 105 · 2016: 20
- FWCI
- 58.39
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Max Planck Digital Library (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1427425, 1212798, N000141110688)
- Defense Advanced Research Projects Agency
- Multidisciplinary University Research Initiative (N000141110688)
- Air Force Research Laboratory
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- visual question answering, visual grounding, multimodal fusion, outer product, attention mechanism, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.