Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years.However, tasks such as visual question answering require combining these vector representations with each other.Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations.We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors.As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features.We extensively evaluate MCB on the visual question answering and grounding tasks.We consistently show the benefit of MCB over ablations without MCB.For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation.This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, the PJ-X backbone MCB [16] (which is crucial for the answering performance) preceded the publication of the CLEVR dataset.”
From CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We replace the concatenation of the visual representation and the encoded phrase in Grounder with MCB to combine both modalities.”
MethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“, 2015) that is pretrained on ImageNet data (Deng et al., 2009).”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Also, for this model, we concatenated the learned word embedding with pretrained GloVe vectors (Pennington et al., 2014).”
MethodGirshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Research works printed but not linked: 3
- Ross Girshick. 2015. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
- Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014. Multimodal neural language mod- els. In Proceedings of the International Conference on Machine Learning (ICML), pages 595-603.
- Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems (NIPS).
Cited by
Items 201–300 of 1,002
Exploiting Unlabeled Data with Vision and Language Models for Object Detection
Zhao, S., Zhang, Z.-X., Schulter, S., Zhao, L., Kumar, B. G. V., Stathopoulos, A., Chandraker, M. K., & Metaxas, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20077-9_10 · Cited by 165
Ishmam, M. F., Shovon, M. S. H., Mridha, M. F., & Dey, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102270 · Cited by 130
“Anderson et al. [223] introduced bottom-up top-down attention.”
BackgroundFocal Visual-Text Attention for Visual Question Answering
Liang, J., Jiang, L., Cao, L., Li, L.-J., & Hauptmann, A. G. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00642 · Cited by 103
Overcoming Language Priors in VQA via Decomposed Linguistic Representations
Jing, C., Wu, Y., Zhang, X., Jia, Y., & Wu, Q. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6776 · Cited by 122
“Most VQA models (Fukui et al. 2016; Yang et al. 2016; Anderson et al. 2018) consist of three parts: extracting informative representations for both images and questions, fusing these representations to obtain joint embeddings of images and questions, and predicting final answers with the joint…”
BackgroundMRA-Net: Improving VQA Via Multi-Modal Relation Attention Network
Peng, L., Yang, Y., Wang, Z., Huang, Z. H., & Shen, H. T. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3004830 · Cited by 111
Learning Visual Question Answering by Bootstrapping Hard Attention
Malinowski, M., Doersch, C., Santoro, A., & Battaglia, P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01231-1_1 · Cited by 119
Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds
Jain, A., Gkanatsios, N., Mediratta, I., & Fragkiadaki, K. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_24 · Cited by 183
Rodis, N., Sardianos, C., Grammatikis, P. I. R., Sarigiannidis, P. G., Varlamis, I., & Papadopoulos, G. T. (2024) · IEEE Access · doi:10.1109/access.2024.3467062 · Cited by 123
“MU Intrinsic Concept-based [15], fusion/graph-based [67], attention [54, 25, 27, 28, 29, 30, 46, 71, 36, 37, 98, 43, 42, 99], attention/gradient-based [26], fusion/attention [70]”
MethodMultimodal machine translation through visuals and speech
Sulubacak, U., Caglayan, O., Grönroos, S.-A., Rouhe, A., Elliott, D., Specia, L., & Tiedemann, J. (2019) · Machine Translation · doi:10.1007/s10590-020-09250-0 · Cited by 105
“Delbrouck and Dupont (2017b) propose a different fusion operation based on compact bilinear pooling (Fukui et al. 2016), to efficiently realise the computationally expensive outer product.”
BackgroundLi, X., Lei, L., Sun, Y., Li, M., & Kuang, G. (2020) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2020.2975252 · Cited by 106
Exploring Human-Like Attention Supervision in Visual Question Answering
Qiao, T., Dong, J., & Xu, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12272 · Cited by 100
Grassmann Pooling as Compact Homogeneous Bilinear Pooling for Fine-Grained Visual Classification
Wei, X., Zhang, Y., Gong, Y., Zhang, J., & Zheng, N. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01219-9_22 · Cited by 105
“Thereafter, second-order models have been studied in several computer vision problems, such as object detection [22], semantic segmentation [35], fine-grained classification [21], visual question answering [36], etc.”
MethodA Read-Write Memory Network for Movie Story Understanding
Na, S., Lee, S., Kim, J., & Kim, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.80 · Cited by 100
“To transform the memory M into a query-dependent memory Mq ∈ Rm×d×3, we apply the CBP [6] between each memory cell of M and the query embedding u as”
MethodInfluentialLearning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos
Chen, S., Jiang, W., Liu, W., & Jiang, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58548-8_20 · Cited by 127
DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering
Wang, J., Bao, B., & Xu, C. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3097171 · Cited by 114
Multi-Modal Machine Learning in Engineering Design: A Review and Future Directions
Song, B., Zhou, R., & Ahmed, F. (2023) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4063954 · Cited by 121
Multi-modal Circulant Fusion for Video-to-Language and Backward
Wu, A., & Han, Y. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/143 · Cited by 126
Second-Order Pooling for Graph Neural Networks
Wang, Z., & Ji, S. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2999032 · Cited by 123
“recent advances of deep learning, second-order pooling is also used in CNNs for fine-grained visual recognition [28] and visual question answering [29], [30], [31].”
Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis
Tsai, Y.-H. H., Ma, M., Yang, M., Salakhutdinov, R., & Morency, L.-P. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.143 · Cited by 98
Supervised and Unsupervised Transfer Learning for Question Answering
Chung, Y.-A., Lee, H.-Y., & Glass, J. R. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1143 · Cited by 104
“…have proven to be excellent feature extractors for a broad range of visual tasks such as image captioning (Lu et al., 2017; Karpathy and Fei-Fei, 2015; Fang et al., 2015) and visual question answering (Xu and Saenko, 2016; Fukui et al., 2016; Yang et al., 2016; Antol et al., 2015), among others.”
BackgroundHypergraph Attention Networks for Multimodal Learning
Kim, E., Kang, W. Y., On, K.-W., Heo, Y., & Zhang, B. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01459 · Cited by 156
“The conventional VQA models [1, 36, 16, 17, 23, 6, 38, 15] learn the joint embedding of a pair of the question and the image with two stages.”
BackgroundRe-Attention for Visual Question Answering
Guo, W., Zhang, Y., Yang, J., & Yuan, X. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3097180 · Cited by 98
Yu, J., Zhang, W., Lu, Y., Qin, Z., Hu, Y., Tan, J., & Wu, Q. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2972830 · Cited by 105
“Besides straight-forward solutions, existing work applies bilinear pooling [29], [30] or more complex fusion methods [31].”
MethodYuan, Z., Mou, L., Wang, Q., & Zhu, X. X. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3173811 · Cited by 127
“[35], Kimet al. [36], and Ben et al .”
BackgroundVQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions
Li, Q., Tao, Q., Joty, S., Cai, J., & Luo, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_34 · Cited by 123
Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization
Liang, P. P., Liu, Z., Tsai, Y.-H. H., Zhao, Q., Salakhutdinov, R., & Morency, L.-P. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1152 · Cited by 95
“…to prior work which found tensor representations to capture higher-order interactions from multimodal data (Liu et al., 2018; Zadeh et al., 2017; Fukui et al., 2016), we form tensors via outer products of the individual representations through time (as shown in Figure 2): M = T ∑ t=1 [h t ` 1…”
BackgroundResultMarioQA: Answering Questions by Watching Gameplay Videos
Mun, J., Seo, P. H., Jung, I., & Han, B. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.312 · Cited by 94
“ImageQA Because ImageQA needs to handle two input modalities, i.e ., image and question, [6] presents a method of fusing two modalities to obtain rich multi-modal representations.”
BackgroundMethodInfluentialFluency-Guided Cross-Lingual Image Captioning
Lan, W., Li, X., & Dong, J. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123366 · Cited by 92
“LSTM [15], for its capability of modeling long-term word dependency in natural language text, has been used to learn a meaningful and compact representation for a given sentence [12, 22].”
BackgroundAdversarial attack and defense technologies in natural language processing: A survey
Qiu, S., Liu, Q., Zhou, S., & Huang, W. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.04.020 · Cited by 94
Visual Reference Resolution using Attention Memory for Visual DialogPreprint
Seo, P. H., Lehrmann, A., Han, B., & Sigal, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.07992 · Cited by 114
“Attention for Visual Reference Resolution: While visual dialog is a recent task, VQA has been studied extensively and attention models have been known to be beneficial for answering independent questions [10–16].”
BackgroundModality-Agnostic Attention Fusion for visual search with text feedbackPreprint
Dodds, E. M., Culpepper, J., Herdade, S., Zhang, Y., & Boakye, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00145 · Cited by 112
“A similar model has also been trained directly on image captioning and VQA [29].”
BackgroundMultimodality in meta-learning: A comprehensive survey
Ma, Y., Zhao, S., Wang, W., Li, Y., & King, I. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108976 · Cited by 99
Contextual Translation Embedding for Visual Relationship Detection and Scene Graph Generation
Hung, Z.-S., Mallya, A., & Lazebnik, S. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2992222 · Cited by 102
“Such phrase or relationship detections were used in applications such as object recognition [28], [29], [30], image classification [31], and text grounding [32], [33].”
MethodSun, L., Lian, Z., Liu, B., & Tao, J. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102382 · Cited by 74
Fooling Vision and Language Models Despite Localization and Attention Mechanism
Xu, X., Chen, X., Liu, C., Rohrbach, A., Darrell, T., & Song, D. X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00520 · Cited by 126
“We experiment with two state-of-the-art models for open-ended visual question answering, namely the MCB model [16], which is the winner of the VQA challenge in 2016, and the compositional model N2NMN [25].”
BackgroundMethodSelf-citationInformation Maximizing Visual Question Generation
Krishna, R., Bernstein, M., & Fei-Fei, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00211 · Cited by 84
“Successive models have improved performance by stacking attention [62, 37], modularizing components [1, 26, 21], adding relation networks [49], augmenting memory [59], and adding proxy tasks [13, 57].”
Progressive Attention Memory Network for Movie Story Question Answering
Kim, J., Ma, M., Kim, K., Kim, S.-J., & Yoo, C. D. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00853 · Cited by 99
“Fusing multi-modal data by bilinear operations [4, 7, 14] often requires heavy computation or large number of parameters.”
BackgroundMethodInfluentialThe VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
Wang, P., Wu, Q., Shen, C., & van den Hengel, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.416 · Cited by 97
Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos
Huang, D.-A., Buch, S., Dery, L. M., Garg, A., Fei-Fei, L., & Niebles, J. C. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00623 · Cited by 106
“We follow prior work [12, 43] and compute accuracy as the ratio of phrases for which the grounded bounding box overlaps with the groundTable 2: Generalizability to unseen instructional video classes (RoboWatch).”
MethodExplicit Bias Discovery in Visual Question Answering Models
Manjunatha, V., Saini, N., & Davis, L. S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00979 · Cited by 87
“Many models have attempted to solve the problem of VQA with varying degrees of success, but among them, the vast majority still attempt to solve the VQA task by exploiting biases in the dataset [25, 37, 2, 17, 7, etc], while a smaller minority address the individual problem types [4, 38, 11, etc].”
BackgroundLu, P., Li, H., Zhang, W., Wang, J., & Wang, X. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12240 · Cited by 96
Boosting Visual Question Answering with Context-aware Knowledge Aggregation
Li, G., Wang, X. Y., & Zhu, W. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413943 · Cited by 115
“For the conventional visual questions, the widely adopted solution is jointly embedding the visual and question features into a common space using advanced attention mechanism [1, 15] or multimodal fusion techniques [3, 6, 12], then feeding them into a classifier over candidate answers.”
MethodDifferential Attention for Visual Question Answering
Patro, B. N., & Namboodiri, V. P. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00801 · Cited by 91
Answer Them All! Toward Universal Visual Question Answering Models
Shrestha, R., Kafle, K., & Kanan, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01072 · Cited by 83
“Spatial attention mechanisms [5, 32, 37, 13, 9] are one of the most widely used methods for natural language VQA.”
MethodMulti-Modality Latent Interaction Network for Visual Question Answering
Peng, G., You, H., Zhang, Z., Wang, X., & Li, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00592 · Cited by 65
“Previous approaches solve the Visual Question Answering (VQA) by designing better features [25, 44, 13, 17, 1], better bilinear fusion approaches [10, 7, 22, 3, 52] or better attention mechanisms [48, 29, 49, 45, 36].”
MethodMultimedia Intelligence: When Multimedia Meets Artificial Intelligence
Zhu, W., Wang, X. Y., & Gao, W. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2969791 · Cited by 95
“Multimodal Compact Bilinear Pooling, figure from [5].”
Visual Grounding with Transformers
Du, Y., Fu, Z., Liu, Q., & Wang, Y. (2022) · 2022 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859880 · Cited by 71
Lossy Intermediate Deep Learning Feature Compression and Evaluation
Chen, Z., Fan, K., Wang, S., Duan, L., Lin, W., & Kot, A. C. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350849 · Cited by 96
“0%) captioning [15], QA [12] pool4 / tracking [37]”
High-Order Attention Models for Visual Question Answering
Schwartz, I., Schwing, A. G., & Hazan, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.04323 · Cited by 69
“[7] via Multimodal Compact Bilinear Pooling (MCB) in the pairwise setting or Multimodal Compact Trilinear Pooling (MCT), an extension of MCB that pools data from three modalities.”
BackgroundMethodResultInfluentialLearning Cross-Modal Context Graph for Visual Grounding
Liu, Y., Wan, B., Zhu, X., & He, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6833 · Cited by 99
Reducing Language Biases in Visual Question Answering with Visually-Grounded Question Encoder
Gouthaman, K. V., & Mittal, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58601-0_2 · Cited by 103
A Real-Time Global Inference Network for One-Stage Referring Expression Comprehension
Zhou, Y., Ji, R., Luo, G., Sun, X., Su, J., Ding, X., Lin, C., & Tian, Q. C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3090426 · Cited by 103
Modality Shifting Attention Network for Multi-Modal Video Question Answering
Kim, J., Ma, M., Pham, T. X., Kim, K., & Yoo, C. D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01012 · Cited by 84
Rich Visual Knowledge-Based Augmentation Network for Visual Question Answering
Zhang, L., Liu, S., Liu, D., Zeng, P., Li, X., Song, J., & Gao, L. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3017530 · Cited by 92
Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
Kuang, J., Shen, Y., Xie, J., Luo, H., Xu, Z., Li, R., Li, Y., Cheng, X., Lin, X., & Han, Y. (2025) · ACM Computing Surveys · doi:10.1145/3711680 · Cited by 177
Loss Re-Scaling VQA: Revisiting the Language Prior Problem From a Class-Imbalance View
Guo, Y., Nie, L., Cheng, Z., Tian, Q., & Zhang, M. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3128322 · Cited by 101
(2019) · neural information processing systems · Cited by 83
An enhanced SSD with feature fusion and visual reasoning for object detection
Leng, J., & Liu, Y. (2018) · Neural Computing and Applications · doi:10.1007/s00521-018-3486-1 · Cited by 83
Visual question answering: a state-of-the-art review
Manmadhan, S., & Kovoor, B. C. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09832-7 · Cited by 95
“Fukui et al. (2016) proposed an end-to-end model with a Multimodal Compact Bilin-ear Pooling (MCB) layer (see Fig.”
MethodInfluentialLoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval
Tan, R., Xu, H., Saenko, K., & Plummer, B. A. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00213 · Cited by 107
VQAMix: Conditional Triplet Mixup for Medical Visual Question Answering
Gong, H., Chen, G., Mao, M., Li, Z., & Li, G. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3185008 · Cited by 100
“The representative cross-modal reasoning modal includes SAN [35], BAN [24], MCB [23], etc. .”
BackgroundMethodInfluentialInformation fusion in visual question answering: A Survey
Zhang, D., Cao, R., & Wu, S. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.03.005 · Cited by 87
Li, Y., Wang, H., & Luo, Y. (2020) · 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm49941.2020.9313289 · Cited by 81
A Question-Centric Model for Visual Question Answering in Medical Imaging
Vu, M. H., Löfstedt, T., Nyholm, T., & Sznitman, R. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.2978284 · Cited by 81
“The features are combined using a multi-glimpse attention mechanism [9] and a novel fusion scheme for global image features and global question features.”
BackgroundMethodResultInfluentialHierarchical deep multi-modal network for medical visual question answering
Gupta, D., Suman, S., & Ekbal, A. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113993 · Cited by 89
Multimodal Learning For Hateful Memes Detection
Zhou, Y., Chen, Z., & Yang, H. (2021) · IEEE International Conference on Multimedia & Expo Workshops (ICMEW) · doi:10.1109/icmew53276.2021.9455994 · Cited by 112
Bilinear Graph Networks for Visual Question Answering
Guo, D., Xu, C., & Tao, D. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3104937 · Cited by 99
“MCB [3] is approaching this process by calculating the count sketch of two features and convolving them in faster Fourier transform (FFT) space.”
BackgroundMethodInfluentialDeep Multi-Modality Adversarial Networks for Unsupervised Domain Adaptation
Ma, X., Zhang, T., & Xu, C. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2902100 · Cited by 83
“sive studied in addressing visual question answering task [8], [55].”
BackgroundToward Visual Grounding: A Survey
Xiao, L., Yang, X., Lan, X., Wang, Y., & Xu, C. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3630635 · Cited by 115
Attentive Explanations: Justifying Decisions and Pointing to the Evidence
Park, D. H., Hendricks, L. A., Akata, Z., Schiele, B., Darrell, T. J., & Rohrbach, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.04757 · Cited by 51
“Similar to [15]”
BackgroundMethodInfluentialSelf-citationPathVQA: 30000+ Questions for Medical Visual Question Answering
He, X., Zhang, Y., Mou, L., Xing, E., & Xie, P. (2020) · arXiv · doi:10.48550/arxiv.2003.10286 · Cited by 79
Cross-Domain Image Retrieval with Attention Modeling
Ji, X., Wang, W., Zhang, M., & Yang, Y. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123429 · Cited by 88
“, via a multi-layer perceptron (MLP) [2], inner-product [23], outer-product [9], etc.”
Question-Guided Hybrid Convolution for Visual Question Answering
Gao, P., Li, H., Li, S., Lu, P., Li, Y., Hoi, S. C. H., & Wang, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_29 · Cited by 86
Gao, Z., Lu, Z., Wang, J., Ying, S., & Shi, J. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3153671 · Cited by 89
Two Can Play This Game: Visual Dialog with Discriminative Question Generation and Answering
Jain, U., Schwing, A., & Lazebnik, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00603 · Cited by 76
R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answering
Lu, P., Ji, L., Zhang, W., Duan, N., Zhou, M., & Wang, J. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3220036 · Cited by 95
“es perform attention mechanism to improve model capacity. Current attention methods [36, 38] for VQA mainly perform visual attention to learn image regions relevant to the question. Some recent works [8, 9, 13, 14] integrate effective multimodal feature embedding with visual attention to further improve VQA performanc…”
BackgroundMethodInfluentialImage-Question-Answer Synergistic Network for Visual Dialog
Guo, D., Xu, C., & Tao, D. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01068 · Cited by 79
“However, MCB needs to sam- ple features, which is computationally intractable and has very large projected dimension, so low-rank bilinear pooling (MLB) [15] and multi-factor bilinear pooling (MFB) [32] have been proposed to project the two features into a common low-rank space, and the bilinear attention network (BAN…”
MethodCompact Trilinear Interaction for Visual Question Answering
Do, T., Tran, H., Do, T.-T., Tjiputra, E., & Tran, Q. D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00048 · Cited by 78
“For MC VQA, we compare with the state-of-the-art methods on Visual7W dataset, including BAN2 [18], SAN [43], MLP [16], MCB [8], STL [41], and fPMC [14]).”
BackgroundMethodInfluentialLEAF-QA: Locate, Encode & Attend for Figure Question Answering
Chaudhry, R., Shekhar, S., Gupta, U., Maneriker, P., Bansal, P., & Joshi, A. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093269 · Cited by 132
“Correspondingly, methods using attention [29, 30, 15], neural modules [1] and compositional modeling [6] have been explored.”
MethodMultimodal Dual Attention Memory for Video Story Question Answering
Kim, K.-M., Choi, S.-H., Kim, J.-H., & Zhang, B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01267-0_41 · Cited by 83
Relationship-Embedded Representation Learning for Grounding Referring Expressions
Yang, S., Li, G., & Yu, Y. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2973983 · Cited by 87
“The study of the two problems can be integrated and learned from each other [42], [53].”
BackgroundMethodInfluentialMIntRec: A New Dataset for Multimodal Intent Recognition
Zhang, H., Xu, H., Wang, X. Y., Zhou, Q., Zhao, S., & Teng, J. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547906 · Cited by 136
“Traditional methods, such as MCB [14] and TFN [55] obtained representations by learning intra-modality and inter-modality relations.”
MethodSee the Sound, Hear the Pixels
Ramaswamy, J., & Das, S. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093616 · Cited by 75
“AVFB uses a hybrid of neural network (LSTM) [25, 26, 32, 33, 49] and multi-modal bilinear pooling [7, 10, 22, 41, 42, 50] to fuse the feature representations obtained after extraction using pre-trained CNNs from the two modalities (that is, from audio of single-channel and video which are broken…”
MethodImproving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation
Wang, L., Huang, J. L., Li, Y., Xu, K., Yang, Z., & Yu, D. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01387 · Cited by 104
“Building on these datasets, recent approaches learn a similarity function between regions and phrases by using the ground-truth region-phrase pairs [38, 11].”
BackgroundMethodSarkar, A., Chowdhury, T., Murphy, R. R., Gangopadhyay, A., & Rahnemoonfar, M. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3276293 · Cited by 93
“Attention-based VQA models [1], [2], [7], [15], [16] showed remarkable performance on many ground imagery-based VQA datasets.”
BackgroundStructured Matching for Phrase Localization
Wang, M., Azab, M., Kojima, N., Mihalcea, R. F., & Deng, J. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46484-8_42 · Cited by 86
From Pixels to Objects: Cubic Visual Attention for Visual Question Answering
Song, J., Zeng, P., Gao, L., & Shen, H. T. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/126 · Cited by 77
An Improved Attention for Visual Question Answering
Rahman, T., Chou, S.-H., Sigal, L., & Carenini, G. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00181 · Cited by 75
An End-to-End Multimodal Voice Activity Detection Using WaveNet Encoder and Residual Networks
Ariav, I., & Cohen, I. (2019) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2901195 · Cited by 81
“Here, we adopt ideas from [50] to the multimodal case for audio and visual modalities, and use MCB to fuse the two embeddings Za and Zv .”
BackgroundMethodShapeWorld - A new test methodology for multimodal language understanding
Kuhnle, A., & Copestake, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.04517 · Cited by 44
“…VQA setup (i.e., yes/no questions), it neither requires the evaluated model to generate answers nor to rephrase the problem to fit it into a classification task of some sort – for instance, over the 1000/3000 most common answers, as is common practice recently (Lu et al., 2016; Fukui et al., 2016).”
BackgroundMethodFaithful Multimodal Explanation for Visual Question Answering
Wu, J., & Mooney, R. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4812 · Cited by 73
“VQS pretraining.”
MethodInfluentialStructured Two-Stream Attention Network for Video Question Answering
Gao, L., Zeng, P., Song, J., Li, Y.-F., Liu, W., Mei, T., & Shen, H. T. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016391 · Cited by 81
“Existing methods can be divided into two categories: 1) image-based approaches including VIT+LSTM (Ren, Kiros, and Zemel 2015a) and VQA-MCB (Fukui et al. 2016); and 2) video-based approaches including ST (Jang et al. 2017) and Co-memory (Gao et al. 2018a).”
MethodMaking History Matter: History-Advantage Sequence Training for Visual Dialog
Yang, T., Zha, Z.-J., & Zhang, H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00265 · Cited by 88
“A number of approaches[39, 40, 5, 11, 33] have been proposed that apply question-based attention module on image features.”
BackgroundLearning Visual Context for Group Activity Recognition
Yuan, H., & Ni, D. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16437 · Cited by 90
“2019) and visual question answering (Fukui et al. 2016; Kim et al. 2016; Yu et al. 2018b).”
MethodMulti-Level Multimodal Common Semantic Space for Image-Phrase Grounding
Akbari, H., Karaman, S., Bhargava, S., Chen, B., Vondrick, C., & Chang, S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01276 · Cited by 96
“This ability to link text and image content is a key component of many visual semantic tasks such as image captioning [10, 21, 18], visual question answering [2, 30, 48, 52, 11], text-based image retrieval [12, 40], and robotic navigation [44].”
BackgroundMethodRadial Graph Convolutional Network for Visual Question Generation
Xu, X., Wang, T., Yang, Y., Hanjalić, A., & Shen, H. T. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2986029 · Cited by 84
“Besides, some other studies adopt external knowledge bases [31] and better visual-linguistic information fusion schemes [32]–[34] in the encoder–decoder framework to explore more informative relevance between images and questions.”
BackgroundExploring Diverse Methods in Visual Question Answering
Li, P., Yang, Q., Geng, X., Zhou, W., Ding, Z., & Nian, Y. (2024) · International Conference on Electronic Communication and Artificial Intelligence (ICECAI) · doi:10.1109/icecai62591.2024.10674838 · Cited by 91
“Most existing approaches (such as MCB [14]) utilize a fixed method to embed the question and image features together.”
MethodTraining Recurrent Answering Units with Joint Loss Minimization for VQA
Noh, H., & Han, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.03647 · Cited by 54
MSAF: Multimodal Split Attention FusionPreprint
Su, L., Hu, C., Li, G., & Cao, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.07175 · Cited by 91
Metadata
- Authors
- Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M.
- Affiliations
- Sony (Japan) (JP)
- University of California, Berkeley (US)
- Max Planck Institute for Informatics (DE)
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 457-468
- OpenAlex
- W2963383024
- Cited by
- 1,582
- References
- 60
- Citations by year
- 2026: 42 · 2025: 105 · 2024: 126 · 2023: 165 · 2022: 189 · 2021: 222 · 2020: 225 · 2019: 201 · 2018: 179 · 2017: 105 · 2016: 20
- FWCI
- 58.39
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Max Planck Digital Library (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1427425, 1212798, N000141110688)
- Defense Advanced Research Projects Agency
- Multidisciplinary University Research Initiative (N000141110688)
- Air Force Research Laboratory
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- visual question answering, visual grounding, multimodal fusion, outer product, attention mechanism, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.