Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years.However, tasks such as visual question answering require combining these vector representations with each other.Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations.We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors.As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features.We extensively evaluate MCB on the visual question answering and grounding tasks.We consistently show the benefit of MCB over ablations without MCB.For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation.This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, the PJ-X backbone MCB [16] (which is crucial for the answering performance) preceded the publication of the CLEVR dataset.”
From CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations (2022) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We replace the concatenation of the visual representation and the encoded phrase in Grounder with MCB to combine both modalities.”
MethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“, 2015) that is pretrained on ImageNet data (Deng et al., 2009).”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Also, for this model, we concatenated the learned word embedding with pretrained GloVe vectors (Pennington et al., 2014).”
MethodGirshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Research works printed but not linked: 3
- Ross Girshick. 2015. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
- Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014. Multimodal neural language mod- els. In Proceedings of the International Conference on Machine Learning (ICML), pages 595-603.
- Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems (NIPS).
Cited by
Items 101–200 of 1,002
Motion-Appearance Co-memory Networks for Video Question Answering
Gao, J., Ge, R., Chen, K., & Nevatia, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00688 · Cited by 250
Audiovisual SlowFast Networks for Video Recognition
Xiao, F., Lee, Y. J., Grauman, K., Malik, J., & Feichtenhofer, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.08740 · Cited by 146
“Beyond audio and visual modalities, extensive research has been conducted in various other instantiations of multi-modal learning, including vision and motion [62, 16, 77, 9], vision and language [4, 18], and learning from physiological data [44].”
BackgroundA Joint Speaker-Listener-Reinforcer Model for Referring Expressions
Yu, L., Tan, H., Bansal, M., & Berg, T. L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.375 · Cited by 242
“Besides concatenation, the elementwise product or compact bilinear pooling can also be applied [6].”
MethodKembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., & Hajishirzi, H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.571 · Cited by 244
“The winner of the recent VQA workshop employed Multimodal Compact Bilinear Pooling [8] at the attention layer instead of the commonly used element wise product/concatenation mechanisms.”
MethodLearning a Text-Video Embedding from Incomplete and Heterogeneous Data
Miech, A., Ivan Laptev, A., & Šivic, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.02516 · Cited by 161
“Another related application is visual question answering [1], [2], [28], [29].”
BackgroundRevisiting Visual Question Answering Baselines
Jabri, A., Joulin, A., & van der Maaten, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_44 · Cited by 218
Multi-level Attention Networks for Visual Question Answering
Yu, D., Fu, J., Mei, T., & Rui, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.446 · Cited by 230
“[9] achieve a comparable performance with ours when they add glove tricks and additional training data.”
BackgroundA Survey of Evaluation Metrics Used for NLG Systems
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2023) · ACM Computing Surveys · doi:10.1145/3485766 · Cited by 235
“The second method is to concatenate the image and reference first, and then combine it with the candidate caption using ‘Compact Bilinear Pooling’ [47], which has been shown to effectively combine the heterogeneous information of image and text [45].”
MethodIterative Answer Prediction With Pointer-Augmented Multimodal Transformers for TextVQA
Hu, R., Singh, A., Darrell, T., & Rohrbach, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01001 · Cited by 217
Self-citationMulti-modal Sensor Fusion for Auto Driving Perception: A SurveyPreprint
Huang, K., Shi, B., Li, X., Li, X., Huang, S., & Li, Y. (2022) · arXiv (Cornell University) · doi:10.4230/lipics.ecrts.2025.6 · Cited by 355
Just Ask: Learning to Answer Questions from Millions of Narrated Videos
Yang, A., Miech, A., Sivic, J., Laptev, I., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00171 · Cited by 219
End-to-End Concept Word Detection for Video Captioning, Retrieval, and Question Answering
Yu, Y., Ko, H., Choi, J., & Kim, G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.347 · Cited by 219
“encoding. To measure a similarity score S k;l between a movie k las follows (see Figure3(c)): p = (W p)T(s N;q N) (25) f = maxout(p) (26) S k;l= (W s) Tf (27) The function notation means MCB layer [2], which captures the interactions between different modalities better than simple concatenation. That is, we learn the…”
BackgroundMethodInfluentialMedical visual question answering: A survey
Lin, Z., Zhang, D., Tao, Q., Shi, D., Haffari, G., Wu, Q., He, M., & Ge, Z. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102611 · Cited by 177
“Multi-modal Compact Bilinear (MCB) pooling [27] is a typical method for pooling.”
MethodOvercoming Data Limitation in Medical Visual Question Answering
Nguyen, B. D., Do, T.-T., Nguyen, B. X., Do, T., Tjiputra, E., & Tran, Q. D. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32251-9_57 · Cited by 195
Deep Learning Approaches on Image Captioning: A Review
Ghandi, T., Pourreza, H., & Mahyar, H. (2024) · ACM Computing Surveys · doi:10.1145/3617592 · Cited by 194
“…complexities, attracting a significant amount of work with extensive applications across various domains such as human-computer interaction [ 32 , 71 , 143 ], medical image captioning and prescription [ 9 , 58 , 96 ], traffic data analysis [ 69 ], quality control in industry [ 83 ], and…”
BackgroundPubMedCLIP: How Much Does CLIP Benefit Visual Question Answering in the Medical Domain?
Eslami, S., Meinel, C., & de Melo, G. (2023) · Findings of the Association for Computational Linguistics: EACL 2023 · doi:10.18653/v1/2023.findings-eacl.88 · Cited by 167
“Bilinear attention networks (Kim et al., 2018), multimodal compact bilinear pooling (Fukui et al., 2016), stacked attention networks (Yang et al., 2016), and element-wise production are popular as multimodal pooling approaches in MedVQA.”
MethodExplainable Neural Computation via Stack Neural Module Networks
Hu, R., Andreas, J., Darrell, T., & Saenko, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_4 · Cited by 216
Self-citationMueller, S. T., Hoffman, R. R., Clancey, W., Emrey, A., & Klein, G. (2019) · arXiv · doi:10.48550/arxiv.1902.01876 · Cited by 215
“…Doshi-Velez and Kim, 2017; Elenberg, Dimakis, Feldman, and Karbasi, 2017; Erhan, Bengio, Courville, and Vincent, 2009; Féraud and Clérot, 2002; Fukui et al., 2016; Gan, Wang, Yang, Yeung, and Hauptmann, 2015; Goyal, et al., 2016a,b,c; Hendricks et al., 2016; Hinton, et al., 2015; Jain et al.,…”
KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA
Marino, K., Chen, X., Parikh, D., Gupta, A., & Rohrbach, M. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01389 · Cited by 185
“[24, 33]) or self-attention ( e.g .”
Self-citationPythia v0.1: the Winning Entry to the VQA Challenge 2018
Jiang, Y. L., Natarajan, V., Chen, X., Rohrbach, M., Batra, D., & Parikh, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.09956 · Cited by 150
“We follow the same procedure as [4] to extract grid-level features from ResNet152 [7].”
MethodSelf-citationA Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204
“[116], [117], which lacks exploiting the multilinear fusion power [118].”
BackgroundVersatile Diffusion: Text, Images and Variations All in One Diffusion Model
Xu, X., Wang, Z., Zhang, E., Wang, K., & Shi, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00713 · Cited by 339
“Another topic, VQA [2, 20], conducted cross-modal reasoning that transferred visual concepts into linguistic answers.”
BackgroundTransparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
Mascharka, D., Tran, P., Soklaski, R., & Majumdar, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00519 · Cited by 203
“While a wide variety of models have recently been proposed for the VQA task [6, 10, 19, 21, 28, 30], neural module networks [2, 3, 10, 14] are among the most intuitive.”
BackgroundVisual Grounding via Accumulated Attention
Deng, C., Wu, Q., Wu, Q., Hu, F., Lyu, F., & Tan, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00808 · Cited by 190
“Recently, the attention mechanism has been widely used in many applications such as Natural Language Processing [2, 23], Image Captioning [29, 30] and VQA [4, 14, 16, 24] due to its effectiveness in dealing with information redundancies.”
BackgroundMethodInfluentialControllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network
Wang, B., Ma, L., Zhang, W., Jiang, W., Wang, J., & Liu, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00273 · Cited by 199
“To further demonstrate the effectiveness of the proposed fusion strategy, we also compare the gated fusion network with more complex fusion algorithms, such as MCB [13], Model B@4 M R C”
MethodA Crossmodal Multiscale Fusion Network for Semantic Segmentation of Remote Sensing Data
Ma, X., Zhang, X., & Pun, M.-O. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3165005 · Cited by 179
“…multimodal fusion techniques have been developed in the literature, such as feature extraction, feature alignment, and feature fusion for various applications such as visual question answering [22]–[24], sentiment analysis [25], medical image processing [26], [27], and remote sensing [21], [28].”
BackgroundDumoulin, V., Perez, E., Schucher, N., Strub, F., Vries, H., Courville, A., & Bengio, Y. (2018) · Distill · doi:10.23915/distill.00011 · Cited by 160
MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare
Choi, E., Xiao, C., Stewart, W. F., & Sun, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.09593 · Cited by 157
“(3) was inspired by recent developments in bilinear pooling technique [37, 21, 19, 24], which we discuss in more detail in Appendix A.”
BackgroundMethodKhoo, L. S., McNaney, R., Lim, M. K., & Chong, C. Y. (2024) · Sensors · doi:10.3390/s24020348 · Cited by 242
Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering
Narasimhan, M., Lazebnik, S., & Schwing, A. G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00538 · Cited by 61
“The initial VQA models [1,2,4,11,16,17,19,24,32,33,35,41,44,53,57,59,60,65,68] combined the LSTM encoding of the question and the CNN encoding of the image using a deep network which finally predicted the answer.”
BackgroundMethodGoal-Oriented Gaze Estimation for Zero-Shot Learning
Liu, Y., Zhou, L., Bai, X., Huang, Y., Gu, L., Zhou, J., & Harada, T. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00379 · Cited by 171
Jaume, G., Vaidya, A., Chen, R. J., Williamson, D. F. K., Liang, P. P., & Mahmood, F. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01100 · Cited by 150
“Multimodal baselines: (a) Late fusion: We combine ABMIL [30], AMISL [85], and TransMIL [59] with an S-MLP using concatenation [48], denoted as ABMIL (Cat) , AMISL (Cat) , and TransMIL (Cat) , and Kronecker product [9, 21, 78, 86], denoted as ABMIL (KP) , AMISL (KP) , and TransMIL (KP) .”
MethodEmbraceNet: A robust deep learning architecture for multimodal classification
Choi, J.-H., & Lee, J.-S. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.02.010 · Cited by 156
“In addition, a L2 normalization is applied to each modality before calculating its count sketch, as in the previous work [39, 53].”
MethodRSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data
Zhan, Y., Xiong, Z., & Yuan, Y. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3250471 · Cited by 179
“Multimodal compact bilinear (MCB) pooling is first proposed in [33] to fuse the multimodal features.”
BackgroundEfficient Large-Scale Multi-Modal Classification
Kiela, D., Grave, E., Joulin, A., & Mikolov, T. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11945 · Cited by 177
Visual Question Generation as Dual Task of Visual Question Answering
Li, Y., Duan, N., Zhou, B., Chu, X., Ouyang, W., Wang, X., & Zhou, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00640 · Cited by 170
Talk2Car: Taking Control of Your Self-Driving Car
Deruyttere, T., Vandenhende, S., Grujicic, D., Van Gool, L., & Moens, M.-F. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1215 · Cited by 106
“Early work (Kafle and Kanan, 2016; Zhou et al., 2015; Fukui et al., 2016) tried to solve the task by fusing image features extracted by a convolutional neural network (CNN), together with an encoding of the question.”
BackgroundMatching Images and Text with Multi-modal Tensor Fusion and Re-ranking
Wang, T., Xu, X., Yang, Y., Hanjalic, A., Shen, H. T., & Song, J. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3350875 · Cited by 161
“lassification-basedmethodandourMTFNembeddingonthepartofMSCOCO test set (8000 image-text pairs) with the learned linear SVM boundary. on different fusion schemes. Following the experimental setting in [7], each combination of model has similar amount of model parameters by combining with multiple fully connected layers…”
BackgroundHYDRO-3D: Hybrid Object Detection and Tracking for Cooperative Perception Using 3D LiDAR
Meng, Z., Xia, X., Xu, R., Liu, W., & Ma, J. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3282567 · Cited by 166
“Remark 2: It should be noted that the concatenation method used in this work is not unique and can be replaced by other dedi-catedfusionmethodssuchasself-attention[63]andmulti-model bilinearpooling[64],aslongastheyarecapabletofusethe differentorheterogeneousfeaturesforobjectdetection.”
MethodCounting Everyday Objects in Everyday Scenes
Chattopadhyay, P., Vedantam, R., Selvaraju, R. R., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.471 · Cited by 151
“For [21], there were 11 non-numerical answers and for [15] there were 3 (e.”
BackgroundMethodInfluentialPhrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cues
Plummer, B. A., Mallya, A., Cervantes, C. M., Hockenmaier, J., & Lazebnik, S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.213 · Cited by 161
“Today’s deep features can give reliable signals about a broad range of content in natural images, leading to advances in image-language tasks such as automatic captioning [6, 14, 16, 17, 42] and visual question answering [1, 8, 44].”
BackgroundMethodInfluentialZero-Shot Grounding of Objects From Natural Language Queries
Sadhu, A., Chen, K., & Nevatia, R. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00479 · Cited by 191
“Multi-modal feature representation has many flavors like linear transformation, concatenation, hadamard product [21], bilinear pooling [28] and have shown success in vision-language tasks like VQA [3, 4, 13, 47, 48], Scene Graph Generations [25,43] and Image Captioning [30].”
MethodDid the Model Understand the Question?
Mudrakarta, P. K., Taly, A., Sundararajan, M., & Dhamdhere, K. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1176 · Cited by 166
“1 % accuracy on the validation set (the state of the art (Fukui et al., 2016) achieves 66 .”
BackgroundDeepStory: Video Story QA by Deep Embedded Memory Networks
Kim, K.-M., Heo, M.-O., Choi, S.-H., & Zhang, B.-T. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/280 · Cited by 145
Language-Conditioned Graph Networks for Relational Reasoning
Hu, R., Rohrbach, A., Darrell, T., & Saenko, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01039 · Cited by 160
“[8, 29]) to pooling over pair-wise relationships ( e.g .”
BackgroundSelf-citationMedical Visual Question Answering via Conditional Reasoning
Zhan, L.-M., Liu, B., Fan, L., Chen, J., & Wu, X.-M. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413761 · Cited by 154
“SAN [23, 43] (baseline) MCB [10, 23] (baseline) BAN [20, 30] (baseline) MEVF+SAN [30] (baseline) MEVF+BAN [”
BackgroundMethodInfluentialINCLUDE: A Large Scale Dataset for Indian Sign Language Recognition
Sridhar, A., Ganesan, R. G., Kumar, P., & Khapra, M. M. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413528 · Cited by 227
“This has led to a continued focus on using deep learning for tasks on video data such as video classification [30], object detection [42, 53], object tracking [62], action recognition [9], visual question answering [20] and video encoding [43].”
BackgroundVisual Coreference Resolution in Visual Dialog Using Neural Module Networks
Kottur, S., Moura, J. M. F., Parikh, D., Batra, D., & Rohrbach, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01267-0_10 · Cited by 160
Self-citationCLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical ReasoningPreprint
Lindström, A. D., & Abraham, S. S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.05358 · Cited by 186
Vision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155
Comprehension-Guided Referring Expressions
Luo, R., & Shakhnarovich, G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.333 · Cited by 144
“Later, [23] achieves a slight improvement by replacing the concatenation layer with a compact bilinear pooling layer.”
BackgroundAgarwal, V., Shetty, R., & Fritz, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00971 · Cited by 142
Chen, H., Zhang, H., Chen, P., Yi, J., & Hsieh, C. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1241 · Cited by 183
“…Bansal, 2017; Venu-gopalan et al., 2015), visual dialog (Das et al., 2017; De Vries et al., 2017), visual question answering (Antol et al., 2015; Fukui et al., 2016; Lu et al., 2016; Zhu et al., 2017), visual story-telling (Huang et al., 2016), natural question generation (Mostafazadeh et al.,…”
BackgroundJoint embedding VQA model based on dynamic word vector
Ma, Z., Zheng, W., Chen, X., & Yin, L. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.353 · Cited by 155
“As shown in the Table 1, the existing joint embedding models (Anderson et al., 2018; Antol et al., 2015; Fukui et al., 2016; Ma, Lu & Li, 2015; Noh, Hongsuck Seo & Han, 2016; Teney et al., 2018; Yu et al., 2019; Zhou et al., 2015) use different combinations of image characterization, text…”
MethodLearning to Evaluate Image Captioning
Cui, Y., Yang, G., Veit, A., Huang, X., & Belongie, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00608 · Cited by 135
On the diversity of multi-head attention
Li, J., Wang, X., Tu, Z., & Lyu, M. R.-T. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.04.038 · Cited by 165
“For example, Fukui et al. [16] and Ben et al. [17] succeed on fusing multi-modal features (e.g., visual features and textual features) more effectively via employing the higher-order bilinear pooling instead of vector concatenation or element-wise operations.”
BackgroundMethodAMFB: Attention based multimodal Factorized Bilinear Pooling for multimodal Fake News Detection
Kumari, R., & Ekbal, A. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115412 · Cited by 166
Toward Intelligent Sensing: Intermediate Deep Feature Compression
Chen, Z., Fan, K., Wang, S., Duan, L., Lin, W., & Kot, A. C. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2941660 · Cited by 168
“object detection [5], [6], [44], [45], visual retrieval [8], [46], image QA [47], [48].”
BackgroundMethodBiomedical Question Answering: A Survey of Approaches and Challenges
Jin, Q., Yuan, Z., Xiong, G., Yu, Q., Ying, H., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2023) · ACM Computing Surveys · doi:10.1145/3490238 · Cited by 118
“Multi-modal compact bilinear pooling [56], multi-modal factorized bilinear pooling [221], and multi-modal factorized high-order pooling [222] are often used for feature fusion in VQA.”
MethodTrends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
Mogadala, A., Kalimuthu, M., & Klakow, D. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11688 · Cited by 47
“961 MCB (Fukui et al., 2016) - 26.”
MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition
Chen, J., & Ho, C. M. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00086 · Cited by 176
Rethinking Diversified and Discriminative Proposal Generation for Visual Grounding
Yu, Z., Yu, J., Xiang, C., Zhao, Z., Tian, Q., & Tao, D. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/155 · Cited by 126
Multiple Meta-model Quantifying for Medical Visual Question Answering
Do, T., Nguyen, B. X., Tjiputra, E., Tran, M. Q., Tran, Q. D., & Nguyen, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87240-3_7 · Cited by 186
Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text
Gao, D., Li, K., Wang, R., Shan, S., & Chen, X. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01276 · Cited by 124
Multimodal Categorization of Crisis Events in Social Media
Abavisani, M., Wu, L., Hu, S., Tetreault, J., & Jaimes, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01469 · Cited by 188
Learning to Contrast the Counterfactual Samples for Robust Visual Question Answering
Liang, Z., Jiang, W., Hu, H., & Zhu, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.265 · Cited by 175
“Although the current state-of-the-art methods (Fukui et al., 2016; Anderson et al., 2018; Cadene et al., 2019a) can achieve good results on the VQA benchmarks such as VQA v2 (Goyal et al., 2017), recent researches (Agrawal et al., 2016; Kafle and Kanan, 2017; Agrawal et al., 2018) have found that…”
MethodManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation
Li, X., Zhang, M., Geng, Y., Geng, H., Long, Y., Shen, Y. F., Zhang, R., Liu, J., & Dong, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01710 · Cited by 288
“Existing advancements in Multimodal Large Language Models (MLLMs)[1, 19, 22, 38] highlight their proficiency in common sense reasoning and remarkable generalization in vision tasks [2, 8].”
BackgroundCreativity: Generating Diverse Questions Using Variational Autoencoders
Jain, U., Zhang, Z., & Schwing, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.575 · Cited by 123
“When considering generation of text from images, caption and paragraph generation [32, 4, 11, 18, 19, 20, 13, 33, 37, 40, 48, 60, 64, 69], as well as visual question answering [2, 22, 47, 52, 57, 67, 68, 71, 21, 34, 74, 1, 15, 31, 72, 73, 66, 46, 45, 31] come to mind immediately.”
Self-Critical Reasoning for Robust Visual Question Answering
Wu, J., & Mooney, R. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09998 · Cited by 110
“The state-of-the-art VQA systems [8, 2, 1, 3, 12, 33, 25, 29, 14, 15, 21] achieve high performance when the training and test question-answer (QA) pairs are sampled from the same distribution.”
BackgroundTRAR: Routing the Attention Spans in Transformer for Visual Question Answering
Zhou, Y., Ren, T., Zhu, C., Sun, X., Liu, J., Ding, X., Xu, M., & Ji, R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00208 · Cited by 173
“The rapid development of VQA has been supported by the emergence of various benchmark datasets [1, 3, 14, 28, 29, 32, 52] and methods [13, 59, 66, 72, 54, 46, 77, 78, 79, 80].”
BackgroundDynamic Computational Time for Visual Attention
Li, Z., Yang, Y., Liu, X., Zhou, F., Wen, S., & Xu, W. (2017) · 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw.2017.145 · Cited by 130
Recursive Visual Attention in Visual Dialog
Niu, Y., Zhang, H., Zhang, M., Zhang, J., Lu, Z., & Wen, J.-R. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00684 · Cited by 150
“Different from one-round VQA, visual dialog is a multi-round conversation about an image.”
BackgroundInfluentialConvolutional Self-Attention Networks
Yang, B., Wang, L., Wong, D. F., Chao, L. S., & Tu, Z. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1407 · Cited by 128
“The vanilla SAN models linearly aggregate features from different heads, and this procedure limits the extent of abstraction (Fukui et al., 2016; Li et al., 2019).”
BackgroundLink the Head to the "Beak": Zero Shot Learning from Noisy Text Description at Part Precision
Elhoseiny, M., Zhu, Y., Zhang, H., & Elgammal, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.666 · Cited by 121
“, [35, 18]), which requires object-(text phrase) annotations during training and has been mainly studied at the object level/not at part-level.”
BackgroundTRIE: End-to-End Text Reading and Information Extraction for Document Understanding
Zhang, P., Xu, Y., Cheng, Z., Pu, S., Lu, J., Liang, Q., Niu, Y., & Wu, F. Q. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413900 · Cited by 149
“lation) are mined and processed from previously recognized plain text for diverse Natural Language Processing (NLP) tasks, such as Named Entity Recognition (NER) [24, 36, 48] and Question-Answer (QA) [1, 13, 57]. Note that in existing routines, the two stages are separately executed. It means that the former recognize…”
BackgroundConditional Image-Text Embedding Networks
Plummer, B. A., Kordas, P., Kiapour, M. H., Zheng, S., Piramuthu, R., & Lazebnik, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01258-8_16 · Cited by 137
Query-Guided Regression Network with Context Policy for Phrase Grounding
Chen, K., Kovvuri, R., & Nevatia, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.95 · Cited by 114
“It is an important building block in computer vision with natural language interaction, which can be utilized in high-level tasks, such as image retrieval [3, 26], image captioning [1, 7] and visual question answering [2, 4, 8].”
InfluentialPerception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning ModelsPreprint
Li, Y., Liu, Z., Li, Z., Zhang, X., Xu, Z., Chen, X., Shi, H., Jiang, S., Wang, X., Wang, J., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.04921 · Cited by 148
Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering
Mallya, A., & Lazebnik, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_25 · Cited by 109
Structured Attentions for Visual Question Answering
Zhu, C., Zhao, Y., Huang, S., Tu, K., & Ma, Y. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.145 · Cited by 121
“There have been many different directions for improving VQA performance, including predicting answer types [14], utilizing task-specific submodules [1, 2], and better multimodal feature pooling methods [8, 17].”
BackgroundMethodInfluentialMultimodal research in vision and language: A review of current and emerging trends
Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermann, R., & Zadeh, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.009 · Cited by 150
Cross-Modal Self-Attention with Multi-Task Pre-Training for Medical Visual Question Answering
Gong, H., Chen, G., Liu, S., Yu, Y., & Li, G. (2021) · Proceedings of the 2021 International Conference on Multimedia Retrieval · doi:10.1145/3460426.3463584 · Cited by 104
“With the prosperity of deep learning, VQA has received extensive attention in recent years and has made great progress, mainly benefiting from the strong bottom-up feature representation based on deep CNNs[4] and the cross-modal feature alignment and fusion techniques[4, 8, 13, 26, 28].”
BackgroundMethodMulti-modal cross-attention network for Alzheimer’s disease diagnosis with multi-modality data
Zhang, J., He, X., Liu, Y., Cai, Q., Chen, H., & Qing, L. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107050 · Cited by 117
Ye, J., Tian, J., Yan, M., Yang, X., Wang, X., Zhang, J., He, L., & Lin, X. A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01506 · Cited by 154
“Other methods [14, 34, 35] maintain the spatial structure to perform interaction but have a high computational cost.”
BackgroundMethodGan, C., Li, Y., Li, H., Sun, C., & Gong, B. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.201 · Cited by 116
“Dev Standard Two-layer LSTM [3] 62.7 63.1 Region selection [38] 62.4 62.4 DPPNet [32] 62.5 62.7 MCB [9] 65.4 − Co-Attention [26] 65.9 66.1 MRN [19] 66.3 66.3 MLB [20] − 68.9 MLP + ResNet [17] 67.”
InfluentialData Augmentation for Visual Question Answering
Kafle, K., Yousefhussien, M., & Kanan, C. (2017) · International Conference on Natural Language Generation · doi:10.18653/v1/w17-3529 · Cited by 139
“Next, to demonstrate that the data augmentation scheme also helps improve more complex models, we train the state-of-the-art MCB model with attention (MCB+Att.+GloVe) (Fukui et al., 2016) with the template augmentation and compare the accuracy when the same model trained only on the COCO-VQA dataset (Table 3).”
MethodInfluentialVisual Question Answering with Memory-Augmented Networks
Ma, C., Shen, C., Dick, A. R., Wu, Q., Wang, P., van den Hengel, A., & Reid, I. D. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00729 · Cited by 135
Referring Expression Comprehension: A Survey of Methods and Datasets
Qiao, Y., Deng, C., & Wu, Q. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3042066 · Cited by 167
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
Gan, Z., Cheng, Y., Kholy, A., Li, L., Liu, J., & Gao, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1648 · Cited by 109
“The whole process above is denoted in short as: There are also other methods for multimodal fusion, such as MCB [14] and MLB [23].”
BackgroundMethodAsk Your Neurons: A Deep Learning Approach to Visual Question Answering
Malinowski, M., Rohrbach, M., & Fritz, M. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1038-2 · Cited by 105
Self-citationKumar, G. K., & Nandakumar, K. (2022) · Workshop on NLP for Positive Impact (NLP4PI) · doi:10.18653/v1/2022.nlp4pi-1.20 · Cited by 147
Schwartz, I., Yu, S., Hazan, T., & Schwing, A. G. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00214 · Cited by 106
“While attention mechanisms have been applied to visual question answering [14, 33, 42, 26, 55, 58], few works have addressed visual dialog because of the many different data utilities.”
BackgroundMethodVQA-Med: Overview of the Medical Visual Question Answering Task at ImageCLEF 2019
Abacha, A. B., Hasan, S. A., Datla, V. V., Liu, J., Demner‐Fushman, D., & Müller, H. (2024) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.10499038 · Cited by 106
Cephalometric landmark detection in dental x-ray images using convolutional neural networks
Lee, H., Park, M., & Kim, J. (2017) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2255870 · Cited by 122
Deep spatio-temporal feature fusion with compact bilinear pooling for multimodal emotion recognition
Nguyen, D., Nguyen, K., Sridharan, S., Dean, D., & Fookes, C. (2018) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2018.06.005 · Cited by 104
Deep Bilinear Learning for RGB-D Action Recognition
Hu, J., Zheng, W., Pan, J., Lai, J., & Zhang, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_21 · Cited by 110
“Method max mean Linear SVM FCN MCB [9] MKL Ours”
BackgroundMethodInfluentialJiang, J., Chen, Z., Lin, H., Zhao, X., & Gao, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6766 · Cited by 108
“For Image QA, there have been extensive efforts concentrated on (Yang et al. 2016; Xu and Saenko 2016; Wu et al. 2016; Kazemi and Elqursh 2017; Anderson et al. 2018; Gao et al. 2016; Fukui et al. 2016; Ben-Younes et al. 2017) in the last decade.”
Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answering
Narasimhan, M., & Schwing, A. G. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01237-3_28 · Cited by 131
Deep Multimodal Neural Architecture Search
Zhou, Y., Cui, Y., Yu, J., Wang, M., Tao, D., & Tian, Q. C. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413977 · Cited by 122
“, visual captioning [1, 51], visual grounding [43, 53], image-text matching [26, 36], and visual question answering (VQA) [14, 57].”
BackgroundSharma, D., Purushotham, S., & Reddy, C. K. A. (2021) · Scientific Reports · doi:10.1038/s41598-021-98390-1 · Cited by 133
“The size of the combined feature vector is set to be 16, 000 for MCB, 5000 for MFB and MUTAN.”
BackgroundMethodInfluential
Metadata
- Authors
- Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M.
- Affiliations
- Sony (Japan) (JP)
- University of California, Berkeley (US)
- Max Planck Institute for Informatics (DE)
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 457-468
- OpenAlex
- W2963383024
- Cited by
- 1,582
- References
- 60
- Citations by year
- 2026: 42 · 2025: 105 · 2024: 126 · 2023: 165 · 2022: 189 · 2021: 222 · 2020: 225 · 2019: 201 · 2018: 179 · 2017: 105 · 2016: 20
- FWCI
- 58.39
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Max Planck Digital Library (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1427425, 1212798, N000141110688)
- Defense Advanced Research Projects Agency
- Multidisciplinary University Research Initiative (N000141110688)
- Air Force Research Laboratory
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- visual question answering, visual grounding, multimodal fusion, outer product, attention mechanism, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.