Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
Open accessAbstract
In this paper, we propose a novel end-to-end trainable Video Question Answering (VideoQA) framework with three major components: 1) a new heterogeneous memory which can effectively learn global context information from appearance and motion features; 2) a redesigned question memory which helps understand the complex semantics of question and highlights queried subjects; and 3) a new multimodal fusion layer which performs multi-step reasoning by attending to relevant visual and textual hints with self-updated attention. Our VideoQA model firstly generates the global context-aware visual and textual features respectively by interacting current inputs with memory contents. After that, it makes the attentional fusion of the multimodal visual and textual representations to infer the correct answer. Multiple cycles of reasoning can be made to iteratively refine attention weights of the multimodal data and improve the final representation of the QA pair. Experimental results demonstrate our approach achieves state-of-the-art performance on four VideoQA benchmark datasets.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The most common existing VQA methods use attention and memory mechanisms to learn visual artifacts and the semantic correlations needed to answer questions accurately.
“Currently, the most common existing methods use attention [125, 127, 129] and memory [126] mechanisms to efficiently learn visual artifacts and the semantic correlations that allow questions to be answered accurately.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [126]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Self-citationAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
(2017) · Cited by 11,625 · see citing works
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
Stacked Attention Networks for Image Question Answering
Yang, Z., He, X., Gao, J., Deng, L., & Smola, A. J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.10 · Cited by 2,036 · see citing works
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.571 · Cited by 1,810 · see citing works
Meta-learning with memory-augmented neural networks
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., & Lillicrap, T. (2016) · International Conference on Machine Learning · Cited by 1,504 · see citing works
Sukhbaatar, S., Szlam, A., Weston, J., & Fergus, R. (2015) · arXiv · doi:10.48550/arxiv.1503.08895 · Cited by 1,345 · see citing works
Graves, A., Wayne, G., & Danihelka, I. (2014) · arXiv · doi:10.48550/arxiv.1410.5401 · Cited by 1,058 · see citing works
VQA: Visual Question AnsweringPreprint
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Batra, D., & Parikh, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00468 · Cited by 968 · see citing works
Andreas, J., Rohrbach, M., Darrell, T., & Klein, D. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.12 · Cited by 933 · see citing works
Collecting Highly Parallel Data for Paraphrase Evaluation
Chen, D. J., & Dolan, W. B. (2011) · Cited by 898 · see citing works
Memory NetworksPreprint
Weston, J., Chopra, S., & Bordes, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.3916 · Cited by 819 · see citing works
Dynamic Memory Networks for Visual and Textual Question AnsweringPreprint
Xiong, C., Merity, S., & Socher, R. (2016) · International Conference on Machine Learning · doi:10.48550/arxiv.1603.01417 · Cited by 770 · see citing works
Guadarrama, S., Krishnamoorthy, N., Malkarnenkar, G., Venugopalan, S., Mooney, R. J., Darrell, T., & Saenko, K. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.337 · Cited by 500 · see citing works
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
Jang, Y., Song, Y., Yu, Y., Kim, Y., & Kim, G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.149 · Cited by 496 · see citing works
Video Question Answering via Gradually Refined Attention over Appearance and Motion
Xu, D., Zhao, Z., Xiao, J., Wu, F., Zhang, H., He, X., & Zhuang, Y. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123427 · Cited by 438 · see citing works
A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain InputPreprint
Malinowski, M., & Fritz, M. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0210 · Cited by 432 · see citing works
Attention-Based Multimodal Fusion for Video Description
Hori, C., Hori, T., Lee, T.-Y., Zhang, Z., Harsham, B., Hershey, J. R., Marks, T. K., & Sumi, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.450 · Cited by 401 · see citing works
Motion-Appearance Co-memory Networks for Video Question Answering
Gao, J., Ge, R., Chen, K., & Nevatia, R. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00688 · Cited by 252 · see citing works
Multimodal Residual Learning for Visual QAPreprint
Kim, J.-H., Lee, S., Kwak, D., Heo, M.-O., Kim, J., Ha, J.-W., & Zhang, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01455 · Cited by 250 · see citing works
Mayer, G. (2017) · De Gruyter eBooks · doi:10.1515/9783110499469-009 · Cited by 235 · see citing works
TGIF: A New Dataset and Benchmark on Animated GIF Description
Li, Y., Song, Y., Cao, L., Tetreault, J., Goldberg, L., Jaimes, A., & Luo, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.502 · Cited by 231 · see citing works
End-to-End Concept Word Detection for Video Captioning, Retrieval, and Question Answering
Yu, Y., Ko, H., Choi, J., & Kim, G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.347 · Cited by 223 · see citing works
Leveraging Video Descriptions to Learn Video Question Answering
Zeng, K.-H., Chen, T.-H., Chuang, C.-Y., Liao, Y.-H., Niebles, J. C., & Sun, M. (2017) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v31i1.11238 · Cited by 162 · see citing works
Visual Question Answering with Memory-Augmented Networks
Ma, C., Shen, C., Dick, A., Wu, Q., Wang, P., van den Hengel, A., & Reid, I. D. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00729 · Cited by 113 · see citing works
Video Question Answering via Attribute-Augmented Attention Network Learning
Ye, Y., Zhao, Z., Li, Y., Chen, L., Xiao, J., & Zhuang, Y. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080655 · Cited by 98 · see citing works
TGIF: A New Dataset and Benchmark on Animated GIF DescriptionPreprint
Li, Y., Song, Y., Cao, L., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.02748 · Cited by 49 · see citing works
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question AnsweringPreprint
Jang, Y., Song, Y., Yu, Y., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.04497 · Cited by 36 · see citing works
A Taxonomy for Neural Memory Networks
Ma, Y., & Príncipe, J. C. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2926466 · Cited by 32 · see citing works
End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question AnsweringPreprint
Yu, Y., Ko, H., Choi, J., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.02947 · Cited by 19 · see citing works
Flagged first, then by citations
1 of the 38 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Learning Memory-Guided Normality for Anomaly Detection
Park, H., Noh, J., & Ham, B. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01438 · Cited by 998 · see citing works
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling
Lei, J., Li, L., Zhou, L., Gan, Z., Berg, T. L., Bansal, M., & Liu, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00725 · Cited by 606 · see citing works
The multi-modal fusion in visual question answering: a review of attention mechanisms
Lu, S., Liu, M., Yin, L., Yin, Z., Liu, X., & Zheng, W. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1400 · Cited by 478 · see citing works
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering
Liu, Y., Li, G., & Lin, L. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3284038 · Cited by 354 · see citing works
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions
Xiao, J., Shang, X., Yao, A., & Chua, T. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00965 · Cited by 315 · see citing works
Hierarchical Conditional Relation Networks for Video Question Answering
Le, T. M., Le, V., Venkatesh, S., & Tran, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00999 · Cited by 260 · see citing works
CLEVRER: CoLlision Events for Video REpresentation and ReasoningPreprint
Yi, K., Gan, C., Li, Y., Kohli, P., Wu, J., Torralba, A. B., & Tenenbaum, J. B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.01442 · Cited by 283 · see citing works
MeGA-CDA: Memory Guided Attention for Category-Aware Unsupervised Domain Adaptive Object Detection
Vibashan, V. S., Gupta, V., Oza, P., Sindagi, V. A., & Patel, V. M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00449 · Cited by 225 · see citing works
Just Ask: Learning to Answer Questions from Millions of Narrated Videos
Yang, A., Miech, A., Šivic, J., Laptev, I., & Schmid, C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00171 · Cited by 220 · see citing works
Divide-and-Assemble: Learning Block-wise Memory for Unsupervised Anomaly Detection
Hou, J., Zhang, Y., Zhong, Q., Xie, D., Pu, S., & Zhou, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00867 · Cited by 202 · see citing works
Reasoning with Heterogeneous Graph Alignment for Video Question Answering
Pin, J., & Han, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6767 · Cited by 191 · see citing works
URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark
Seo, S., Lee, J., & Han, B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58555-6_13 · Cited by 182 · see citing works
Align and Prompt: Video-and-Language Pre-training with Entity Prompts
Li, D., Li, J., Li, H., Niebles, J. C., & Hoi, S. C. H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00490 · Cited by 178 · see citing works
Location-Aware Graph Convolutional Networks for Video Question Answering
Huang, D. Y., Chen, P., Zeng, R., Du, Q., Tan, M., & Gan, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6737 · Cited by 174 · see citing works
HVPR: Hybrid Voxel-Point Representation for Single-stage 3D Object Detection
Noh, J., Lee, S., & Ham, B. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01437 · Cited by 170 · see citing works
All in One: Exploring Unified Video-Language Pre-Training
Wang, J., Ge, Y., Yan, R., Ge, Y., Lin, K. Q., Tsutsui, S., Lin, X., Cai, G., Wu, J., Shan, Y., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00638 · Cited by 164 · see citing works
Pan, X., Lu, Y., Zhao, K., Huang, H., Wang, M., & Chen, H. (2021) · Geophysical Research Letters · doi:10.1029/2021gl095302 · Cited by 151 · see citing works
Self-citationEnriching Local and Global Contexts for Temporal Action Localization
Zhu, Z., Tang, W., Wang, L., Zheng, N., & Hua, G. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01326 · Cited by 140 · see citing works
Learning to Answer Questions in Dynamic Audio-Visual Scenarios
Li, G., Wei, Y., Tian, Y., Xu, C., Wen, J.-R., & Hu, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01852 · Cited by 124 · see citing works
Video as Conditional Graph Hierarchy for Multi-Granular Question Answering
Xiao, J., Yao, A., Liu, Z., Li, Y., Ji, W., & Chua, T. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20184 · Cited by 116 · see citing works
Jiang, J., Chen, Z., Lin, H., Zhao, X., & Gao, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6766 · Cited by 107 · see citing works
Bridge to Answer: Structure-aware Graph Interaction Network for Video Question Answering
Park, J., Lee, J., & Sohn, K. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01527 · Cited by 105 · see citing works
Gao, L., Yu, L., Zeng, P., Song, J., Wang, M., & Shen, H. T. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3120867 · Cited by 103 · see citing works
Video Graph Transformer for Video Question Answering
Xiao, J., Zhou, P., Chua, T., & Yan, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_3 · Cited by 99 · see citing works
DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering
Wang, J., Bao, B., & Xu, C. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3097171 · Cited by 96 · see citing works
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token ModelingPreprint
Fu, T.-J., Li, L., Gan, Z., Lin, K., Wang, W. Y., Wang, L., & Liu, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12681 · Cited by 130 · see citing works
Self-citationAGQA: A Benchmark for Compositional Spatio-Temporal Reasoning
Grunde-McLaughlin, M., Krishna, R., & Agrawala, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01113 · Cited by 89 · see citing works
Xu, L., Huang, H., & Liu, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00975 · Cited by 89 · see citing works
Self-citationVideo Question Answering: Datasets, Algorithms and Challenges
Zhong, Y., Ji, W., Xiao, J., Li, Y., Deng, W., & Chua, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.432 · Cited by 83 · see citing works
Modality Shifting Attention Network for Multi-Modal Video Question Answering
Kim, J., Ma, M., Pham, T. X., Kim, K., & Yoo, C. D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01012 · Cited by 82 · see citing works
EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone
Pramanick, S., Song, Y., Nag, S., Lin, K. Q., Shah, H., Shou, M. Z., Chellappa, R., & Zhang, P. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00487 · Cited by 83 · see citing works
Modality-specific and shared generative adversarial network for cross-modal retrieval
Wu, F., Jing, X., Wu, Z., Ji, Y., Dong, X., Luo, X., Huang, Q., & Wang, R. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107335 · Cited by 78 · see citing works
Temporal Memory Attention for Video Semantic Segmentation
Wang, H., Wang, W., & Liu, J. (2021) · Proceedings - International Conference on Image Processing · doi:10.1109/icip42928.2021.9506731 · Cited by 74 · see citing works
Self-citationAVQA: A Dataset for Audio-Visual Question Answering on Videos
Yang, P., Wang, X. Y., Duan, X., Chen, H., Hou, R., Jin, C., & Zhu, W. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548291 · Cited by 64 · see citing works
Hierarchical Object-oriented Spatio-Temporal Reasoning for Video Question Answering
Dang, L. H., Le, T. M., Le, V., & Tran, T. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/88 · Cited by 63 · see citing works
Visual Anomaly Detection via Partition Memory Bank Module and Error Estimation
Xing, P., & Li, Z. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3237562 · Cited by 65 · see citing works
Long-Term Video Question Answering via Multimodal Hierarchical Memory Attentive Networks
Yu, T., Yu, J., Zhou, Y., Huang, Q., & Tian, Q. C. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.2995959 · Cited by 62 · see citing works
Li, J., Niu, L., & Zhang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02059 · Cited by 63 · see citing works
Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering
Seo, A., Kang, G.-C., Park, J., & Zhang, B. (2021) · doi:10.18653/v1/2021.acl-long.481 · Cited by 61 · see citing works
Contrastive Video Question Answering via Video Graph Transformer
Xiao, J., Zhou, P., Yao, A., Li, Y., Hong, R., Yan, S., & Chua, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292266 · Cited by 62 · see citing works
Look Before you Speak: Visually Contextualized Utterances
Seo, P. H., Nagrani, A., & Schmid, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01660 · Cited by 55 · see citing works
HAIR: Hierarchical Visual-Semantic Relational Reasoning for Video Question Answering
Liu, F., Liu, J., Wang, N., & Lu, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00172 · Cited by 55 · see citing works
An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling
Fu, T.-J., Li, L., Gan, Z., Lin, K., Wang, W. Y., Wang, L., & Liu, Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02193 · Cited by 57 · see citing works
Self-citationDiscovering the Real Association: Multimodal Causal Reasoning in Video Question Answering
Zang, C., Wang, H., Pei, M., & Liang, W. G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01824 · Cited by 53 · see citing works
Action-Centric Relation Transformer Network for Video Question Answering
Zhang, J., Shao, J., Cao, R., Gao, L., Xu, X., & Shen, H. T. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3048440 · Cited by 50 · see citing works
Cross-Modal Dynamic Networks for Video Moment Retrieval With Text Query
Wang, G., Xu, X., Shen, F., Lu, H., Ji, Y., & Shen, H. T. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3142420 · Cited by 49 · see citing works
Progressive Graph Attention Network for Video Question Answering
Peng, L., Yang, S., Bin, Y., & Wang, G. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475193 · Cited by 48 · see citing works
Clover: Towards A Unified Video-Language Alignment and Fusion Model
Huang, J., Li, Y., Feng, J., Wu, X., Sun, X., & Ji, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01427 · Cited by 52 · see citing works
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization
Tang, Z., Lei, J., & Bansal, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.193 · Cited by 48 · see citing works
Progressive Spatio-temporal Perception for Audio-Visual Question Answering
Li, G., Hou, W., & Hu, D. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612293 · Cited by 43 · see citing works
Adaptive Spatio-Temporal Graph Enhanced Vision-Language Representation for Video QA
Jin, W., Zhao, Z., Cao, X., Zhu, J., He, X., & Zhuang, Y. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3076556 · Cited by 41 · see citing works
Memorial GAN With Joint Semantic Optimization for Unpaired Image Captioning
Song, P., Guo, D., Zhou, J., Xu, M., & Wang, M. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3175012 · Cited by 41 · see citing works
Cross-Attentional Spatio-Temporal Semantic Graph Networks for Video Question Answering
Liu, Y., Zhang, X., Huang, F., Zhang, B., & Li, Z. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3142526 · Cited by 40 · see citing works
Self-citationEnd-to-End Video Question-Answer Generation With Generator-Pretester Network
Su, H.-T., Chang, C.-H., Shen, P.-W., Wang, Y.-S., Chang, Y.-L., Chang, Y.-C., Cheng, P.-J., & Hsu, W. H. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3051277 · Cited by 38 · see citing works
Self-trained prediction model and novel anomaly score mechanism for video anomaly detection
Guo, A., Guo, L., Zhang, R., Wang, Y., & Gao, S. (2022) · Image and Vision Computing · doi:10.1016/j.imavis.2022.104391 · Cited by 38 · see citing works
Video Question Answering With Prior Knowledge and Object-Sensitive Learning
Zeng, P., Zhang, H., Gao, L., Song, J., & Shen, H. T. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3205212 · Cited by 38 · see citing works
Adaptive Memory Broad Learning System for Unsupervised Time Series Anomaly Detection
Zhong, Z., Yu, Z., Fan, Z., Chen, C. L. P., & Yang, K. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3415621 · Cited by 38 · see citing works
(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering
Cherian, A., Hori, C., Marks, T. K., & Roux, J. L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19922 · Cited by 37 · see citing works
Contextual Attention Network for Emotional Video Captioning
Song, P., Guo, D., Cheng, J. S., & Wang, M. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3183402 · Cited by 36 · see citing works
Visual Commonsense-Aware Representation Network for Video Captioning
Zeng, P., Zhang, H., Gao, L., Li, X., Qian, J., & Shen, H. T. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3323491 · Cited by 37 · see citing works
Recurrent Relational Memory Network for Unsupervised Image Captioning
Guo, D., Wang, Y., Song, P., & Wang, M. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/128 · Cited by 35 · see citing works
Counting Bites and Recognizing Consumed Food from Videos for Passive Dietary Monitoring
Qiu, J., Lo, F. P.-W., Jiang, S., Tsai, Y.-Y., Sun, Y., & Lo, B. P. L. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3022815 · Cited by 35 · see citing works
Equivariant and Invariant Grounding for Video Question Answering
Li, Y., Wang, X., Xiao, J., & Chua, T. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548035 · Cited by 34 · see citing works
Locate Before Answering: Answer Guided Question Localization for Video Question Answering
Qian, T., Cui, R., Chen, J., Peng, P., Guo, X., & Jiang, Y. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3323878 · Cited by 33 · see citing works
MELTR: Meta Loss Transformer for Learning to Fine-tune Video Foundation Models
Ko, D., Choi, J., Choi, H. K., On, K.-W., Roh, B., & Kim, H. J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01925 · Cited by 33 · see citing works
Visual Causal Scene Refinement for Video Question Answering
Wei, Y.-S., Liu, Y., Yan, H., Li, G., & Lin, L. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611873 · Cited by 34 · see citing works
BiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialogues
Le, H., Sahoo, D., Chen, N. F., & Hoi, S. C. H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.145 · Cited by 32 · see citing works
Multilevel Hierarchical Network with Multiscale Sampling for Video Question Answering
Peng, M., Wang, C., Gao, Y., Shi, Y., & Zhou, X.-D. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/178 · Cited by 31 · see citing works
Learning from Inside: Self-driven Siamese Sampling and Reasoning for Video Question Answering
Yu, W., Zheng, H., Li, M., Ji, L., Wu, L., Xiao, N., & Duan, N. (2021) · neural information processing systems · Cited by 30 · see citing works
EgoVQA - An Egocentric Video Question Answering Benchmark Dataset
Fan, C. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00536 · Cited by 30 · see citing works
Self-citationLong video question answering: A Matching-guided Attention Model
Wang, W., Huang, Y., & Wang, L. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107248 · Cited by 29 · see citing works
Self-citationFake News Detection via Multi-Modal Topic Memory Network
Ying, L., Yu, H. L., Wang, J., Ji, Y., & Qian, S. (2021) · IEEE Access · doi:10.1109/access.2021.3113981 · Cited by 28 · see citing works
Chadha, A. R., Arora, G., & Kaloty, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.07735 · Cited by 29 · see citing works
IntentQA: Context-aware Video Intent Reasoning
Li, J., Wei, P., Han, W., & Fan, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01099 · Cited by 28 · see citing works
Visual Question Answering on Image Sets
Bansal, A., Zhang, Y., & Chellappa, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58589-1_4 · Cited by 28 · see citing works
Hierarchical Conditional Relation Networks for Multimodal Video Question Answering
Le, T. M., Le, V., Venkatesh, S., & Tran, T. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01514-3 · Cited by 27 · see citing works
Learning to Answer Visual Questions From Web Videos
Yang, A., Miech, A., Šivic, J., Ivan Laptev, A., & Schmid, C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3173208 · Cited by 27 · see citing works
Video Question Answering With Semantic Disentanglement and Reasoning
Liu, J., Wang, G., Xie, J., Zhou, F., & Xu, H. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3317447 · Cited by 28 · see citing works
Weakly Supervised Segmentation of Small Buildings with Point Labels
Lee, J.-H., Kim, C., & Sull, S. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00731 · Cited by 26 · see citing works
Geometry Sensitive Cross-Modal Reasoning for Composed Query Based Image Retrieval
Zhang, F., Xu, M., & Xu, C. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3138302 · Cited by 26 · see citing works
Question-Aware Global-Local Video Understanding Network for Audio-Visual Question Answering
Chen, Z., Wang, L., Wang, P., & Gao, P. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3318220 · Cited by 26 · see citing works
Discovering Spatio-Temporal Rationales for Video Question Answering
Li, Y., Xiao, J., Feng, C., Wang, X., & Chua, T. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01275 · Cited by 27 · see citing works
Embedded Heterogeneous Attention Transformer for Cross-Lingual Image Captioning
Song, Z., Hu, Z., Zhou, Y., Zhao, Y., Hong, R., & Wang, M. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3384678 · Cited by 25 · see citing works
Multi-Scale Progressive Attention Network for Video Question Answering
Guo, Z., Zhao, J., Jiao, L., Liu, X., & Li, L. (2021) · doi:10.18653/v1/2021.acl-short.122 · Cited by 25 · see citing works
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning
Amrani, E., Ben‐Ari, R., Rotman, D., & Bronstein, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16822 · Cited by 29 · see citing works
COCA: COllaborative CAusal Regularization for Audio-Visual Question Answering
Lao, M., Pu, N., Liu, Y., He, K., Bakker, E. M., & Lew, M. S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26527 · Cited by 24 · see citing works
3-D Relation Network for visual relation recognition in videos
Cao, Q., Huang, H., Shang, X., Wang, B., & Chua, T. (2021) · Neurocomputing · doi:10.1016/j.neucom.2020.12.029 · Cited by 23 · see citing works
Self-citationLiVLR: A Lightweight Visual-Linguistic Reasoning Framework for Video Question Answering
Jiang, J., Liu, Z., & Zheng, N. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3185900 · Cited by 23 · see citing works
ERM: Energy-Based Refined-Attention Mechanism for Video Question Answering
Zhang, F., Wang, R., Zhou, F., & Luo, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3212463 · Cited by 23 · see citing works
Multi-Granularity Interaction and Integration Network for Video Question Answering
Wang, Y., Liu, M., Wu, J., & Nie, L. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3278492 · Cited by 23 · see citing works
Video Question Answering: a Survey of Models and Datasets
Sun, G., Liang, L., Li, T., Yu, B., Wu, M., & Zhang, B. (2021) · Mobile Networks and Applications · doi:10.1007/s11036-020-01730-0 · Cited by 24 · see citing works
Dynamic Spatio-Temporal Graph Reasoning for VideoQA With Self-Supervised Event Recognition
Nie, J., Wang, X. Y., Hou, R., Li, G., Chen, H., & Zhu, W. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3411448 · Cited by 23 · see citing works
Robust video question answering via contrastive cross-modality representation learning
Yang, X., Zeng, J., Guo, D., Wang, S., Dong, J., & Wang, M. (2024) · Science China Information Sciences · doi:10.1007/s11432-023-4084-6 · Cited by 22 · see citing works
Document Collection Visual Question Answering
Tito, R., Karatzas, D., & Valveny, E. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86331-9_50 · Cited by 24 · see citing works
Khurana, K., & Deshpande, U. A. (2021) · IEEE Access · doi:10.1109/access.2021.3058248 · Cited by 23 · see citing works
Stereoscopic Vision Recalling Memory for Monocular 3D Object Detection
Kim, J. U., Kim, H.-I., & Ro, Y. M. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3274479 · Cited by 21 · see citing works
Reducing Vision-Answer Biases for Multiple-Choice VQA
Zhang, X., Zhang, F., & Xu, C. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3302162 · Cited by 21 · see citing works
Self-citationTem-adapter: Adapting Image-Text Pretraining for Video Question Answer
Chen, G., Liu, X., Wang, G., Zhang, K., Torr, P. H. S., Zhang, X.-P. S., & Tang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01282 · Cited by 21 · see citing works
Self-citationBoosting Audio Visual Question Answering via Key Semantic-Aware Cues
Li, G., Du, H., & Hu, D. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680803 · Cited by 20 · see citing works
- Authors
- Fan, C., Zhang, X., Zhang, S., Wang, W., Zhang, C., & Huang, H.
- Affiliations
- Jingdong (China) (CN)
- University of Pittsburgh (US)
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 1999-2007
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2954199749
- Cited by
- 293
- References
- 38
- Citations by year
- 2026: 16 · 2025: 23 · 2024: 49 · 2023: 65 · 2022: 46 · 2021: 59 · 2020: 31 · 2019: 4
- FWCI
- 11.42
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- video question answering, heterogeneous memory, multimodal attention, multimodal fusion, global context modeling, appearance and motion features, multi-step reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
