ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
Open accessAbstract
Recent developments in modeling language and vision have been successfully applied to image question answering. It is both crucial and natural to extend this research direction to the video domain for video question answering (VideoQA). Compared to the image domain where large scale and fully annotated benchmark datasets exists, VideoQA datasets are limited to small scale and are automatically generated, etc. These limitations restrict their applicability in practice. Here we introduce ActivityNet-QA, a fully annotated and large scale VideoQA dataset. The dataset consists of 58,000 QA pairs on 5,800 complex web videos derived from the popular ActivityNet dataset. We present a statistical analysis of our ActivityNet-QA dataset and conduct extensive experiments on it by comparing existing VideoQA baselines. Moreover, we explore various video representation strategies to improve VideoQA performance, especially for long videos.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
[128] obtained promising results on the TVQA and ActivityNetQA datasets.
“Overall, they obtained promising results on the TVQA [129] and ActivityNetQA [130] datasets.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [130]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2× · see citing works
“E-VQA is the extension of an ImageQA baseline (Antol et al. 2015), where one long-short term memory (LSTM) network (Hochreiter and Schmidhuber 1997) is used to encode all words in the question and another different LSTM network is used to encode the frames in the video.”
BackgroundMethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · cited 2× · see citing works
“As we have obtained the sampled video unit set, for each unit ui, we used the VGG-16 network pre-trained on the ImageNet dataset (Simonyan and Zisserman 2014) to extract the appearance feature (the fc7 feature xi ∈ R(4096)) given the central frame of the unit, and the C3D network pretrained on the Sport-1M dataset (Tr…”
MethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1× · see citing works
“All models use the pre-trained 300-dimensional GloVe embedding (Pennington, Socher, and Manning 2014) to initialize the question embedding layer.”
MethodLearning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · cited 2× · see citing works
“As we have obtained the sampled video unit set, for each unit ui, we used the VGG-16 network pre-trained on the ImageNet dataset (Simonyan and Zisserman 2014) to extract the appearance feature (the fc7 feature xi ∈ R(4096)) given the central frame of the unit, and the C3D network pretrained on the Sport-1M dataset (Tr…”
MethodShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · cited 2× · see citing works
“These advances stimulated research into bridging the semantic connections between vision and language, such as in visual captioning (Donahue et al. 2015; Xu et al. 2015), visual grounding (Rohrbach et al. 2016; Chen, Kovvuri, and Nevatia 2017; Yu et al. 2018b) and visual question answering…”
BackgroundBottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · cited 1× · see citing works
“…shown recent success, with many approaches proposed to investigate the key components of this task, e.g. , discriminative feature representation (Anderson et al. 2018), multi-modal fusion (Kim et al. 2017a; Yu et al. 2017; Yu et al. 2018a) and visual reasoning (Nam, Ha, and Kim 2016; Lu et al.…”
BackgroundVisual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · cited 3× · see citing works
“datasets, such as Visual Genome (Krishna et al. 2016) and VQA (Antol et al.”
BackgroundMethodCIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 1× · see citing works
“We evaluate the average similarities of each two natural language sentences using CIDEr score (Vedantam, Lawrence Zitnick, and Parikh 2015), setting an empirical threshold to the average CIDEr score, and manually checking samples that did not reach the threshold.”
MethodVQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
ActivityNet: A large-scale video benchmark for human activity understanding
Heilbron, F. C., Escorcia, V., Ghanem, B., & Niebles, J. C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298698 · Cited by 2,578 · see citing works
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287 · see citing works
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781 · see citing works
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · see citing works
Long-Term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2599174 · Cited by 1,547 · see citing works
Hierarchical Question-Image Co-Attention for Visual Question AnsweringPreprint
Lu, J., Yang, J., Batra, D., & Parikh, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00061 · Cited by 1,303 · cited 2× · see citing works
“2017; 2018a) and visual reasoning (Nam, Ha, and Kim 2016; Lu et al. 2016; Johnson et al. 2017b).”
BackgroundDense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,279 · see citing works
Describing Videos by Exploiting Temporal Structure
Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., & Courville, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.512 · Cited by 1,035 · cited 2× · see citing works
“E-SA is the extension of the soft attention model (Yao et al. 2015) for ImageQA, where the question are fist encoded using a LSTM network.”
MethodMulti-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering
Zhou, Y., Yu, J., Jianping Fan, B., & Tao, D. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.202 · Cited by 796 · see citing works
Self-citationDual Attention Networks for Multimodal Reasoning and Matching
Nam, H., Ha, J.-W., & Kim, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.232 · Cited by 749 · cited 1× · see citing works
“…many approaches proposed to investigate the key components of this task, e.g. , discriminative feature representation (Anderson et al. 2018), multi-modal fusion (Kim et al. 2017a; Yu et al. 2017; Yu et al. 2018a) and visual reasoning (Nam, Ha, and Kim 2016; Lu et al. 2016; Johnson et al. 2017b).”
BackgroundMovieQA: Understanding Stories in Movies through Question-Answering
Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A. B., Urtasun, R., & Fidler, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.501 · Cited by 646 · cited 3× · see citing works
“Several VideoQA datasets have been compiled for different scenarios, such as MovieQA (Tapaswi et al. 2016), TGIF-QA (Jang et al.”
BackgroundMethodAsk Your Neurons: A Neural-Based Approach to Answering Questions about Images
Malinowski, M., Rohrbach, M., & Fritz, M. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.9 · Cited by 633 · cited 2× · see citing works
“The WUPS score with the threshold γ is given by where w and v are the words in the each predicted answer and ground-truth answer respectively. µ γ ( w, v ) is given by otherwise (4) Following the setting in (Malinowski, Rohrbach, and Fritz 2015), we choose two thresholds γ = 0 .”
BackgroundMethodExploring Models and Data for Image Question AnsweringPreprint
Ren, M., Kiros, R., & Zemel, R. S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.02074 · Cited by 618 · cited 2× · see citing works
“Referring to the taxonomy in existing VQA datasets (Ren, Kiros, and Zemel 2015; Antol et al. 2015), we manually categorize the samples into the following six classes by their answer types: Yes/No , Number , Color , Object , Location and Other .”
MethodHadamard Product for Low-rank Bilinear PoolingPreprint
Kim, J.-H., On, K.-W., Lim, W., Kim, J., Ha, J.-W., & Zhang, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.04325 · Cited by 604 · see citing works
VQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570 · cited 11× · see citing works
“E-VQA is the extension of an ImageQA baseline (Antol et al. 2015), where one long-short term memory (LSTM) network (Hochreiter and Schmidhuber 1997) is used to encode all words in the question and another different LSTM network is used to encode the frames in the video.”
BackgroundMethodResultInfluentialBeyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering
Zhou, Y., Yu, J., Xiang, C., Jianping Fan, B., & Tao, D. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2817340 · Cited by 536 · cited 2× · see citing works
“…many approaches proposed to investigate the key components of this task, e.g. , discriminative feature representation (Anderson et al. 2018), multi-modal fusion (Kim et al. 2017a; Yu et al. 2017; Yu et al. 2018a) and visual reasoning (Nam, Ha, and Kim 2016; Lu et al. 2016; Johnson et al. 2017b).”
BackgroundSelf-citationTGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
Jang, Y., Song, Y., Yu, Y., Kim, Y., & Kim, G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.149 · Cited by 496 · cited 3× · see citing works
“Existing approaches mainly focus on the temporal attention mechanism (Jang et al. 2017; Xu et al. 2017) or memory mechanism (Na et al. 2017; Kim et al. 2017b; Zhao et al. 2018a).”
BackgroundMethodInferring and Executing Programs for Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Hoffman, J., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.325 · Cited by 487 · cited 7× · see citing works
“E-VQA is the extension of an ImageQA baseline (Antol et al. 2015), where one long-short term memory (LSTM) network (Hochreiter and Schmidhuber 1997) is used to encode all words in the question and another different LSTM network is used to encode the frames in the video.”
BackgroundResultInfluentialMaking the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., & Parikh, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1116-0 · Cited by 487 · cited 1× · see citing works
“…and perform spatio-temporal reasoning on the visual contents of the video ( i.e. , frames in red border and objects in blue box). been facilitated by large scale and well annotated training datasets, such as Visual Genome (Krishna et al. 2016) and VQA (Antol et al. 2015; Goyal et al. 2017).”
BackgroundVideo Question Answering via Gradually Refined Attention over Appearance and Motion
Xu, D., Zhao, Z., Xiao, J., Wu, F., Zhang, H., He, X., & Zhuang, Y. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123427 · Cited by 438 · cited 12× · see citing works
“2017) MSVD Automatic OE 1,970 50,505 10s MSRVTT-QA (Xu et al. 2017) MSRVTT Automatic OE 10,000 243,680 15s TGIF-QA (Jang et al.”
BackgroundMethodInfluentialSelf-citationA Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain InputPreprint
Malinowski, M., & Fritz, M. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0210 · Cited by 432 · cited 6× · see citing works
“WUPS is a generalization of the accuracy measure that accounts for word-level ambiguities in the answer words.”
MethodInfluentialGrounding of Textual Phrases in Images by Reconstruction
Rohrbach, A., Rohrbach, M., Hu, R., Darrell, T., & Schiele, B. (2016) · Max Planck Digital Library · Cited by 422 · see citing works
Describing Videos by Exploiting Temporal StructurePreprint
Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., & Courville, A. C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.08029 · Cited by 292 · cited 2× · see citing works
“E-SA is the extension of the soft attention model (Yao et al. 2015) for ImageQA, where the question are fist encoded using a LSTM network.”
MethodFast Temporal Activity Proposals for Efficient Detection of Human Actions in Untrimmed Videos
Heilbron, F. C., Niebles, J. C., & Ghanem, B. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.211 · Cited by 279 · see citing works
Motion-Appearance Co-memory Networks for Video Question Answering
Gao, J., Ge, R., Chen, K., & Nevatia, R. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00688 · Cited by 252 · see citing works
Ask Your Neurons: A Neural-based Approach to Answering Questions about ImagesPreprint
Malinowski, M., Rohrbach, M., & Fritz, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.01121 · Cited by 181 · cited 2× · see citing works
“The WUPS score with the threshold γ is given by where w and v are the words in the each predicted answer and ground-truth answer respectively. µ γ ( w, v ) is given by otherwise (4) Following the setting in (Malinowski, Rohrbach, and Fritz 2015), we choose two thresholds γ = 0 .”
BackgroundMethodLeveraging Video Descriptions to Learn Video Question Answering
Zeng, K.-H., Chen, T.-H., Chuang, C.-Y., Liao, Y.-H., Niebles, J. C., & Sun, M. (2017) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v31i1.11238 · Cited by 162 · cited 3× · see citing works
“2016) Movies Human MC 6,771 6,462 200s Video-QA (Zeng et al. 2017) Jukinmedia Automatic OE 18,100 174,775 45s”
MethodDeepStory: Video Story QA by Deep Embedded Memory Networks
Kim, K.-M., Heo, M.-O., Choi, S.-H., & Zhang, B. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/280 · Cited by 145 · see citing works
Rethinking Diversified and Discriminative Proposal Generation for Visual Grounding
Zhou, Y., Yu, J., Xiang, C., Zhao, Z., Tian, Q. C., & Tao, D. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/155 · Cited by 134 · see citing works
Self-citationQuery-Guided Regression Network with Context Policy for Phrase Grounding
Chen, K., Kovvuri, R., & Nevatia, R. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.95 · Cited by 123 · cited 1× · see citing works
“…semantic connections between vision and language, such as in visual captioning (Donahue et al. 2015; Xu et al. 2015), visual grounding (Rohrbach et al. 2016; Chen, Kovvuri, and Nevatia 2017; Yu et al. 2018b) and visual question answering (Malinowski, Rohrbach, and Fritz 2015; Fukui et al. 2016).”
BackgroundMulti-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question AnsweringPreprint
Zhou, Y., Yu, J., Fan, J., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.01471 · Cited by 101 · see citing works
Self-citationA Read-Write Memory Network for Movie Story Understanding
Na, S., Lee, S., Kim, J., & Kim, G. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.80 · Cited by 101 · cited 3× · see citing works
“Existing approaches mainly focus on the temporal attention mechanism (Jang et al. 2017; Xu et al. 2017) or memory mechanism (Na et al. 2017; Kim et al. 2017b; Zhao et al. 2018a).”
BackgroundMethodOpen-Ended Long-form Video Question Answering via Adaptive Hierarchical Reinforced Networks
Zhao, Z., Zhang, Z., Xiao, S., Zhou, Y., Yu, J., Cai, D., Wu, F. Q., & Zhuang, Y. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/512 · Cited by 56 · see citing works
Self-citationMulti-Turn Video Question Answering via Multi-Stream Hierarchical Attention Context Network
Zhao, Z., Jiang, X., Cai, D., Xiao, J., He, X., & Pu, S. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/513 · Cited by 43 · cited 2× · see citing works
“Existing approaches mainly focus on the temporal attention mechanism (Jang et al. 2017; Xu et al. 2017) or memory mechanism (Na et al. 2017; Kim et al. 2017b; Zhao et al. 2018a).”
BackgroundMethodSelf-citationDeepStory: Video Story QA by Deep Embedded Memory NetworksPreprint
Kim, K.-M., Heo, M.-O., Choi, S.-H., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.00836 · Cited by 33 · see citing works
Motion-Appearance Co-Memory Networks for Video Question AnsweringPreprint
Gao, J., Ge, R., Chen, K., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.10906 · Cited by 23 · cited 2× · see citing works
“proposed a co-memory network to jointly model and interact with the motion and appearance information (Gao et al. 2018).”
BackgroundMethodRethinking Diversified and Discriminative Proposal Generation for Visual GroundingPreprint
Zhou, Y., Yu, J., Xiang, C., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.03508 · Cited by 3 · cited 3× · see citing works
“2015), visual grounding (Rohrbach et al. 2016; Chen, Kovvuri, and Nevatia 2017; Yu et al. 2018b) and visual question answering (Malinowski, Rohrbach, and Fritz 2015; Fukui et al.”
BackgroundSelf-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Sukhbaatar, S.; Weston, J.; Fergus, R.; et al. 2015. End-toend memory networks. In NIPS , 2440-2448.
Cited by
Sorted by citations
VideoMamba: State Space Model for Efficient Video Understanding
Li, K., Li, X., Wang, Y., He, Y., Wang, Y., Wang, L., & Qiao, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73347-5_14 · Cited by 378 · see citing works
Span-based Localizing Network for Natural Language Video Localization
Zhang, H., Sun, A., Jing, W., & Zhou, J. T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.585 · Cited by 326 · see citing works
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions
Xiao, J., Shang, X., Yao, A., & Chua, T. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00965 · Cited by 315 · see citing works
“Unlike previous works [18, 56] which focus on reasoning temporal actions of a single object in a question, we emphasize more on object interactions.”
BackgroundMethodInfluentialMVBench: A Comprehensive Multi-modal Video Understanding Benchmark
Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Lou, P., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02095 · Cited by 300 · see citing works
“The evaluations show that, our VideoChat2 significantly surpasses the top-performing VideoChat [42] by over 15% accuracy on MVBench, and also achieves the new state-of-the-art results on video conversation [51] and zero-shot QA benchmarks [88, 98].”
BackgroundMethodJust Ask: Learning to Answer Questions from Millions of Narrated Videos
Yang, A., Miech, A., Šivic, J., Laptev, I., & Schmid, C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00171 · Cited by 220 · see citing works
MERLOT: Multimodal Neural Script Knowledge ModelsPreprint
Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J., Farhadi, A., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02636 · Cited by 224 · see citing works
“We report results on 12 video reasoning tasks: TVQA [64], TVQA(+) [65], VLEP [66], MSRVTT-QA [117], MSRVTT-Multichoice [121], LSMDC-Multichoice, LSMDC fill-in-the-blank QA [110, 92], ActivityNetQA [122, 45], TGIFQA [52], and DramaQA [23].”
MethodInfluentialVideo-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.02245 · Cited by 291 · see citing works
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding
Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Chi, H., Guo, X., Ye, T., Zhang, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01725 · Cited by 176 · see citing works
“The evaluation pipeline covers crucial metrics (including Correctness of Information , Detailed Orientation , Contextual Understanding , Temporal Understanding and Consistency ) and assigns relative scores to the generated predictions on a scale of 1-5.”
MethodEnd-to-end Generative Pretraining for Multimodal Video Captioning
Seo, P. H., Nagrani, A., Arnab, A., & Schmid, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01743 · Cited by 173 · see citing works
Otter: A Multi-Modal Model With In-Context Instruction Tuning
Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Cahyono, J. A., Yang, J., Li, C., & Liu, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3571946 · Cited by 253 · see citing works
Vision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155 · see citing works
Unmasked Teacher: Towards Training-Efficient Video Foundation Models
Li, K., Wang, Y., Li, Y., Wang, Y., He, Y., Wang, L., & Qiao, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01826 · Cited by 143 · see citing works
“We further validate our model on two mainstream video-language tasks, including video-text retrieval (MSRVTT [84], DiDeMo [1], ActivityNet [38], LSMDC [63], MSVD [13] and Something-Something [40]) and video question-answering (ActivityNet-QA [94], MSRVTT-QA [82], MSRVTT-MC [93] and MSVD-QA [82]).”
BackgroundMethodInfluentialRevisiting the “Video” in Video-Language Understanding
Buch, S., Eyzaguirre, C., Gaidon, A., Wu, J., Fei-Fei, L., & Niebles, J. C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00293 · Cited by 139 · see citing works
“Standard video-language tasks include both discriminative tasks, such as video question answering [12, 19, 27, 28, 30, 54, 56, 58, 59], text-to-video/moment retrieval [16,23,43,56,62], and generative tasks, such as video cap-tioning [4, 23] and open-ended VQA [54, 56].”
BackgroundMulla, N., & Gharpure, P. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00295-9 · Cited by 132 · see citing works
Jin, P., Takanobu, R., Zhang, W., Cao, X., & Li, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01300 · Cited by 146 · see citing works
“4, we show the zero-shot video-question answering performance on several commonly used open-ended question-answer datasets, including MSRVTT-QA [62], MSVD-QA [62], TGIF-QA FrameQA [22], and ActivityNet-QA [67].”
BackgroundLearning to Answer Questions in Dynamic Audio-Visual Scenarios
Li, G., Wei, Y., Tian, Y., Xu, C., Wen, J.-R., & Hu, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01852 · Cited by 124 · see citing works
“The audio information in most released datasets ( e.g ., ActivityNet-QA [54] and AVSD [1]) is usually accompanied by severe noise that sound and visual objects in the video do not match ( e.g . background music), which makes them difficult to explore the association between different modalities.”
BackgroundMethodInfluentialVideo Understanding With Large Language Models: A Survey
Tang, Y., Bi, J., Xu, S., Song, L., Liang, S., Wang, T., Zhang, D., An, J., Lin, J., Zhu, R., et al. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3566695 · Cited by 174 · see citing works
“VALLEY’s performance is evaluated using datasets like MSVD-QA [160], MSRVTT-QA [160], ActivityNet-QA [161] for video understanding, and ScienceQA [162], MemeCap [163] for image understanding.”
MethodJiang, J., Chen, Z., Lin, H., Zhao, X., & Gao, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6766 · Cited by 107 · see citing works
“(Jang et al. 2017; Xu et al. 2017; Gao et al. 2018; Li et al. 2019; Yu et al. 2019; Fan et al. 2019) introduce extra temporal reasoning modules to answer questions which involve changing on temporal dimension like action or state transition.”
Visual question answering model based on visual relationship detection
Xi, Y., Zhang, Y., Ding, S., & Wan, S. (2019) · Signal Processing Image Communication · doi:10.1016/j.image.2019.115648 · Cited by 91 · see citing works
Natural Language Video Localization: A Revisit in Span-based Question Answering Framework
Zhang, H., Sun, A., Jing, W., Zhen, L., Zhou, J. T., & Goh, R. S. M. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3060449 · Cited by 90 · see citing works
AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning
Grunde-McLaughlin, M., Krishna, R., & Agrawala, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01113 · Cited by 89 · see citing works
Xu, L., Huang, H., & Liu, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00975 · Cited by 89 · see citing works
Video Question Answering: Datasets, Algorithms and Challenges
Zhong, Y., Ji, W., Xiao, J., Li, Y., Deng, W., & Chua, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.432 · Cited by 83 · see citing works
Information fusion in visual question answering: A Survey
Zhang, D., Cao, R., & Wu, S. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.03.005 · Cited by 81 · see citing works
Bi-Modal Transformer-Based Approach for Visual Question Answering in Remote Sensing Imagery
Bazi, Y., Rahhal, M. M. A., Mekhalfi, M. L., Zuair, M., & Melgani, F. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3192460 · Cited by 79 · see citing works
Pano-AVQA: Grounded Audio-Visual Question Answering on 360° Videos
Yun, H., Yu, Y., Yang, W., Lee, K., & Kim, G. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00204 · Cited by 78 · see citing works
Revealing Single Frame Bias for Video-and-Language Learning
Lei, J., Berg, T. L., & Bansal, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.29 · Cited by 90 · see citing works
VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset
Liu, J., Chen, S., He, X., Guo, L., Zhu, X., Wang, W., & Tang, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3479776 · Cited by 73 · see citing works
“For open-ended question answering tasks, we evaluate on 6 public datasets, including VQA (MSVD-QA [94], MSRVTT-QA [94], ActivityNet-QA [95], TGIF-Frame QA [96], and VQAv2 [97]) and AVQA (MUSIC-AVQA [60]).”
SPRNet: Single-Pixel Reconstruction for One-Stage Instance Segmentation
Yu, J., Yao, J., Zhang, J., Zhou, Y., & Tao, D. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2969046 · Cited by 76 · see citing works
“Just like what happens in the fields of image recognition [4], [5]; image captioning [6]; and visual question answering [7]–[9], the state-of-the-art models applied to instance segmentation are based on deep neural networks.”
BackgroundSelf-citationVisionZip: Longer is Better but Not Necessary in Vision Language Models
Yang, S., Chen, Y., Tian, Z., Wang, C., Li, J., Yu, B., & Jia, J. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01843 · Cited by 93 · see citing works
“Specifically, EVLGen [21] utilizes the token merging strategy, while LLaVA-PruMerge [44] employs a clustering method to reduce the tokens.”
HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training
Ye, Q., Xu, G., Yan, M., Xu, H., Qian, Q., Zhang, J., & Huang, F. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01413 · Cited by 70 · see citing works
AVQA: A Dataset for Audio-Visual Question Answering on Videos
Yang, P., Wang, X. Y., Duan, X., Chen, H., Hou, R., Jin, C., & Zhu, W. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548291 · Cited by 64 · see citing works
“Although most of the video question answering datasets [2, 4, 9, 24, 25, 37, 41, 44, 46] provide access to features of three modalities (audio, visual, and text), few of them have carefully considered the information from the audio modality in the creating process, thus most of the questions in these datasets could be…”
BackgroundInfluentialVindLU: A Recipe for Effective Video-and-Language Pretraining
Cheng, F., Wang, X., Lei, J., Crandall, D., Bansal, M., & Bertasius, G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01034 · Cited by 65 · see citing works
Long-Term Video Question Answering via Multimodal Hierarchical Memory Attentive Networks
Yu, T., Yu, J., Zhou, Y., Huang, Q., & Tian, Q. C. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.2995959 · Cited by 62 · see citing works
“Based upon the question-answering format of ActivityNet-QA [31], the inputs of MHMAN are (i) a question vector q ∈ R d q , which is encoded by a GloVe word embeddings layer [32], followed by a long short-term memory (LSTM) network [33] with d q hidden units, and (ii) a temporally aligned video with…”
MethodSelf-citationVisual Semantic Role Labeling for Video Understanding
Sadhu, A., Gupta, T., Yatskar, M., Nevatia, R., & Kembhavi, A. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00554 · Cited by 63 · see citing works
Li, J., Niu, L., & Zhang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02059 · Cited by 63 · see citing works
“QA [17] and ActivityNet-QA [53] manually annotated short and long web video respectively to explore the evidence reasoning, especially spatio-temporal reasoning.”
MethodAttend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering
Seo, A., Kang, G.-C., Park, J., & Zhang, B. (2021) · doi:10.18653/v1/2021.acl-long.481 · Cited by 61 · see citing works
“In particular, video question answering (video QA) tasks (Xu et al., 2016; Jang et al., 2017; Lei et al., 2018; Yu et al., 2019; Choi et al., 2020) have been proposed with the goal of reasoning over higher-level vision-”
Frame Augmented Alternating Attention Network for Video Question Answering
Zhang, W., Tang, S., Cao, Y., Pu, S., Wu, F. Q., & Zhuang, Y. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2935678 · Cited by 59 · see citing works
“Such cross-media [1], [2] nature makes VQA different from the previous studies in Multimedia research community, such as Visual Content Understanding [3]–[6] and QA Systems [7]–[10] that mostly rely on the analysis of the data from a single modal.”
Self-citationLongVLM: Efficient Long Video Understanding via Large Language Models
Weng, Y., Han, M., He, H., Chang, X., & Zhuang, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73414-4_26 · Cited by 63 · see citing works
Look Before you Speak: Visually Contextualized Utterances
Seo, P. H., Nagrani, A., & Schmid, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01660 · Cited by 55 · see citing works
“Further, we demonstrate that our model trained for this task on unlabelled videos achieves state-of-the-art performance on a number of downstream VideoQA benchmarks such as MSRVTT-QA, MSVD-QA, ActivityNet-QA and How2QA.”
MethodInfluentialOn Scaling Up a Multilingual Vision and Language Model
Chen, X., Djolonga, J., Padlewski, P., Mustafa, B., Changpinyo, S., Wu, J., Riquelme, C., Goodman, S., Wang, X., Tay, Y., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01368 · Cited by 140 · see citing works
“We fine-tune and evaluate the PaLI-X model on 4 video captioning (MSR-VTT [50], VATEX [51], ActivityNet Cap-tions [52], Spoken Moments in Time [53]) and 3 video question answering benchmarks (NExT-QA [54], MSR-VTT-QA [55], ActivityNet-QA [56]).”
MethodCompositional Attention Networks With Two-Stream Fusion for Video Question Answering
Yu, T., Yu, J., Zhou, Y., & Tao, D. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2940677 · Cited by 50 · see citing works
“Following the existing works [60], [62], we evaluated the”
BackgroundMethodInfluentialSelf-citationAction-Centric Relation Transformer Network for Video Question Answering
Zhang, J., Shao, J., Cao, R., Gao, L., Xu, X., & Shen, H. T. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3048440 · Cited by 50 · see citing works
MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding
Cao, X., Zhou, T. H., Ma, Y., Ye, W., Cui, C., Tang, K., Cao, Z., Liang, K., Wang, Z., Rehg, J. M., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02061 · Cited by 49 · see citing works
“Since the task of visual question answering (VQA) was first proposed by [2], there have been plenty of VQA datasets for different research areas [5, 24, 36, 79, 80].”
BackgroundRoboVQA: Multimodal Long-Horizon Reasoning for Robotics
Sermanet, P., Ding, T., Zhao, J. W., Xia, F., Dwibedi, D., Gopalakrishnan, K., Chan, C., Dulac-Arnold, G., Maddineni, S., Joshi, N. J., et al. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610216 · Cited by 49 · see citing works
“Recently many large-scale video datasets have been introduced [23, 24, 25, 26, 27, 28, 29, 30] that include videos of humans performing tasks with text narrations or question-answer annotations.”
Parallel Attention Network with Sequence Matching for Video Grounding
Zhang, H., Sun, A., Jing, W., Zhen, L., Zhou, J. T., & Goh, S. M. R. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.69 · Cited by 54 · see citing works
“Video grounding is a fundamental and challenging problem in vision-language understanding research area (Hu et al., 2019; Yu et al., 2019; Zhu and Yang, 2020).”
BackgroundNVILA: Efficient Frontier Visual Language Models
Liu, Z., Zhu, L., Shi, B., Zhang, Z., Lou, Y., Yang, S. W., Xi, H., Cao, S., Gu, Y., Li, D., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00390 · Cited by 73 · see citing works
“…[135] Text-only FLAN-1M [136], MathInstruct [137], Dolly [138], GSM8K-ScRel-SFT [139] Knowledge WordART [140], WIT [141], STEM-QA [142] Medical PathVQA [39], Slake [143], MedVQA [143] Video ActivityNet-QA [57], MSRVTT-QA [105], iVQA [144], Youcook2 [145], VaTeX [146], ShareG-PTVideo [147]”
Progressive Spatio-temporal Perception for Audio-Visual Question Answering
Li, G., Hou, W., & Hu, D. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612293 · Cited by 43 · see citing works
“The uni-modal question answering tasks represented by AQA [7, 22, 44] and VQA [2, 5, 9, 19, 38, 40, 46] have been extensively studied.”
Graph-Based Multi-Interaction Network for Video Question Answering
Gu, M., Zhao, Z., Jin, W., Hong, R., & Wu, F. Q. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3051756 · Cited by 43 · see citing works
“Compared to image question answering, video question answering [46]–[49] is relatively less explored.”
BackgroundSelf-citationAn Image Grid Can Be Worth a Video: Zero-Shot Video Question Answering Using a VLM
Kim, W., Choi, C., Lee, W., & Rhee, W. (2024) · IEEE Access · doi:10.1109/access.2024.3517625 · Cited by 50 · see citing works
“For open-ended VQA, our evaluation includes benchmarks of MSVD-QA [46], MSRVTT-QA [46], ActivityNet-QA [53], TGIF-Frame-QA [24], and the Text Generation Performance benchmark [31].”
MethodCross-Attentional Spatio-Temporal Semantic Graph Networks for Video Question Answering
Liu, Y., Zhang, X., Huang, F., Zhang, B., & Li, Z. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3142526 · Cited by 40 · see citing works
Vision Language Models are blind
Rahmanzadehgervi, P., Bolton, L., Taesiri, M. R., & Nguyen, A. T. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-981-96-0917-8_17 · Cited by 54 · see citing works
End-to-End Video Question-Answer Generation With Generator-Pretester Network
Su, H.-T., Chang, C.-H., Shen, P.-W., Wang, Y.-S., Chang, Y.-L., Chang, Y.-C., Cheng, P.-J., & Hsu, W. H. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3051277 · Cited by 38 · see citing works
Large Language Models are Temporal and Causal Reasoners for Video Question Answering
Ko, D., Lee, J. E., Kang, W., Roh, B., & Kim, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.261 · Cited by 48 · see citing works
Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering
Lipping, S., Sudarsanam, P., Drossos, K., & Virtanen, T. I. (2022) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco55093.2022.9909680 · Cited by 34 · see citing works
Locate Before Answering: Answer Guided Question Localization for Video Question Answering
Qian, T., Cui, R., Chen, J., Peng, P., Guo, X., & Jiang, Y. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3323878 · Cited by 33 · see citing works
“In recent years, several datasets have emerged for VideoQA with untrimmed long-term videos, such as NExT-QA [14], ActivityNet-QA [15], AGQA [19], and TVQA [27].”
BackgroundMethodInfluentialM-LLM Based Video Frame Selection for Efficient Video Understanding
Hu, K., Gao, F., Nie, X., Peng, Z., Tran, S., Neiman, T., Wang, L., Shah, M., Hamid, R., Yin, B., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01279 · Cited by 34 · see citing works
“Model Size NExT-QA SlowFast-LLaVA [ Table 1, Table 2, Table 3 and Table 4 present the comparison on ActivityNet-QA [58], NExT-QA [54] and EgoSchema [32], VideoMME [9] respectively.”
Temporal Textual Localization in Video via Adversarial Bi-Directional Interaction Networks
Zhang, Z., Zhao, Z., Zhang, Z., Lin, Z., Wang, Q., & Hong, R. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3023339 · Cited by 32 · see citing works
Self-citationLearning from Inside: Self-driven Siamese Sampling and Reasoning for Video Question Answering
Yu, W., Zheng, H., Li, M., Ji, L., Wu, L., Xiao, N., & Duan, N. (2021) · neural information processing systems · Cited by 30 · see citing works
Multichannel Attention Refinement for Video Question Answering
Zhuang, Y., Xu, D., Yan, X., Cheng, W., Zhao, Z., Pu, S., & Xiao, J. (2020) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3366710 · Cited by 29 · see citing works
“Most of the existing works either encode the question as a single semantic vector [9, 61] or fuse the extracted appearance and motion features via average pooling or concating [17, 55].”
BackgroundMethodInfluentialSelf-citationLearning to Answer Visual Questions From Web Videos
Yang, A., Miech, A., Šivic, J., Ivan Laptev, A., & Schmid, C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3173208 · Cited by 27 · see citing works
Where did I leave my keys? — Episodic-Memory-Based Question Answering on Egocentric Videos
Bärmann, L., & Waibel, A. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00162 · Cited by 23 · see citing works
“While this is not problematic for current VideoQA datasets like [37, 38, 40] due to their short video lengths ranging from a few seconds to a few minutes, it prohibits future realistic deployment scenarios involving long-horizon episodic data, e.g ., a memorization helper device recording…”
BackgroundEnv-QA: A Video Question Answering Benchmark for Comprehensive Understanding of Dynamic Environments
Gao, D., Wang, R., Bai, Z., & Chen, X. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00170 · Cited by 23 · see citing works
“In the last decades, tremendous works [9, 30, 45, 17, 58, 10, 49] have brought revolutionary advancements to computer vision systems for understanding web data, e.”
BackgroundRetrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models
Pan, J., Lin, Z., Ge, Y., Zhu, X., Zhang, R., Wang, Y., Qiao, Y., & Li, H. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00035 · Cited by 25 · see citing works
“Except for one case (BERT-base on ActivityNet-QA), we observe that the retrieved contexts sig-nificantly improve performance, sometimes even doubling the accuracy on all LMs. Notably, our models benefit more from stronger language models, as we observe larger gains with the increase in language model size.”
Video Question Answering: a Survey of Models and Datasets
Sun, G., Liang, L., Li, T., Yu, B., Wu, M., & Zhang, B. (2021) · Mobile Networks and Applications · doi:10.1007/s11036-020-01730-0 · Cited by 24 · see citing works
Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-Based Large Language Models
Ning, M., Zhu, B., Xie, Y., Lin, B., Cui, J., Yuan, L., Chen, D., & Yuan, L. (2025) · Computational Visual Media · doi:10.26599/cvm.2025.9450516 · Cited by 36 · see citing works
Deep Episodic Memory for Verbalization of Robot Experience
Bärmann, L., Peller-Konrad, F., Constantin, S., Asfour, T., & Waibel, A. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3085166 · Cited by 21 · see citing works
“VQA task complexity varies from choosing among a multiple-choice-set of answer sentences [18] to choosing a single output word [19] up to open ended answers sentences constructed using a decoder network [20].”
BackgroundSViTT: Temporal Learning of Sparse Video-Text Transformers
Li, Y., Min, K., Tripathi, S., & Vasconcelos, N. M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01814 · Cited by 20 · see citing works
“Question answering is evaluated on MSRVTT-QA [49], ActivityNet-QA [10,54] and AGQA [21], with top-1 accuracy of the answers.”
MethodTESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding
Ren, S., Chen, S., Li, S., Sun, X., & Hou, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.66 · Cited by 20 · see citing works
Hong, W., Cheng, Y., Yang, Z., Wang, W., Wang, L., Gu, X., Huang, S., Dong, Y., & Tang, J. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00791 · Cited by 22 · see citing works
“Traditional benchmarks like MSRVTT-QA [43] and ActivityNet-QA [48] primarily focus on basic action recognition and video question answering with short clips.”
DyCoke : Dynamic Compression of Tokens for Fast Video Large Language Models
Tao, K., Qin, C., You, H., Sui, Y., & Wang, H. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01769 · Cited by 19 · see citing works
“For benchmarks such as PerceptionTest [34], VideoMME [8], NeXTQA, VideoDetailCaption [31], and ActivityNet-QA [50], we use the LMMs-Eval [17, 53] for evaluation, while MVBench [23] is evaluated using the official code.”
Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs
Zhang, Q. F., Cheng, A., Lu, M., Zhang, R., Zhuo, Z., Cao, J., Guo, S., She, Q., & Zhang, S. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01939 · Cited by 46 · see citing works
CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios
Ye, Q., Yu, Z., Shao, R., Xie, X., Torr, P. H. S., & Cao, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72684-2_9 · Cited by 24 · see citing works
Self-citationHan, S., Huang, W., Shi, H., Zhuo, L., Su, X., Zhang, S., Zhou, X., Qi, X., Liao, Y., & Liu, S. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.02438 · Cited by 22 · see citing works
“Current VideoQA datasets [16, 45, 52, 57] rely on costly manual annotations that often lack the granularity needed for detailed understanding, limiting scalability.”
VoCo-LLaMA: Towards Vision Compression with Large Language Models
Ye, X., Gan, Y., Huang, X., Ge, Y., & Tang, Y. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02777 · Cited by 18 · see citing works
“ActivityNet-QA [56] is a fully annotated video QA dataset containing 58K question-answering pairs derived from 5,800 complex web videos from the ActivityNet dataset [18].”
Multi-Granularity Relational Attention Network for Audio-Visual Question Answering
Li, L., Jin, T., Wang, L., Jiang, H., Pan, W., Wang, J., Xiao, S., Xia, Y., Jiang, W., & Zhao, Z. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3264524 · Cited by 18 · see citing works
“Considerable amount of previous studies [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15] on VideoQA have been remarkably successful.”
BackgroundSelf-citationVideo Question Answering with Iterative Video-Text Co-tokenization
Piergiovanni, A., Morton, K., Kuo, W., Ryoo, M. S., & Angelova, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_5 · Cited by 18 · see citing works
Foundation Models for Video Understanding: A SurveyPreprint
Madan, N., Møgelmose, A., Modi, R., Rawat, Y. S., & Moeslund, T. B. (2024) · doi:10.36227/techrxiv.171769139.99464428/v1 · Cited by 18 · see citing works
CAT+: Investigating and Enhancing Audio-Visual Understanding in Large Language Models
Ye, Q., Yu, Z., Shao, R., Cui, Y., Kang, X., Liu, X., Torr, P. H. S., & Cao, X. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3582389 · Cited by 17 · see citing works
“4%accuracyintheMSRVTT-QA[71]andActivityNet-QA[76]benchmarks,respectively.”
Self-citationLifeQA: A Real-life Dataset for Video Question Answering
Castro, S., Azab, M., Stroud, J., Noujaim, C., Wang, R., Deng, J., & Mihalcea, R. F. (2020) · Language Resources and Evaluation · Cited by 17 · see citing works
Recent Advances in Video Question Answering: A Review of Datasets and Methods
Patel, D., Parikh, R., & Shastri, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68790-8_27 · Cited by 20 · see citing works
Referring Atomic Video Action Recognition
Peng, K., Fu, J., Yang, K., Wen, D., Chen, Y., Liu, R., Zheng, J., Zhang, J., Sarfraz, M. S., Stiefelhagen, R., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72655-2_10 · Cited by 16 · see citing works
MovieChat+: Question-Aware Sparse Memory for Long Video Question Answering
Song, E., Chai, W., Tian, Y., Hwang, J., Li, X., & Wang, G. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3604614 · Cited by 24 · see citing works
“However,merelyadjustingthepromptfortheLLMcannot effectivelyaddressthisissue.”
3MASSIV: Multilingual, Multimodal and Multi-Aspect dataset of Social Media Short Videos
Gupta, V., Mittal, T., Mathur, P., Mishra, V., Maheshwari, M., Bera, A., Mukherjee, D., & Manocha, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02039 · Cited by 16 · see citing works
“…: Videos which are short in duration [48] (5-6 secs) or are clipped out from longer videos [15,46,85], b) Long Narrative : Longer videos [1,10,78], usually more than 1-2 minutes, which tell a more detailed narrative or story c) Short Narrative : These are longer than micro-videos (typically…”
BackgroundPERCEIVER-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention
Tang, Z., Cho, J., Lei, J., & Bansal, M. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00439 · Cited by 16 · see citing works
“These methods have improved the performance of various tasks, such as text-based image/video retrieval [8, 58, 81, 63, 7, 2, 43, 40] and visual question answering [3, 29, 80, 42, 83].”
MethodMirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities
Piergiovanni, A., Noble, I., Kim, D., Ryoo, M. S., Gomes, V., & Angelova, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02531 · Cited by 17 · see citing works
“ActivityNet-QA [66] contains longer videos of about 160 seconds per video.”
BackgroundVideoQA in the Era of LLMs: An Empirical Study
Xiao, J., Huang, N., Qin, H., Li, D., Li, Y., Zhu, F., Tao, Z., Yu, J., Lin, L., Chua, T., et al. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02385-8 · Cited by 17 · see citing works
“OEQA treats each answer as a category and requires the models to predict a correct answer from a predefined answer list (Xu et al., 2017; Yu et al., 2019).”
InfluentialSelf-citationQuestion-Guided Erasing-Based Spatiotemporal Attention Learning for Video Question Answering
Liu, F., Liu, J., Hong, R., & Lu, H. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3105280 · Cited by 16 · see citing works
“We evaluate our model on four popular VideoQA datasets, namely, TGIF-QA [13], MSVD-QA [12], MSRVTT-QA [12], and ActivityNet-QA [65].”
MethodCross-modal Causal Relation Alignment for Video Question Grounding
Chen, W.-X., Liu, Y., Chen, B., Su, J.-D., Zheng, Y.-S., & Lin, L. (2025) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.02243 · Cited by 30 · see citing works
“1 can be reformulated as: which the probability of P ( a | do ( M = v t ) , do ( L )) can be formulated via back-door intervention applied at M ← V ← Z → a as: where ˆ v is the feature selected from V to represent the over-all distribution of the dataset.”
Yu, T., Fu, K., Zhang, J., Huang, Q., & Yu, J. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3390984 · Cited by 16 · see citing works
“Detailed statistics are presented in Table I. ActivityNet-QA [41] targets comprehending long-term videos through manually curated question-answer pairs.”
BackgroundSelf-citationLearning visual language models for video understanding
Yang, A. (2023) · theses.fr (ABES) · doi:10.70675/802872a7z8092z4556za1fdz2c7af45f0e72 · Cited by 15 · see citing works
Measuring Compositional Consistency for Video Question Answering
Gandhi, M., Gul, M. O., Prakash, E., Grunde-McLaughlin, M., Krishna, R., & Agrawala, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00499 · Cited by 14 · see citing works
“Despite the popularity of video question answering as a benchmark task [10, 12, 15, 23, 39, 46, 47], questions in several prominent benchmarks rely on dialogue and plot summaries instead of a video’s visual contents [18, 23, 39, 49], focus on short video clips or only a handful of objects [29, 45], or suffer from bias…”
BackgroundTowards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval
Lin, X., Tiwari, S., Huang, S., Li, M., Shou, M. Z., Ji, H., & Chang, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01426 · Cited by 14 · see citing works
“We select 5 commonly used multi-channel retrieval-style datasets: iVQA [40], How2QA [13], ActivityNet-QA [45], YouCook II [49], and VATEX [37] as the main evaluation datasets.”
BackgroundMethodInfluentialCAD - Contextual Multi-modal Alignment for Dynamic AVQA
Nadeem, A., Hilton, A., Dawes, R., Thomas, G., & Mustafa, A. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00709 · Cited by 15 · see citing works
“Recent advances address this limitation by enhancing the spatio-temporal reasoning ability through the integration of video-centered dynamics into the question-answering task [23,45,55,104,112].”
BackgroundMethodDistilling Vision-Language Models on Millions of Videos
Zhao, Y., Zhao, L., Zhou, X., Wu, J., Chu, C.-T., Miao, H., Schroff, F., Adam, H., Liu, T., Gong, B., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01245 · Cited by 15 · see citing works
“…(train) +3K (test) CIDEr MSR-VTTQA[65] QA 6.5K (train) +3K (test) Accuracy ANet-Captions[27] CAP 31K (train) +14K (test) CIDEr S-MiT [40] CAP 480K (train) +3K (test) CIDEr ANet-QA [73] QA 32K (train) +8K (test) Accuracy NExT-OE-QA[64] QA 37K (train) +9K (test) Wu-PalmerSimilarity(WUPS) Table 1.”
MethodVideo Question Answering with Procedural Programs
Choudhury, R., Niinuma, K., Kitani, K., & Jeni, L. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72920-1_18 · Cited by 14 · see citing works
Explore Video Clip Order With Self-Supervised and Curriculum Learning for Video Applications
Xiao, J., Li, L., Xu, D., Long, C., Shao, J., Zhang, S., Pu, S., & Zhuang, Y. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3025661 · Cited by 14 · see citing works
“Considering the balance between model complexity and training time, we adopt the extended soft-attention (E-SA) model mentioned in [56], and conduct experiments on the ActivityNet-QA dataset [57].”
MethodSelf-citationVideo Dialog via Multi-Grained Convolutional Self-Attention Context Multi-Modal Networks
Gu, M., Zhao, Z., Jin, W., Cai, D., & Wu, F. Q. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2957309 · Cited by 14 · see citing works
“Compared to image question answering, video question answering [30], [31] is relatively less explored.”
BackgroundSelf-citationDynamic self-attention with vision synchronization networks for video question answering
Liu, Y., Zhang, X., Huang, F., Shen, S., Tian, P., Li, L., & Li, Z. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108959 · Cited by 13 · see citing works
FunQA: Towards Surprising Video Comprehension
Xie, B., Zhang, S., Zhou, Z., Li, B., Zhang, Y., Hessel, J., Yang, J., & Liu, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73232-4_3 · Cited by 13 · see citing works
- Authors
- Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., & Tao, D.
- Affiliations
- Zhejiang University (CN)
- Hangzhou Dianzi University (CN)
- The University of Sydney (AU)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2019-07-17
- Volume
- 33
- Issue
- 01
- Pages
- 9127-9134
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2964220823
- Cited by
- 281
- References
- 43
- Citations by year
- 2026: 44 · 2025: 73 · 2024: 47 · 2023: 39 · 2022: 26 · 2021: 32 · 2020: 15 · 2019: 5
- FWCI
- 6.36
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- www.aaai.org/ (unspecified)
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61602405, Grant 61622205, 61622205, 61702143, 61836002)
- China Knowledge Centre for Engineering Sciences and Technology
- Australian Research Council (FL170100117, IH180100002, DP180103424)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Keywords
- video question answering, annotated dataset, online video, ActivityNet, video representation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
