TVQA: Localized, Compositional Video Question Answering
Open accessAbstract
Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks.However, due to data limitations, there has been much less work on video-based QA.In this paper, we present TVQA, a largescale video QA dataset based on 6 popular TV shows.TVQA consists of 152,545 QA pairs from 21,793 clips, spanning over 460 hours of video.Questions are designed to be compositional in nature, requiring systems to jointly localize relevant moments within a clip, comprehend subtitle-based dialogue, and recognize relevant visual concepts.We provide analyses of this new dataset as well as several baselines and a multi-stream end-to-end trainable neural network framework for the TVQA task.The dataset is publicly available at http://tvqa.cs.unc.edu.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
[128] obtained promising results on the TVQA and ActivityNetQA datasets.
“Overall, they obtained promising results on the TVQA [129] and ActivityNetQA [130] datasets.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [129]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2× · see citing works
“We also use ResNet101 (He et al., 2016) trained on ImageNet (Deng et al., 2009) to extract whole image features.”
MethodImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 5× · see citing works
“ImageNet Features: We extract the pooled 2048D feature of the last block of ResNet101.”
BackgroundMethodInfluentialFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · cited 5× · see citing works
“Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al., 2017), etc. Addit…”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 4× · see citing works
“A subtitle S, which contains a set of sentences, is flattened into a long sequence of words and GloVe (Pennington et al., 2014) is used to embed the words.”
BackgroundMethodInfluentialSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 2× · see citing works
“With the same goal in mind, Rajpurkar et al. (2016) introduced the SQuAD dataset, but their answers are specific spans from long passages.”
BackgroundVQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., & Gool, L. V. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46484-8_2 · Cited by 4,164 · see citing works
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,779 · cited 1× · see citing works
“Text Question Answering: The related task of text-based question answering has been extensively explored (Richardson et al., 2013; Weston et al., 2015; Rajpurkar et al., 2016; Hermann et al., 2015; Hill et al., 2015).”
BackgroundBidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01603 · Cited by 2,322 · cited 2× · see citing works
“The context matching module is adopted from the context-query attention layer from previous works (Seo et al., 2017; Yu et al., 2018a).”
MethodCLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781 · see citing works
Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S. (2015) · arXiv · doi:10.48550/arxiv.1506.06726 · Cited by 1,293 · cited 3× · see citing works
“We embed sentences into vectors using TFIDF, SkipThought (Kiros et al., 2015), or averaged GloVe (Pennington et al.”
MethodDense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,279 · see citing works
QANet: Combining Local Convolution with Global Self-Attention for Reading ComprehensionPreprint
Yu, A. W., Dohan, D., Luong, M.-T., Zhao, R., Chen, K., Norouzi, M., & Le, Q. V. (2018) · International Conference on Learning Representations · doi:10.48550/arxiv.1804.09541 · Cited by 1,169 · cited 3× · see citing works
“The context matching module is adopted from the context-query attention layer from previous works (Seo et al., 2017; Yu et al., 2018a).”
BackgroundMethod(2016) · doi:10.1007/978-3-319-46475-6_5 · Cited by 1,081 · cited 2× · see citing works
“, 2016) and referring expression comprehension (Hu et al., 2016; Yu et al., 2016; Nagaraja et al., 2016; Yu et al., 2017, 2018b).”
BackgroundSelf-citationDeep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048 · see citing works
Localizing Moments in Video with Natural Language
Hendricks, L. A., Wang, O., Shechtman, E., Šivic, J., Darrell, T., & Russell, B. C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.618 · Cited by 983 · cited 1× · see citing works
“Most recently, moment localization was proposed in (Hendricks et al., 2017; Gao et al., 2017), where the goal is to localize a short moment from a long video sequence given a query description.”
BackgroundTALL: Temporal Activity Localization via Language Query
Gao, J., Sun, C., Yang, Z., & Nevatia, R. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.563 · Cited by 884 · cited 1× · see citing works
“Most recently, moment localization was proposed in (Hendricks et al., 2017; Gao et al., 2017), where the goal is to localize a short moment from a long video sequence given a query description.”
BackgroundMAttNet: Modular Attention Network for Referring Expression Comprehension
Yu, L., Lin, Z., Shen, X., Yang, J., Lu, X., Bansal, M., & Berg, T. L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00142 · Cited by 857 · see citing works
Self-citationVisual7W: Grounded Question Answering in Images
Zhu, Y., Groth, O., Bernstein, M., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.540 · Cited by 794 · cited 5× · see citing works
“Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al., 2017), etc. Addit…”
BackgroundInfluentialLearning Deep Structure-Preserving Image-Text Embeddings
Wang, L., Li, Y., & Lazebnik, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.541 · Cited by 788 · cited 1× · see citing works
“For image-based object grounding, there has been much work on phrase grounding (Plummer et al., 2015; Wang et al., 2016b; Rohrbach et al., 2016) and referring expression comprehension (Hu et al., 2016; Yu et al., 2016; Nagaraja et al., 2016; Yu et al., 2017, 2018b).”
BackgroundMCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C., & Renshaw, E. (2013) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 778 · cited 2× · see citing works
“Text Question Answering: The related task of text-based question answering has been extensively explored (Richardson et al., 2013; Weston et al., 2015; Rajpurkar et al., 2016; Hermann et al., 2015; Hill et al., 2015).”
BackgroundMethodTowards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015) · arXiv · doi:10.48550/arxiv.1502.05698 · Cited by 670 · cited 2× · see citing works
“Weston et al. (2015) designed a set of tasks with automatically generated QAs to evaluate the textual reasoning ability of artificial agents and Hermann et al. (2015); Hill et al. (2015) constructed the cloze dataset on top of an existing corpus.”
BackgroundMovieQA: Understanding Stories in Movies through Question-Answering
Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A. B., Urtasun, R., & Fidler, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.501 · Cited by 646 · cited 5× · see citing works
“MovieQA (Tapaswi et al., 2016) is most similar to our dataset, with both multiple choice questions and timestamp annotation.”
BackgroundMethodInfluentialExploring Models and Data for Image Question AnsweringPreprint
Ren, M., Kiros, R., & Zemel, R. S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.02074 · Cited by 618 · cited 3× · see citing works
“Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al., 2017), etc. Addit…”
BackgroundMethodVQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570 · cited 5× · see citing works
“Alternative methods include sampling negative answers from other questions’ correct answers, either based on semantic similarity (Das et al., 2017; Jang et al., 2017) or randomly (Antol et al., 2015; Das et al., 2017).”
BackgroundMethodInfluentialNatural Language Object Retrieval
Hu, R., Xu, H., Rohrbach, M., Feng, J., Saenko, K., & Darrell, T. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.493 · Cited by 534 · cited 1× · see citing works
“For image-based object grounding, there has been much work on phrase grounding (Plummer et al., 2015; Wang et al., 2016b; Rohrbach et al., 2016) and referring expression comprehension (Hu et al., 2016; Yu et al., 2016; Nagaraja et al., 2016; Yu et al., 2017, 2018b).”
BackgroundTGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
Jang, Y., Song, Y., Yu, Y., Kim, Y., & Kim, G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.149 · Cited by 496 · cited 6× · see citing works
“Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al., 2017), etc. Addit…”
BackgroundMethodInfluentialThe Goldilocks Principle: Reading Children's Books with Explicit Memory RepresentationsPreprint
Hill, F., Bordes, A., Chopra, S., & Weston, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.02301 · Cited by 478 · cited 2× · see citing works
“Weston et al. (2015) designed a set of tasks with automatically generated QAs to evaluate the textual reasoning ability of artificial agents and Hermann et al. (2015); Hill et al. (2015) constructed the cloze dataset on top of an existing corpus.”
BackgroundA dataset for Movie Description
Rohrbach, A., Rohrbach, M., Tandon, N., & Schiele, B. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298940 · Cited by 450 · cited 3× · see citing works
“, 2017), LSMDC (Rohrbach et al., 2015), TRECVID (Over et al.”
MethodModeling Context Between Objects for Referring Expression Understanding
Nagaraja, V., Morariu, V. I., & Davis, L. S. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46493-0_48 · Cited by 450 · cited 2× · see citing works
“, 2016) and referring expression comprehension (Hu et al., 2016; Yu et al., 2016; Nagaraja et al., 2016; Yu et al., 2017, 2018b).”
Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question AnsweringPreprint
Gao, H., Mao, J., Zhou, J., Huang, Z., Wang, L., & Xu, W. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.05612 · Cited by 437 · cited 4× · see citing works
“, 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al.”
BackgroundInfluentialDas, A., Kottur, S., Gupta, K., Singh, A., Jain, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · Cited by 432 · see citing works
A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain InputPreprint
Malinowski, M., & Fritz, M. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0210 · Cited by 432 · cited 3× · see citing works
“Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al., 2017), etc. Addit…”
BackgroundMethodGrounding of Textual Phrases in Images by Reconstruction
Rohrbach, A., Rohrbach, M., Hu, R., Darrell, T., & Schiele, B. (2016) · Max Planck Digital Library · Cited by 422 · see citing works
A Joint Speaker-Listener-Reinforcer Model for Referring Expressions
Yu, L., Tan, H., Bansal, M., & Berg, T. (2016) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2017.375 · Cited by 297 · cited 1× · see citing works
“For image-based object grounding, there has been much work on phrase grounding (Plummer et al., 2015; Wang et al., 2016b; Rohrbach et al., 2016) and referring expression comprehension (Hu et al., 2016; Yu et al., 2016; Nagaraja et al., 2016; Yu et al., 2017, 2018b).”
BackgroundSelf-citationTowards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2016) · International Conference on Learning Representations · Cited by 261 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and VQA.Preprint
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2017) · arXiv (Cornell University) · Cited by 252 · see citing works
Uncovering the Temporal Context for Video Question Answering
Zhu, L., Xu, Z., Yang, Y., & Hauptmann, A. G. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1033-7 · Cited by 210 · cited 4× · see citing works
“Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al., 2017), etc. Addit…”
InfluentialDeepStory: Video Story QA by Deep Embedded Memory Networks
Kim, K.-M., Heo, M.-O., Choi, S.-H., & Zhang, B. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/280 · Cited by 145 · cited 5× · see citing works
“Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al., 2017), etc. Addit…”
BackgroundMethodInfluentialNPCEditor: Creating Virtual Human Dialogue Using Information Retrieval Techniques
Leuski, A., & Traum, D. R. (2011) · AI Magazine · doi:10.1609/aimag.v32i2.2347 · Cited by 102 · cited 1× · see citing works
“Similar approaches have also been used in dialogue systems (Jafarpour et al., 2010; Leuski and Traum, 2011), picking the appropriate responses to an utterance from a predefined human conversational corpus.”
MethodMarioQA: Answering Questions by Watching Gameplay Videos
Mun, J., Seo, P. H., Jung, I., & Han, B. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.312 · Cited by 94 · cited 3× · see citing works
“Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al., 2017), etc. Addit…”
MethodMaharaj, T., Ballas, N., Rohrbach, A., Courville, A. C., & Pal, C. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.778 · Cited by 89 · cited 5× · see citing works
“Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al., 2017), etc. Addit…”
MethodInfluentialVisual Madlibs: Fill in the blank Image Generation and Question AnsweringPreprint
Yu, L., Park, E., Berg, A. C., & Berg, T. L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00278 · Cited by 75 · cited 5× · see citing works
“Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al., 2017), etc. Addit…”
BackgroundInfluentialSelf-citationObject Referring in Videos with Language and Human Gaze
Vasudevan, A. B., Dai, D., & Gool, L. V. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00434 · Cited by 62 · see citing works
Obj2Text: Generating Visually Descriptive Language from Object Layouts
Yin, X., & Ordóñez, V. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1017 · Cited by 60 · cited 1× · see citing works
“Visual Concept Features: Recent work (Yin and Ordonez, 2017) found that using detected object”
BackgroundFilter, Rank, and Transfer the Knowledge: Learning to Chat
Jafarpour, S. (2010) · Cited by 50 · see citing works
Flagged first, then by citations
3 of the 49 references couldn’t be matched to a known paper.
Research works printed but not linked: 8
- Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question an- swering. ICCV, pages 2425-2433.
- Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. 2016. Natu- ral language object retrieval. CVPR.
- Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron Courville, and Christopher Pal. 2017b. A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering. CVPR.
- Paul Over, Jon Fiscus, Greg Sanders, David Joy, Mar- tial Michel, George Awad, Alan Smeaton, Wessel Kraaij, and Georges Quénot. 2014. Trecvid 2014-an overview of the goals, tasks, data, evaluation mech- anisms and metrics. In Proceedings of TRECVID, page 52.
- Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- to-sentence models. In ICCV.
- Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018b. Mat- tnet: Modular attention network for referring expres- sion comprehension. In CVPR.
- Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016. Modeling context in referring expressions. In ECCV.
- Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. 2017. A joint speakerlistener-reinforcer model for referring expressions. In CVPR.
Cited by
Sorted by citations
From Recognition to Cognition: Visual Commonsense Reasoning
Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00688 · Cited by 769 · see citing works
“While past work has asked humans to write several counterfactual choices for each correct answer [75, 46], this process is expensive.”
BackgroundLess is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling
Lei, J., Li, L., Zhou, L., Gan, Z., Berg, T. L., Bansal, M., & Liu, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00725 · Cited by 606 · see citing works
“Popular video-andlanguage tasks include text-to-video retrieval [75, 28, 54], video captioning [75, 82, 28, 54, 40], video question answering [74, 23, 33], and moment retrieval [2, 18, 35].”
BackgroundInfluentialSelf-citationLearning 2D Temporal Adjacent Networks for Moment Localization with Natural Language
Zhang, S., Peng, H., Fu, J., & Luo, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6984 · Cited by 486 · see citing works
Zhao, F., Zhang, C., & Geng, B. (2024) · ACM Computing Surveys · doi:10.1145/3649447 · Cited by 480 · see citing works
A Corpus for Reasoning about Natural Language Grounded in Photographs
Suhr, A., Zhou, S., Zhang, A. Q., Zhang, I., Bai, H., & Artzi, Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1644 · Cited by 487 · see citing works
“, 2018) Visual Question Answering TVQA (Lei et al., 2018) Video Question Answering 4 4”
BackgroundActBERT: Learning Global-Local Video-Text Representations
Zhu, L., & Yang, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00877 · Cited by 414 · see citing works
“, video question answering [36, 10, 18, 54], video captioning [46, 52], textvideo retrieval [47, 41, 25], video grounding [50].”
BackgroundHERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
Li, L., Chen, Y.-C., Cheng, Y., Gan, Z., Yu, L., & Liu, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.161 · Cited by 408 · see citing works
Self-citationCross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering
Liu, Y., Li, G., & Lin, L. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3284038 · Cited by 354 · see citing works
“First, existing visual question answering methods usually focus on relatively simple events like movie [21], TVshow [22] or synthetic videos [23] where temporal understanding and causality discovery are simply not required to perform well, and ignore more complex and challenging events that require in-depth understand…”
BackgroundInfluentialSwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning
Lin, K., Li, L., Lin, C.-C., Ahmed, F., Gan, Z., Liu, Z., Lu, Y., & Wang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01742 · Cited by 321 · see citing works
“Although it has shown great success in video-and-language understanding tasks, such as video question answering [24] and text-tovideo retrieval [25, 56], it remains unclear whether these sparsely sampled video frames are sufficient to generate rich and descriptive captions.”
BackgroundVL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
Sung, Y.-L., Cho, J., & Bansal, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00516 · Cited by 329 · see citing works
“Table 6 shows the performance of the representative approaches used in Table 2 on TVQA, How2QA, TVC, and YC2C.”
MethodInfluentialSelf-citationLocal-Global Video-Text Interactions for Temporal Grounding
Mun, J., Cho, M., & Han, B. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01082 · Cited by 314 · see citing works
“Furthermore, with the recent advances of deep learning on top of large-scale datasets [2, 16, 17, 23], research on video content understanding is moving towards multi-modal problems (e.”
BackgroundNExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions
Xiao, J., Shang, X., Yao, A., & Chua, T. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00965 · Cited by 315 · see citing works
“Regarding visual feature representations, motion feature are important but naively concatenating appearance and motion features usually results in sub-optimal results (refer to EVQA [2], PSAC+ [26] and STVQA [27]).”
BackgroundMethodInfluentialVLP: A Survey on Vision-language Pre-training
Chen, F., Zhang, D., Han, M., Chen, X.-Y., Shi, J. C., Xu, S., & Xu, B. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1369-5 · Cited by 295 · see citing works
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Lou, P., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02095 · Cited by 300 · see citing works
“More results on NExT-QA [84], STAR [82], TVQA [35], EgoSchema [52] and IntentQA [40] can be found in the appendix.”
BackgroundResultTVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval
Lei, J., Yu, L., Berg, T. L., & Bansal, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58589-1_27 · Cited by 275 · see citing works
Self-citationHierarchical Conditional Relation Networks for Video Question Answering
Le, T. M., Le, V., Venkatesh, S., & Tran, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00999 · Cited by 260 · see citing works
BERT applications in natural language processing: a review
Gardazi, N. M., Daud, A. Y., Malik, M. K., Bukhari, A. A., Alsahfi, T., & Alshemaimri, B. K. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11162-5 · Cited by 307 · see citing works
Unifying Vision-and-Language Tasks via Text GenerationPreprint
Cho, J., Lei, J., Tan, H. X., & Bansal, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.02779 · Cited by 268 · see citing works
“, 2018b) in a wide range of discriminative (Goyal et al., 2019; Hudson & Manning, 2019; Lei et al., 2018; Mao et al., 2016; Xu et al., 2016; Zhou et al., 2018) and generative tasks (Chen et al.”
BackgroundSelf-citationVid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
Yang, A., Nagrani, A., Seo, P. H., Miech, A., Pont-Tuset, J., Laptev, I., Šivic, J., & Schmid, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01032 · Cited by 266 · see citing works
CLEVRER: CoLlision Events for Video REpresentation and ReasoningPreprint
Yi, K., Gan, C., Li, Y., Kohli, P., Wu, J., Torralba, A. B., & Tenenbaum, J. B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.01442 · Cited by 283 · see citing works
“The VCR dataset (Zellers (Tapaswi et al., 2016) × × × MarioQA (Mun et al., 2017) × × × TVQA (Lei et al., 2018) × × × × Social-IQ (Zadeh et al., 2019) × × × ×”
BackgroundInfluentialLarge-scale Multi-modal Pre-trained Models: A Comprehensive Survey
Wang, X., Chen, G., Qian, G., Gao, P., Wei, X.-Y., Wang, Y., Tian, Y., & Gao, W. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1410-8 · Cited by 245 · see citing works
“• TVQA [90] is build based on six longrunning TV shows from 3 genres, including sitcoms, medical dramas, and crime drama.”
BackgroundJust Ask: Learning to Answer Questions from Millions of Narrated Videos
Yang, A., Miech, A., Šivic, J., Laptev, I., & Schmid, C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00171 · Cited by 220 · see citing works
MERLOT: Multimodal Neural Script Knowledge ModelsPreprint
Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J., Farhadi, A., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02636 · Cited by 224 · see citing works
“The tasks we report on are: TVQA [53], TVQA(+) [54], VLEP [55], MSRVTT-QA [105], MSRVTTMultichoice [109], LSMDC-Multichoice, LSMDC fill-in-the-blank QA [97, 81], ActivityNetQA [110, 37], TGIFQA [44], and DramaQA [18]5.”
MethodInfluentialFoundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Liang, P. P., Zadeh, A., & Morency, L. (2024) · ACM Computing Surveys · doi:10.1145/3656580 · Cited by 306 · see citing works
“…recent progress in large-scale generative models has yielded impressive results in text-to-image [273, 278], text-to-video [302], audio-to-image [146], text-to-speech [276], speech-to-gesture [10], speaker-to-listener [240], language to pose [11], and speech and music generation [7, 72, 245].”
BackgroundVideo-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.02245 · Cited by 291 · see citing works
Reasoning with Heterogeneous Graph Alignment for Video Question Answering
Pin, J., & Han, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6767 · Cited by 191 · see citing works
VideoChat: chat-centric video understanding
Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, L., Wang, L., & Qiao, Y. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4321-9 · Cited by 426 · see citing works
“Please use the sequence adverbs "first", "next", "then" and "finally" to describe this video in detail, but don’t mention the specific time.”
TVQA+: Spatio-Temporal Grounding for Video Question Answering
Lei, J., Yu, L., Berg, T. L., & Bansal, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.730 · Cited by 197 · see citing works
Self-citationAlign and Prompt: Video-and-Language Pre-training with Entity Prompts
Li, D., Li, J., Li, H., Niebles, J. C., & Hoi, S. C. H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00490 · Cited by 178 · see citing works
“Most prior methods on text-video retrieval [13,31,42,52,54] and videoQA [11, 14, 24, 26] employ pre-trained visual backbones and extract video features for each frame densely yet offline.”
MethodNuScenes-QA: A Multi-Modal Visual Question Answering Benchmark for Autonomous Driving Scenario
Qian, T., Chen, J., Zhuo, L., Jiao, Y., & Jiang, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28253 · Cited by 177 · see citing works
Location-Aware Graph Convolutional Networks for Video Question Answering
Huang, D. Y., Chen, P., Zeng, R., Du, Q., Tan, M., & Gan, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6737 · Cited by 174 · see citing works
“Based on the visual sources, we can classify the VQA tasks into two categories: image QA (Goyal et al. 2017; Gan et al. 2017) and video QA (Lei et al. 2018; Yi et al. 2019).”
BackgroundMethodAudio Visual Scene-Aware Dialog
Alamri, H., Cartillier, V., Das, A., Wang, J., Cherian, A., Essa, I. A., Batra, D., Marks, T. K., Hori, C., Anderson, P., et al. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00774 · Cited by 172 · see citing works
“While AVSD has fewer unique video clips compared to TVQA and MovieQA, which are curated from television and film, our videos are more naturalistic.”
BackgroundSupport-set bottlenecks for video-text representation learning
Patrick, M., Huang, P.-Y., Asano, Y. M., Metze, F., Hauptmann, A. G., Henriques, J., & Vedaldi, A. (2020) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.48550/arxiv.2010.02824 · Cited by 192 · see citing works
“0 JSFusion (Yu et al., 2018) 10 .”
BackgroundBoundary Proposal Network for Two-stage Natural Language Video Localization
Xiao, S., Chen, L., Zhang, S., Ji, W., Shao, J., Ye, L., & Xiao, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16406 · Cited by 164 · see citing works
QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
Rogers, A., Gardner, M., & Augenstein, I. (2022) · ACM Computing Surveys · doi:10.1145/3560260 · Cited by 177 · see citing works
“, [132, 330]), and the source of text used for generating those question matters a lot: the audio descriptions tend to focus on visual features, and text summaries focus on the plot [159].”
BackgroundMethodMERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound
Zellers, R., Lu, J., Lu, X., Yu, Y., Zhao, Y., Salehi, M., Kusupati, A., Hessel, J., Farhadi, A., & Choi, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01589 · Cited by 163 · see citing works
“[74] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg.”
BackgroundMethodInfluentialVision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155 · see citing works
Interventional Video Grounding with Dual Contrastive Learning
Nan, G., Qiao, R., Xiao, Y., Liu, J., Leng, S., Zhang, H., & Lu, W. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00279 · Cited by 149 · see citing works
Revisiting the “Video” in Video-Language Understanding
Buch, S., Eyzaguirre, C., Gaidon, A., Wu, J., Fei-Fei, L., & Niebles, J. C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00293 · Cited by 139 · see citing works
“Standard video-language tasks include both discriminative tasks, such as video question answering [12, 19, 27, 28, 30, 54, 56, 58, 59], text-to-video/moment retrieval [16,23,43,56,62], and generative tasks, such as video cap-tioning [4, 23] and open-ended VQA [54, 56].”
BackgroundMulla, N., & Gharpure, P. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00295-9 · Cited by 132 · see citing works
“Overall, they obtained promising results on the TVQA [129] and ActivityNetQA [130] datasets.”
ResultKaleido-BERT: Vision-Language Pre-training on Fashion Domain
Zhuge, M., Gao, D., Fan, D.-P., Jin, L., Chen, B., Zhou, H., Qiu, M., & Shao, L. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01246 · Cited by 128 · see citing works
Learning to Answer Questions in Dynamic Audio-Visual Scenarios
Li, G., Wei, Y., Tian, Y., Xu, C., Wen, J.-R., & Hu, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01852 · Cited by 124 · see citing works
“Most video QA datasets, like ActivityNet-QA [54], TVQA [29], only contain visual question and provide limited possibility to explore audio-visual correlation.”
MethodInfluentialVideo Understanding With Large Language Models: A Survey
Tang, Y., Bi, J., Xu, S., Song, L., Liang, S., Wang, T., Zhang, D., An, J., Lin, J., Zhu, R., et al. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3566695 · Cited by 174 · see citing works
“TVQA [197].”
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment
Yang, J., Bisk, Y., & Gao, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01136 · Cited by 115 · see citing works
“Realistic application scenarios around videos have prompted emergence of various video-language tasks, such as text-video retrieval [29, 54, 52], video question answering [20, 26], video captioning [53, 58], etc.”
BackgroundCausal Inference Meets Deep Learning: A Comprehensive Survey
Jiao, L., Wang, Y., Liu, X., Li, L., Liu, F., Ma, W., Guo, Y., Chen, P., Yang, S., & Hou, B. (2024) · Research · doi:10.34133/research.0467 · Cited by 122 · see citing works
BERT Representations for Video Question Answering
Yang, Z., García, N., Chu, C., Otani, M., Nakashima, Y., & Takemura, H. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093596 · Cited by 108 · see citing works
“For comparison, we report results on TVQA [21], STAGE [22] and MDAM [17].”
BackgroundMethodInfluentialUncertainty Guided Collaborative Training for Weakly Supervised Temporal Action Detection
Yang, W., Zhang, T., Yu, X., Tian, Q. C., Zhang, Y., & FengWu, F. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00012 · Cited by 110 · see citing works
“Because of its broad application potentials in video summarization [22, 16], video surveillance [17], visual question answering [23], and others, it has attracted more and more attention from both academia and industry in recent years.”
BackgroundBridge to Answer: Structure-aware Graph Interaction Network for Video Question Answering
Park, J., Lee, J., & Sohn, K. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01527 · Cited by 105 · see citing works
Video Graph Transformer for Video Question Answering
Xiao, J., Zhou, P., Chua, T., & Yan, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_3 · Cited by 99 · see citing works
Yu, J., Zhang, W., Lu, Y., Qin, Z., Hu, Y., Tan, J., & Wu, Q. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2972830 · Cited by 96 · see citing works
Ishmam, M. F., Shovon, M. S. H., Mridha, M. F., & Dey, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102270 · Cited by 100 · see citing works
“Being a multi-modal domain, visual question answering naturally works with visual and textual modalities but also works with auditory input as seen in multimodal video question answering [5].”
BackgroundMethodInfluentialSocial-IQ: A Question Answering Benchmark for Artificial Social Intelligence
Zadeh, A., Chan, M., Liang, P. P., Tong, E., & Morency, L. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00901 · Cited by 98 · see citing works
“In machine learning, this form of probing has a well established precedence in multiple different areas ranging from understanding books and text [25], to understanding events in the movies [26].”
BackgroundMethodInfluentialMIST : Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering
Gao, D., Zhou, L., Ji, L., Zhu, L., Yang, Y., & Shou, M. Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01419 · Cited by 96 · see citing works
“Some others, such as TVQA [25], MovieQA [39], and Social-IQ [50], mainly focus on long videos cropped from movies or TV series for evaluating the understanding of the plot and social interactions, where subtitles play an essential role.”
BackgroundVIOLET : End-to-End Video-Language Transformers with Masked Visual-token ModelingPreprint
Fu, T.-J., Li, L., Gan, Z., Lin, K., Wang, W. Y., Wang, L., & Liu, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12681 · Cited by 130 · see citing works
Fine-grained Iterative Attention Network for Temporal Language Localization in Videos
Qu, X., Tang, P., Zou, Z., Cheng, Y., Dong, J., Zhou, P., & Xu, Z. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3414053 · Cited by 92 · see citing works
“rategy to better predict the temporal boundary. Extensive experiments on three real-world datasets validate the effectiveness of our method. Thefutureworkincludesapply FIANtomorecomplicatedbenchmarks [17, 20]. Combining our method with some pre-training model [18] could also further boost the performance. 7 ACKNOWLEDG…”
MethodTrends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
Mogadala, A., Kalimuthu, M., & Klakow, D. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11688 · Cited by 128 · see citing works
“2 TVQA The TVQA28 (Lei et al., 2018) is created using six different TV shows such as Friends, The Big Bang Theory, How I Met Your Mother, House M.”
MethodProgressive Attention Memory Network for Movie Story Question Answering
Kim, J., Ma, M., Kim, K., Kim, S.-J., & Yoo, C. D. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00853 · Cited by 90 · see citing works
“To this end, various video story QA benchmarks such as PororoQA [16], MeMexQA [12], TVQA [17] and MovieQA [26] have been suggested.”
BackgroundMethodAGQA: A Benchmark for Compositional Spatio-Temporal Reasoning
Grunde-McLaughlin, M., Krishna, R., & Agrawala, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01113 · Cited by 89 · see citing works
Xu, L., Huang, H., & Liu, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00975 · Cited by 89 · see citing works
“In this manner, generally, a fixed computation architecture [29, 12, 21, 10] can be applied over the whole video for tackling each question.”
BackgroundMethodInfluentialGrounding-Tracking-Integration
Yang, Z., Kumar, T., Chen, T., Su, J., & Luo, J. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3038720 · Cited by 88 · see citing works
Video Question Answering: Datasets, Algorithms and Challenges
Zhong, Y., Ji, W., Xiao, J., Li, Y., Deng, W., & Chua, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.432 · Cited by 83 · see citing works
Modality Shifting Attention Network for Multi-Modal Video Question Answering
Kim, J., Ma, M., Pham, T. X., Kim, K., & Yoo, C. D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01012 · Cited by 82 · see citing works
Reading-Strategy Inspired Visual Representation Learning for Text-to-Video Retrieval
Dong, J., Wang, Y., Chen, X., Qu, X., Li, X., He, Y., & Wang, X. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3150959 · Cited by 84 · see citing works
“Additionally, comparing the models pre-trained on TV dataset [71]”
MethodInformation fusion in visual question answering: A Survey
Zhang, D., Cao, R., & Wu, S. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.03.005 · Cited by 81 · see citing works
Loss Re-Scaling VQA: Revisiting the Language Prior Problem From a Class-Imbalance View
Guo, Y., Nie, L., Cheng, Z., Tian, Q., & Zhang, M. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3128322 · Cited by 79 · see citing works
Dual-Evidential Learning for Weakly-supervised Temporal Action Localization
Chen, M., Gao, J., Yang, S., & Xu, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19772-7_12 · Cited by 80 · see citing works
“It has attracted significant attention from both academia and industry, due to the great potential for video retrieval [9, 41], summarization [24], surveillance [18, 51], anomaly detection [50], visual question answering [25], to name a few.”
BackgroundPano-AVQA: Grounded Audio-Visual Question Answering on 360° Videos
Yun, H., Yu, Y., Yang, W., Lee, K., & Kim, G. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00204 · Cited by 78 · see citing works
VideoLLM-online: Online Video Large Language Model for Streaming Video
Chen, J., Lv, Z., Wu, S., Lin, K. Q., Song, C., Gao, D., Liu, J.-W., Gao, Z., Mao, D., & Shou, M. Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01742 · Cited by 83 · see citing works
“Typical video understanding benchmarks, such as action recognition [10], temporal action localization [31], video question answering [41], and video dialogue [2], typically allow models to access entire video frames to make predictions, a setting referred to as ”offline”.”
BackgroundFrozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
Bain, M., Nagrani, A., Varol, G., & Zisserman, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00175 · Cited by 76 · see citing works
“Joint visual-text models have become increasingly popular as they enable a wide suite of downstream tasks, including text-to-visual retrieval [29, 31, 36, 59], visual captioning [24, 58, 66], and visual question and answering [4, 27].”
BackgroundWebQA: Multihop and Multimodal QA
Chang, Y., Cao, G., Narang, M., Gao, J., Suzuki, H., & Bisk, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01600 · Cited by 82 · see citing works
“We provide annotators with four snippets to prevent and allow them to contribute facts they researched to help answer the question.”
MethodRevealing Single Frame Bias for Video-and-Language Learning
Lei, J., Berg, T. L., & Bansal, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.29 · Cited by 90 · see citing works
Self-citationLearnable Aggregating Net with Diversity Learning for Video Question Answering
Li, X., Gao, L., Wang, X., Liu, W., Xu, X., Shen, H. T., & Song, J. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350971 · Cited by 72 · see citing works
“Firstly, we extend single-path co-attention to multi-path pyramid co-attention with diversity learning, which explicitly encourage attended more diverse.”
BackgroundKnowIT VQA: Answering Knowledge-Based Questions about Videos
García, N., Otani, M., Chu, C., & Nakashima, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6713 · Cited by 4 · see citing works
TEVAD: Improved video anomaly detection with captions
Chen, W., Ma, K. T., Yew, Z. J., Hur, M., & Khoo, D. A.-A. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00587 · Cited by 77 · see citing works
“More recently, with the success of transformer models [10, 52] in natural language processing fields, the computer vision community has tried to apply the ideas on different downstream tasks and achieved promising results [5, 18, 28, 29, 33, 67].”
BackgroundTarget Adaptive Context Aggregation for Video Scene Graph Generation
Teng, Y., Wang, L., Li, Z., & Wu, G. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01343 · Cited by 74 · see citing works
“This structured video representation, known as video scene graph [27], can contribute to more accurate action recognition [10] and allow for our vision models to tackle high-level and complex inference tasks, such as video caption [7, 36], video retrieval [4, 16], and video question answering [33, 15].”
BackgroundLearning Audio-Video Modalities from Image Captions
Nagrani, A., Seo, P. H., Seybold, B., Hauth, A., Manén, S., Sun, C., & Schmid, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19781-9_24 · Cited by 68 · see citing works
Proposal-Based Multiple Instance Learning for Weakly-Supervised Temporal Action Localization
Ren, H., Yang, W., Zhang, T., & Zhang, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00237 · Cited by 69 · see citing works
“TAL has recently received increasing attention from the research community due to its broad application potentials, such as intelligent surveillance [44], video summarization [22], highlight detection [49], and visual question answering [23].”
Multi-Scale 2D Temporal Adjacency Networks for Moment Localization With Natural Language
Zhang, S., Peng, H., Fu, J., Lu, Y., & Luo, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3120745 · Cited by 67 · see citing works
“Video moment localization with natural language has a wide range of applications, such as video question answering [14], video content retrieval [15], dense video captioning [16], aswell as video storytelling [17].”
VLG-Net: Video-Language Graph Matching Network for Video Grounding
Soldan, M., Xu, M., Qu, S., Tegner, J. N., & Ghanem, B. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00361 · Cited by 67 · see citing works
“This novel inter-disciplinary task has gained momentum within the vision and language communities for its relevance and possible applications in video retrieval [13, 52, 75], video question answering [21, 28], human-computer interaction [82], and video storytelling [19].”
BackgroundCan I Trust Your Answer? Visually Grounded Video Question Answering
Xiao, J., Yao, A., Li, Y., & Chua, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01254 · Cited by 72 · see citing works
“TVQA [25]) or limited to few objects ( e.g .”
BackgroundMethodInfluentialVindLU: A Recipe for Effective Video-and-Language Pretraining
Cheng, F., Wang, X., Lei, J., Crandall, D., Bansal, M., & Bertasius, G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01034 · Cited by 65 · see citing works
“In Table 6, we also present our results for the video question-answering task on ActivityNet-QA [85], MSRVTT-QA [75], MSRVTTMC [84] and TVQA [33].”
BackgroundResultInfluentialSelf-citationMultimodal Emotion-Cause Pair Extraction in Conversations
Wang, F., Ding, Z., Xia, R., Li, Z., & Yu, J. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3226559 · Cited by 66 · see citing works
“For example, exploring visual commonsense reasoning [42], [43], [44], a new direction in computer vision, might be helpful for MECPE.”
BackgroundSlideVQA: A Dataset for Document Visual Question Answering on Multiple Images
Tanaka, R., Nishida, K., Nishida, K., Hasegawa, T., Saito, I., & Saito, K. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26598 · Cited by 63 · see citing works
Violin: A Large-Scale Dataset for Video-and-Language Inference
Liu, J., Chen, W., Cheng, Y., Gan, Z., Yu, L., Yang, Y., & Liu, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01091 · Cited by 62 · see citing works
Self-citationVisual Semantic Role Labeling for Video Understanding
Sadhu, A., Gupta, T., Yatskar, M., Nevatia, R., & Kembhavi, A. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00554 · Cited by 63 · see citing works
“Segments ActivityNet[24], Vatex[77], YouCook[13], MSR-VTT [82] , LSMDC [60] Video QA Q/A, Subtitle or Script (optional) MSRVTT-QA[81], VideoQA[86], ActivityNetQA[85], TVQA[37], MovieQA[70] Text to Video Retrieval Text Query, ASR output (optional) HowTo100M[50], TVR[38], DiDeMo[25], Charades-STA[17] Video Object Ground…”
MethodLi, J., Niu, L., & Zhang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02059 · Cited by 63 · see citing works
“This is different from the popular TVQA [27], MovieQA [40], and CLEVRER [52] dataset that are based on some unrealistic scenes (e.”
MethodInfluentialVideo Face Clustering With Unknown Number of Clusters
Tapaswi, M., Law, M. T., & Fidler, S. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00513 · Cited by 62 · see citing works
“Understanding characters also has a direct influence on important research such as video captioning [34, 35], question-answering [22, 44], studying social situations [45] and 4D effects [56].”
BackgroundVideo2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning
Fang, Z., Gokhale, T., Banerjee, P., Baral, C. R., & Yang, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.61 · Cited by 63 · see citing works
Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering
Seo, A., Kang, G.-C., Park, J., & Zhang, B. (2021) · doi:10.18653/v1/2021.acl-long.481 · Cited by 61 · see citing works
“In particular, video question answering (video QA) tasks (Xu et al., 2016; Jang et al., 2017; Lei et al., 2018; Yu et al., 2019; Choi et al., 2020) have been proposed with the goal of reasoning over higher-level vision-”
Learning Interactions and Relationships Between Movie Characters
Kukleva, A., Tapaswi, M., & Laptev, I. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00987 · Cited by 63 · see citing works
Document Understanding Dataset and Evaluation (DUDE)
Landeghem, J. V., Powalski, R., Tito, R., Jurkiewicz, D., Blaschko, M. B., Borchmann, Ł., Coustaty, M., Moens, S., Pietruszka, M., Ackaert, B., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01789 · Cited by 70 · see citing works
“For example, the question-answering approach is combined with modalities such as videos [44, 13, 14, 28, 17], images [98, 3, 29, 68, 7, 8], speech [99, 43], knowledge graphs [93, 84, 80, 22, 37], and maps [70, 15].”
MethodEmoSen: Generating Sentiment and Emotion Controlled Responses in a Multimodal Dialogue System
Firdaus, M., Chauhan, H., Ekbal, A., & Bhattacharyya, P. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3015491 · Cited by 59 · see citing works
“Similar to [53], we sample dialogue from a given scene while constraining video clip length to 60 seconds for the comedy based TV series and 90 seconds for drama TV series.”
BackgroundMethodA Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC
Yatskar, M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1241 · Cited by 56 · see citing works
“Other work considers multi-modal contexts where interactions are a single turn (Tapaswi et al., 2016; Antol et al., 2015; Lei et al., 2018) or multi-turn (Das et al.”
BackgroundVX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs
Lin, X., Bertasius, G., Wang, J., Chang, S., Parikh, D., & Torresani, L. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00693 · Cited by 57 · see citing works
“Prime examples include datasets for image or video captioning [10, 38, 51, 24, 56, 28], question answering (QA) [5, 13, 54, 58, 19, 36, 46, 26], as well as audio-visual dialog [11, 1].”
BackgroundMethodResultInfluentialA Survey on Relation Extraction
Detroja, K., Bhensdadia, C. K., & Bhatt, B. (2022) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2023.200244 · Cited by 57 · see citing works
Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Grounding
Zhou, H., Zhang, C., Luo, Y., Chen, Y., & Hu, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00834 · Cited by 54 · see citing works
“video captioning [26, 41, 5], video object segmentation [7, 13] and video question answering [19, 14, 35].”
BackgroundRelation-aware Video Reading Comprehension for Temporal Language Grounding
Gao, J., Sun, X., Xu, M., Zhou, X., & Ghanem, B. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.324 · Cited by 60 · see citing works
“It has also drawn great attention from industry due to its various applications such as video question answering (Huang et al., 2020; Lei et al., 2018), video content retrieval (Dong et al.”
An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling
Fu, T.-J., Li, L., Gan, Z., Lin, K., Wang, W. Y., Wang, L., & Liu, Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02193 · Cited by 57 · see citing works
“Researchers have explored such capability on VidL tasks including text-to-video retrieval [80, 30, 59, 38, 40], video question answering [26, 79, 36, 37], moment retrieval [24, 20, 30, 38], and video captioning [74, 87, 80, 59].”
BackgroundSupport-Set Based Cross-Supervision for Video Grounding
Ding, X., Wang, N., Zhang, S., Cheng, D., Li, X., Huang, Z., Tang, M., & Gao, X. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01137 · Cited by 53 · see citing works
“Some works use the contrastive loss to improve video-text representations based on HowTo100M for tasks such as video caption [53], video retrieval [2] and video question answering [24].”
MethodInteractive Question Answering Systems: Literature Review
Biancofiore, G. M., Deldjoo, Y., Noia, T. D., Sciascio, E. D., & Narducci, F. (2024) · ACM Computing Surveys · doi:10.1145/3657631 · Cited by 57 · see citing works
“[63] to be compositional in nature, requiring systems to recognize relevant moments within clips, comprehend subtitles and visual concepts to adequately respond.”
Background
- Authors
- Lei, J., Yu, L., Bansal, M., & Berg, T.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 1369-1379
- OpenAlex
- W2963541336
- Cited by
- 541
- References
- 49
- Citations by year
- 2026: 39 · 2025: 50 · 2024: 65 · 2023: 93 · 2022: 90 · 2021: 101 · 2020: 70 · 2019: 32 · 2018: 1
- FWCI
- 15.13
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1405822, 1562098, 1633295)
- Army Research Office (W911NF)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- video question answering, video localization, television series, end-to-end learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
