The DOI matches its Crossref record; 47 of the 49 references it cites were found in a registry or an index.
TVQA: Localized, Compositional Video Question Answering
Open accessAbstract
Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks.However, due to data limitations, there has been much less work on video-based QA.In this paper, we present TVQA, a largescale video QA dataset based on 6 popular TV shows.TVQA consists of 152,545 QA pairs from 21,793 clips, spanning over 460 hours of video.Questions are designed to be compositional in nature, requiring systems to jointly localize relevant moments within a clip, comprehend subtitle-based dialogue, and recognize relevant visual concepts.We provide analyses of this new dataset as well as several baselines and a multi-stream end-to-end trainable neural network framework for the TVQA task.The dataset is publicly available at http://tvqa.cs.unc.edu.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Overall, they obtained promising results on the TVQA [129] and ActivityNetQA [130] datasets.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [129]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“We also use ResNet101 (He et al., 2016) trained on ImageNet (Deng et al., 2009) to extract whole image features.”
MethodFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 5×
“Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al., 2017), etc. Addit…”
BackgroundMethodInfluentialFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 5×
“ImageNet Features: We extract the pooled 2048D feature of the last block of ResNet101.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 4×
“A subtitle S, which contains a set of sentences, is flattened into a long sequence of words and GloVe (Pennington et al., 2014) is used to embed the words.”
BackgroundMethodInfluentialSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 2×
“With the same goal in mind, Rajpurkar et al. (2016) introduced the SQuAD dataset, but their answers are specific spans from long passages.”
BackgroundVQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., & Gool, L. V. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46484-8_2 · Cited by 4,232
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855 · cited 1×
“Text Question Answering: The related task of text-based question answering has been extensively explored (Richardson et al., 2013; Weston et al., 2015; Rajpurkar et al., 2016; Hermann et al., 2015; Hill et al., 2015).”
BackgroundCLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Flagged first, then by citations
2 of the 49 references couldn’t be matched to a known paper.
Research works printed but not linked: 8
- Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question an- swering. ICCV, pages 2425-2433.
- Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. 2016. Natu- ral language object retrieval. CVPR.
- Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron Courville, and Christopher Pal. 2017b. A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering. CVPR.
- Paul Over, Jon Fiscus, Greg Sanders, David Joy, Mar- tial Michel, George Awad, Alan Smeaton, Wessel Kraaij, and Georges Quénot. 2014. Trecvid 2014-an overview of the goals, tasks, data, evaluation mech- anisms and metrics. In Proceedings of TRECVID, page 52.
- Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- to-sentence models. In ICCV.
- Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018b. Mat- tnet: Modular attention network for referring expres- sion comprehension. In CVPR.
- Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016. Modeling context in referring expressions. In ECCV.
- Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. 2017. A joint speakerlistener-reinforcer model for referring expressions. In CVPR.
Cited by
Sorted by citations
Items 101–200 of 534
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding EvaluationPreprint
Li, L., Lei, J., Gan, Z., Yu, L., Chen, Y.-C., Pillai, R., Cheng, Y., Zhou, L., Wang, X. E., Wang, W. Y., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.04632 · Cited by 90
“To construct a comprehensive evaluation benchmark, we include recent datasets collected on multi-channel videos: TVR [35], How2R [36], TVQA [32], How2QA [36], VIOLIN [40], VLEP [34] and TVC [35].”
BackgroundMethodInfluentialSelf-citationA dataset for medical instructional video classification and question answering
Gupta, D., Attal, K., & Demner‐Fushman, D. (2023) · Scientific Data · doi:10.1038/s41597-023-02036-y · Cited by 62
“In recent years there has been an increasing interest in video question-answering [7, 8] tasks, where given a video, the systems are expected to retrieve the answer to a natural language question about the content in the video.”
BackgroundComputational Media Intelligence: Human-Centered Machine Analysis of Media
Somandepalli, K., Guha, T., Martínez, V. R., Kumar, N., Adam, H., & Narayanan, S. S. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2020.3047978 · Cited by 51
Temporal Reasoning via Audio Question Answering
Fayek, H. M., & Johnson, J. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3010650 · Cited by 57
CONQUER: Contextual Query-aware Ranking for Video Corpus Moment Retrieval
Hou, Z., Ngo, C., & Chan, W. K. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475281 · Cited by 50
“Since the deep-learning era, especially due to the popularity of attention mechanism [29] and transformer [38], learning query-aware or question-aware features have been explored, for example, in multimedia question-answering [2, 18].”
Compositional Attention Networks With Two-Stream Fusion for Video Question Answering
Yu, T., Yu, J., Zhou, Y., & Tao, D. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2940677 · Cited by 51
“The rich annotation question was compositional and the second part of question was used to localize relevant moments that make TVQA dataset supports a secondary task of temporal localization [54].”
BackgroundAction-Centric Relation Transformer Network for Video Question Answering
Zhang, J., Shao, J., Cao, R., Gao, L., Xu, X., & Shen, H. T. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3048440 · Cited by 50
Survey: Transformer based video-language pre-training
Ruan, L., & Jin, Q. (2022) · AI Open · doi:10.1016/j.aiopen.2022.01.001 · Cited by 56
An Efficient Framework for Dense Video Captioning
Suin, M., & Rajagopalan, A. N. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6881 · Cited by 49
“Additionally, following (Lei et al. 2018), we explore the use of object information in feature encoding by adding another semantically rich object embedding.”
MethodA Survey on Temporal Sentence Grounding in Videos
Lan, X., Yuan, Y.-T., Wang, X., Wang, Z., & Zhu, W.-W. (2021) · ACM Trans. Multim. Comput. Commun. Appl. · doi:10.1145/3532626 · Cited by 70
“Moreover, it could also serve as an intermediate task for various downstream vision-and-language tasks such as video question answering [28, 48, 50, 107] and video summarization [23, 66, 81, 123, 140].”
BackgroundVideo Question Answering with Spatio-Temporal Reasoning
Jang, Y., Song, Y., Kim, C. D., Yu, Y., Kim, Y., & Kim, G. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01189-x · Cited by 46
“For our text-video architecture (STVQA-TextVideo), we set it to the last hidden state of the text encoder, hqT (explained next).”
BackgroundInfluentialTowards Visual-Prompt Temporal Answer Grounding in Instructional Video
Shutao Li, A., Li, B., Sun, B., & Weng, Y. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3411045 · Cited by 55
“Among numerous tasks in computer vision, the video question answering (VQA) [9] and the temporal sentence grounding in general video (TSGV) [10] are the most similar ones to the TAGV.”
RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
Sermanet, P., Ding, T., Zhao, J. W., Xia, F., Dwibedi, D., Gopalakrishnan, K., Chan, C., Dulac-Arnold, G., Maddineni, S., Joshi, N. J., et al. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610216 · Cited by 108
“Recently many large-scale video datasets have been introduced [23, 24, 25, 26, 27, 28, 29, 30] that include videos of humans performing tasks with text narrations or question-answer annotations.”
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization
Tang, Z., Lei, J., & Bansal, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.193 · Cited by 48
Self-citationWhat is More Likely to Happen Next? Video-and-Language Future Event Prediction
Lei, J., Yu, L., Berg, T. L., & Bansal, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.706 · Cited by 63
Self-citationNatural Language Video Localization with Learnable Moment Proposals
Xiao, S., Chen, L., Shao, J., Zhuang, Y., & Xiao, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.327 · Cited by 46
“Moreover, NLVL is helpful to numerous downstream video understanding tasks, e.g. , content retrieval (Shao et al., 2018), relation detection (Gao et al., 2021), and VQA (Lei et al., 2018; Ye et al., 2017).”
BackgroundMMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering
Urooj, A., Mazaheri, A., Lobo, N. D. V., & Shah, M. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.417 · Cited by 49
MSAF: Multimodal Split Attention FusionPreprint
Su, L., Hu, C., Li, G., & Cao, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.07175 · Cited by 72
“rks. 1. Introduction Multimodal learning has been explored in numerous machine learning applications such as audio-visual speech recognition [33], action recognition [2], and video question answering [21], where each modality contains useful information from a different perspective. Although these tasks can benefit fro…”
BackgroundA Universal Quaternion Hypergraph Network for Multimodal Video Question Answering
Guo, Z., Zhao, J., Jiao, L., Liu, X., & Liu, F. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3120544 · Cited by 45
“The TVQA [3] dataset is based on 6 TV shows: BBT, Friends, HIMYM,Grey,House,andCastle.”
MethodAn Image Grid Can Be Worth a Video: Zero-Shot Video Question Answering Using a VLM
Kim, W., Choi, C., Lee, W., & Rhee, W. (2024) · IEEE Access · doi:10.1109/access.2024.3517625 · Cited by 101
“For multiple-choice VQA, we assess performance on NExT-QA [45], STAR [44], TVQA [17], IntentQA [18], and EgoSchema [32].”
MethodAdaptive Spatio-Temporal Graph Enhanced Vision-Language Representation for Video QA
Jin, W., Zhao, Z., Cao, X., Zhu, J., He, X., & Zhuang, Y. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3076556 · Cited by 41
“Lei et al. [44] introduced a multi-stream end-to-end network and utilized a recurrent neural network to fuse them.”
BackgroundVideo Object Grounding Using Semantic Roles in Language Description
Sadhu, A., Chen, K., & Nevatia, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01043 · Cited by 36
“It is a building block for downstream grounded vision+language tasks such as Grounded-VQA [14,31,32,74,77], Grounded-Captioning [35–37 In this work, we address the task of Video Object Grounding (VOG): given a video and its natural language description we aim to localize each referred object.”
MethodDeep Vision Multimodal Learning: Methodology, Benchmark, and Trend
Chai, W., & Wang, G. (2022) · Applied Sciences · doi:10.3390/app12136588 · Cited by 48
End-to-End Video Question-Answer Generation With Generator-Pretester Network
Su, H.-T., Chang, C.-H., Shen, P.-W., Wang, Y.-S., Chang, Y.-L., Chang, Y.-C., Cheng, P.-J., & Hsu, W. H. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3051277 · Cited by 38
Video Question Answering With Prior Knowledge and Object-Sensitive Learning
Zeng, P., Zhang, H., Gao, L., Song, J., & Shen, H. T. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3205212 · Cited by 39
“[14], [15] propose a temporal attention mech-160 anism to select the key information through questions as guidance.”
BackgroundKnowledge-Based Video Question Answering with Unsupervised Scene Descriptions
García, N., & Nakashima, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58523-5_34 · Cited by 40
Large Language Models are Temporal and Causal Reasoners for Video Question Answering
Ko, D., Lee, J. E., Kang, W., Roh, B., & Kim, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.261 · Cited by 66
ManyModalQA: Modality Disambiguation and QA over Diverse Inputs
Hannan, D., Jain, A., & Bansal, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6294 · Cited by 37
Self-citationVideo-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models
Ning, M., Zhu, B., Xie, Y., Lin, B., Cui, J., Yuan, L., Chen, D., & Yuan, L. (2023) · arXiv · doi:10.48550/arxiv.2311.16103 · Cited by 36
Self-supervised Pre-training and Contrastive Representation Learning for Multiple-choice Video QA
Kim, S., Jeong, S., Kim, E., Kang, I.-H., & Kwak, N. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17556 · Cited by 36
Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering
Lipping, S., Sudarsanam, P., Drossos, K., & Virtanen, T. I. (2022) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco55093.2022.9909680 · Cited by 34
Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA
Kim, H., Tang, Z., & Bansal, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.435 · Cited by 34
Self-citationMultimodal Conversational AI: A Survey of Datasets and Approaches
Sundar, A., & Heck, L. P. (2022) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2022.nlp4convai-1.12 · Cited by 42
Locate Before Answering: Answer Guided Question Localization for Video Question Answering
Qian, T., Cui, R., Chen, J., Peng, P., Guo, X., & Jiang, Y. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3323878 · Cited by 37
“In recent years, several datasets have emerged for VideoQA with untrimmed long-term videos, such as NExT-QA [14], ActivityNet-QA [15], AGQA [19], and TVQA [27].”
BackgroundX-VARS: Introducing Explainability in Football Refereeing with Multi-Modal Large Language Models
Held, J., Itani, H., Cioppa, A., Giancola, S., Ghanem, B., & Droogenbroeck, M. V. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00332 · Cited by 40
AutoAD II: The Sequel – Who, When, and What in Movie Audio Description
Han, T., Bain, M., Nagrani, A., Varol, G., Xie, W., & Zisserman, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01255 · Cited by 39
“Dense labelling of characters in movies and TV shows enables the modeling of interactions, relationships, and intentions – which can be formulated into classification [27], question answering [29], or captioning [30] tasks.”
BackgroundBiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialogues
Le, H., Sahoo, D., Chen, N. F., & Hoi, S. C. H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.145 · Cited by 33
Robust Referring Video Object Segmentation with Cyclic Structural Consensus
Li, X., Wang, J., Xu, X., Li, X., Raj, B., & Lu, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02032 · Cited by 32
“Different from other vision-language tasks, e.g., image-text retrieval [22, 24, 31] and video question answering [13, 36], R-VOS needs to construct object-level multimodal semantic consensus in a dense visual representation.”
BackgroundOn Modality Bias in the TVQA Dataset
Winterbottom, T., Xiao, S., McLean, A., & Moubayed, N. A. (2020) · British Machine Vision Conference (BMVC) · doi:10.5244/c.34.122 · Cited by 39
MMCoQA: Conversational Question Answering over Text, Tables, and Images
Li, Y., Li, W., & Nie, L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.290 · Cited by 37
“Besides, Textbook QA (Kemb-havi et al., 2017) and TVQA (Lei et al., 2018) were also explored under specific scenes.”
BackgroundA Qualitative Comparison of CoQA, SQuAD 2.0 and QuACPreprint
Yatskar, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10735 · Cited by 46
“Other work considers multi-modal contexts where interactions are a single turn (Tapaswi et al., 2016; Antol et al., 2015; Lei et al., 2018) or multi-turn (Das et al.”
BackgroundOmniNet: A unified architecture for multi-modal multi-task learningPreprint
Pramanik, S., Agrawal, P., & Hussain, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.07804 · Cited by 50
“Similarly, multi-modal learning has been essential for solving a broad range of interesting problems such as Visual Question Answering [19, 20] and Video Question Answering [21].”
Learning from Inside: Self-driven Siamese Sampling and Reasoning for Video Question Answering
Yu, W., Zheng, H., Li, M., Ji, L., Wu, L., Xiao, N., & Duan, N. (2021) · neural information processing systems · Cited by 30
Cross-modal text and visual generation: A systematic review. Part 1: Image to text
Żelaszczyk, M., & Mańdziuk, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.01.008 · Cited by 31
Uncertainty-Aware Dual-Evidential Learning for Weakly-Supervised Temporal Action Localization
Chen, M., Gao, J., & Xu, C. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3308571 · Cited by 29
“…its vast potential in video retrieval [5], [6], summarization [7], surveillance [8], [9], anomaly detection [10], [11], visual question answering [12], to mention a few, numerous TAL algorithms have been developed and have achieved out-standing performance under the fully-supervised setting,…”
BackgroundYe, S., Chen, D., Han, S., & Liao, J. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3225327 · Cited by 38
“…standard VQA, there are some extensions of VQA to other sub-areas, such as diagrams and document analysis [8], [9], [12], [15], video understanding [10], [11], [13], [14], multiview and viewpoint selection [23], knowledge-based question answering [14], [16], [17], visual commonsense reasoning…”
BackgroundAdversarial Multimodal Network for Movie Story Question Answering
Yuan, Z., Sun, S., Duan, L., Li, C., Wu, X. Y., & Xu, C. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3002667 · Cited by 32
“This work focuses on multimodal movie story question answering [20], [22]–[25], which requires the agent to comprehend the story of a movie (or TV show) based on aligned visual clips and textual subtitles to infer the correct answer from multiple candidate choices, as shown in Fig.”
BackgroundMethodInfluentialChadha, A. R., Arora, G., & Kaloty, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.07735 · Cited by 36
“Learned Proposals B@3 B@4 M B@3 B@4 M Seen full dataset Krishna et al. [27] 4 We train and evaluate iPerceive VideoQA using the TVQA dataset [29] which consists of video frames, subtitles and QA pairs from six TV shows.”
MethodM5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining
Dong, X. M., Zhan, X., Wu, Y., Wei, Y., Kampffmeyer, M., Wei, X.-Y., Lu, M., Wang, Y., & Liang, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02057 · Cited by 36
“Among these, LJ Speech [19] and SQuAD [25] are classical audio/text datasets and used for voice synthesis and audio Q & A, while most video/text datasets [2, 20, 24, 32, 46, 47, 51, 57] are used for video Q & A.”
MethodHierarchical Conditional Relation Networks for Multimodal Video Question Answering
Le, T. M., Le, V., Venkatesh, S., & Tran, T. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01514-3 · Cited by 27
“in the experiment S. + Q. (w/ MUL. + w/ LSTM) to evaluate the effect of sequential modeling in textual stream. 16 Le et al. Exp. Model Validation w/o ts w/ ts State-of-the-art methods 1 TVQA S.+Q.+V.[2][46] 64.7 67.7 2 VideoQABERT[46] 63.1 72.1 Textual stream only 3 (B) Q. 41.6 41.6 4 (B) S. + Q. (w/o pre-selection) 4…”
InfluentialLearning to Answer Visual Questions From Web Videos
Yang, A., Miech, A., Šivic, J., Laptev, I., & Schmid, C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3173208 · Cited by 36
“In contrast to some previous works using additional video features such as subtitles [16], [45], [46], [48], [53], [54], [56], [62], [89], [95], [105], our video representation is exclusively based on visual information, as we focus on the detailed visual understanding of videos.”
Concept Parser With Multimodal Graph Learning for Video Captioning
Wu, B., Liu, B., Huang, P., Bao, J., Peng, X., & Yu, J. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3277827 · Cited by 28
“answers [52], video-textual retrieval [23].”
Video Understanding as Machine TranslationPreprint
Korbar, B., Petroni, F., Girdhar, R., & Torresani, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07203 · Cited by 30
“We use the TVQA [23] benchmark to assess question answering performance.”
Gaining Extra Supervision via Multi-task learning for Multi-Modal Video Question Answering
Kim, J., Ma, M., Kim, K., Kim, S.-J., & Yoo, C. D. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852087 · Cited by 25
“For a fair comparison, we followed the same video and text feature extraction procedure used in previous work [15] and fixed them during training.”
InfluentialStructured Co-reference Graph Attention for Video-grounded Dialogue
Kim, J., Yoon, S., Kim, D., & Yoo, C. D. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16273 · Cited by 25
Episodic Memory Question Answering
Datta, S., Dharur, S., Cartillier, V., Desai, R., Khanna, M., Batra, D., & Parikh, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01853 · Cited by 25
“VideoQA has witnessed a rich history of prior work with the introduction of datasets sampled from ”open-world” domains (movies [27], TV shows [19], cooking recipes [10, 34]) and tasks involving detecting/localizing actions and answering”
BackgroundObject-Aware Adaptive-Positivity Learning for Audio-Visual Question Answering
Li, Z., Guo, D., Zhou, J., Zhang, J., & Wang, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i4.28116 · Cited by 25
Adaptive Hierarchical Graph Reasoning with Semantic Coherence for Video-and-Language Inference
Li, J., Tang, S., Zhu, L., Shi, H., Huang, X., Wu, F. Q., Yang, Y., & Zhuang, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00188 · Cited by 24
“Several large-scale video datasets [1, 5, 8, 23, 49] and video-and-language tasks have been proposed, such as video captioning [16, 44, 57, 58, 48, 52, 26, 13, 41, 50], text-based video moment retrieval [15, 3, 6, 29, 36], and video question answering [27, 28, 47, 43, 24, 37].”
BackgroundMethodSelective Query-Guided Debiasing for Video Corpus Moment Retrieval
Yoon, S., Hong, J. W., Yoon, E., Kim, D., Kim, J., Yoon, H. S., & Yoo, C. D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_11 · Cited by 27
“As one of these video search technologies, video moment retrieval (VMR) [8,1] serves as essential building block to underpin many frontier interactive AI systems, including video/image captioning [31,26], video/image question answering [23,14] and visual dialog [3].”
BackgroundMUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration
Hayes, T., Zhang, S., Yin, X. P., Pang, G., Sheng, S., Yang, H. C., Ge, S., Hu, Q., & Parikh, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20074-8_25 · Cited by 28
Auto-captions on GIF: A Large-scale Video-sentence Dataset for Vision-language Pre-trainingPreprint
Pan, Y., Li, Y., Luo, J., Xu, J., Yao, T., & Mei, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02375 · Cited by 40
“The pre-trained generic representation or structure can better reflect the cross-modal interaction in a free way and thus benefit a series of downstream video-language tasks, such as video captioning [14, 21, 33], sentence localization in videos [3], sentence-to-video generation [20], and video question answering [11].”
BackgroundMMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question AnsweringPreprint
Khan, A. U., Mazaheri, A., Lobo, N. D. V., & Shah, M. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14095 · Cited by 36
“4653 Figure 3: Accuracy comparison with respect to the question family between Two-stream (Lei et al., 2018), WACV20 (Yang et al.”
InfluentialWhere did I leave my keys? — Episodic-Memory-Based Question Answering on Egocentric Videos
Bärmann, L., & Waibel, A. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00162 · Cited by 29
“There exist plenty of VideoQA datasets [3,20,33,37,38,40], of which Tab.”
BackgroundEnv-QA: A Video Question Answering Benchmark for Comprehensive Understanding of Dynamic Environments
Gao, D., Wang, R., Bai, Z., & Chen, X. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00170 · Cited by 26
“Some of the video QA datasets, such as TVQA [32] and MovieQA [49], evaluate the model’s understanding of movies, TV series or YouTube videos.”
BackgroundCONQUER: Contextual query-aware ranking for video corpus moment retrieval
HOU, Z., Ngo, C., & Chan, W. K. (2021) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 23
LiVLR: A Lightweight Visual-Linguistic Reasoning Framework for Video Question Answering
Jiang, J., Liu, Z., & Zheng, N. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3185900 · Cited by 26
“2) Comparisons on KnowIT VQA: We next compare LiVLR with the latest reported results on KnowIT VQA (four different settings of ROCK [24] and TVQA [7]) and ROLL [9].”
BackgroundMethodResultInfluentialReddy, R. G., Rui, X., Li, M., Lin, X., Wen, H., Cho, J., Huang, L., Bansal, M., Sil, A., Chang, S., et al. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21370 · Cited by 28
Self-citationERM: Energy-Based Refined-Attention Mechanism for Video Question Answering
Zhang, F., Wang, R., Zhou, F., & Luo, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3212463 · Cited by 23
“• TVQA [52]: a method designs a multi-stream end-to-end trainable neural network for multi-modal VideoQA.”
MethodInfluentialYouMakeup: A Large-Scale Domain-Specific Multimodal Dataset for Fine-Grained Semantic Comprehension
Wang, W., Wang, Y., Chen, S., & Jin, Q. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1517 · Cited by 24
“In recent years, video semantic comprehension has attracted much research attention, with a number of datasets and tasks being proposed such as activity recognition (Xu et al., 2017), dense video captioning (Krishna et al., 2017a), visual question answering (Lei et al., 2018, 2019) etc.”
BackgroundMethodVideo Question Answering: a Survey of Models and Datasets
Sun, G., Liang, L., Li, T., Yu, B., Wu, M., & Zhang, B. (2021) · Mobile Networks and Applications · doi:10.1007/s11036-020-01730-0 · Cited by 25
“Region-Level (Object-Level) Faster-RCNN [22] CMSA [32] K-PSTANet [33, 34] MGTA-Net [35] MGTA-Net [36] STAGE [37]”
MethodInfluentialRobust video question answering via contrastive cross-modality representation learning
Yang, X., Zeng, J., Guo, D., Wang, S., Dong, J., & Wang, M. (2024) · Science China Information Sciences · doi:10.1007/s11432-023-4084-6 · Cited by 25
Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-Based Large Language Models
Ning, M., Zhu, B., Xie, Y., Lin, B., Cui, J., Yuan, L., Chen, D., & Yuan, L. (2025) · Computational Visual Media · doi:10.26599/cvm.2025.9450516 · Cited by 121
Meta Spatio-Temporal Debiasing for Video Scene Graph Generation
Li, X., Qu, H., Kuen, J., Gu, J., & Liu, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19812-0_22 · Cited by 22
Khurana, K., & Deshpande, U. A. (2021) · IEEE Access · doi:10.1109/access.2021.3058248 · Cited by 26
“TVQA dataset [122] comprises 21,793 video clips extracted from 925 episodes of 6 popular TV shows, which belong to 3 genres: medical dramas, sitcoms, and crime shows.”
BackgroundSTAR: A Benchmark for Situated Reasoning in Real-World VideosPreprint
Wu, B., Yu, S., Chen, Z., Tenenbaum, J. B., & Gan, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.09711 · Cited by 135
“Various benchmarks [11, 44, 20] have been constructed by asking annotators to provide designed questions [1, 44] or synthesizing questions with pre-defined templates [15, 24, 14, 6].”
BackgroundMulti-Turn Video Question Generation via Reinforced Multi-Choice Attention Network
Guo, Z., Zhao, Z., Jin, W., Wei, Z., Yang, M., Wang, N., & Yuan, N. J. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3014775 · Cited by 21
CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding
Hou, Z., Zhong, W., Ji, L., Gao, D., Yan, K., Chan, W. K., Ngo, C., Shou, M. Z., & Duan, N. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.445 · Cited by 21
Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer
Chen, G., Liu, X., Wang, G., Zhang, K., Torr, P. H. S., Zhang, X.-P. S., & Tang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01282 · Cited by 28
“Accuracy Year Human 95.4 – VIS+LSTM [61] 29.9 2015 TVQA [35] 35.2 2018 HCRN [32] 36.5 2020 BERT-VQA [80] 33.7 2020 Eclipse [76] 37.1 2021 CLIP [56] 32.3 2021 CLIP+HCRN[56, 32] 41.9 2021 ATP [2] 35.6 2022 Tem-adapter 46.0 gin on both two datasets.”
MethodInfluentialA comparative study of language transformers for video question answering
Yang, Z., García, N., Chu, C., Otani, M., Nakashima, Y., & Takemura, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.02.092 · Cited by 21
Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval
Zhao, C., Xiong, C., Boyd-Graber, J. L., & Daumé, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.368 · Cited by 25
GEB+: A Benchmark for Generic Event Boundary Captioning, Grounding and Retrieval
Wang, Y., Gao, D., Yu, L., Lei, W., Feiszli, M., & Shou, M. Z. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_41 · Cited by 20
“In the experiment, we compare several state-of-the-art methods [17,22,43,41,3] along with many variants on our datasets to analyze the limitation of current methods and show the challenges of the proposed tasks.”
MethodSelf-citationMIMOQA: Multimodal Input Multimodal Output Question Answering
Singh, H., Nasery, A., Mehta, D., Agarwal, A., Lamba, J., & Srinivasan, B. V. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.418 · Cited by 23
“While RecipeQA aims reasoning over recipes and the associated pictures, TVQA involves multimodal comprehension over videos and their subtitles.”
BackgroundQuestion Answering Survey: Directions, Challenges, Datasets, Evaluation MatricesPreprint
Pandya, H. A., & Bhatt, B. (2021) · Preprints.org · doi:10.20944/preprints202112.0136.v1 · Cited by 20
NORMSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly
Fung, Y. R., Chakrabarty, T., Guo, H., Rambow, O., Muresan, S., & Ji, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.941 · Cited by 49
Han, S., Huang, W., Shi, H., Zhuo, L., Su, X., Zhang, S., Zhou, X., Qi, X., Liao, Y., & Liu, S. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.02438 · Cited by 22
“Current VideoQA datasets [16, 45, 52, 57] rely on costly manual annotations that often lack the granularity needed for detailed understanding, limiting scalability.”
Yang, W., Zhang, T., Zhang, Y., & Wu, F. H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3200399 · Cited by 18
“As an important research area in computer vision, it has great application potentials in many video related applications, such as video summarization [3], [4], video surveillance [5] and visual question answering [6].”
BackgroundProgressive Attention on Multi-Level Dense Difference Maps for Generic Event Boundary Detection
Tang, J., Liu, Z., Qian, C., Wu, W., & Wang, L. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.00335 · Cited by 30
Multi-Granularity Relational Attention Network for Audio-Visual Question Answering
Li, L., Jin, T., Wang, L., Jiang, H., Pan, W., Wang, J., Xiao, S., Xia, Y., Jiang, W., & Zhao, Z. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3264524 · Cited by 18
“To deal with VideoQA task, two-stream or multi-stream attention network [10], [11], [18], [21], graph convolution We show three informative frames and the speech of the video.”
BackgroundMethodInfluentialKeyword-Aware Relative Spatio-Temporal Graph Networks for Video Question Answering
Cheng, Y., Fan, H., Lin, D., Sun, Y., Kankanhalli, M., & Lim, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3345172 · Cited by 20
VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Control
Hu, Z.-Y., Li, Y., Lyu, M. R.-T., & Wang, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00281 · Cited by 20
“For completeness, we also transfer VL-PET modules to four video-text tasks, including video question answering (TVQA [24] and How2QA [29]) and video captioning (TVC [25] and YC2C [57]).”
BackgroundMethodMELINDA: A Multimodal Dataset for Biomedical Experiment Method Classification
Wu, T.-L., Singh, S., Paul, S., Burns, G., & Peng, N. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17657 · Cited by 19
Video Question Answering with Iterative Video-Text Co-tokenization
Piergiovanni, A., Morton, K., Kuo, W., Ryoo, M. S., & Angelova, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_5 · Cited by 18
Instance-sequence reasoning for video question answering
Liu, R., & Han, Y. (2022) · Frontiers of Computer Science · doi:10.1007/s11704-021-1248-1 · Cited by 17
Sparse autoencoder for social image understanding
Liu, J., Wang, S., & Yang, W. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.08.083 · Cited by 17
LifeQA: A Real-life Dataset for Video Question Answering
Castro, S., Azab, M., Stroud, J., Noujaim, C., Wang, R., Deng, J., & Mihalcea, R. F. (2020) · Language Resources and Evaluation · Cited by 17
Recent Advances in Video Question Answering: A Review of Datasets and Methods
Patel, D., Parikh, R., & Shastri, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68790-8_27 · Cited by 20
AssistQ: Affordance-Centric Question-Driven Task Completion for Egocentric Assistant
Wong, B., Chen, J., Wu, Y., Lei, S. W., Mao, D., Gao, D., & Shou, M. Z. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_28 · Cited by 22
Holistic Multi-Modal Memory Network for Movie Question Answering
Wang, A., Luu, A. T., Foo, C.-S., Zhu, H., Tay, Y., & Chandrasekhar, V. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2931534 · Cited by 17
“For the video modality, we follow [15] to use visual concepts (vcpt) detected from frames to represent the video modality.”
BackgroundMethodInfluentialVision-and-Language Navigation via Latent Semantic Alignment Learning
Wu, S., Fu, X., Wu, F. H., & Zha, Z.-J. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3358112 · Cited by 19
3MASSIV: Multilingual, Multimodal and Multi-Aspect dataset of Social Media Short Videos
Gupta, V., Mittal, T., Mathur, P., Mishra, V., Maheshwari, M., Bera, A., Mukherjee, D., & Manocha, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02039 · Cited by 16
“[41] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg.”
Background
Metadata
- Authors
- Lei, J., Yu, L., Bansal, M., & Berg, T.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 1369-1379
- OpenAlex
- W2963541336
- Cited by
- 541
- References
- 49
- Citations by year
- 2026: 39 · 2025: 50 · 2024: 65 · 2023: 93 · 2022: 90 · 2021: 101 · 2020: 70 · 2019: 32 · 2018: 1
- FWCI
- 15.13
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1405822, 1562098, 1633295)
- Army Research Office (W911NF)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- video question answering, video localization, television series, end-to-end learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
