The DOI matches its Crossref record; 47 of the 49 references it cites were found in a registry or an index.
TVQA: Localized, Compositional Video Question Answering
Open accessAbstract
Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks.However, due to data limitations, there has been much less work on video-based QA.In this paper, we present TVQA, a largescale video QA dataset based on 6 popular TV shows.TVQA consists of 152,545 QA pairs from 21,793 clips, spanning over 460 hours of video.Questions are designed to be compositional in nature, requiring systems to jointly localize relevant moments within a clip, comprehend subtitle-based dialogue, and recognize relevant visual concepts.We provide analyses of this new dataset as well as several baselines and a multi-stream end-to-end trainable neural network framework for the TVQA task.The dataset is publicly available at http://tvqa.cs.unc.edu.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Overall, they obtained promising results on the TVQA [129] and ActivityNetQA [130] datasets.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [129]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“We also use ResNet101 (He et al., 2016) trained on ImageNet (Deng et al., 2009) to extract whole image features.”
MethodFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 5×
“Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al., 2017), etc. Addit…”
BackgroundMethodInfluentialFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 5×
“ImageNet Features: We extract the pooled 2048D feature of the last block of ResNet101.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 4×
“A subtitle S, which contains a set of sentences, is flattened into a long sequence of words and GloVe (Pennington et al., 2014) is used to embed the words.”
BackgroundMethodInfluentialSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 2×
“With the same goal in mind, Rajpurkar et al. (2016) introduced the SQuAD dataset, but their answers are specific spans from long passages.”
BackgroundVQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., & Gool, L. V. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46484-8_2 · Cited by 4,232
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855 · cited 1×
“Text Question Answering: The related task of text-based question answering has been extensively explored (Richardson et al., 2013; Weston et al., 2015; Rajpurkar et al., 2016; Hermann et al., 2015; Hill et al., 2015).”
BackgroundCLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Research works printed but not linked: 8
- Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question an- swering. ICCV, pages 2425-2433.
- Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. 2016. Natu- ral language object retrieval. CVPR.
- Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron Courville, and Christopher Pal. 2017b. A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering. CVPR.
- Paul Over, Jon Fiscus, Greg Sanders, David Joy, Mar- tial Michel, George Awad, Alan Smeaton, Wessel Kraaij, and Georges Quénot. 2014. Trecvid 2014-an overview of the goals, tasks, data, evaluation mech- anisms and metrics. In Proceedings of TRECVID, page 52.
- Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- to-sentence models. In ICCV.
- Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018b. Mat- tnet: Modular attention network for referring expres- sion comprehension. In CVPR.
- Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016. Modeling context in referring expressions. In ECCV.
- Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. 2017. A joint speakerlistener-reinforcer model for referring expressions. In CVPR.
Cited by
Sorted by citations
Items 301–400 of 534
VideoExpert: Augmented LLM for Temporal-Sensitive Video Understanding
Zhao, H., Ji, G.-P., Yan, R., Xiong, H., & Li, Z. (2026) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2026.3653742 · Cited by 18
“Can a model that excels in temporal-sensitive tasks also perform well in text generation?”
PSAM: Parameter-Free Spatiotemporal Attention Mechanism for Video Question Answering
Zhang, F., Wang, R., Zhou, F., Luo, Y., & Li, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3333192 · Cited by 6
Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamic Audio-Visual Scenarios
Jiang, Y., & Yin, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.630 · Cited by 6
Kim, M. J., Song, S., Lee, Y., Jang, H., & Lim, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29798 · Cited by 7
“TVQA specializes in six TV show domains and contains 152,000 question-answer pairs from 925 episodes.”
MethodCompositional Physical Reasoning of Objects and Events From Videos
Chen, Z., Dong, S., Yi, K., Li, Y., Ding, M., Torralba, A. B., Tenenbaum, J. B., & Gan, C. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3574322 · Cited by 62
“Several benchmarks have been posited to address the task of video question answering, such as MarioQA [31], TVQA [32], and AGQA [15].”
Zhao, C., Xiong, C., Boyd-Graber, J. L., & Daumé, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.756 · Cited by 8
“, 2020), multi-model reasoning (Lei et al., 2018), or combination of these skills (Bartolo et al.”
BackgroundResearch Progress on Vision–Language Multimodal Pretraining Model Technology
Wang, H., Huang, R., & Zhang, J. (2022) · Electronics · doi:10.3390/electronics11213556 · Cited by 6
Difference-Masking: Choosing What to Mask in Continued Pretraining
Wilf, A., Akter, S., Mathur, L., Liang, P. P., Mathew, S., Shou, M., Nyberg, E., & Morency, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.881 · Cited by 6
Himakunthala, V., Ouyang, A., Rose, D., He, R., Mei, A., Lu, Y., Sonar, C., Saxon, M., & Wang, W. Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.15 · Cited by 19
“For example, Video Question Answering (Lei et al., 2018) and Video Summarization (Xu et al., 2016) are two existing datasets that empirically evaluate video models.”
BackgroundYTCommentQA: Video Question Answerability in Instructional Videos
Yang, S., Park, S., Jang, Y., & Lee, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29906 · Cited by 11
Learning to Reason with Relational Video Representation for Question Answering.
Le, T. M., Le, V., Venkatesh, S., & Tran, T. (2019) · Cited by 6
Cut-Based Graph Learning Networks to Discover Compositional Structure of Sequential Video Data
On, K.-W., Kim, E.-S., Heo, Y., & Zhang, B.-T. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5978 · Cited by 7
What Gives the Answer Away? Question Answering Bias Analysis on Video QA DatasetsPreprint
Yang, J., Zhu, Y., Wang, Y., Yi, R., Zadeh, A., & Morency, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.03626 · Cited by 15
“We hypothesize w118 may have asked many questions that are similar to other 1 We thank the authors of TVQA for sharing this information.”
BackgroundInfluentialCUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation LearningPreprint
Zhou, L., Liu, J., Yu, C., Gan, Z., & Zhang, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.00285 · Cited by 8
“Thus, the final representation for Method Curated Data TVR TVQA R@1 R@10 R@100 Acc. video v i S with clip embedding vectors stacked horizontally as a matrix M i S ∈ R d × Q .”
BackgroundMethodInfluentialLow-Fidelity End-to-End Video Encoder Pre-training for Temporal Action LocalizationPreprint
Xu, M., Pérez-Rúa, J.-M., Zhu, X., Ghanem, B., & Martínez, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.15233 · Cited by 6
“Video analysis has become an increasingly important area of research, encompassing multiple relevant problems such as action recognition [8, 13], temporal action localization [68, 7, 12, 22, 59, 60, 55, 67, 15], video grounding[36, 65, 64, 66, 49, 16, 24], and video question answering [25, 31].”
BackgroundMulti-Step Reasoning Over Unstructured Text with Beam Dense RetrievalPreprint
Zhao, C., Xiong, C., Boyd-Graber, J. L., & Daumé, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.05883 · Cited by 6
“, 2015), connect concepts across modalities (Lei et al., 2018), or to connect related questions to each other (Elgohary et al.”
BackgroundM5Product: A Multi-modal Pretraining Benchmark for E-commercial Product Downstream TasksPreprint
Dong, X. M., Zhan, X., Wu, Y., Wei, Y., & Liang, X. (2021) · arXiv (Cornell University) · doi:10.13140/rg.2.2.35620.81281 · Cited by 6
Temporal Pyramid Transformer with Multimodal Interaction for Video Question AnsweringPreprint
Peng, M., Wang, C., Gao, Y., Shi, Y., & Zhou, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04735 · Cited by 6
“Most existing methods (Lei et al. 2018; Gao et al. 2019; Jiang et al. 2020; Liu et al. 2020; Cai et al. 2020; Jiang and Han 2020) use recurrent neural networks (RNNs) and their Figure 1: The interaction of the question and the visual content usually happens at multiple temporal scales, as…”
MethodComplex QA and language models hybrid architectures, SurveyPreprint
Daull, X., Bellot, P., Bruno, E., Martin, V., & Murisasco, É. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.09051 · Cited by 32
Neuro-Symbolic AI: Bringing a new era of Machine Learning
Kishor, R. (2022) · International Journal of Research Publication and Reviews · doi:10.55248/gengpi.2022.31271 · Cited by 6
Video Question Answering: A survey of the state-of-the-art
Jeshmol, P. J., & Kovoor, B. C. (2024) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2024.104320 · Cited by 9
Scene-Text Grounding for Text-Based Video Question Answering
Zhou, S., Xiao, J., Yang, X., Song, P., Guo, D., Yao, A., Wang, M., & Chua, T. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3639956 · Cited by 9
“TVQA [24] and TVQA + [25] pioneer in study grounded VideoQA, yet in full supervision.”
InfluentialKangaroo: A Powerful Video-Language Model Supporting Long-context Video Input
Liu, J., Wang, Y., Ma, H., Wu, X., Ma, X., Wei, X., Jiao, J., Wu, E., & Hu, J. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02620-2 · Cited by 132
“Videos with excessively large face coverage in these frames are excluded.”
Video Language Co-Attention with Multimodal Fast-Learning Feature Fusion for VideoQA
Abdessaied, A., Sood, E., & Bulling, A. (2022) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2022.repl4nlp-1.15 · Cited by 5
“However, there still is a semantic gap between the visual and language channels (Lei et al., 2018; Sun et al., 2021; Song et al., 2018) that prior work has tried to close by leveraging external memory (Kim et al., 2018, 2019; Fan et al., 2019).”
DoSSIER at MedVidQA 2022: Text-based Approaches to Medical Video Answer Localization Problem
Kusa, W., Peikos, G., Espitia, Ó., Hanbury, A., & Pasi, G. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.43 · Cited by 5
“The introduction of large-scale multimodal datasets covering both language and vision enabled the development of efficient deep neural network techniques that bridge the gap between language, and visual understanding (Lei et al., 2018, 2019; Tapaswi et al., 2016).”
BackgroundCascaded MPN: Cascaded Moment Proposal Network for Video Corpus Moment Retrieval
Yoon, S., Kim, D., Kim, J., & Yoo, C. D. (2022) · IEEE Access · doi:10.1109/access.2022.3183106 · Cited by 5
“Numerous respectful works have made great strides in bridging computer vision and natural language processing including video/image captioning [30], [34], video moment retrieval [1], [8], video/image question answering [12], [25].”
BackgroundVideo question answering supported by a multi-task learning objective
Falcon, A., Serra, G., & Lanz, O. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14333-0 · Cited by 5
“In particular, two types of tasks are often linked to VideoQA: the “open-ended” (e.g. in [5, 15, 55, 58]) and the “multiple choice” task (e.g. [5, 15, 19, 25, 45]).”
BackgroundMethodInfluentialAssistSR: Task-oriented Video Segment Retrieval for Personal AI Assistant
Lei, W., Gao, D., Wang, Y., Mao, D., Liang, Z., Ran, L., & Shou, M. Z. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.24 · Cited by 5
“Video Question Answering.”
BackgroundMethodResultInfluentialT2V2T: Text-to-Video-to-Text Fusion for Text-to-Video Retrieval
Kim, J., Lee, Y., & Moon, J. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00594 · Cited by 5
“Recently, text-to-video retrieval methods have leveraged video-language pre-training (VLP) [2, 5, 7, 9, 12, 17, 24–26, 33] since VLP has shown superior performance in downstream tasks such as text-to-video retrieval [1,11,28], video question answering [14, 27, 30, 32], and video captioning [11, 28, 34].”
MethodNguyen, T. T., Wu, X., Dong, X., Nguyen, C.-D. T., Ng, S.-K., & Luu, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.235 · Cited by 5
“However, TLG has received increasing attention in CV and NLP communities because it provides myriad usage for further down-stream tasks, e.g. VQA (Lei et al., 2018; Ye et al., 2017; Wang et al., 2019a), relation extraction (Gao et al., 2021b), and information retrieval (Ghosh et al., 2019).”
BackgroundMethodQuestion-Answering Dense Video Events
Qin, H., Xiao, J., & Yao, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729945 · Cited by 11
“Also, we limit the number of “who” and “where” questions to keep them in a low percentage of the dataset, as they can be well-answered without the need for video -level understanding (Xu et al. 2017; Lei et al. 2018).”
Referring to Objects in Videos Using Spatio-Temporal Identifying Descriptions
Wiriyathammabhum, P., Shrivastava, A., Morariu, V. I., & Davis, L. S. (2019) · doi:10.18653/v1/w19-1802 · Cited by 5
“, 2018) and TVQA (Lei et al., 2018) use a compositional format for words like before or after to specify temporal rela-”
Learning Question-Guided Video Representation for Multi-Turn Video Question Answering
Chao, G.-L., Rastogi, A., Yavuz, S., Hakkani-Tür, D. Z., Chen, J., & Lane, I. R. (2019) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w19-5926 · Cited by 6
“Similarly, Lei et al. (2018) use Faster R-CNN (Ren et al., 2015b) trained with the Visual Genome (Krishna et al., 2017) dataset to detect object and attribute regions in each frame, which are used as input features to the question answering model.”
BackgroundMethodInfluentialDramaQA: Character-Centered Video Story Understanding with Hierarchical QA
Choi, S.-H., On, K.-W., Heo, Y., Jang, Y., Seo, A., Lee, S.-C., Lee, M., & Zhang, B. (2021) · 정보과학회 컴퓨팅의 실제 논문지 · doi:10.5626/ktcp.2021.27.1.7 · Cited by 6
3DVQA: Visual Question Answering for 3D Environments
Etesam, Y., Kochiev, L., & Chang, A. X. (2022) · Canadian Conference on Computer and Robot Vision · doi:10.1109/crv55824.2022.00038 · Cited by 5
“The introduction of VQA has spurred work in QA for figures and charts [13, 22, 23], document images [35], videos [44, 30].”
BackgroundDense but Efficient VideoQA for Intricate Compositional Reasoning
Lee, J., Kang, W., & Kim, E. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00117 · Cited by 5
“Moreover, DSR achieves the state-of-the-art result in the V+Q setting of TVQA, where subtitles and timestamps are not used for training.”
BackgroundInfluential2nd Place Solution to the GQA Challenge 2019Preprint
Geng, S., Zhang, J., Zhang, H., Elgammal, A., & Metaxas, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.06794 · Cited by 5
“Table 1 presents results using our detected features, statistical features and ground truth features with two reasoning models: MAC [4] and multi-stream cross attentional model (MS-CA) from TVQA [6].”
MethodLoss re-scaling VQA: Revisiting the LanguagePrior Problem from a Class-imbalance ViewPreprint
Guo, Y., Nie, L., Cheng, Z., Tian, Q. C., & Zhang, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.16010 · Cited by 5
Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Models
Liu, Z., Xie, C.-W., Li, P., Zhao, L., Tang, L., Zheng, Y. P., Liu, C., & Xie, H. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00801 · Cited by 13
“Inspired by [51, 72], we collect 2.68M video instruction data for instruction tuning, including 1.6M from LLaVA-Video [72], 292K from VideoChat2-IT [23] subset, 255K from M4-Instruct-Data [21], 11K from Cha-rades [45], 114K from NTU RGB+D [44], 122K from TVQA [18], and 290K from MiT [38].”
Video question answering via traffic knowledge database and question classification
Sun, X., Dai, Y., Wang, Y., Ma, W., & Lin, X. (2024) · Multimedia Systems · doi:10.1007/s00530-023-01240-5 · Cited by 4
“TVQA [49] is a multi-stream network that combines input features from different modalities to answer questions.”
End-to-End Recurrent Cross-Modality Attention for Video Dialogue
Chu, Y.-W., Lin, K.-Y., Hsu, C.-C., & Ku, L. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3065852 · Cited by 4
“Furthermore, some research [18], [31], [65] provide video datasets from movie or TV series for systems to output an accurate answer from possible multiple choice.”
BackgroundModality attention fusion model with hybrid multi-head self-attention for video understanding
Zhuang, X., Liu, F., Hou, J. L., Hao, J., & Cai, X. (2022) · PLoS ONE · doi:10.1371/journal.pone.0275156 · Cited by 5
“[23]) based on a bi-directional LSTM to encode both textual and visual sequences.”
BackgroundXie, B., & Park, C. H. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00331 · Cited by 4
“In multiple previous studies, the long short-term memory (LSTM) model has been investigated for its capacity to summarize or fuse multi-modal information in order to provide answers to questions in videos [32, 10, 33, 9].”
BackgroundNo Head Left Behind – Multi-Head Alignment Distillation for Transformers
Zhao, T., Singh, K. Y., Appalaraju, S., Tang, P., Mahadevan, V., Manmatha, R., & Wu, Y. N. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i7.28583 · Cited by 6
“…et al. 2020a; Cho et al. 2020; Zhang et al. 2021; Sun et al. 2019; Zhu and Yang 2020; Miech et al. 2020; Radford et al. 2021; Appalaraju et al. 2024) for visual-linguistic tasks (Goyal et al. 2019; Hudson and Manning 2019; Lei et al. 2018; Mao et al. 2016; Xu et al. 2016; Zhou, Xu, and Corso 2018).”
BackgroundLee, J.-H., & Kang, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01299 · Cited by 4
“Their effectiveness is acknowledged across various down-stream tasks, including video question answering [21, 25], video retrieval [17, 38], and video captioning [32, 47].”
BackgroundMethodCompositional Video Understanding with Spatiotemporal Structure-based Transformers
Yun, H., Ahn, J., Kim, M., & Kim, E. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01774 · Cited by 4
“Recently, a number of benchmark datasets such as CLEVRER [40], TVQA [21], and AGQA [11] have been suggested for compositional video question answering tasks.”
BackgroundSurvey of Large Multimodal Model Datasets, Application Categories and Taxonomy
Pattnayak, P., Patel, H. L., Kumar, B., Agarwal, A., Banerjee, I., Panda, S., & Kumar, T. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-14041-8_4 · Cited by 30
Knowledge Graph Extraction from Videos
Mahon, L., Giunchiglia, E., Li, B., & Lukasiewicz, T. (2020) · International Conference on Machine Learning and Applications · doi:10.1109/icmla51294.2020.00014 · Cited by 4
“Promising results have been produced in the tasks of video annotation (or captioning) using natural language (see, e.g., [1]–[3]) and video question answering (see, e.g., [4]–[6]).”
Music Question Answering:Cognize and Perceive Music
Gao, W., Li, X., Jin, C., & Tie, Y. (2022) · IEEE International Conference on Multimedia and Expo workshops · doi:10.1109/icmew56448.2022.9859499 · Cited by 4
“MovieQA [16], TVQA [17] and KnowITVQA [18].”
FHGN: Frame-Level Heterogeneous Graph Networks for Video Question Answering
Qi, J., Li, F., Bai, T., Cao, C., Song, W., Hu, Y., & Wu, B. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859784 · Cited by 4
“In Table.1, we compare our FHGN model against the state-of-the-art (SOTA) methods on TVQA+ dataset.”
BackgroundInfluentialA novel temporal moment retrieval model for apron surveillance video
Lyu, Z., & Zhang, Y. (2023) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2023.108616 · Cited by 4
Araujo, V., Soto, Á., & Moens, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.830 · Cited by 4
External Commonsense Knowledge as a Modality for Social Intelligence Question-Answering
Natu, S., Sural, S., & Sarkar, S. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00327 · Cited by 4
“Datasets such as TVQA [10], Social-IQ [22] and HOW2QA [12] are popular datasets in the broad domain of video-based QA.”
MethodAttention Based Multi-Modal Fusion Architecture for Open-Ended Video Question Answering Systems
Pendurkar, S., Kolpekwar, S., Dhoot, S., Haribhakta, Y. V., & Banerjee, B. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.04.047 · Cited by 4
Context-Aware Question-Answer for Interactive Media Experiences
Jorgensen, K., Zhao, Z., Wang, H., Wang, M., & He, Z. (2021) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3452918.3458795 · Cited by 5
“To this end, natural language processing models have been proposed [14, 29, 31].”
BackgroundLi, Z., Zhu, X., Lei, Z., & Zhang, Z. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i2.20044 · Cited by 4
“Visual Reasoning People also study the task of visual reasoning in order to better assess model’s capability of dynamics modelling (Ehrhardt et al. 2019; Finn, Goodfellow, and Levine 2016; Fraccaro et al. 2017; Hafner et al. 2019; Lei et al. 2018; Ha and Schmidhuber 2018; Finn and Levine 2017).”
BackgroundConstructing Hierarchical Q&A Datasets for Video Story UnderstandingPreprint
Heo, Y., On, K.-W., Choi, S.-H., Lim, J., Kim, J., Ryu, J.-K., Bae, B.-C., & Zhang, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.00623 · Cited by 4
“For the TVQA, all questions and answers are attached to 60-90 seconds video clips.”
InfluentialVideo2Commonsense: Generating Commonsense Descriptions to Enrich Video CaptioningPreprint
Fang, Z., Gokhale, T., Banerjee, P., Baral, C. R., & Yang, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.05162 · Cited by 4
“Results: MSR-VTT QA (Xu et al., 2017) is as a good baseline since it is trained on a conventional videoQA task on the MSR-VTT videos, and only takes video and query as input, unlike recent video understanding models (Lei et al., 2018) that take additional supervision, such as subtitles.”
Hopper: Multi-hop Transformer for Spatiotemporal ReasoningPreprint
Zhou, H., Kadav, A., Lai, F., Niculescu-Mizil, A., Min, M. R., Kapadia, M., & Graf, H. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.10574 · Cited by 10
“There have been also numerous datasets that are based on real-world videos and human-generated questions such as MovieQA (Tapaswi et al., 2016), TGIF-QA (Jang et al., 2017), TVQA (Lei et al., 2018) and Social-IQ (Zadeh et al., 2019).”
BackgroundCoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and DenoisingPreprint
Luo, J., Li, Y., Pan, Y., Yao, T., Chao, H., & Mei, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.07515 · Cited by 4
“, TV dataset [24] and our newly collected ACTION dataset [35]).”
MethodInfluentialMultiple visual relationship forecasting and arrangement in videos
Ouyang, W., Hu, Y., Ou, Y., & Chen, Z. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126274 · Cited by 3
Open-Ended Multi-Modal Relational Reasoning for Video Question Answering
Luo, H., Qin, R., Xu, C., Ye, G., & Luo, Z. (2023) · IEEE RO-MAN · doi:10.1109/ro-man57019.2023.10309342 · Cited by 5
Fully Authentic Visual Question Answering Dataset from Online Communities
Chen, C., Liu, M., Codella, N., Li, Y., Yuan, L., & Gurari, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73195-2_15 · Cited by 11
Shwe, L. L., Matayong, S., & Witosurapot, S. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127244 · Cited by 3
Temporal-Guided Mixture-of-Experts for Zero-Shot Video Question Answering
Qin, Y., Zhao, L., Gao, L., Zhang, H., Zeng, P., & Shen, H. T. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3556422 · Cited by 5
“We evaluate our model on seven VideoQA datasets, including four open-ended VideoQA datasets: MSRVTT-QA [21], MSVD-QA [21], iVQA [14] and TGIF-FrameQA [51], and three multi-choice VideoQA datasets: NExT-QA [18], STAR [52] and TVQA [53].”
Winterbottom, T. M., Xiao, S. H., McLean, A., & Moubayed, N. A. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.974 · Cited by 7
“(2019) find various BLP fusions perform worse than their ‘dynamic modality fusion’ mechanism on TVQA (Lei et al., 2018) andMovieQA (Tapaswi et al.”
BackgroundMethodInfluentialCali-NCE: Boosting Cross-modal Video Representation Learning with Calibrated Alignment
Zhao, N., Jiao, J., Xie, W., & Lin, D. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00672 · Cited by 3
“A series of works have explored the modeling on such cross-modal tasks including text-to-video retrieval [41, 67], video captioning [31,63,74], and video question answering [3,39].”
BackgroundUnderstanding Video Scenes through Text: Insights from Text-based Video Question Answering
Jahagirdar, S., Mathew, M., Karatzas, D., & Jawahar, C. V. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00500 · Cited by 3
Counterfactual contrastive learning for weakly supervised temporal sentence grounding
Xu, Y., Xu, W., & Miao, Z. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129508 · Cited by 3
Descriptor: Open-Domain Long-Form Context-Aware Question-Answering Dataset (DragonVerseQA)
Lahiri, A. K., & Hu, Q. (2025) · IEEE data descriptions. · doi:10.1109/ieeedata.2025.3562173 · Cited by 3
Temporal Attention and Consistency Measuring for Video Question Answering
Zhang, L., & Radke, R. J. (2020) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3382507.3418886 · Cited by 3
“constructed the TVQA dataset [18] from multiple TV shows and proposed a multistream Recurrent Neural Network (RNN) with a context matching module to jointly model the question, answer and data.”
MethodCoSIm: Commonsense Reasoning for Counterfactual Scene Imagination
Kim, H., Zala, A., & Bansal, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.66 · Cited by 8
Self-citationHysia: Serving DNN-based video-to-retail applications in cloud
Zhang, H., Li, Y., Ai, Q., Luo, Y., Wen, Y., Jin, Y., & Nguyen Binh Duong, T. A. (2020) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 3
Continuous Language Generative Flow
Tang, Z., Zhang, S., Kim, H., & Bansal, M. (2021) · doi:10.18653/v1/2021.acl-long.355 · Cited by 4
“We test QG and QA data augmentation on two large-scale QA datasets: (a) SQuAD (Rajpurkar et al., 2016), a widely explored textual QA and QG dataset and (b) TVQA (Lei et al., 2018), a large-scale multimodal video-dialogue QA task.”
MethodSelf-citationTrUMAn: Trope Understanding in Movies and Animations
Su, H.-T., Shen, P.-W., Tsai, B., Cheng, W., Wang, K.-J., & Hsu, W. H. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482018 · Cited by 4
“Follow previous work [12, 13], we use LSTM network to encode visual and audio features.”
MethodInfluentialA Multi -level Alignment Training Scheme for Video-and-Language Grounding
Zhang, Y., Niu, F., Ping, Q., & Thattai, G. (2022) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw58026.2022.00124 · Cited by 3
“Popular video-and-language tasks include text-video retrieval [2, 3, 33, 41, 50, 53], video moment retrieval [30, 39, 60, 62], video captioning [3, 24, 50, 53, 60], video question answering [17, 21, 28, 30, 47, 58], and video-and-language inference [35].”
BackgroundEnhancing Augmented Reality Dialogue Systems with Multi-Modal Referential Information
He, Z., & Cai, Z. (2023) · China Automation Congress (CAC) · doi:10.1109/cac59555.2023.10450983 · Cited by 3
“Researchers introduce modalities and datasets like IGC [17], Visual7W [18], TVQA [19], MovieQA [20], and SIMMC [1], [21], laying the foundation for training multi-modal dialogue systems.”
MethodMultimodal Transformer with Pointer Network for the DSTC8 AVSD ChallengePreprint
Le, H., & Chen, N. F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10695 · Cited by 3
“ery may refer to previous information mentioned in previous dialogue turns and such references must be learned to answer the query correctly. (2) Secondly, most of video QA models (Jang et al. 2017) (Lei et al. 2018) (Kim et al. 2019) are more suitable for the retrieval-based setting. In this setting, the model is typ…”
Multi-Scale 2D Temporal Adjacent Networks for Moment Localization with Natural LanguagePreprint
Zhang, S., Peng, H., Fu, J., Lu, Y., & Luo, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.02646 · Cited by 3
“Video moment localization with natural language has a wide range of applications, such as video question answering [14], video content retrieval [15], dense video captioning [16], aswell as video storytelling [17].”
Mounting Video Metadata on Transformer-based Language Model for Open-ended Video Question AnsweringPreprint
Lee, D., Choi, S.-H., Jang, Y., & Zhang, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.05158 · Cited by 4
“Attention mechanisms allow the model to selectively focus on segments of input text it predicts to be the most relevant.”
InfluentialShin, M., Mun, J., On, K.-W., Kang, W., Han, G., & Kim, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06476 · Cited by 4
Maintaining Common Ground in Dynamic Environments
Udagawa, T., & Aizawa, A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00409 · Cited by 2
“This includes video QA (Lei et al., 2018; Yu et al., 2019; Castro et al., 2020), video object grounding (Zhou et al.”
Mao, Y., Lin, X. A., Ni, Q., & He, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i1.27814 · Cited by 2
HSSHG: Heuristic Semantics-Constrained Spatio-Temporal Heterogeneous Graph for VideoQA
Wang, R., Luo, Y., Zhang, F., Liu, M., & Luo, X. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3443661 · Cited by 3
“Since video frames and questions are in sequence form, early works [4], [26] effectively achieve spatio-temporal reasoning of video through an encoder-decoder model.”
Nawaz, H. S., Shi, D., & Zhang, X. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112667 · Cited by 2
Cross-Task Relation-Aware Consistency for Weakly Supervised Temporal Action Detection
Yang, W.-F., Ren, H., Zhang, T., Zhang, Z., Zhang, Y., & Wu, F. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3594178 · Cited by 3
Liu, M., Wang, R., Zhou, F., & Lin, G. (2022) · Symmetry · doi:10.3390/sym14061133 · Cited by 2
Shot Retrieval and Assembly with Text Script for Video Montage Generation
Yang, G., Lu, H., Sun, Z., & Lu, Z. (2023) · International Conference on Multimedia Retrieval · doi:10.1145/3591106.3592247 · Cited by 2
“Visual Semantic Embeddings (VSE) [7, 9] are commonly adopted in multi-modal tasks to learn vision-language joint representations [2, 10, 14, 17, 30].”
BackgroundA Video Question Answering Model Based on Knowledge Distillation
Shao, Z., Wan, J., & Zong, L. (2023) · Information · doi:10.3390/info14060328 · Cited by 2
“Film and television datasets comprise video clips from movies and TV shows, including datasets such as MovieQA [13] and TVQA [14].”
MethodSpatio-Temporal Two-stage Fusion for video question answering
Xu, F., Zhu, Y., Wang, C., Cao, Y., Zhong, Z., & Li, X. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103821 · Cited by 2
Fung, Y. R., Wang, H., Wang, T., Kebarighotbi, A., Bansal, M., Ji, H., & Natarajan, P. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.221 · Cited by 3
“Pre-existing video/movie/TV show QA datasets fall into two extremes of either focusing on too narrow and minor scene-specific details such as an object on the couch that is irrelevant to the storyline development (Lei et al., 2018), or overly brief and simple details from summaries (Tapaswi et al.”
BackgroundInfluentialSelf-citationZhou, Z., Liu, J., Cheng, S., Luo, H., Gu, Y., & Ye, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44699-3_20 · Cited by 2
Weakly supervised temporal action localization: a survey
Li, R., Zhang, T., & Zhang, R. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18554-9 · Cited by 2
“TAL techniques have been widely used in video surveillance [88–90], video summarization, action retrieval, anomaly detection [91], video questions and answers [92], multi-camera temporal event localization [93], and audio-visual video parsing [94].”
Qi, J., Xu, Y., & Wu, B. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650620 · Cited by 2
A Survey of Video Datasets for Grounded Event Understanding
Sanders, K., & Durme, B. V. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00727 · Cited by 2
Learning Video Context as Interleaved Multimodal Sequences
Lin, K. Q., Zhang, P., Gao, D., Xia, X., Chen, J., Gao, Z., Xie, J., Xiao, X., & Shou, M. Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72967-6_21 · Cited by 5
Weakly-Supervised Spatio-Temporal Video Grounding with Variational Cross-Modal Alignment
Yang, J., & Mu, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73195-2_24 · Cited by 2
“…how language description relates to video content in spatial-temporal dimensions is paramount for video surveillance [10], video question answering [18], etc. Recently, spatio-temporal video grounding (STVG) was introduced in [43], which requires localizing a spatio-temporal tube (i.e., a sequence…”
It’s Just Another Day: Unique Video Captioning by Discriminitive Prompting
Perrett, T., Han, T., Damen, D., & Zisserman, A. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0908-6_16 · Cited by 2
Perceive. Query & Reason: Enhancing Video QA with Question-Guided Temporal Queries
Amoroso, R., Zhang, G., Koner, R., Baraldi, L., Cucchiara, R., & Volker Tresp, C. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00858 · Cited by 2
“The development of video QA has been driven by diverse benchmark datasets spanning multiple domains, including movies and TV shows [17,18, 32,40], as well as wild, online videos [10,37,38,40,48].”
Enhancing Visual Question Answering for Multiple Choice Questions
Goel, R., Nandwani, H., Shah, E., Nayak, A., & Kumar, A. P. (2025) · IEEE Access · doi:10.1109/access.2025.3572529 · Cited by 2
Metadata
- Authors
- Lei, J., Yu, L., Bansal, M., & Berg, T.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 1369-1379
- OpenAlex
- W2963541336
- Cited by
- 541
- References
- 49
- Citations by year
- 2026: 39 · 2025: 50 · 2024: 65 · 2023: 93 · 2022: 90 · 2021: 101 · 2020: 70 · 2019: 32 · 2018: 1
- FWCI
- 15.13
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1405822, 1562098, 1633295)
- Army Research Office (W911NF)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- video question answering, video localization, television series, end-to-end learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
