The DOI matches its Crossref record; 47 of the 49 references it cites were found in a registry or an index.
TVQA: Localized, Compositional Video Question Answering
Open accessAbstract
Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks.However, due to data limitations, there has been much less work on video-based QA.In this paper, we present TVQA, a largescale video QA dataset based on 6 popular TV shows.TVQA consists of 152,545 QA pairs from 21,793 clips, spanning over 460 hours of video.Questions are designed to be compositional in nature, requiring systems to jointly localize relevant moments within a clip, comprehend subtitle-based dialogue, and recognize relevant visual concepts.We provide analyses of this new dataset as well as several baselines and a multi-stream end-to-end trainable neural network framework for the TVQA task.The dataset is publicly available at http://tvqa.cs.unc.edu.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Overall, they obtained promising results on the TVQA [129] and ActivityNetQA [130] datasets.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [129]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“We also use ResNet101 (He et al., 2016) trained on ImageNet (Deng et al., 2009) to extract whole image features.”
MethodFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 5×
“Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al., 2017), etc. Addit…”
BackgroundMethodInfluentialFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 5×
“ImageNet Features: We extract the pooled 2048D feature of the last block of ResNet101.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 4×
“A subtitle S, which contains a set of sentences, is flattened into a long sequence of words and GloVe (Pennington et al., 2014) is used to embed the words.”
BackgroundMethodInfluentialSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 2×
“With the same goal in mind, Rajpurkar et al. (2016) introduced the SQuAD dataset, but their answers are specific spans from long passages.”
BackgroundVQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., & Gool, L. V. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46484-8_2 · Cited by 4,232
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855 · cited 1×
“Text Question Answering: The related task of text-based question answering has been extensively explored (Richardson et al., 2013; Weston et al., 2015; Rajpurkar et al., 2016; Hermann et al., 2015; Hill et al., 2015).”
BackgroundCLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Flagged first, then by citations
2 of the 49 references couldn’t be matched to a known paper.
Research works printed but not linked: 8
- Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question an- swering. ICCV, pages 2425-2433.
- Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. 2016. Natu- ral language object retrieval. CVPR.
- Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron Courville, and Christopher Pal. 2017b. A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering. CVPR.
- Paul Over, Jon Fiscus, Greg Sanders, David Joy, Mar- tial Michel, George Awad, Alan Smeaton, Wessel Kraaij, and Georges Quénot. 2014. Trecvid 2014-an overview of the goals, tasks, data, evaluation mech- anisms and metrics. In Proceedings of TRECVID, page 52.
- Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- to-sentence models. In ICCV.
- Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018b. Mat- tnet: Modular attention network for referring expres- sion comprehension. In CVPR.
- Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016. Modeling context in referring expressions. In ECCV.
- Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. 2017. A joint speakerlistener-reinforcer model for referring expressions. In CVPR.
Cited by
Sorted by citations
Items 401–500 of 534
DistinctAD: Distinctive Audio Description Generation in Contexts
Fang, B., Wu, W., Wu, Q., Song, Y., & Chan, A. B. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01267 · Cited by 2
“TV-AD [82] is a recently proposed AD dataset based on TVQA [30], which contains ∼ 31k ADs for training and ∼ 3k ADs for evaluation.”
LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answering
Zhang, H., Dong, L., Liu, Y., Huang, Y., Wang, L., Wang, L., & Qiao, Y. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02555-8 · Cited by 48
“For the number of words, on average, both questions and answers in our LvBench have a higher word count compared to previous datasets, especially the answers, which are approximately 3 × longer than those in TVQA [19] and MovieQA [48].”
InfluentialAffective question answering on video
Ruwa, N., Mao, Q., Wang, L., & Gou, J. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.06.046 · Cited by 2
McOmet: Multimodal Fusion Transformer for Physical Audiovisual Commonsense Reasoning
Zong, D., & Sun, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i5.25813 · Cited by 2
“2019) and TVQA (Lei et al. 2018) by constructing two multimodal sequences using all input modalities.”
RetractedContext-aware network with foreground recalibration for grounding natural language in video
Chen, C., & Gu, X. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-05807-z · Cited by 2
“Several researches such as temporal proposal generation [1], video summarization [2, 3], video question answering [4], video caption [5, 6] and grounding natural language in video [7] have been proposed for different scenarios.”
BackgroundMulti-Granularity Semantic Clues Extraction for Video Question Answering
Zhou, F., & Han, Y. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859764 · Cited by 2
“TVQA [11] exploited object-level relation via a Bi-LSTM, which is affected by the order of the input.”
MethodExtending Compositional Attention Networks for Social Reasoning in Videos
Sartzetaki, C., Paraskevopoulos, G., & Potamianos, A. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10858 · Cited by 2
“The VQA task can also be formulated with video content (Video QA) [10, 11, 12], where the input has a temporal dimension and may include audio and dialogue transcript.”
BackgroundRelation Enhanced Vision Language Pre-Training
Lee, J.-H., & Kang, J. (2022) · Proceedings - International Conference on Image Processing · doi:10.1109/icip46576.2022.9897623 · Cited by 2
“TR VCA VQA COCO [18] COCO [18] COCO [10] VQA 2.0 [12] TVQA [19] R@5 R@10 R@5 R@10 BLEU-4 METEOR CIDEr SPICE ACC.”
BackgroundMethodInfluentialA Multi-Layer Attention Network for Visual Commonsense Reasoning
Zhang, W., Gao, Y., Qian, H., & Lyu, H. (2022) · International Conference on Data Science and Information Technology (DSIT) · doi:10.1109/dsit55514.2022.9943834 · Cited by 2
“Extensive experiments show that the method can significantly improve the performance of VQA.”
BackgroundInfluentialSST-VLM: Sparse Sampling-Twice Inspired Video-Language Model
Gao, Y., & Lu, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26316-3_32 · Cited by 2
Cheng, S., Zhou, Z., Liu, J., Ye, J., Luo, H., & Gu, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44699-3_18 · Cited by 2
Multimodal Learning for Accurate Visual Question Answering: An Attention-based Approach
Bhardwaj, J., Balakrishnan, A., Pathak, S., Unnarkar, I., Gawande, A., & Ahmadnia, B. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_020 · Cited by 2
“The first step in pre-processing the MSCOCO VQA V2 dataset is data cleaning (Lei et al., 2018).”
MethodLocating Visual Explanations for Video Question Answering
Chen, X., Liu, R. Z. Z., Song, X., & Han, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67832-6_24 · Cited by 2
ManyModalQA: Modality Disambiguation and QA over Diverse InputsPreprint
Hannan, D., Jain, A., & Bansal, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.08034 · Cited by 6
Self-citationOpen-Ended Multi-Modal Relational Reason for Video Question Answering.Preprint
Luo, H., & Qin, R. (2020) · arXiv (Cornell University) · Cited by 2
Trying Bilinear Pooling in Video-QAPreprint
Winterbottom, T., Xiao, S. H., McLean, A., & Moubayed, N. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.10285 · Cited by 2
“Since there are multiple objects detected per frame (we cap it at 20 per frame), it is difficult to efficiently represent this in time sequences [28].”
BackgroundMethodInfluentialFill-in-the-blank as a Challenging Video Understanding Evaluation Framework.Preprint
Castro, S., Wang, R., Huang, P., Stewart, I., Liu, N., Stroud, J., & Mihalcea, R. F. (2021) · arXiv (Cornell University) · Cited by 2
Attend What You Need: Motion-Appearance Synergistic Networks for Video Question AnsweringPreprint
Seo, A., Kang, G.-C., Park, J., & Zhang, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10446 · Cited by 10
“In particular, video question answering (video QA) tasks (Xu et al., 2016; Jang et al., 2017; Lei et al., 2018; Yu et al., 2019; Choi et al., 2020) have been proposed with the goal of reasoning over higher-level vision-”
Toward a Human-Level Video Understanding IntelligencePreprint
Heo, Y., Lee, M. S., Choi, S.-H., Choi, W. S., Shin, M., Jung, M., Ryu, J.-K., & Zhang, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04203 · Cited by 2
“In addition, large-scale datasets have been built and publicly available to facilitate the researches (Alamri et al. 2019; Lei et al. 2018, 2020; Choi et al. 2021).”
MethodAssistSR: Affordance-centric Question-driven Video Segment RetrievalPreprint
Lei, S. W., Wang, Y., Mao, D., Gao, D., & Shou, M. Z. (2021) · arXiv (Cornell University) · Cited by 2
Episodic Question Answering for Cognitive AgentsPreprint
Singh, S., & Seshadri, S. (2024) · Research Square · doi:10.21203/rs.3.rs-4351479/v1 · Cited by 2
Temporal-aware Mechanism with Bidirectional Complementarity for Video Q&A
Luo, Y., Wang, R., Zhang, F., Zhou, F., & Lin, S. (2022) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53654.2022.9945377 · Cited by 1
“Traffic-QA Setting-1/4 ↑ Setting-1/2 ↑ BERT-VQA [23] 33.68 63.50 TVQA [24] 35.16 63.15 HCRN [13] 36.49 63.79 Eclipse [22] 37 module, we design an ablation experiment with parameter R att .”
MethodInfluentialGated Multi-modal Fusion with Cross-modal Contrastive Learning for Video Question Answering
Lyu, C., Li, W., Ji, T., Zhou, L., & Gurrin, C. G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44195-0_35 · Cited by 1
Task parse tree: Learning task policy from videos with task-irrelevant components
Wang, W., You, M., Zhou, H., & He, B. (2023) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2023.104552 · Cited by 1
“Previously on…” from Recaps to Story Summarization
SINGH, A. K., Srivastava, D., & Tapaswi, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01294 · Cited by 2
“Thus, for a given shot from the recap, we obtain all matching shots in the episode that are localized to a certain region of high-scoring similarity (see Fig.”
MethodMultimedia-Aware Question Answering: A Review of Retrieval and Cross-Modal Reasoning Architectures
Raja, R., & Vats, A. (2025) · doi:10.1145/3746274.3760393 · Cited by 3
“Furthermore, attention-based fusion is applied over different modality embeddings: This fused representation z is then used for downstream QA prediction.”
Valor32k-AVQA v2.0: Open-Ended Audio-Visual Question Answering Dataset and Benchmark
Riahi, I., Radman, A., Guo, Z., Hedjam, R., & Laaksonen, J. T. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3758261 · Cited by 1
“Existing AVQA datasets can be grouped into two main categories: domain-specific datasets [1, 19, 21, 37] and open-activity datasets [36].”
InfluentialEnd-to-End VideoQA with Frame Scoring Mechanisms and Adaptive Sampling
Liang, J., Meng, X., Wang, Y., Liu, C., Liu, Q., & Zhao, D. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3346-6_11 · Cited by 1
Image and Video Captioning and Visual Question Answering Using Machine Learning Models
Singh, S. K., Sharma, A. P., Saghir, U., Madan, K., Lal, R., & Singh, D. (2025) · International Conference on New Frontiers in Communication, Automation, Management and Security (ICCAMS) · doi:10.1109/iccams65118.2025.11233939 · Cited by 1
“Since deep learning (DL) was introduced, the qualitative and quantitative performance of VC systems has significantly improved [8], [9], and [10].”
Generating Tennis Action Instruction Based on a Large Language Model
Wang, Z., Xia, Z., Zhao, G., Lu, K., Xia, H., & Xia, S. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00683 · Cited by 1
Multi-emotion and intensity-driven response generation for richer multimodal dialogue
Singh, A., Shree, R., Pandey, D., & Singh, V. (2026) · Scientific Reports · doi:10.1038/s41598-026-41034-z · Cited by 1
“Given the wealth of information available in movie datasets, recent studies have explored the use of movies and TV series for various applications, including question-answering systems 28,29 that incorporate multiple modalities such as text, audio, and video.”
Coffee-Mate: Assisting keyframe selection via self-distillation for video question answering
Yuan, B., & You, S. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113740 · Cited by 1
DART: Difficulty-Adaptive Routing for Zero-Shot Video Temporal Grounding
Zhang, Z., Huang, H., Tu, Z., & Yang, M.-H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37095-2_5 · Cited by 2
Adaptive Proposal on Spatio-Temporal Graph in Audio-Visual Question Answering
Xie, Z., Yang, Y., Yu, Y., Wang, J., Peng, B., Li, X., & Liu, Y. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3714320 · Cited by 1
Fusing Temporally Distributed Multi-Modal Semantic Clues for Video Question Answering
Zhang, F., Wang, R., Xu, S., & Zhou, F. (2021) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme51207.2021.9428225 · Cited by 1
“TVQA[14] is based on six TV shows with of 152,545 question-answer pairs (Q/A pairs) from 21,800 clips.”
MethodInferential Knowledge-Enhanced Integrated Reasoning for Video Question Answering
Mao, J., Jiang, W., Liu, H., Wang, X., & Lyu, Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26570 · Cited by 1
“Prior works on VLQ-VideoQA mainly adopted the paradigm of cross-modal encoding and matching (Lei et al. 2018, 2020; Kim et al. 2019b, 2021).”
BackgroundMethodInfluentialLyu, C., Ji, T., Graham, Y., & Foster, J. (2023) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2023.sustainlp-1.12 · Cited by 1
Video Turing Test: A first step towards human‐level AI
Lee, M., Heo, Y., Choi, S.-H., Choi, W. S., & Zhang, B. (2023) · AI Magazine · doi:10.1002/aaai.12128 · Cited by 3
Dataset Bias Mitigation in Multiple-Choice Visual Question Answering and Beyond
Wang, Z., Chen, L., You, H., Xu, K., He, Y., Li, W., Codella, N., Chang, K.-W., & Chang, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.576 · Cited by 4
DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence Understanding
Guo, X., Li, Y.-F., & Haf, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.191 · Cited by 8
LocoMotion: Learning Motion-Focused Video-Language Representations
Doughty, H., Thoker, F. M., & Snoek, C. G. M. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0908-6_1 · Cited by 1
HouseOfTheDragonQA: Open-Domain Long-form Context-Aware QA Pairs for TV Series
Lahiri, A. K., & Hu, Q. (2024) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat62293.2024.00028 · Cited by 1
“Early influential works like the MovieQA [7] and TVQA [8] datasets enhanced models’ capabilities to understand and answer questions related to movie and TV series narratives.”
MARS2 2025 Challenge on Multimodal Reasoning: Datasets, Methods, Results, Discussion, and Outlook
Xu, P., Xiong, S., Zhang, J., Chen, Y., Zhou, B., Loy, C. C., Clifton, D. A., Lee, K. M., Gool, L. V., He, R., et al. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00681 · Cited by 1
“In contrast to short-duration factual datasets such as MSVD-QA [24] or TGIF-QA [50], and long-duration narrative datasets such as TVQA [64] and MovieQA [115], creative advertising videos are shorter in duration yet contain richer abstract cues.”
Segment-Then-Rank: Non-Factoid Question Answering on Instructional Videos
Lee, K., Duan, N., Ji, L., Li, J., & Hwang, S.-W. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6327 · Cited by 1
“Recently, with the creation of various datasets, such as TVQA (Lei et al. 2018), TGIF-QA (Jang et al. 2017), and MovieQA (Tapaswi et al. 2016), techniques for video-based QA have been advanced rapidly.”
BackgroundMethodCharacterizing Datasets for Social Visual Question Answering, and the New TinySocial Dataset
Chen, Z., Li, S., Rashedi, R. N., Zi, X., Elrod-Erickson, M., Hollis, B., Maliakal, A., Shen, X., Zhao, S., & Kunda, M. (2020) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/icdl-epirob48136.2020.9278057 · Cited by 1
“…video VQA datasets include social questions in addition to other more factual questions, such as the TVQA dataset that uses videos from six well-known TV shows [16]; PororoQA, that uses videos from a children’s animated television show [17]; and MovieQA that contains questions about movies [18].”
BackgroundQuestion Difficulty Estimation with Directional Modality Association in Video Question Answering
Kim, B.-M., & Park, S.-B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-08530-7_24 · Cited by 1
Semantic-Aware Dynamic Retrospective-Prospective Reasoning for Event-Level Video Question Answering
Lyu, C., Ji, T., Graham, Y., & Foster, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-srw.7 · Cited by 1
Video Question Generation via Cross-Modal Self-Attention Networks LearningPreprint
Wang, Y.-S., Su, H.-T., Chang, C.-H., Liu, Z.-Y., & Hsu, W. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.03049 · Cited by 1
“Moreover, the *: Both authors contributed equally to this research. ability to ask can also aid the development in popular vision-language fields, such as Visual Dialogue task[2], Video Question Answering[3], and Embodied Question Answering[4] by decreasing the expensive questions annotation cost.”
BackgroundMethodInfluentialYouMakeup VQA Challenge: Towards Fine-grained Action Understanding in Domain-Specific VideosPreprint
Chen, S., Wang, W., Ruan, L., Yao, L., & Jin, Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05573 · Cited by 1
“In recent years, video semantic understanding has attracted increasing research attentions [3, 6, 17, 8, 15, 19].”
Xu, L., Huang, H., & Liu, J. (2021) · arXiv (Cornell University) · Cited by 1
Pawar, K., Halbe, S., Patil, V., Gupta, V., & Attar, V. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4880715 · Cited by 1
Two-stream Spatiotemporal Feature for Video QA TaskPreprint
Song, C., Im, W., & Yoon, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.05006
“We test three baseline results adopted from the work of Lei et al. [3], which are tested with three kinds of video features: image indicates the ImageNet [18] pre-trained ResNet101 [24] features, which are extracted from convolutional layer5 after pooling and has 2048 dimension, region uses the…”
BackgroundMethodInfluentialInvestigating the Decoders of Maximum Likelihood Sequence Models: A Look-ahead ApproachPreprint
Wang, Y.-S., Kuo, Y., & Katz, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.03716 · Cited by 2
Knowledge-Based Visual Question Answering in VideosPreprint
García, N., Otani, M., Chu, C., & Nakashima, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.08385
“To address these limitations, video question answering (VideoQA) [7, 5] and knowledgebased visual question answering (KBVQA) [9, 8] have emerged as two independent fields.”
InfluentialCo-attentional Transformers for Story-Based Video UnderstandingPreprint
Bebensee, B., & Zhang, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14104
“One such dataset for video question answering is the TVQA dataset which is build around short 60 to 90 seconds long video clips and questions bridging vision and language clues [7].”
BackgroundAnalysis on Image Set Visual Question AnsweringPreprint
Khattar, A., Joshi, A., Singh, H. S., Goel, P., & Barnwal, R. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.00107 · Cited by 1
VGNMN: Video-grounded Neural Module Network to Video-Grounded Language TasksPreprint
Le, H., Chen, N. F., & Hoi, S. C. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07921
“Extended from image-grounded tasks, e.g. (Antol et al., 2015), recently Jang et al. (2017); Lei et al. (2018) propose to use video as the grounding features.”
BackgroundFast Temporal Information Retrieval In Videos With Visual Memory
Shin, J., & Moon, J. (2021) · International Conference on Artificial Intelligence in Information and Communication (ICAIIC) · doi:10.1109/icaiic51459.2021.9415226
MTVR: Multilingual Moment Retrieval in VideosPreprint
Lei, J., Berg, T. L., & Bansal, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.00061 · Cited by 3
“The TVR (Lei et al., 2020) dataset contains 108,965 high-quality English queries from 21,793 videos from 6 long-running TV shows (provided by TVQA (Lei et al., 2018)).”
BackgroundSelf-citationZhu, W., & Wang, X. Y. (2021) · Springer eBooks · doi:10.1007/978-3-030-88132-0_4
Video2Subtitle: Matching Weakly-Synchronized Sequences via Dynamic Temporal Alignment
Ben, X., Liu, C., & Mu, Y. (2022) · International Conference on Multimedia Retrieval · doi:10.1145/3512527.3531371
“For example, [26, 28, 42] were collected from real-world videos, while [23, 24, 35] were collected from movies and TV series.”
BackgroundResultInfluentialWu, Q., Wang, P., Wang, X. Y., He, X., & Zhu, W. (2022) · Advances in computer vision and pattern recognition · doi:10.1007/978-981-19-0964-1_8
Wu, Q., Wang, P., Wang, X. Y., He, X., & Zhu, W. (2022) · Advances in computer vision and pattern recognition · doi:10.1007/978-981-19-0964-1_1
A Multi-Stream Approach for Video Understanding
Kunam, L., Rossetto, L., & Bernstein, A. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3551567
“They employ a multimodal transformer that is pre-trained on large video captioning [17] and video question answering [15] datasets.”
BackgroundVideo in 10 Bits: Few-Bit VideoQA for Efficiency and Privacy
Huang, S., Piramuthu, R., Chang, S., & Sigurdsson, G. A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25072-9_49
Weight-based multi-stream model for Multi-Modal Video Question Answering
Rajesh, M., Sridhar, S., Kulkarni, C., Shah, A., & Natarajan, S. (2023) · Proceedings of the ... International Florida Artificial Intelligence Research Society Conference · doi:10.32473/flairs.36.133306
“The table below illustrates a comparison between the proposed model and other works(Lei et al. 2019; Le et al. 2021; Li et al. 2020; Zellers et al. 2021 78.70% By assigning varying weights to various video streams and effectively utilising the pre-trained model for each stream, it can be observed…”
MethodInfluentialQuestion difficulty estimation via enhanced directional modality association transformer
Kim, B.-M., Park, G.-M., & Park, S.-B. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04988-5
Mulan: A Multi-Level Alignment Model for Video Question Answering
Fu, Y., Cao, C., Yang, Y., Lu, Y., Yuan, F., Wang, D., & Liu, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.364
“In the realm of visual and language fusion methodologies, exploratory methods, as exemplified by the multi-stream approach(Lei et al., 2018), entail the utilization of a unified encoder to merge various information streams.”
MethodA Coarse and Fine Grained Masking Approach for Video-Grounded Dialogue
Xu, F., Zhou, W., Sun, T., Lu, J., Yu, Z., & Li, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-53308-2_30
Luo, Y., Wang, R., Zhang, F., Zhou, F., Liu, M., & Feng, J. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09482-8
IMF-MF: Interactive moment localization with adaptive multimodal fusion and self-attention
Singh, P., Kushwaha, A. K. S., & Varshney, N. (2024) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-233071
Leveraging Task-Specific Pre-Training to Reason across Images and Videos
Sadhu, A., & Nevatia, R. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00569
“We opt for multiple-choice due to ease of evaluation similar to previous work in common-sense reasoning [19,48].”
ResultLatipov, I.-A., Borevskiy, A., & Kertész‐Farkas, A. (2024) · International Conference on Computer Vision and Computational Intelligence (CVCI) · doi:10.1117/12.3027602
“Some datasets combines QA with video question answering (KnowIT-VQA [19] or TVQA [20]) which is more complex task as require reasoning through time.”
Traffic Incident Reasoning Through Multi-Grained Conceptual Graph Networks
Sattarzadeh, A. R., & Pathirana, P. N. (2024) · International Conference on Soft Computing & Machine Intelligence (ISCMI) · doi:10.1109/iscmi63661.2024.10851673
Lee, D., Shim, J. Y., Yoon, S., & Yoo, C. D. (2025) · Lecture notes in computer science · doi:10.1007/978-981-97-8702-9_8
Audiopedia: Audio QA with Knowledge
Penamakuri, A. S., Chhatre, K., & Jain, A. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889814 · Cited by 2
“…(ii) Visual Question Answering (VQA) [22]–[25]: question to be answered over an image, (iii) Audio Question Answering (AQA) [4]–[7]: question to be answered in the context of an audio, and (iv) Video Question Answering (VideoQA) [26]–[28]: question to be answered in the context of an video.”
ViCo: A Multitask Video-enhanced and Cognition-preserving Modality Alignment Training Framework
Yu, Z., Chen, J., Shen, J., Zhou, L., Fang, H., Zang, X., Ban, C., Chen, J., He, Z., Sun, H., et al. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888636 · Cited by 1
VFE: A large-scale video future event description dataset for evaluating video temporal prediction
Lai, C., & Wang, H. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06547-6
“Against this back-drop, applications such as video question answering [3] and video summarization [4] have been empirically evaluated using existing datasets, demonstrating the effectiveness of video models.”
Integration of Video Semantic Understanding and Chatbot Systems: A Review
Cao, L. (2025) · Lecture notes in electrical engineering · doi:10.1007/978-981-96-4967-9_114
Actions and Objects Pathways for Domain Adaptation in Video Question Answering
Mohamud, S. A. M., & Jung, H. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-6969-1_1
Leveraging LLM’s Mastery in Video Question Answering
Rakshitha, R. T., Desai, P., Sujatha, C., & Gangisetty, S. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-2700-4_38
ThriveBuddy: Towards Student Well-being Through AI-Powered Digital Mentorship
Palahan, S. (2024) · IIAI International Congress on Advanced Applied Informatics (IIAI-AAI) · doi:10.1109/iiai-aai-winter65925.2024.00023
“The TVQA dataset [15] was also presented for video-based question answering, focusing on spatio-temporal reasoning.”
Explicit-implicit feature modeling in drama videos: dataset and benchmark
Du, Z., & Wu, X. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06699-5
SRIN: Structured Reasoning Integration Network for Robust Video Question AnsweringPreprint
Yamada, K. (2025) · Preprints.org · doi:10.20944/preprints202508.1438.v1
Multimodal Large Language Models for Video Understanding
Wang, Y., Yu, J., He, Y., Wang, L., & Qiao, Y. (2025) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-031-94969-2_3
ICT-QA: Question Answering Over Multi-Modal Contexts Including Image, Chart, and Text Modalities
Jang, Y., Kong, H., Kim, G., Lee, Y., Choi, J., & Bae, K. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00019
A Needle in a Haystack: Finding Contextual Knowledge for Video Question Answering
Ichimiya, S., & Nakashima, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4398-4_25
Through the Theory of Mind’s Eye: Reading Minds with Multimodal Video Large Language Models
Chen, Z., Wang, T., Wang, Y., Kosiński, M., Zhang, X. D., Fu, Y., & Li, S. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11229224 · Cited by 10
“A notable example is the “Movie Time Social Learning” program [14], which uses movie discussions to assist children with autism in grasping social contexts and understanding others’ points of view.”
LEARNet: A Learning Entropy-Aware Representation Network for Educational Video Understanding
Chitrakala, S., Nivedha, V. V., & R, N. S. (2025) · Entropy · doi:10.3390/e28010003
“[64], HowTo100M [65], and COIN [66] focus on dynamic content with limited visual annotations, and specialized educational systems like LectureNet [67] and EduNet [68] offer basic keyframe extraction without detailed region-level labels, EVUD-2M addresses the critical gap for comprehensive…”
Semantic-Driven Video Question Answering with Fine-Grained Spatio-Temporal Interaction
Zhang, J., Yang, Q., Pang, S., Shen, Y., Wang, S., & Zhang, F. (2024) · doi:10.1109/isdh64927.2024.00057
“Similarly, Lei et al [21]. proposed a method called structured two-stream network, which aims to capture answers related to the question by separately modeling the multimodal information in the video.”
VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflection
Wang, Q., Gao, X., He, Y., Han, J., Li, J., Dong, S., Ma, Z., & Gong, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37592-6_30
Wang, X., Lan, X., & Zhu, W. (2025) · Springer eBooks · doi:10.1007/978-3-031-94837-4_1
Wang, X., Lan, X., & Zhu, W. (2025) · Springer eBooks · doi:10.1007/978-3-031-94837-4_3
It’s Just Another Day: Unique Video Captioning by Discriminitive Prompting
Perrett, T., Han, T., Damen, D., & Zisserman, A. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02664-4
“Long-video is again being studied, due to the rise of large-scale instructional (Zhou et al., 2018; Miech et al., 2019) and egocentric datasets (Damen et al., 2022; Grauman et al., 2022), and the ability of models to operate on larger temporal windows (Wu et al., 2022; Wang, 2022; Liu et al., 2023).”
TBGA: Tripartite and Bipartite Graph Attention for Multi-modal Video Question Answering
Kim, J. (2025) · International Conference on Electrical, Control and Instrumentation Engineering (ICECIE) · doi:10.1109/icecie66637.2025.11363814
“To tackle the complexities of multi-modal reasoning, recent studies [5,6,7,9,10,40,41] have focused on developing sophisticated deep architectures.”
InfluentialLexical Distractor Mining Network for Causal Video Question Answering
Kim, J., & Yoon, S. (2026) · IEEE Access · doi:10.1109/access.2026.3666880
FuzzySeek: Multimodal Refinement of Imprecise Video Queries for Moment Retrieval
Mishra, A., Niinuma, K., & Gupta, A. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789095
“State-of-the-art video–retrieval systems are evaluated on datasets with carefully phrased, single–ground-truth queries (e.g., [6, 23, 25, 48, 50, 52]), deliberately minimizing imprecision.”
Metadata
- Authors
- Lei, J., Yu, L., Bansal, M., & Berg, T.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 1369-1379
- OpenAlex
- W2963541336
- Cited by
- 541
- References
- 49
- Citations by year
- 2026: 39 · 2025: 50 · 2024: 65 · 2023: 93 · 2022: 90 · 2021: 101 · 2020: 70 · 2019: 32 · 2018: 1
- FWCI
- 15.13
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1405822, 1562098, 1633295)
- Army Research Office (W911NF)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- video question answering, video localization, television series, end-to-end learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
