Hierarchical Conditional Relation Networks for Video Question Answering
Open accessAbstract
Video question answering (VideoQA) is challenging as it requires modeling capacity to distill dynamic visual artifacts and distant relations and to associate them with linguistic concepts. We introduce a general-purpose reusable neural unit called Conditional Relation Network (CRN) that serves as a building block to construct more sophisticated structures for representation and reasoning over video. CRN takes as input an array of tensorial objects and a conditioning feature, and computes an array of encoded output objects. Model building becomes a simple exercise of replication, rearrangement and stacking of these reusable units for diverse modalities and contextual information. This design thus supports high-order relational and multi-step reasoning. The resulting architecture for VideoQA is a CRN hierarchy whose branches represent sub-videos or clips, all sharing the same question as the contextual condition. Our evaluations on well-known datasets achieved new SoTA results, demonstrating the impact of building a general-purpose reasoning unit on complex domains such as VideoQA.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The most common existing VQA methods use attention and memory mechanisms to learn visual artifacts and the semantic correlations needed to answer questions accurately.
“Currently, the most common existing methods use attention [125, 127, 129] and memory [126] mechanisms to efficiently learn visual artifacts and the semantic correlations that allow questions to be answered accurately.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [127]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · see citing works
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2015) · arXiv · doi:10.48550/arxiv.1511.07289 · Cited by 4,513 · see citing works
A Closer Look at Spatiotemporal Convolutions for Action Recognition
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., & Paluri, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00675 · Cited by 3,672 · see citing works
Film: Visual reasoning with a general conditioning layer
Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11671 · Cited by 2,490 · see citing works
Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
Hara, K., Kataoka, H., & Satoh, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00685 · Cited by 2,300 · see citing works
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.571 · Cited by 1,810 · see citing works
Temporal Relational Reasoning in Videos
Zhou, B., Andonian, A., Oliva, A., & Torralba, A. B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_49 · Cited by 1,104 · see citing works
(2017) · International Conference on Machine Learning · Cited by 758 · see citing works
MovieQA: Understanding Stories in Movies through Question-Answering
Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A. B., Urtasun, R., & Fidler, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.501 · Cited by 646 · see citing works
TVQA: Localized, Compositional Video Question Answering
Lei, J., Yu, L., Bansal, M., & Berg, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1167 · Cited by 541 · see citing works
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
Jang, Y., Song, Y., Yu, Y., Kim, Y., & Kim, G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.149 · Cited by 496 · see citing works
Long-Term Feature Banks for Detailed Video Understanding
Wu, C.-Y., Feichtenhofer, C., Fan, H., He, K., Krähenbühl, P., & Girshick, R. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00037 · Cited by 454 · see citing works
Video Question Answering via Gradually Refined Attention over Appearance and Motion
Xu, D., Zhao, Z., Xiao, J., Wu, F., Zhang, H., He, X., & Zhuang, Y. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123427 · Cited by 438 · see citing works
Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning
Pan, P., Xu, Z., Yang, Y., Wu, F. Q., & Zhuang, Y. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.117 · Cited by 417 · see citing works
Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
Fan, C., Zhang, X., Zhang, S., Wang, W., Zhang, C., & Huang, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00210 · Cited by 293 · see citing works
Beyond RNNs: Positional Self-Attention with Co-Attention for Video Question Answering
Li, X., Song, J., Gao, L., Liu, X., Huang, W., He, X., & Gan, C. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018658 · Cited by 265 · see citing works
Motion-Appearance Co-memory Networks for Video Question Answering
Gao, J., Ge, R., Chen, K., & Nevatia, R. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00688 · Cited by 252 · see citing works
Uncovering the Temporal Context for Video Question Answering
Zhu, L., Xu, Z., Yang, Y., & Hauptmann, A. G. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1033-7 · Cited by 210 · see citing works
Hierarchical Boundary-Aware Neural Encoder for Video Captioning
Baraldi, L., Grana, C., & Cucchiara, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.339 · Cited by 198 · see citing works
Leveraging Video Descriptions to Learn Video Question Answering
Zeng, K.-H., Chen, T.-H., Chuang, C.-Y., Liao, Y.-H., Niebles, J. C., & Sun, M. (2017) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v31i1.11238 · Cited by 162 · see citing works
DeepStory: Video Story QA by Deep Embedded Memory Networks
Kim, K.-M., Heo, M.-O., Choi, S.-H., & Zhang, B. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/280 · Cited by 145 · see citing works
Focal Visual-Text Attention for Visual Question Answering
Liang, J., Jiang, L., Cao, L., Li, L.-J., & Hauptmann, A. G. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00642 · Cited by 105 · see citing works
A Read-Write Memory Network for Movie Story Understanding
Na, S., Lee, S., Kim, J., & Kim, G. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.80 · Cited by 101 · see citing works
Video Question Answering via Attribute-Augmented Attention Network Learning
Ye, Y., Zhao, Z., Li, Y., Chen, L., Xiao, J., & Zhuang, Y. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080655 · Cited by 98 · see citing works
Progressive Attention Memory Network for Movie Story Question Answering
Kim, J., Ma, M., Kim, K., Kim, S.-J., & Yoo, C. D. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00853 · Cited by 90 · see citing works
Multimodal Dual Attention Memory for Video Story Question Answering
Kim, K.-M., Choi, S.-H., Kim, J.-H., & Zhang, B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01267-0_41 · Cited by 78 · see citing works
Learnable Aggregating Net with Diversity Learning for Video Question Answering
Li, X., Gao, L., Wang, X., Liu, W., Xu, X., Shen, H. T., & Song, J. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350971 · Cited by 72 · see citing works
Abstracting home video automatically
Lienhart, R. W. (1999) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/319878.319888 · Cited by 60 · see citing works
Multi-interaction Network with Object Relation for Video Question Answering
Jin, W., Zhao, Z., Gu, M., Yu, J., Xiao, J., & Zhuang, Y. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3351065 · Cited by 60 · see citing works
Explore Multi-Step Reasoning in Video Question Answering
Song, X., Shi, Y., Chen, X., & Han, Y. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240563 · Cited by 59 · see citing works
Temporal Modeling Approaches for Large-scale Youtube-8M Video UnderstandingPreprint
Li, F., Gan, C., Liu, X., Bian, Y., Long, X., Li, Y., Li, Z., Zhou, J., & Wen, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.04555 · Cited by 51 · see citing works
Movie Question Answering: Remembering the Textual Cues for Layered Visual Contents
Wang, B., Xu, Y., Han, Y., & Hong, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12253 · Cited by 46 · see citing works
Multi-Turn Video Question Answering via Multi-Stream Hierarchical Attention Context Network
Zhao, Z., Jiang, X., Cai, D., Xiao, J., He, X., & Pu, S. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/513 · Cited by 43 · see citing works
Long-Form Video Question Answering via Dynamic Hierarchical Reinforced Networks
Zhao, Z., Zhang, Z., Xiao, S., Xiao, Z., Yan, X., Yu, J., Cai, D., & Wu, F. Q. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2922062 · Cited by 39 · see citing works
Non-local NetVLAD Encoding for Video Classification
Tang, Y., Zhang, X., Wang, J., Chen, S., Ma, L., & Jiang, Y. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11018-5_20 · Cited by 39 · see citing works
Hierarchical Relational Attention for Video Question Answering
Chowdhury, M. I. H., Nguyen, K., Sridharan, S., et al. (2018) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2018.8451103 · Cited by 37 · see citing works
Hierarchical Video Frame Sequence Representation with Deep Convolutional Graph Network
Mao, F., Wu, X., Xue, H., & Zhang, R. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11018-5_24 · Cited by 37 · see citing works
Question-Aware Tube-Switch Network for Video Question Answering
Yang, T., Zha, Z.-J., Xie, H., et al. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350969 · Cited by 24 · see citing works
Holistic Multi-Modal Memory Network for Movie Question Answering
Wang, A., Luu, A. T., Foo, C.-S., Zhu, H., Tay, Y., & Chandrasekhar, V. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2931534 · Cited by 17 · see citing works
Learning Deep Matrix RepresentationsPreprint
Do, K., Tran, T., & Venkatesh, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01454 · Cited by 9 · see citing works
Self-citationLearning to Reason with Relational Video Representation for Question Answering.
Le, T. M., Le, V., Venkatesh, S., & Tran, T. (2019) · Cited by 6 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 5
- Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bilin- ear attention networks. In Advances in Neural Information Processing Systems, pages 1564-1574, 2018. 2
- Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio- temporal representation with pseudo-3d residual networks. In Proceedings of the IEEE International Conference on Com- puter Vision, pages 5533-5541, 2017. 2
- Gursimran Singh, Leonid Sigal, and James J Little. Spatio- temporal relational reasoning for video question answering. In BMVC, 2019. 2
- Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. Multi- modal factorized bilinear pooling with co-attention learning for visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 1821- 1830, 2017. 2
- Zhou Zhao, Qifan Yang, Deng Cai, Xiaofei He, and Yueting Zhuang. Video question answering via hierarchical spatio- temporal attention networks. In IJCAI, pages 3518-3524, 2017. 2
Cited by
Sorted by citations
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
- Authors
- Le, T. M., Le, V., Venkatesh, S., & Tran, T.
- Affiliations
- Deakin University (AU)
- Venue
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2020-06
- Pages
- 9969-9978
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3008341877
- Cited by
- 260
- References
- 44
- Citations by year
- 2026: 20 · 2025: 25 · 2024: 52 · 2023: 76 · 2022: 41 · 2021: 42 · 2020: 4
- FWCI
- 12.57
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- video question answering, hierarchical neural architecture, multi-step reasoning, video representation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
