Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers
Open accessAbstract
Given an input video, its associated audio, and a brief caption, the audio-visual scene aware dialog (AVSD) task requires an agent to indulge in a question-answer dialog with a human about the audio-visual content. This task thus poses a challenging multi-modal representation learning and reasoning scenario, advancements into which could influence several human-machine interaction applications. To solve this task, we introduce a semantics-controlled multi-modal shuffled Transformer reasoning framework, consisting of a sequence of Transformer modules, each taking a modality as input and producing representations conditioned on the input question. Our proposed Transformer variant uses a shuffling scheme on their multi-head outputs, demonstrating better regularization. To encode fine-grained visual information, we present a novel dynamic scene graph representation learning pipeline that consists of an intra-frame reasoning layer producing spatio-semantic graph representations for every frame, and an inter-frame aggregation module capturing temporal cues. Our entire pipeline is trained end-to-end. We present experiments on the benchmark AVSD dataset, both on answer generation and selection tasks. Our results demonstrate state-of-the-art performances on all evaluation metrics.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Vision Language Pretraining follows two general pipelines: two-stage (needs object detector) and end-to-end.
“Vision Language Pretraining (VLP) follows two general pipelines: two-stage (need object detector, e.g., Faster R-CNN [202]) (e.g., LXMERT [103], ViLBert [102], VL-Bert [105], UNITER [106]) and end-to-end (e.g., Pixel-Bert [113], SOHO [203], KD-VLP [204], Simvlm [199]).”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [202]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Graph Attention NetworksPreprint
Veličković, P., Casanova, A., Liò, P., Cucurull, G., Romero, A., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.17863/cam.48429 · Cited by 12,099 · see citing works
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596 · see citing works
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01083 · Cited by 9,982 · see citing works
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00716 · Cited by 9,982 · see citing works
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Dynamic Graph CNN for Learning on Point Clouds
Wang, Y., Sun, Y., Liu, Z., Sarma, S. E., Bronstein, M. M., & Solomon, J. M. (2019) · ACM Transactions on Graphics · doi:10.1145/3326362 · Cited by 7,199 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · see citing works
CNN architectures for large-scale audio classification
Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F., Jansen, A., Moore, R. C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., et al. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952132 · Cited by 2,604 · see citing works
MMDetection: Open MMLab Detection Toolbox and Benchmark
Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., et al. (2019) · arXiv · doi:10.48550/arxiv.1906.07155 · Cited by 2,227 · see citing works
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614 · see citing works
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · see citing works
Sequence to Sequence -- Video to Text
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., & Saenko, K. (2015) · IEEE International Conference on Computer Vision · doi:10.1109/iccv.2015.515 · Cited by 1,507 · see citing works
Structural-RNN: Deep Learning on Spatio-Temporal Graphs
Jain, A., Zamir, A. R., Savarese, S., & Saxena, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.573 · Cited by 1,133 · see citing works
Visual Relationship Detection with Language Priors
Lu, C., Krishna, R., Bernstein, M. S., & Fei-Fei, L. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46448-0_51 · Cited by 1,118 · see citing works
Image retrieval using scene graphs
Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D. A., Bernstein, M. S., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298990 · Cited by 1,101 · see citing works
VQA: Visual Question AnsweringPreprint
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Batra, D., & Parikh, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00468 · Cited by 968 · see citing works
Self-Attention Graph PoolingPreprint
Lee, J., Lee, I., & Kang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.08082 · Cited by 869 · see citing works
When does label smoothing help?
Müller, R., Kornblith, S., & Hinton, G. (2019) · arXiv · doi:10.48550/arxiv.1906.02629 · Cited by 859 · see citing works
Auto-Encoding Scene Graphs for Image Captioning
Yang, X., Tang, K., Zhang, H., & Cai, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01094 · Cited by 852 · see citing works
Videos as Space-Time Region Graphs
Wang, X., & Gupta, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01228-1_25 · Cited by 786 · see citing works
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
Jang, Y., Song, Y., Yu, Y., Kim, Y., & Kim, G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.149 · Cited by 496 · see citing works
Das, A., Kottur, S., Gupta, K., Singh, A., Jain, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · Cited by 432 · see citing works
Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering
Gao, P., Jiang, Z., You, H., Lu, P., Hoi, S. C. H., Wang, X., & Li, H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00680 · Cited by 415 · see citing works
Self-citationAttention-Based Multimodal Fusion for Video Description
Hori, C., Hori, T., Lee, T.-Y., Zhang, Z., Harsham, B., Hershey, J. R., Marks, T. K., & Sumi, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.450 · Cited by 401 · see citing works
Self-citationMultimodal Compact Bilinear Pooling for Visual Question Answering and Visual GroundingPreprint
Fukui, A., Park, D. H., Yang, D., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01847 · Cited by 388 · see citing works
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs
Ji, J., Krishna, R., Fei-Fei, L., & Niebles, J. C. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01025 · Cited by 326 · see citing works
Clotho: an Audio Captioning Dataset
Drossos, K., Lipping, S., & Virtanen, T. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9052990 · Cited by 285 · see citing works
Spatio-Temporal Graph for Video Captioning With Knowledge Distillation
Pan, B., Cai, H., Huang, D.-A., Lee, K.-H., Gaidon, A., Adeli, E., & Niebles, J. C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01088 · Cited by 281 · see citing works
Deep Audio-visual Learning: A Survey
Zhu, H., Luo, M., Wang, R., Zheng, A., & He, R. (2021) · Machine Intelligence Research · doi:10.1007/s11633-021-1293-0 · Cited by 185 · see citing works
Know More Say Less: Image Captioning Based on Scene Graphs
Li, X., & Jiang, S. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2896516 · Cited by 181 · see citing works
Audio Visual Scene-Aware Dialog
Alamri, H., Cartillier, V., Das, A., Wang, J., Cherian, A., Essa, I. A., Batra, D., Marks, T. K., Hori, C., Anderson, P., et al. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00774 · Cited by 172 · see citing works
Self-citationLearning Conditioned Graph Structures for Interpretable Visual Question AnsweringPreprint
Norcliffe-Brown, W., Vafeias, E., & Parisot, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07243 · Cited by 148 · see citing works
Talk2Car: Taking Control of Your Self-Driving Car
Deruyttere, T., Vandenhende, S., Grujicic, D., Gool, L. V., & Moens, M.-F. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1215 · Cited by 135 · see citing works
Are You Talking to Me? Reasoned Visual Dialog Generation Through Adversarial Learning
Wu, Q., Wang, P., Shen, C., Reid, I. D., & van den Hengel, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00639 · Cited by 131 · see citing works
End-to-end Audio Visual Scene-aware Dialog Using Multimodal Attention-based Video Features
Hori, C., Alamri, H., Wang, J., Wichern, G., Hori, T., Cherian, A., Marks, T. K., Cartillier, V., Lopes, R. G., Das, A., et al. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682583 · Cited by 112 · see citing works
Self-citationMulti-step Reasoning via Recurrent Dual Attention for Visual Dialog
Gan, Z., Cheng, Y., Kholy, A. E., Li, L., Liu, J., & Gao, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1648 · Cited by 108 · see citing works
Schwartz, I., Yu, S., Hazan, T., & Schwing, A. G. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00214 · Cited by 108 · see citing works
Video Relationship Reasoning Using Gated Spatio-Temporal Energy Graph
Tsai, Y.-H. H., Divvala, S., Morency, L., Salakhutdinov, R., & Farhadi, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01067 · Cited by 98 · see citing works
Video Action Transformer Network
Girdhar, R., Carreira, J., Doersch, C., & Zisserman, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00033 · Cited by 93 · see citing works
Multimodal transformer networks for end-to-end video-grounded dialogue systems
LE, H., SAHOO, D., Chen, N. F., & Hoi, S. C. H. (2019) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 89 · see citing works
Yang, G. H., Chaisorn, L., Zhao, Y., Neo, S.-Y., & Chua, T. (2003) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/957013.957146 · Cited by 84 · see citing works
Multi-Modality Latent Interaction Network for Visual Question Answering
Gao, P., You, H., Zhang, Z., Wang, X., & Li, H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00592 · Cited by 84 · see citing works
Self-citationSpatio-Temporal Action Graph Networks
Herzig, R., Levi, E., Xu, H., Gao, H., Brosh, E., Wang, X., Globerson, A., & Darrell, T. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00288 · Cited by 79 · see citing works
Ghosh, S., Burachas, G., Ray, A., & Ziskind, A. (2019) · arXiv.org · doi:10.48550/arxiv.1902.05715 · Cited by 69 · see citing works
A Simple Baseline for Audio-Visual Scene-Aware Dialog
Schwartz, I., Schwing, A. G., & Hazan, T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01283 · Cited by 69 · see citing works
Improving Grounded Natural Language Understanding through Human-Robot Dialog
Thomason, J., Padmakumar, A., Sinapov, J., Walker, N., Jiang, Y., Yedidsion, H., Hart, J., Stone, P., & Mooney, R. J. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794287 · Cited by 63 · see citing works
Multi-modality Latent Interaction Network for Visual Question AnsweringPreprint
Gao, P., You, H., Zhang, Z., Wang, X., & Li, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04289 · Cited by 55 · see citing works
Self-citationSpatio-Temporal Graph for Video Captioning with Knowledge DistillationPreprint
Pan, B., Cai, H., Huang, D.-A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.13942 · Cited by 30 · see citing works
BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detection
Ben-younes, H., Cadène, R., Thome, N., & Cord, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018102 · Cited by 27 · see citing works
Multi-step Reasoning via Recurrent Dual Attention for Visual DialogPreprint
Gan, Z., Cheng, Y., Kholy, A. E., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00579 · Cited by 27 · see citing works
VideoQA: Question answering on news video
Yang, H., Chaisorn, L., Zhao, Y., Neo, S.-Y., & Chua, T.-S. (2003) · Cited by 27 · see citing works
Video-Grounded Dialogues with Pretrained Generation Language Models
Le, H., & Hoi, S. C. H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.518 · Cited by 25 · see citing works
Graphical Contrastive Losses for Scene Graph Generation.Preprint
Zhang, J., Shih, K. J., Elgammal, A., Tao, A., & Catanzaro, B. (2019) · arXiv (Cornell University) · Cited by 22 · see citing works
Multimodal Attention for Fusion of Audio and Spatiotemporal Features for Video Description
Hori, C., Hori, T., Wichern, G., Wang, J., Lee, T., Cherian, A., & Marks, T. K. (2018) · Cited by 21 · see citing works
Self-citationContrastive Visual-Linguistic PretrainingPreprint
Shi, L., Shuang, K., Geng, S., Su, P., Jiang, Z., Gao, P., Fu, Z., de Melo, G., & Su, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.13135 · Cited by 19 · see citing works
Self-citationAction Genome: Actions as Composition of Spatio-temporal Scene GraphsPreprint
Ji, J., Krishna, R., Fei-Fei, L., & Niebles, J. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.06992 · Cited by 18 · see citing works
Are You Talking to Me? Reasoned Visual Dialog Generation through Adversarial LearningPreprint
Wu, Q., Wang, P., Shen, C., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.07613 · Cited by 16 · see citing works
Deep Audio-Visual Learning: A SurveyPreprint
Zhu, H., Luo, M., Wang, R., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.04758 · Cited by 16 · see citing works
Multi-Layer Content Interaction Through Quaternion Product for Visual Question Answering
Shi, L., Geng, S., Shuang, K., Hori, C., Liu, S., Gao, P., & Su, S. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053595 · Cited by 14 · see citing works
Self-citationCharacter Matters: Video Story Understanding with Character-Aware RelationsPreprint
Geng, S., Zhang, J., Fu, Z., Gao, P., Zhang, H., & de Melo, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.08646 · Cited by 11 · see citing works
Self-citationReactive Multi-Stage Feature Fusion for Multimodal Dialogue ModelingPreprint
Yeh, Y. T., Lin, T.-C., Cheng, H.-H., Deng, Y.-H., Su, S., & Chen, Y.-N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.05067 · Cited by 10 · see citing works
2nd Place Solution to the GQA Challenge 2019Preprint
Geng, S., Zhang, J., Zhang, H., Elgammal, A., & Metaxas, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.06794 · Cited by 5 · see citing works
Self-citationMulti-Layer Content Interaction Through Quaternion Product For Visual Question AnsweringPreprint
Shi, L., Geng, S., Shuang, K., Hori, C., Liu, S., Gao, P., & Su, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.05840 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
Fast Convergence of DETR with Spatially Modulated Co-Attention
Gao, P., Zheng, M., Wang, X., Dai, J., & Li, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00360 · Cited by 386 · see citing works
Self-citationMultimodal Human–Robot Interaction for Human‐Centric Smart Manufacturing: A Survey
Wang, T., Zheng, P., Li, S., & Wang, L. (2023) · Advanced Intelligent Systems · doi:10.1002/aisy.202300359 · Cited by 204 · see citing works
Video Graph Transformer for Video Question Answering
Xiao, J., Zhou, P., Chua, T., & Yan, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_3 · Cited by 99 · see citing works
Path Language Modeling over Knowledge Graphsfor Explainable Recommendation
Geng, S., Fu, Z., Tan, J., Ge, Y., de Melo, G., & Zhang, Y. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511937 · Cited by 89 · see citing works
Self-citationContrastive Video Question Answering via Video Graph Transformer
Xiao, J., Zhou, P., Yao, A., Li, Y., Hong, R., Yan, S., & Chua, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292266 · Cited by 62 · see citing works
Constructing Holistic Spatio-Temporal Scene Graph for Video Semantic Role Labeling
Zhao, Y., Fei, H., Cao, Y., Li, B., Zhang, M., Wei, J., Zhang, M., & Chua, T. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612096 · Cited by 46 · see citing works
Container: Context Aggregation NetworkPreprint
Gao, P., Lu, J., Li, H., Mottaghi, R., & Kembhavi, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01401 · Cited by 63 · see citing works
Self-citationEfficient Non-Sampling Knowledge Graph Embedding
Li, Z., Ji, J., Fu, Z., Ge, Y., Xu, S., Chen, C., & Zhang, Y. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449859 · Cited by 39 · see citing works
Self-citation(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering
Cherian, A., Hori, C., Marks, T. K., & Roux, J. L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19922 · Cited by 37 · see citing works
Self-citationRobust Knowledge Adaptation for Dynamic Graph Neural Networks
Li, H., Li, C., Feng, K., Yuan, Y., Wang, G., & Hongyuan Zha, B. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3388453 · Cited by 35 · see citing works
Self-citationMultimodal Conversational AI: A Survey of Datasets and Approaches
Sundar, A., & Heck, L. P. (2022) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2022.nlp4convai-1.12 · Cited by 37 · see citing works
Deep Representation Learning: Fundamentals, Technologies, Applications, and Open Challenges
Payandeh, A., Baghaei, K. T., Fayyazsanavi, P., Ramezani, S. B., Chen, Z., & Rahimi, S. (2023) · IEEE Access · doi:10.1109/access.2023.3335196 · Cited by 41 · see citing works
Shah, A., Geng, S., Gao, P., Cherian, A., Hori, T., Marks, T. K., Roux, J. L., & Hori, C. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746481 · Cited by 18 · see citing works
Self-citationAchieving EEG-based depression recognition using Decentralized-Centralized structure
Shao, X., Ying, M., Zhu, J., Li, X., & Hu, B. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106402 · Cited by 19 · see citing works
Contrastive Visual-Linguistic PretrainingPreprint
Shi, L., Shuang, K., Geng, S., Su, P., Jiang, Z., Gao, P., Fu, Z., de Melo, G., & Su, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.13135 · Cited by 19 · see citing works
Self-citationCAD - Contextual Multi-modal Alignment for Dynamic AVQA
Nadeem, A., Hilton, A., Dawes, R., Thomas, G., & Mustafa, A. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00709 · Cited by 15 · see citing works
Fast Convergence of DETR with Spatially Modulated Co-AttentionPreprint
Gao, P., Zheng, M., Wang, X., Dai, J., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.07448 · Cited by 12 · see citing works
Self-citationAction recognition in compressed domains: A survey
Ming, Y., Zhou, J., Hu, N., Feng, F., Zhao, P., Lyu, B., & Yu, H. L. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127389 · Cited by 11 · see citing works
Video Dialog as Conversation About Objects Living in Space-Time
Pham, H.-A., Le, T. M., Le, V., Phuong, T. M., & Tran, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19842-7_41 · Cited by 11 · see citing works
Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue
Yoon, S., Yoon, E., Yoon, H. S., Kim, J., & Yoo, C. D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.280 · Cited by 11 · see citing works
DialogMCF: Multimodal Context Flow for Audio Visual Scene-Aware Dialog
Chen, Z., Liu, H., & Wang, Y. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3284511 · Cited by 10 · see citing works
RomeBERT: Robust Training of Multi-Exit BERTPreprint
Geng, S., Gao, P., Fu, Z., & Zhang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.09755 · Cited by 18 · see citing works
Self-citationSpatio-Temporal Graph Convolution Transformer for Video Question Answering
Tang, J., Hu, J., Huang, W., Shen, S., Pan, J., Wang, D.-M., & Ding, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3445636 · Cited by 5 · see citing works
A Survey on Multimodal Dialogue Systems: Recent Advances and New Frontiers
Liu, G., Wang, S., Yu, J., & Yin, J. J. (2022) · International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE) · doi:10.1109/aemcse55572.2022.00170 · Cited by 6 · see citing works
Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generation
Zhao, X., Wang, Y., Tao, C., Wang, C., & Zhao, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.442 · Cited by 5 · see citing works
CE-BART: Cause-and-Effect BART for Visual Commonsense Generation
Kim, J., Hong, J. W., Yoon, S., & Yoo, C. D. (2022) · Sensors · doi:10.3390/s22239399 · Cited by 3 · see citing works
Multi-Pass Transformer for Machine TranslationPreprint
Gao, P., Hori, C., Geng, S., Hori, T., & Roux, J. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.11382 · Cited by 3 · see citing works
Self-citationLin, Z., Wu, Y., Chen, J., & Wang, S. (2024) · Guidance Navigation and Control · doi:10.1142/s2737480724410073 · Cited by 2 · see citing works
Scene-aware Graph-enhanced Multimodal Collaboration for Video-grounded Dialogue
Du, S.-S., & Wang, H. (2025) · International Symposium on Machine Learning and Media Computing (MLMC) · doi:10.1109/mlmc65154.2025.11189313 · Cited by 2 · see citing works
Harnessing Representative Spatial-Temporal Information for Video Question Answering
Wang, Y., Liu, M., Song, X., & Nie, L. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3675399 · Cited by 2 · see citing works
HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue
Yoon, S., Kim, D., Yoon, E., Yoon, H. S., Kim, J.-Y., & Yoo, C. D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.797 · Cited by 2 · see citing works
UWAV: Uncertainty-Weighted Weakly-Supervised Audio-Visual Video Parsing
Lai, Y.-H., Ebbers, J., Wang, Y.-C. F., Germain, F. G., Jones, M., & Chatterjee, M. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01266 · Cited by 1 · see citing works
Self-citationVisual Scene-Aware Dialogue System for Cross-Modal Intelligent Human-Machine Interaction
Liu, F., Guo, B., Wang, H., & Liu, Y. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-2385-4_25 · Cited by 1 · see citing works
Dense Contrastive Visual-Linguistic PretrainingPreprint
Shi, L., Shuang, K., Geng, S., Gao, P., Fu, Z., de Melo, G., Chen, Y., & Su, S. (2022) · doi:10.31219/osf.io/cfj3n
Self-citationSG-Shuffle: Multi-aspect Shuffle Transformer for Scene Graph Generation
Bui, A. D., Han, S. C., & Poon, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-22695-3_7
Grounding is All You Need? Dual Temporal Grounding for Video Dialog
Qin, Y., Ji, W., Lan, X., Fei, H., Yang, X., Guo, D., Zimmermann, R., & Liao, L. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3668656
Uncovering Hidden Connections: Iterative Search and Reasoning for Video-grounded Dialog
ZHANG, H., Liu, M., Feng, Y., Wang, Y., Guan, W., & Nie, L. (2026) · ACM Transactions on Information Systems · doi:10.1145/3808220
- Authors
- Geng, S., Gao, P., Chatterjee, M., Hori, C., Le Roux, J., Zhang, Y., Li, H., & Cherian, A.
- Affiliations
- Rutgers, The State University of New Jersey (US)
- Chinese University of Hong Kong (HK)
- University of Illinois Urbana-Champaign (US)
- Mitsubishi Electric (United States) (US)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2021-05-18
- Volume
- 35
- Issue
- 2
- Pages
- 1415-1423
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W3134294468
- Cited by
- 38
- References
- 71
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 7 · 2023: 8 · 2022: 12 · 2021: 5 · 2020: 2
- FWCI
- 2.62
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Mitsubishi Electric Research Laboratories
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- multimodal representation learning, graph representation learning, temporal cues, question answering, human-machine interaction, end-to-end learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
