Attention-Based Multimodal Fusion for Video Description
Open accessAbstract
Current methods for video description are based on encoder-decoder sentence generation using recurrent neural networks (RNNs). Recent work has demonstrated the advantages of integrating temporal attention mechanisms into these models, in which the decoder network predicts each word in the description by selectively giving more weight to encoded features from specific time frames. Such methods typically use two different types of features: image features (from an object classification model), and motion features (from an action recognition model), combined by naive concatenation in the model input. Because different feature modalities may carry task-relevant information at different times, fusing them by naive concatenation may limit the model's ability to dynamically determine the relevance of each type of feature to different parts of the description. In this paper, we incorporate audio features in addition to the image and motion features. To fuse these three modalities, we introduce a multimodal attention model that can selectively utilize features from different modalities for each word in the output description. Combining our new multimodal attention model with standard temporal attention outperforms state-of-the-art methods on two standard datasets: YouTube2Text and MSR-VTT.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The attention mechanism has gained wide adoption in computer vision due to its spectral capabilities for multimodal applications such as video description and salient object detection.
“Since its introduction, the AM has gained wide adoption in the computer vision community due to its spectral capabilities for many multimodal applications such as video description [233, 234], salient object detection [235], etc.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [233]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653 · see citing works
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014) · SSST@EMNLP · doi:10.3115/v1/w14-4012 · Cited by 7,251 · see citing works
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
ADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
Bleu: a method for automatic evaluation of machine translation
Papineni, K. (2024) · doi:10.57702/ljteqmrb · Cited by 3,682 · see citing works
Action recognition by dense trajectories
Wang, H., Klaser, A., Schmid, C., & Liu, C.-L. (2011) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2011.5995407 · Cited by 2,459 · see citing works
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
Denkowski, M., & Lavie, A. (2014) · WMT@ACL · doi:10.3115/v1/w14-3348 · Cited by 2,108 · see citing works
Attention-Based Models for Speech Recognition
Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1506.07503 · Cited by 2,021 · see citing works
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.571 · Cited by 1,810 · see citing works
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614 · see citing works
End-to-end attention-based large vocabulary speech recognition
Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., & Bengio, Y. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472618 · Cited by 1,129 · see citing works
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)Preprint
Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., & Yuille, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6632 · Cited by 1,039 · see citing works
Describing Videos by Exploiting Temporal Structure
Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., & Courville, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.512 · Cited by 1,035 · see citing works
Translating Videos to Natural Language Using Deep Recurrent Neural Networks
Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R. J., & Saenko, K. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1173 · Cited by 996 · see citing works
(2014) · arXiv (Cornell University) · Cited by 942 · see citing works
Jointly Modeling Embedding and Translation to Bridge Video and Language
Pan, Y., Mei, T., Yao, T., Li, H., & Rui, Y. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.497 · Cited by 573 · see citing works
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
Yu, H., Wang, J., Huang, Z., Yang, Y., & Xu, W. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.496 · Cited by 568 · see citing works
Guadarrama, S., Krishnamoorthy, N., Malkarnenkar, G., Venugopalan, S., Mooney, R. J., Darrell, T., & Saenko, K. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.337 · Cited by 500 · see citing works
The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems
Lowe, R., Pow, N., Serban, I. V., & Pineau, J. (2015) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w15-4640 · Cited by 282 · see citing works
Visionary Speech: Looking Ahead to Practical Speechreading Systems
Hennecke, M. E., Stork, D. G., & Prasad, K. V. (1996) · NATO ASI series. Series F : Computer and system sciences · doi:10.1007/978-3-662-13015-5_25 · Cited by 106 · see citing works
Video Paragraph Captioning Using Hierarchical Recurrent Neural NetworksPreprint
Yu, H., Wang, J., Huang, Z., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.07712 · Cited by 86 · see citing works
Maximum entropy and MCE based HMM stream weight estimation for audio-visual ASR
Gravier, G., Axelrod, S., Potamianos, G., & Neti, C. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2002.5743873 · Cited by 44 · see citing works
Robust Sensor Fusion: Analysis and Application to Audio Visual Speech Recognition
Movellan, J. R., & Mineiro, P. (1998) · Machine Learning · doi:10.1023/a:1007468413059 · Cited by 29 · see citing works
Dialog state tracking with attention-based sequence-to-sequence learning
Hori, T., Wang, H., Hori, C., Watanabe, S., Harsham, B., Roux, J. L., Hershey, J. R., Koji, Y., Yi, J., Zhu, Z., et al. (2016) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2016.7846317 · Cited by 18 · see citing works
Self-citationDriver confusion status detection using recurrent neural networks
Hori, C., Watanabe, S., Hori, T., Harsham, B., JohnR., H., Koji, Y., Fujii, Y., & Furumoto, Y. (2016) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2016.7552966 · Cited by 17 · see citing works
Self-citationGenerating Natural Video Descriptions via Multimodal Processing
Jin, Q., Liang, J., & Lin, X. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-380 · Cited by 10 · see citing works
Flagged first, then by citations
2 of the 38 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Dynamic Neural Networks: A Survey
Han, Y., Huang, G., Song, S., Yang, L., Wang, H., & Wang, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3117837 · Cited by 848 · see citing works
Deep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535 · see citing works
ECO: Efficient Convolutional Network for Online Video Understanding
Zolfaghari, M., Singh, K., & Brox, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_43 · Cited by 541 · see citing works
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
STAT: Spatial-Temporal Attention Mechanism for Video Captioning
Yan, C. C., Tu, Y., Wang, X., Zhang, Y., Hao, X., Zhang, Y., & Dai, Q. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2924576 · Cited by 402 · see citing works
Attentive Moment Retrieval in Videos
Liu, M., Wang, X., Nie, L., He, X., Chen, B., & Chua, T. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3210003 · Cited by 313 · see citing works
Attention, please! A survey of neural attention models in deep learning
Correia, A. D. S., & Colombini, E. L. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10148-x · Cited by 330 · see citing works
Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
Fan, C., Zhang, X., Zhang, S., Wang, W., Zhang, C., & Huang, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00210 · Cited by 293 · see citing works
Deep learning based multimodal biomedical data fusion: an overview and comparative review
Duan, J., Xiong, J., Li, Y., & Ding, W. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102536 · Cited by 284 · see citing works
SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videos
Giancola, S., Amine, M., Dghaily, T., & Ghanem, B. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00223 · Cited by 261 · see citing works
Hierarchical LSTMs with Adaptive Attention for Visual Captioning
Gao, L., Li, X., Song, J., & Shen, H. T. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2894139 · Cited by 249 · see citing works
Less Is More: Picking Informative Frames for Video Captioning
Chen, Y., Wang, S., Zhang, W., & Huang, Q. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01261-8_22 · Cited by 222 · see citing works
Multi-Horizon Time Series Forecasting with Temporal Attention Learning
Fan, C., Zhang, Y., Pan, Y., Li, X., Zhang, C., Yuan, R., Wu, D., Wang, W., Pei, J., & Huang, H. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330662 · Cited by 216 · see citing works
Object-Aware Aggregation With Bidirectional Temporal Graph for Video Captioning
Zhang, J., & Peng, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00852 · Cited by 207 · see citing works
Multi-modal Dense Video Captioning
lashin, V., & Rahtu, E. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00487 · Cited by 200 · see citing works
Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition
Shu, X.-B., Yang, J., Yan, R., & Song, Y. (2021) · IEEE transactions on circuits and systems for video technology (Print) · doi:10.1109/tcsvt.2022.3142771 · Cited by 191 · see citing works
MFAS: Multimodal Fusion Architecture Search
Perez-Rua, J.-M., Vielzeuf, V., Pateux, S., Baccouche, M., & Jurie, F. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00713 · Cited by 193 · see citing works
A systematic survey on multimodal emotion recognition using learning algorithms
Ahmed, N., Aghbari, Z. A., & Girija, S. (2023) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2022.200171 · Cited by 178 · see citing works
Neural Multimodal Cooperative Learning Toward Micro-Video Understanding
Wei, Y., Wang, X., Guan, W., Nie, L., Lin, Z., & Chen, B. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2923608 · Cited by 177 · see citing works
Selective Sensor Fusion for Neural Visual-Inertial Odometry
Chen, C., Rosa, S., Miao, Y., Lu, C. X., Wu, W., Markham, A., & Trigoni, N. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01079 · Cited by 178 · see citing works
Multi-modal fusion network with complementarity and importance for emotion recognition
Liu, S., Gao, P., Li, Y., Fu, W., & Ding, W. (2022) · Information Sciences · doi:10.1016/j.ins.2022.11.076 · Cited by 171 · see citing works
You, H., Feng, Y., Ji, R., & Gao, Y. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240702 · Cited by 161 · see citing works
Bao, X., Chen, L., Lopes, A. M., Li, X., Xie, S., Li, P., & Chen, Y. (2023) · Energy · doi:10.1016/j.energy.2023.127734 · Cited by 152 · see citing works
CAM-RNN: Co-Attention Model Based RNN for Video Captioning
Zhao, B., Li, X., & Lu, X. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2916757 · Cited by 147 · see citing works
Motion Guided Spatial Attention for Video Captioning
Chen, S., & Jiang, Y. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018191 · Cited by 134 · see citing works
Xue, Z., & Marculescu, R. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00256 · Cited by 134 · see citing works
Li, Y., Wang, H., & Sun, M. (2023) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2023.109850 · Cited by 137 · see citing works
Move Forward and Tell: A Progressive Generator of Video Descriptions
Xiong, Y., Dai, B., & Lin, D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_29 · Cited by 120 · see citing works
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
Mogadala, A., Kalimuthu, M., & Klakow, D. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11688 · Cited by 128 · see citing works
Dual-Stream Recurrent Neural Network for Video Captioning
Xu, N., Liu, A.-A., Wong, Y., Zhang, Y., Nie, W., Su, Y., & Kankanhalli, M. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2867286 · Cited by 120 · see citing works
Hierarchical Modular Network for Video Captioning
Ye, H., Li, G., Qi, Y., Wang, S., Huang, Q., & Yang, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01741 · Cited by 121 · see citing works
A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer
Iashin, V., & Rahtu, E. (2020) · British Machine Vision Conference (BMVC) · doi:10.5244/c.34.29 · Cited by 126 · see citing works
End-to-end Audio Visual Scene-aware Dialog Using Multimodal Attention-based Video Features
Hori, C., Alamri, H., Wang, J., Wichern, G., Hori, T., Cherian, A., Marks, T. K., Cartillier, V., Lopes, R. G., Das, A., et al. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682583 · Cited by 112 · see citing works
Self-citationWang, P.-Y., Chen, C.-T., Su, J.-W., Wang, T.-Y., & Huang, S.-H. (2021) · IEEE Access · doi:10.1109/access.2021.3071306 · Cited by 105 · see citing works
Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentation
Robert, D., Vallet, B., & Landrieu, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00549 · Cited by 100 · see citing works
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos
Chen, S., Jiang, W., Liu, W., & Jiang, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58548-8_20 · Cited by 98 · see citing works
AudioCaps: Generating Captions for Audios in The Wild
Kim, C. D., Kim, B., Lee, H., & Kim, G. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1011 · Cited by 96 · see citing works
Watch, Listen and Tell: Multi-Modal Weakly Supervised Dense Event Captioning
Rahman, T., Xu, B., & Sigal, L. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00900 · Cited by 95 · see citing works
Progressive Attention Memory Network for Movie Story Question Answering
Kim, J., Ma, M., Kim, K., Kim, S.-J., & Yoo, C. D. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00853 · Cited by 90 · see citing works
Multimodal transformer networks for end-to-end video-grounded dialogue systems
LE, H., SAHOO, D., Chen, N. F., & Hoi, S. C. H. (2019) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 89 · see citing works
Sports Video Captioning via Attentive Motion Representation and Group Relationship Modeling
Qi, M., Wang, Y., Li, A., & Luo, J. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2921655 · Cited by 86 · see citing works
Non-Autoregressive Coarse-to-Fine Video Captioning
Yang, B., Zou, Y., Liu, F., & Zhang, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16421 · Cited by 87 · see citing works
Multimodal Triplet Attention Network for Brain Disease Diagnosis
Zhu, Q., Wang, H., Xu, B., Zhang, Z., Shao, W., & Zhang, D. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3199032 · Cited by 82 · see citing works
Self-citationMotion Guided Region Message Passing for Video Captioning
Chen, S., & Jiang, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00157 · Cited by 80 · see citing works
MINI-Net: Multiple Instance Ranking Network for Video Highlight Detection
Hong, F.-T., Huang, X., Li, W., & Zheng, W. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58601-0_21 · Cited by 79 · see citing works
Gesture Recognition in Robotic Surgery With Multimodal Attention
van Amsterdam, B., Funke, I., Edwards, E., Speidel, S., Collins, J. W., Sridhar, A. N., Kelly, J. D., Clarkson, M. J., & Stoyanov, D. V. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3147640 · Cited by 79 · see citing works
Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification
Bai, X., Yang, M., Lyu, P., Xu, Y., & Luo, J. (2018) · IEEE Access · doi:10.1109/access.2018.2878899 · Cited by 88 · see citing works
Multimodal Local-Global Attention Network for Affective Video Content Analysis
Ou, Y., Chen, Z., & Wu, F. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3014889 · Cited by 78 · see citing works
A Survey on Multi-modal Summarization
Jangra, A., Mukherjee, S., Jatowt, A., Saha, S., & Hasanuzzaman, M. (2023) · ACM Computing Surveys · doi:10.1145/3584700 · Cited by 80 · see citing works
Wang, X. E., Wang, Y.-F., & Wang, W. Y. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2125 · Cited by 77 · see citing works
Towards Bridging Event Captioner and Sentence Localizer for Weakly Supervised Dense Event Captioning
Chen, S., & Jiang, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00832 · Cited by 78 · see citing works
Self-Attentive Feature-Level Fusion for Multimodal Emotion Detection
Hazarika, D., Gorantla, S., Poria, S., & Zimmermann, R. (2018) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr.2018.00043 · Cited by 75 · see citing works
SibNet: Sibling Convolutional Encoder for Video Captioning
Liu, S., Ren, Z., & Yuan, J. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1145/3240508.3240667 · Cited by 151 · see citing works
Video Captioning With Object-Aware Spatio-Temporal Correlation and Aggregation
Zhang, J., & Peng, Y. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2988435 · Cited by 73 · see citing works
Enhancing the alignment between target words and corresponding frames for video captioning
Tu, Y., Zhou, C. S., Guo, J., Gao, S., & Yu, Z. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107702 · Cited by 73 · see citing works
SemAttNet: Toward Attention-Based Semantic Aware Guided Depth Completion
Nazir, D., Pagani, A., Liwicki, M., Stricker, D., & Afzal, M. Z. (2022) · IEEE Access · doi:10.1109/access.2022.3214316 · Cited by 71 · see citing works
End-to-End Video Captioning With Multitask Reinforcement Learning
Li, L., & Gong, B. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00042 · Cited by 70 · see citing works
Mallol-Ragolta, A., Zhao, Z., Stappen, L., Cummins, N., & Schuller, B. W. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2036 · Cited by 69 · see citing works
Exploiting Multi-Modal Features from Pre-Trained Networks for Alzheimer’s Dementia Recognition
Koo, J., Lee, J. H., Pyo, J., Jo, Y., & Lee, K. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-3153 · Cited by 68 · see citing works
SBAT: Video Captioning with Sparse Boundary-Aware Transformer
Jin, T., Huang, S., Chen, M., Li, Y., & Zhang, Z. M. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/88 · Cited by 62 · see citing works
Self-citationDeep Auto-Encoders With Sequential Learning for Multimodal Dimensional Emotion Recognition
Nguyen, D., Nguyen, T. T., Zeng, R., Nguyen, T. T., Tran, S. N., Nguyen, T., Sridharan, S., & Fookes, C. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3063612 · Cited by 60 · see citing works
Temporal Cue Guided Video Highlight Detection with Low-Rank Audio-Visual Fusion
Ye, Q., Shen, X., Gao, Y., Wang, Z., Bi, Q., Li, P., & Yang, G. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00785 · Cited by 60 · see citing works
LocalTrans: A Multiscale Local Transformer Network for Cross-Resolution Homography Estimation
Shao, R., Wu, G., Zhou, Y., Fu, Y., Fang, L., & Liu, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01462 · Cited by 59 · see citing works
Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning
Zhong, X., Li, Z., Chen, S., Jiang, K., Chen, C., & Ye, M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25484 · Cited by 59 · see citing works
Lithium-Ion Battery Calendar Health Prognostics Based on Knowledge-Data-Driven Attention
Hu, T., Ma, H., Liu, K., & Sun, H. (2022) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2022.3148743 · Cited by 58 · see citing works
Deep Reinforcement Polishing Network for Video Captioning
Xu, W., Yu, J. Q., Miao, Z., Wan, L., Tian, Y., & Ji, Q. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3002669 · Cited by 56 · see citing works
Rotation invariant point cloud analysis: Where local geometry meets global topology
Zhao, C., Yang, J., Xiong, X., Zhu, A., Cao, Z., & Li, X. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108626 · Cited by 56 · see citing works
Towards Audio to Scene Image Synthesis Using Generative Adversarial Network
Wan, C.-H., Chuang, S.-P., & Lee, H.-Y. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682383 · Cited by 55 · see citing works
Audio-Driven Talking Face Video Generation With Dynamic Convolution Kernels
Ye, Z., Xia, M., Yi, R., Zhang, J., Lai, Y., Huang, X., Zhang, G., & Liu, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3142387 · Cited by 55 · see citing works
A Survey on Multimodal Knowledge Graphs: Construction, Completion and Applications
Chen, Y., Ge, X., Yang, S., Hu, L., Li, J., & Zhang, J. (2023) · Mathematics · doi:10.3390/math11081815 · Cited by 55 · see citing works
Deep Learning for Video Captioning: A Review
Chen, S., Yao, T., & Jiang, Y. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/877 · Cited by 50 · see citing works
Video description: A comprehensive survey of deep learning approaches
Rafiq, G., Rafiq, M. T., & Choi, G. S. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10414-6 · Cited by 51 · see citing works
An Explainable Multi-Modal Hierarchical Attention Model for Developing Phishing Threat Intelligence
Chai, Y., Zhou, Y., Li, W., & Jiang, Y. (2021) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2021.3119323 · Cited by 46 · see citing works
Generating Video Descriptions With Latent Topic Guidance
Chen, S., Jin, Q., Chen, J., & Hauptmann, A. G. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2896515 · Cited by 45 · see citing works
Multi-modal Attention Network for Handwritten Mathematical Expression Recognition
Wang, J., Du, J., Zhang, J., & Wang, Z.-R. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00191 · Cited by 44 · see citing works
Self-Attention-Based Machine Theory of Mind for Electric Vehicle Charging Demand Forecast
Hu, T., Ma, H., Liu, H., Sun, H., & Liu, K. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3180399 · Cited by 42 · see citing works
Unifying Relational Sentence Generation and Retrieval for Medical Image Report Composition
Wang, F., Liang, X., Xu, L. H., & Lin, L. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3026098 · Cited by 41 · see citing works
Learning Selective Sensor Fusion for State Estimation
Chen, C., Rosa, S., Lu, C. X., Wang, B., Trigoni, N., & Markham, A. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3176677 · Cited by 46 · see citing works
A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions
Hessel, J., Pang, B., Zhu, Z., & Soricut, R. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1039 · Cited by 41 · see citing works
Multimodal Attention Network for Continuous-Time Emotion Recognition Using Video and EEG Signals
Choi, D. Y., Kim, D.-H., & Song, B. C. (2020) · IEEE Access · doi:10.1109/access.2020.3036877 · Cited by 39 · see citing works
Dense Video Captioning With Early Linguistic Information Fusion
Aafaq, N., Mian, A. S., Akhtar, N., Liu, W., & Shah, M. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3146005 · Cited by 40 · see citing works
Concept-Aware Video Captioning: Describing Videos With Effective Prior Information
Yang, B., Cao, M., & Zou, Y. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3307969 · Cited by 39 · see citing works
Learning Hierarchical Modular Networks for Video Captioning
Li, G., Ye, H., Qi, Y., Wang, S., Qing, L., Huang, Q., & Yang, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3327677 · Cited by 39 · see citing works
Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers
Geng, S., Gao, P., Chatterjee, M., Hori, C., Le Roux, J., Zhang, Y., Li, H., & Cherian, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16231 · Cited by 38 · see citing works
Self-citationRelation-aware attention for video captioning via graph learning
Tu, Y., Zhou, C. S., Guo, J., Li, H., Gao, S., & Yu, Z. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.109204 · Cited by 38 · see citing works
Attention-based Densely Connected LSTM for Video Captioning
Zhu, Y., & Jiang, S.-Q. (2019) · ACM Multimedia · doi:10.1145/3343031.3350932 · Cited by 39 · see citing works
How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition
Sterpu, G., Saam, C., & Harte, N. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.2980436 · Cited by 37 · see citing works
Perceived Mental Workload Classification Using Intermediate Fusion Multimodal Deep Learning
Dolmans, T. C., Poel, M., Klooster, J. W. J. R. V. '., & Veldkamp, B. P. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2020.609096 · Cited by 36 · see citing works
Exploring Video Captioning Techniques: A Comprehensive Survey on Deep Learning Methods
Islam, S., Dash, A., Seum, A., Raj, A. H., Hossain, T., & Shah, F. M. (2021) · SN Computer Science · doi:10.1007/s42979-021-00487-x · Cited by 36 · see citing works
End-to-End Modeling and Transfer Learning for Audiovisual Emotion Recognition in-the-Wild
Dresvyanskiy, D., Рюмина, Е. В., Kaya, H., Markitantov, M., Karpov, A. A., & Minker, W. (2022) · Multimodal Technologies and Interaction · doi:10.3390/mti6020011 · Cited by 35 · see citing works
Video ReCap: Recursive Captioning of Hour-Long Videos
Islam, M. M., Ho, N., Yang, X., Nagarajan, T., Torresani, L., & Bertasius, G. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01723 · Cited by 34 · see citing works
Video Captioning with Tube Features
Zhao, B., Li, X., & Lu, X. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/164 · Cited by 34 · see citing works
The Cocktail Fork Problem: Three-Stem Audio Separation for Real-World Soundtracks
Petermann, D., Wichern, G., Wang, Z.-Q., & Roux, J. L. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746005 · Cited by 34 · see citing works
Multimodal Conversational AI: A Survey of Datasets and Approaches
Sundar, A., & Heck, L. P. (2022) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2022.nlp4convai-1.12 · Cited by 37 · see citing works
Improving Multi-Modal Learning with Uni-Modal TeachersPreprint
Du, C., Li, T., Liu, Y., Wen, Z., Hua, T., Wang, Y., & Zhao, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11059 · Cited by 34 · see citing works
Unsupervised Scalable Multimodal Driving Anomaly Detection
Qiu, Y., Misu, T., & Busso, C. (2022) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2022.3160861 · Cited by 33 · see citing works
Bridging Video and Text: A Two-Step Polishing Transformer for Video Captioning
Xu, W., Miao, Z., Yu, J. Q., Tian, Y., Wan, L., & Ji, Q. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3165934 · Cited by 33 · see citing works
Move, Attend and Predict: An attention-based neural model for people’s movement prediction
Al-Molegi, A., Jabreel, M., & Martínez-Ballesté, A. (2018) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.05.015 · Cited by 32 · see citing works
Hierarchical attention-based multimodal fusion for video captioning
Wu, C., Wei, Y., Chu, X., Sun, W., Su, F., & Wang, L. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.07.029 · Cited by 32 · see citing works
Spatio-Temporal Ranked-Attention Networks for Video Captioning
Cherian, A., Wang, J., Hori, C., & Marks, T. K. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093291 · Cited by 32 · see citing works
Self-citation
- Authors
- Hori, C., Hori, T., Lee, T.-Y., Zhang, Z., Harsham, B., Hershey, J. R., Marks, T. K., & Sumi, K.
- Affiliations
- Mitsubishi Electric (United States) (US)
- Aoyama Gakuin University (JP)
- Venue
- 2017 IEEE International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2017-10
- Pages
- 4203-4212
- OpenAlex
- W2584992898
- Cited by
- 401
- References
- 38
- Citations by year
- 2026: 11 · 2025: 24 · 2024: 41 · 2023: 47 · 2022: 54 · 2021: 55 · 2020: 63 · 2019: 71 · 2018: 30 · 2017: 4 · 2012: 1
- FWCI
- 13.22
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- audio description, multimodal fusion, multimodal attention, temporal attention, audio features, imaging features, motion features, video captioning, MSR-VTT
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
