Cited by 401Multimodal Machine Learning ApplicationsVideo Analysis and SummarizationHuman Pose and Action Recognition

The DOI matches its Crossref record; 36 of the 38 references it cites were found in a registry or an index.

Attention-Based Multimodal Fusion for Video Description

Open access

Hori, Chiori, Hori, Takaaki, Lee, Teng-Yok, Zhang, Ziming, Harsham, Bret, Hershey, John R., Marks, Tim K., Sumi, Kazuhiko·2017·2017 IEEE International Conference on Computer Vision (ICCV)

DOI 10.1109/iccv.2017.450

Open this paper in CoChat — free account Open PDF

Abstract

Current methods for video description are based on encoder-decoder sentence generation using recurrent neural networks (RNNs). Recent work has demonstrated the advantages of integrating temporal attention mechanisms into these models, in which the decoder network predicts each word in the description by selectively giving more weight to encoded features from specific time frames. Such methods typically use two different types of features: image features (from an object classification model), and motion features (from an action recognition model), combined by naive concatenation in the model input. Because different feature modalities may carry task-relevant information at different times, fusing them by naive concatenation may limit the model's ability to dynamically determine the relevance of each type of feature to different parts of the description. In this paper, we incorporate audio features in addition to the image and motion features. To fuse these three modalities, we introduce a multimodal attention model that can selectively utilize features from different modalities for each word in the output description. Combining our new multimodal attention model with standard temporal attention outperforms state-of-the-art methods on two standard datasets: YouTube2Text and MSR-VTT.

Integrity signals

Registry

Crossref · 10.1109/iccv.2017.450

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Long Short-Term Memory
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014) · SSST@EMNLP · doi:10.3115/v1/w14-4012 · Cited by 7,251

Flagged first, then by citations

Show all 38 references

2 of the 38 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

chorowski, 2015, Advances in Neural IInformation Processing Systems, 577
tokui, 2015, Chainer: a next-generation open source framework for deep learning, Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Twenty-ninth Annual Conference on Neural Information Processing Systems (NIPS)
tieleman, 2012, Lecture 6.5-RmsProp: Divide the gradient by a running average of its recent magnitude, COURSERA Neural Networks for Machine Learning

Sorted by citations

Dynamic Neural Networks: A Survey
Han, Y., Huang, G., Song, S., Yang, L., Wang, H., & Wang, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3117837 · Cited by 848
Deep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535
ECO: Efficient Convolutional Network for Online Video Understanding
Zolfaghari, M., Singh, K., & Brox, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_43 · Cited by 541
A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456
STAT: Spatial-Temporal Attention Mechanism for Video Captioning
Yan, C. C., Tu, Y., Wang, X., Zhang, Y., Hao, X., Zhang, Y., & Dai, Q. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2924576 · Cited by 412
Attentive Moment Retrieval in Videos
Liu, M., Wang, X., Nie, L., He, X., Chen, B., & Chua, T. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3210003 · Cited by 313
Attention, please! A survey of neural attention models in deep learning
Correia, A. D. S., & Colombini, E. L. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10148-x · Cited by 365
Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
Fan, C., Zhang, X., Zhang, S., Wang, W., Zhang, C., & Huang, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00210 · Cited by 293
Deep learning based multimodal biomedical data fusion: an overview and comparative review
Duan, J., Xiong, J., Li, Y., & Ding, W. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102536 · Cited by 284
SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videos
Giancola, S., Amine, M., Dghaily, T., & Ghanem, B. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00223 · Cited by 289
Hierarchical LSTMs with Adaptive Attention for Visual Captioning
Gao, L., Li, X., Song, J., & Shen, H. T. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2894139 · Cited by 249
Less Is More: Picking Informative Frames for Video Captioning
Chen, Y., Wang, S., Zhang, W., & Huang, Q. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01261-8_22 · Cited by 225
Multi-Horizon Time Series Forecasting with Temporal Attention Learning
Fan, C., Zhang, Y., Pan, Y., Li, X., Zhang, C., Yuan, R., Wu, D., Wang, W., Pei, J., & Huang, H. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330662 · Cited by 228
Object-Aware Aggregation With Bidirectional Temporal Graph for Video Captioning
Zhang, J., & Peng, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00852 · Cited by 207
Multi-modal Dense Video Captioning
lashin, V., & Rahtu, E. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00487 · Cited by 205
Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition
Shu, X., Yang, J., Yan, R., & Song, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3142771 · Cited by 207
MFAS: Multimodal Fusion Architecture Search
Perez-Rua, J.-M., Vielzeuf, V., Pateux, S., Baccouche, M., & Jurie, F. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00713 · Cited by 193
A systematic survey on multimodal emotion recognition using learning algorithms
Ahmed, N., Aghbari, Z. A., & Girija, S. (2023) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2022.200171 · Cited by 207
Neural Multimodal Cooperative Learning Toward Micro-Video Understanding
Wei, Y., Wang, X., Guan, W., Nie, L., Lin, Z., & Chen, B. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2923608 · Cited by 177
Selective Sensor Fusion for Neural Visual-Inertial Odometry
Chen, C., Rosa, S., Miao, Y., Lu, C. X., Wu, W., Markham, A., & Trigoni, N. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01079 · Cited by 178
Multi-modal fusion network with complementarity and importance for emotion recognition
Liu, S., Gao, P., Li, Y., Fu, W., & Ding, W. (2022) · Information Sciences · doi:10.1016/j.ins.2022.11.076 · Cited by 178
PVNet
You, H., Feng, Y., Ji, R., & Gao, Y. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240702 · Cited by 161
Hybrid deep neural network with dimension attention for state-of-health estimation of Lithium-ion Batteries
Bao, X., Chen, L., Lopes, A. M., Li, X., Xie, S., Li, P., & Chen, Y. (2023) · Energy · doi:10.1016/j.energy.2023.127734 · Cited by 152
CAM-RNN: Co-Attention Model Based RNN for Video Captioning
Zhao, B., Li, X., & Lu, X. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2916757 · Cited by 147
Motion Guided Spatial Attention for Video Captioning
Chen, S., & Jiang, Y. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018191 · Cited by 134
Dynamic Multimodal Fusion
Xue, Z., & Marculescu, R. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00256 · Cited by 161
ChatGPT-like large-scale foundation models for prognostics and health management: A survey and roadmaps
Li, Y., Wang, H., & Sun, M. (2023) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2023.109850 · Cited by 149
Move Forward and Tell: A Progressive Generator of Video Descriptions
Xiong, Y., Dai, B., & Lin, D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_29 · Cited by 120
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
Mogadala, A., Kalimuthu, M., & Klakow, D. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11688 · Cited by 146
Dual-Stream Recurrent Neural Network for Video Captioning
Xu, N., Liu, A.-A., Wong, Y., Zhang, Y., Nie, W., Su, Y., & Kankanhalli, M. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2867286 · Cited by 123
Hierarchical Modular Network for Video Captioning
Ye, H., Li, G., Qi, Y., Wang, S., Huang, Q., & Yang, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01741 · Cited by 125
A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer
Iashin, V., & Rahtu, E. (2020) · British Machine Vision Conference (BMVC) · doi:10.5244/c.34.29 · Cited by 126
End-to-end Audio Visual Scene-aware Dialog Using Multimodal Attention-based Video Features
Hori, C., Alamri, H., Wang, J., Wichern, G., Hori, T., Cherian, A., Marks, T. K., Cartillier, V., Lopes, R. G., Das, A., et al. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682583 · Cited by 112
Self-citation
Deep Learning Model for House Price Prediction Using Heterogeneous Data Analysis Along With Joint Self-Attention Mechanism
Wang, P.-Y., Chen, C.-T., Su, J.-W., Wang, T.-Y., & Huang, S.-H. (2021) · IEEE Access · doi:10.1109/access.2021.3071306 · Cited by 105
Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentation
Robert, D., Vallet, B., & Landrieu, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00549 · Cited by 100
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos
Chen, S., Jiang, W., Liu, W., & Jiang, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58548-8_20 · Cited by 100
AudioCaps: Generating Captions for Audios in The Wild
Kim, C. D., Kim, B., Lee, H., & Kim, G. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1011 · Cited by 96
Watch, Listen and Tell: Multi-Modal Weakly Supervised Dense Event Captioning
Rahman, T., Xu, B., & Sigal, L. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00900 · Cited by 95
Progressive Attention Memory Network for Movie Story Question Answering
Kim, J., Ma, M., Kim, K., Kim, S.-J., & Yoo, C. D. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00853 · Cited by 91
Multimodal transformer networks for end-to-end video-grounded dialogue systems
LE, H., SAHOO, D., Chen, N. F., & Hoi, S. C. H. (2019) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 89
Sports Video Captioning via Attentive Motion Representation and Group Relationship Modeling
Qi, M., Wang, Y., Li, A., & Luo, J. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2921655 · Cited by 88
Non-Autoregressive Coarse-to-Fine Video Captioning
Yang, B., Zou, Y., Liu, F., & Zhang, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16421 · Cited by 91
Multimodal Triplet Attention Network for Brain Disease Diagnosis
Zhu, Q., Wang, H., Xu, B., Zhang, Z., Shao, W., & Zhang, D. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3199032 · Cited by 82
Self-citation
Motion Guided Region Message Passing for Video Captioning
Chen, S., & Jiang, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00157 · Cited by 80
MINI-Net: Multiple Instance Ranking Network for Video Highlight Detection
Hong, F.-T., Huang, X., Li, W., & Zheng, W. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58601-0_21 · Cited by 79
Gesture Recognition in Robotic Surgery With Multimodal Attention
van Amsterdam, B., Funke, I., Edwards, E., Speidel, S., Collins, J. W., Sridhar, A. N., Kelly, J. D., Clarkson, M. J., & Stoyanov, D. V. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3147640 · Cited by 79
Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification
Bai, X., Yang, M., Lyu, P., Xu, Y., & Luo, J. (2018) · IEEE Access · doi:10.1109/access.2018.2878899 · Cited by 88
Multimodal Local-Global Attention Network for Affective Video Content Analysis
Ou, Y., Chen, Z., & Wu, F. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3014889 · Cited by 78
A Survey on Multi-modal Summarization
Jangra, A., Mukherjee, S., Jatowt, A., Saha, S., & Hasanuzzaman, M. (2023) · ACM Computing Surveys · doi:10.1145/3584700 · Cited by 88
Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning
Wang, X. E., Wang, Y.-F., & Wang, W. Y. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2125 · Cited by 77
Towards Bridging Event Captioner and Sentence Localizer for Weakly Supervised Dense Event Captioning
Chen, S., & Jiang, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00832 · Cited by 78
Self-Attentive Feature-Level Fusion for Multimodal Emotion Detection
Hazarika, D., Gorantla, S., Poria, S., & Zimmermann, R. (2018) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr.2018.00043 · Cited by 75
SibNet: Sibling Convolutional Encoder for Video Captioning
Liu, S., Ren, Z., & Yuan, J. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1145/3240508.3240667 · Cited by 151
Video Captioning With Object-Aware Spatio-Temporal Correlation and Aggregation
Zhang, J., & Peng, Y. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2988435 · Cited by 73
Enhancing the alignment between target words and corresponding frames for video captioning
Tu, Y., Zhou, C. S., Guo, J., Gao, S., & Yu, Z. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107702 · Cited by 74
SemAttNet: Toward Attention-Based Semantic Aware Guided Depth Completion
Nazir, D., Pagani, A., Liwicki, M., Stricker, D., & Afzal, M. Z. (2022) · IEEE Access · doi:10.1109/access.2022.3214316 · Cited by 71
End-to-End Video Captioning With Multitask Reinforcement Learning
Li, L., & Gong, B. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00042 · Cited by 72
A Hierarchical Attention Network-Based Approach for Depression Detection from Transcribed Clinical Interviews
Mallol-Ragolta, A., Zhao, Z., Stappen, L., Cummins, N., & Schuller, B. W. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2036 · Cited by 71
Exploiting Multi-Modal Features from Pre-Trained Networks for Alzheimer’s Dementia Recognition
Koo, J., Lee, J. H., Pyo, J., Jo, Y., & Lee, K. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-3153 · Cited by 70
SBAT: Video Captioning with Sparse Boundary-Aware Transformer
Jin, T., Huang, S., Chen, M., Li, Y., & Zhang, Z. M. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/88 · Cited by 62
Self-citation
Deep Auto-Encoders With Sequential Learning for Multimodal Dimensional Emotion Recognition
Nguyen, D., Nguyen, T. T., Zeng, R., Nguyen, T. T., Tran, S. N., Nguyen, T., Sridharan, S., & Fookes, C. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3063612 · Cited by 67
Temporal Cue Guided Video Highlight Detection with Low-Rank Audio-Visual Fusion
Ye, Q., Shen, X., Gao, Y., Wang, Z., Bi, Q., Li, P., & Yang, G. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00785 · Cited by 60
LocalTrans: A Multiscale Local Transformer Network for Cross-Resolution Homography Estimation
Shao, R., Wu, G., Zhou, Y., Fu, Y., Fang, L., & Liu, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01462 · Cited by 59
Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning
Zhong, X., Li, Z., Chen, S., Jiang, K., Chen, C., & Ye, M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25484 · Cited by 63
Lithium-Ion Battery Calendar Health Prognostics Based on Knowledge-Data-Driven Attention
Hu, T., Ma, H., Liu, K., & Sun, H. (2022) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2022.3148743 · Cited by 58
Deep Reinforcement Polishing Network for Video Captioning
Xu, W., Yu, J. Q., Miao, Z., Wan, L., Tian, Y., & Ji, Q. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3002669 · Cited by 56
Rotation invariant point cloud analysis: Where local geometry meets global topology
Zhao, C., Yang, J., Xiong, X., Zhu, A., Cao, Z., & Li, X. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108626 · Cited by 83
Towards Audio to Scene Image Synthesis Using Generative Adversarial Network
Wan, C.-H., Chuang, S.-P., & Lee, H.-Y. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682383 · Cited by 55
Audio-Driven Talking Face Video Generation With Dynamic Convolution Kernels
Ye, Z., Xia, M., Yi, R., Zhang, J., Lai, Y., Huang, X., Zhang, G., & Liu, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3142387 · Cited by 55
A Survey on Multimodal Knowledge Graphs: Construction, Completion and Applications
Chen, Y., Ge, X., Yang, S., Hu, L., Li, J., & Zhang, J. (2023) · Mathematics · doi:10.3390/math11081815 · Cited by 76
Deep Learning for Video Captioning: A Review
Chen, S., Yao, T., & Jiang, Y. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/877 · Cited by 50
Video description: A comprehensive survey of deep learning approaches
Rafiq, G., Rafiq, M. T., & Choi, G. S. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10414-6 · Cited by 55
An Explainable Multi-Modal Hierarchical Attention Model for Developing Phishing Threat Intelligence
Chai, Y., Zhou, Y., Li, W., & Jiang, Y. (2021) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2021.3119323 · Cited by 46
Generating Video Descriptions With Latent Topic Guidance
Chen, S., Jin, Q., Chen, J., & Hauptmann, A. G. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2896515 · Cited by 47
Multi-modal Attention Network for Handwritten Mathematical Expression Recognition
Wang, J., Du, J., Zhang, J., & Wang, Z.-R. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00191 · Cited by 44
Self-Attention-Based Machine Theory of Mind for Electric Vehicle Charging Demand Forecast
Hu, T., Ma, H., Liu, H., Sun, H., & Liu, K. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3180399 · Cited by 42
Unifying Relational Sentence Generation and Retrieval for Medical Image Report Composition
Wang, F., Liang, X., Xu, L. H., & Lin, L. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3026098 · Cited by 43
Learning Selective Sensor Fusion for State Estimation
Chen, C., Rosa, S., Lu, C. X., Wang, B., Trigoni, N., & Markham, A. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3176677 · Cited by 46
A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions
Hessel, J., Pang, B., Zhu, Z., & Soricut, R. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1039 · Cited by 41
Multimodal Attention Network for Continuous-Time Emotion Recognition Using Video and EEG Signals
Choi, D. Y., Kim, D.-H., & Song, B. C. (2020) · IEEE Access · doi:10.1109/access.2020.3036877 · Cited by 39
Dense Video Captioning With Early Linguistic Information Fusion
Aafaq, N., Mian, A. S., Akhtar, N., Liu, W., & Shah, M. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3146005 · Cited by 42
Concept-Aware Video Captioning: Describing Videos With Effective Prior Information
Yang, B., Cao, M., & Zou, Y. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3307969 · Cited by 39
Learning Hierarchical Modular Networks for Video Captioning
Li, G., Ye, H., Qi, Y., Wang, S., Qing, L., Huang, Q., & Yang, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3327677 · Cited by 43
Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers
Geng, S., Gao, P., Chatterjee, M., Hori, C., Le Roux, J., Zhang, Y., Li, H., & Cherian, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16231 · Cited by 38
Self-citation
Relation-aware attention for video captioning via graph learning
Tu, Y., Zhou, C. S., Guo, J., Li, H., Gao, S., & Yu, Z. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.109204 · Cited by 38
Attention-based Densely Connected LSTM for Video Captioning
Zhu, Y., & Jiang, S. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350932 · Cited by 39
How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition
Sterpu, G., Saam, C., & Harte, N. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.2980436 · Cited by 37
Perceived Mental Workload Classification Using Intermediate Fusion Multimodal Deep Learning
Dolmans, T. C., Poel, M., Klooster, J. W. J. R. V. '., & Veldkamp, B. P. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2020.609096 · Cited by 38
Exploring Video Captioning Techniques: A Comprehensive Survey on Deep Learning Methods
Islam, S., Dash, A., Seum, A., Raj, A. H., Hossain, T., & Shah, F. M. (2021) · SN Computer Science · doi:10.1007/s42979-021-00487-x · Cited by 43
End-to-End Modeling and Transfer Learning for Audiovisual Emotion Recognition in-the-Wild
Dresvyanskiy, D., Рюмина, Е. В., Kaya, H., Markitantov, M., Karpov, A. A., & Minker, W. (2022) · Multimodal Technologies and Interaction · doi:10.3390/mti6020011 · Cited by 35
Video ReCap: Recursive Captioning of Hour-Long Videos
Islam, M. M., Ho, N., Yang, X., Nagarajan, T., Torresani, L., & Bertasius, G. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01723 · Cited by 34
Video Captioning with Tube Features
Zhao, B., Li, X., & Lu, X. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/164 · Cited by 34
The Cocktail Fork Problem: Three-Stem Audio Separation for Real-World Soundtracks
Petermann, D., Wichern, G., Wang, Z.-Q., & Roux, J. L. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746005 · Cited by 34
Multimodal Conversational AI: A Survey of Datasets and Approaches
Sundar, A., & Heck, L. P. (2022) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2022.nlp4convai-1.12 · Cited by 42
Improving Multi-Modal Learning with Uni-Modal TeachersPreprint
Du, C., Li, T., Liu, Y., Wen, Z., Hua, T., Wang, Y., & Zhao, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11059 · Cited by 34
Unsupervised Scalable Multimodal Driving Anomaly Detection
Qiu, Y., Misu, T., & Busso, C. (2022) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2022.3160861 · Cited by 34
Bridging Video and Text: A Two-Step Polishing Transformer for Video Captioning
Xu, W., Miao, Z., Yu, J. Q., Tian, Y., Wan, L., & Ji, Q. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3165934 · Cited by 33
Move, Attend and Predict: An attention-based neural model for people’s movement prediction
Al-Molegi, A., Jabreel, M., & Martínez-Ballesté, A. (2018) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.05.015 · Cited by 32
Hierarchical attention-based multimodal fusion for video captioning
Wu, C., Wei, Y., Chu, X., Sun, W., Su, F., & Wang, L. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.07.029 · Cited by 32
Spatio-Temporal Ranked-Attention Networks for Video Captioning
Cherian, A., Wang, J., Hori, C., & Marks, T. K. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093291 · Cited by 32
Self-citation

Authors: Hori, C., Hori, T., Lee, T.-Y., Zhang, Z., Harsham, B., Hershey, J. R., Marks, T. K., & Sumi, K.
Affiliations: Mitsubishi Electric (United States) (US)
Aoyama Gakuin University (JP)
ORCID: Hori, Chiori
Hori, Takaaki
Lee, Teng-Yok
Zhang, Ziming
Marks, Tim K.
Sumi, Kazuhiko
Venue: 2017 IEEE International Conference on Computer Vision (ICCV)
Publisher: IEEE
Published: 2017-10
Pages: 4203-4212
DOI: 10.1109/iccv.2017.450
OpenAlex: W2584992898
Cited by: 401
References: 38
Citations by year: 2026: 11 · 2025: 24 · 2024: 41 · 2023: 47 · 2022: 54 · 2021: 55 · 2020: 63 · 2019: 71 · 2018: 30 · 2017: 4
FWCI: 13.22
Type: conference paper
Language: en
Open access: green
arxiv.org
Locations: IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Topics: Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
Keywords: audio description, multimodal fusion, multimodal attention, temporal attention, audio features, imaging features, motion features, video captioning, MSR-VTT

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Attention-Based Multimodal Fusion for Video Description

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks