Attention-Based Multimodal Fusion for Video Description

Hori, Chiori, Hori, Takaaki, Lee, Teng-Yok, Zhang, Ziming, Harsham, Bret, Hershey, John R., Marks, Tim K., Sumi, Kazuhiko·2017·2017 IEEE International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv.2017.450

Cited by (397)

Items 301–397 of 397

  1. Egocentric video description based on temporally-linked sequences
    Marc Bolaños, Álvaro Peris, Francisco Casacuberta, Sergi Soler, Petia I. Radeva · 2017 · Journal of Visual Communication and Image Representation
  2. Multi-modal Multi-scale Speech Expression Evaluation in Computer-Assisted Language Learning
    Jingbei Li, Zhiyong Wu, Runnan Li, Mingxing Xu, Kehua Lei, Lianhong Cai · 2018 · Lecture notes in computer science
  3. Using Multimodal Data for Automated Fidelity Evaluation in Pivotal Response Treatment Videos
    Corey D. C. Heath, Hemanth Venkateswara, Troy L. McDaniel, Sethuraman Panchanathan · 2019 · IEEE Global Conference on Signal and Information Processing
  4. Intro and Recap Detection for Movies and TV Series
    Xiang Hao, Kripa Chettiar, Ben Cheung, Vernon Germano, Raffay Hamid · 2021 · IEEE Winter Conference on Applications of Computer Vision
  5. Trident: Change Point Detection for Multivariate Time Series via Dual-Level Attention Learning
    Ziyi Duan, Haizhou Du, Yang Zheng · 2021 · Lecture notes in computer science
  6. Optimizing Latency for Online Video Captioning Using Audio-Visual Transformers
    Chiori Hori, Takaaki Hori, Jonathan Le Roux · 2021 · Interspeech (USB)
  7. Video Captioning Via Two-stage Attention Model and Generative Adversarial Network
    Wencai Yan, Lin Qi, Yun Tie, Cong Jin · 2021 · International Conference on Computational Science/Intelligence and Applied Informatics (CSII)
  8. SpecTextor: End-to-End Attention-based Mechanism for Dense Text Generation in Sports Journalism
    Indrajeet Ghosh, Matthew Ivler, Sreenivasan Ramasamy Ramamurthy, Nirmalya Roy · 2022 · IEEE International Conference on Smart Computing
  9. Joint multi-scale information and long-range dependence for video captioning
    Zhongyi Zhai, Xiaofeng Chen, Yishuang Huang, Lingzhong Zhao, Bo Hao Cheng, Qian He · 2023 · International Journal of Multimedia Information Retrieval
  10. Deep Motion-Appearance Convolutions for Robust Visual Tracking
    Haojie Li, S. M. Wu, Shuangping Huang, Kin‐Man Lam, Xiaofen Xing · 2019 · IEEE Access
  11. Multi-Head Modularization to Leverage Generalization Capability in Multi-Modal Networks
    Jun-Tae Lee, Hyunsin Park, Sungrack Yun, Simyung Chang · 2022 · Proceedings of the AAAI Conference on Artificial Intelligence
  12. Multimodal Interaction Fusion Network Based on Transformer for Video Captioning
    Hui Xu, Pengpeng Zeng, Abdullah Aman Khan · 2022 · Communications in computer and information science
  13. Multi-Layer Content Interaction Through Quaternion Product For Visual Question Answering
    Lei Shi, Shijie Geng, Kai Shuang, Chiori Hori, Songxiang Liu, Peng Gao, Sen Su · 2020 · arXiv (Cornell University)
  14. Deep Multi-Modal Sets
    Austin Reiter, Menglin Jia, Pu Yang, Ser-Nam Lim · 2020 · arXiv (Cornell University)
  15. Machine Learning for Temporal Data in Finance: Challenges and Opportunities
    Jason D. Wittenbach, d'Alessandro, Brian, C. Bayan Bruss · 2020 · RePEc: Research Papers in Economics
  16. Deep neural network architectures for user-centered design concept generation and evaluation
    Chenxi Yuan · 2022
  17. Sparse Maximum Margin Learning from Multimodal Human Behavioral Patterns
    Ervine Zheng, Qi Yu, Zhi Zheng · 2023 · Proceedings of the AAAI Conference on Artificial Intelligence
  18. ACF-net: appearance-guided content filter network for video captioning
    Min Li, Dongmei Liu, Chunsheng Liu, Faliang Chang, Wenqian Wang, Bin Wang · 2023 · Multimedia Tools and Applications
  19. TRAFMEL: Multimodal Entity Linking Based on Transformer Reranking and Multimodal Co-Attention Fusion
    Xiaoming Zhang, Meng Kaikai, Huiyong Wang · 2024 · International Journal of Software Engineering and Knowledge Engineering
  20. Structured Encoding Based on Semantic Disambiguation for Video Captioning
    Bo Sun, Jinyu Tian, Yong Wu, Lunjun Yu, Yuanyan Tang · 2024 · Cognitive Computation
  21. Method of human emotion recognition through analysis of body motor activity in video stream using neural networks
    Mikhail Uzdiaev, Dmitry Dudarenko, В.Н. Миронов · 2021 · Modelirovanie, optimizaciâ i informacionnye tehnologii.
  22. A Comparison of Late-Fusion Training Strategies for Quad-Modal Joint Embeddings
    Domenic Luca Fürer, Abraham Bernstein, Luca Rossetto · 2024 · International Workshop on Content-Based Multimedia Indexing (CBMI)
  23. Difference Bonds Consistency and Complementarity to Enhance Multimodal Representation Learning
    Chao He, Sensen Song, Zhenhong Jia, Hui Zhao · 2025 · IEEE International Conference on Acoustics Speech and Signal Processing
  24. Recognize-and-tell: Generating video captions with textual cue in scene
    Tao Jin, Lin Wang, Hao Jiang, Jian Wang, Xiao Jin, Zhimeng Zhang, Jingyuan Chen, Zhou Zhao, Zhongfei Mark Zhang · 2025 · Expert Systems with Applications
  25. FruitScope: A Non-Invasive Fruit Ripeness Sensing System via Multi-Resolution FMCW Design and Acoustic Sensing
    Shanmu Wang, Omid Abari · 2026
  26. Attention-Enhanced Sensorimotor Object Recognition
    Spyridon Thermos, Georgios Th. Papadopulos, Petros Daras, Gerasimos Potamianos · 2018 · Proceedings - International Conference on Image Processing
  27. Robust Attentional Pooling via Feature Selection
    Jian Zheng, Teng‐Yok Lee, Chen Feng, Xiaohua Lit, Ziming Zhang · 2018 · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition
  28. VMSG: a video caption network based on multimodal semantic grouping and semantic attention
    Xin Yang, Xiangchen Wang, Xiaohui Ye, Tao Li · 2023 · Multimedia Systems
  29. How will tomorrow’s algorithms fuse multimodal data? The example of the neuroprognosis in Intensive Care
    Théodore Soulier, Olivier Colliot, Nicholas Ayache, Benjamin Rohaut · 2023 · Anaesthesia Critical Care & Pain Medicine
  30. SISTEM REKOMENDASI MUSIK DENGAN MULTIMODAL FUSION ATTENTIVE NETWORKS
    Jacky Chan, Neveline Neveline, Sherina Sherina, Frans Mikael Sinaga, Arwin Halim · 2024 · Jurnal Informatika dan Teknik Elektro Terapan
  31. Cascaded transformer-based networks for wikipedia large-scale image-caption matching
    Nicola Messina, Davide Alessandro Coccomini, Andrea Esuli, Fabrizio Falchi · 2024 · Multimedia Tools and Applications
  32. Multimodality-guided Visual-Caption Semantic Enhancement
    Nan Che, Jiang Liu, Fei Yu, Lechao Cheng, Yuxuan Wang, Yuehua Li, Chenrui Liu · 2024 · Computer Vision and Image Understanding
  33. MultiMF: A Deep Multimodal Academic Resources Recommendation System
    Rafael Tejón, Juan Sanguino, Rubén Manrique · 2024 · Communications in computer and information science
  34. Video Falsity Detection Based on Multimodal Feature Fusion
    Jinlin Guo, Yuequn Gao, Li Zhang, Yancheng Zhao, Yutao Li · 2024 · International Conference on Big Data and Information Analytics (BigDIA)
  35. Presenting an Order-Aware Multimodal Fusion Framework for Financial Advisory Summarization With an Exclusive Video Dataset
    Sarmistha Das, Samrat Ghosh, Abhisek Tiwari, R E Zera Marveen Lyngkhoi, Sriparna Saha, Zak Murad, Alka Maurya · 2025 · IEEE Access
  36. Modality mixer exploiting complementary information for multi-modal action recognition
    Sumin Lee, Sangmin Woo, Muhammad Adi Nugroho, Changick Kim · 2025 · Computer Vision and Image Understanding
  37. Semantic Enhanced Encoder-Decoder Network (SEN) for Video Captioning
    Yuling Gui, Dan Guo, Ye Zhao · 2019
  38. Dynamic Fusion of Eye Movement Data and Verbal Narrations in Knowledge-rich Domains
    Ervine Zheng, Qi Yu, Rui Li, Pengcheng Shi, Anne Reeves Haake · 2020 · neural information processing systems
  39. An Approach to Spatiotemporal Trajectory Clustering Based on Community Detection
    Xin Wang, Xinzheng Niu, Jiahui Zhu, Zuoyan Liu · 2021 · Wireless Communications and Mobile Computing
  40. Sensor Transformation Attention Networks
    Stefan Braun, Daniel Neil, Enea Ceolini, Jithendar Anumula, Shih‐Chii Liu · 2017 · arXiv (Cornell University)
  41. Exploring Context, Attention and Audio Features for Audio Visual Scene-Aware Dialog
    Shachi H Kumar, Eda Okur, Saurav Sahay, Huang, Jonathan, Lama Nachman · 2019 · arXiv (Cornell University)
  42. Sensor Fusion: Gated Recurrent Fusion to Learn Driving Behavior from Temporal Multimodal Data
    Athma Narayanan, Avinash Siravuru, Behzad Dariush · 2019 · arXiv (Cornell University)
  43. Audio captioning and retrieval with improved cross-modal objectives
    Koh, Andrew Jin Jie · 2023 · DR-NTU (Nanyang Technological University)
  44. Deep and statistical learning on audio-visual data for human-machine interface on embedded systems
    Baptiste Pouthier · 2024 · theses.fr (ABES)
  45. CLOSE: Coupled content–semantic embedding
    Junhong Ren, Wensheng Zhang · 2019 · Signal Image and Video Processing
  46. What does a Car-ssette tape tell?
    Xuenan Xu, Heinrich Dinkel, Mengyue Wu, Kai Yu · 2019 · arXiv (Cornell University)
  47. Low-Rank HOCA: Efficient High-Order Cross-Modal Attention for Video Captioning
    Tao Jin, Siyu Huang, Yingming Li, Zhongfei Mark Zhang · 2019 · arXiv (Cornell University)
  48. TCT: A Cross-supervised Learning Method for Multimodal Sequence Representation
    Wubo Li, Zou, Wei, Xiangang Li · 2019 · arXiv (Cornell University)
  49. Object-Oriented Video Captioning with Temporal Graph and Prior Knowledge Building.
    Fangyi Zhu, Jenq–Neng Hwang, Zhanyu Ma, Jun Guo · 2020 · arXiv (Cornell University)
  50. A Tale of Two Modalities for Video Captioning
    Pankaj Joshi, Chitwan Saharia, Vishwajeet Singh, Digvijaysingh Gautam, Ganesh Ramakrishnan, Preethi Jyothi · 2019 · IEEE International Conference on Computer Vision Workshops
  51. Multi-modal Dense Video Captioning
    Vladimir Iashin, Esa Rahtu · 2020 · arXiv (Cornell University)
  52. FDU Participation in TRECVID 2019 VTT Task.
    Shaoxiang Chen, Yu–Gang Jiang · 2019 · TRECVID
  53. KPQA: A Metric for Generative Question Answering Using Word Weights.
    Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Joongbo Shin, Kyomin Jung · 2020 · arXiv (Cornell University)
  54. OVC-Net: Object-Oriented Video Captioning with Temporal Graph and Detail Enhancement
    Fangyi Zhu, Jenq–Neng Hwang, Zhanyu Ma, Guang Chen, Jun Hai Guo · 2020 · arXiv (Cornell University)
  55. Negative Label Guided Discriminative Canonical Correlation Analysis for Semi-Supervised and Semi-Paired Learning
    Xin Guo, Song Wang, Yun Tie, Lin Qi, Ling Guan · 2020 · IEEE International Symposium on Circuits and Systems proceedings
  56. Memory based fusion for multi-modal deep learning
    Darshana Priyasad, Tharindu Fernando, Simon Denman, Sridha Sridharan, Clinton Fookes · 2020 · arXiv (Cornell University)
  57. Caption Alignment for Low Resource Audio-Visual Data
    Vighnesh Reddy Konda, Mayur Warialani, Rakesh Prasanth Achari, Varad Bhatnagar, Jayaprakash Akula, Preethi Jyothi, Ganesh Ramakrishnan, Gholamreza Haffari, Pankaj Pratap Singh · 2020 · Interspeech (USB)
  58. A Multimodal Fusion Model Based on Hybrid Attention Mechanism for Gesture Recognition
    Yajie Li, Yiqiang Chen, Yang Gu, Jianquan Ouyang · 2021 · Lecture notes in computer science
  59. DRDF: Determining the Importance of Different Multimodal Information with Dual-Router Dynamic Framework
    Haiwen Hong, Xuan Hong Jin, Yin Zhang⋆, Yunqing Hu, Jingfeng Zhang, Yuan He, Hui Xue · 2021 · arXiv (Cornell University)
  60. Optimizing Latency for Online Video CaptioningUsing Audio-Visual Transformers
    Chiori Hori, Takaaki Hori, Jonathan Le Roux · 2021 · arXiv (Cornell University)
  61. Deep Neural Network Based Student Response Modeling With Uncertainty, Multimodality and Attention
    Xinyi Ding · 2020 · SMU Scholar (Southern Methodist University)
  62. Image and Video Captioning Using Deep Architectures
    Danny Francis, Benoît Huet · 2012 · Springer eBooks
  63. DRDF
    Haiwen Hong, Xuan Hong Jin, Yin Zhang⋆, Yunqing Hu, Jingfeng Zhang, Yuan He, Hui Xue · 2021 · ACM International Conference on Multimedia (ACM MM)
  64. References
    Hiranmay Ghosh · 2020 · Wiley eBooks
  65. Stratified Alignment Using Attention Mechanism for Video Captioning
    J Vaishnavi, V. Narmatha · 2023 · Lecture notes in electrical engineering
  66. Improving information fusion in deep learning
    Darshana Priyasad Madduma Kankanamalage Don · 2023 · Queensland University of Technology
  67. A Multimodal Spatio-temporal Model for Micro-Video Emotion Classification
    Junxiao Xue, Jie Wang, Yuanxun Zheng, Ke Wang · 2023 · IEEE International Conference on Mobile Data Management (MDM)
  68. Semantic-Guided Multi-feature Fusion for Accurate Video Captioning
    Yunjie Zhang, Tianyang Xu, Xiaoning Song, Zhenghua Feng, Xiao‐Jun Wu · 2023 · Lecture notes in computer science
  69. An Attention-Based Odometry Framework for Multisensory Unmanned Ground Vehicles (UGVs)
    Zhiyao Xiao, Guobao Zhang · 2023 · Drones
  70. WI-FI based Indoor Monitoring Enhanced by Multimodal Fusion
    Chiori Hori, Pu Perry Wang, Mahbub Rahman, Cristian J. Vaca-Rubio, Sameer Khurana, Anoop Cherian, Jonathan Le Roux · 2024 · IEEE International Conference on Acoustics Speech and Signal Processing
  71. TAPER-WE: Transformer-Based Model Attention with Relative Position Encoding and Word Embedding for Video Captioning and Summarization in Dense Environment
    Et al. Himanshu Tyagi · 2023 · International Journal on Recent and Innovation Trends in Computing and Communication
  72. A Hierarchical Graph-Based Approach for Recognition and Description Generation of Bimanual Actions in Videos
    Fatemeh Ziaeetabar, Reza Safabakhsh, Saeedeh Momtazi, Minija Tamošiūnaitė, Florentin Wörgötter · 2024 · SSRN Electronic Journal
  73. Modality Mixer Exploiting Complementary Information for Multi-Modal Action Recognition
    Sumin Lee, Sangmin Woo, Muhammad Adi Nugroho, Changick Kim · 2024 · SSRN Electronic Journal
  74. A Multimodal Fusion Plane Selection Algorithm for a Multidimensional Intelligent Interaction Pen
    Baizhou Liu, Zhiquan Feng · 2024 · Lecture notes in computer science
  75. Deep Learning for Video Captioning
    Zuxuan Wu, Yu–Gang Jiang · 2024 · Wireless networks
  76. Dual-Stream Cross-Modal Feature Fusion Based on Multi-Scale Attention for Industrial Fault Diagnosis
    Penglong Lian, Penghui Shang, Jiyang Zhang, Zhiheng Su, Jianxiao Zou, Shicai Fan · 2024 · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference
  77. Technologies for Service Capability Collaboration Interconnection
    Jianxin Liao, Bo He, Jing Wang, Jingyu Wang, Qi Qi · 2024 · Wireless networks
  78. Exploring Event Misalignment Bias and Segment Focus Bias for Weakly-Supervised Audio-Visual Video Parsing
    Mingchi Li, Songrui Han, Xiaobing Yuan · 2024 · International Conference on Big-data Service and Intelligent Computation (BDSIC)
  79. Improving Behavior-aware Driving Video Captioning through Better Use of In-vehicle Sensors and References
    Hongkuan Zhang, Koichi Takeda, Ryohei Sasano · 2025 · Journal of Information Processing
  80. Toward Sensor-to-Text Generation: Leveraging LLM-Based Video Annotations for Stroke Therapy Monitoring
    Mohammad Akidul Hoque, Shamim Ehsan, Anuradha Choudhury, Peter S. Lum, Monika Akbar, Shashwati Geed, M. Shahriar Hossain · 2025 · Bioengineering
  81. A selective attention approach for cross-modal target perception
    Guangyu Wu, Qihui Wu, Tao JIANG, Haonan Chen, Yuxin ZHAO, Yuan Qu, Jinshui HU, Yao Guo · 2025 · Scientia Sinica Informationis
  82. HAN: Korean Heritage Augmented Narrative Visual-Language Description Dataset
    SungHyun Moon, Aidyn Zhakatayev, SeungJae Lee · 2025 · ACM International Conference on Multimedia (ACM MM)
  83. A Benchmark of Microvideos for Public Opinion Analysis
    Junxiao Xue, Jie Wang, Xiaozhen Liu, Peifu Yang, Xuecheng Wu, Bin Wu · 2025 · IEEE Transactions on Computational Social Systems
  84. LLM-Based Efficient Framework for Dense Video Captioning
    Satyendra Yadav, Vidushi Sharma · 2026 · Lecture notes in networks and systems
  85. Self-Structured Data Representations for Multimodal Systems: An Approach Based on Adaptive-Competitive Dynamics
    Carlos ZARATE-TREJO, Felix Ramos · 2026 · Studies in computational intelligence
  86. MEVE-FN: An Adaptive Learning Framework for Multimodal Fake News Detection using Mixture-of-Experts
    Xiaoyu Kang, Zhixin Shi · 2025 · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics
  87. Interpretable Open-Vocabulary Referring Object Detection with Reverse Contrast Attention
    Dranreb Earl Juanico, Rowel Atienza, Jeffrey Kenneth Go · 2025 · IEEE International Conference on Computer Vision Workshops
  88. Self-Supervised Adaptive Transformer for Surgical Step Recognition in Robotic-Assisted Radical Prostatectomy
    Yiru Ye, Wenlong Wang, Yonghao Long, Chi-Fai Ng, QingYin Zhou, Dongren Yang, Qi Dou, Mengya Xu, Zhifang Pan · 2026 · IEEE Robotics and Automation Letters
  89. Multimodal Multi-Scale Temporal Enhancement Network for Audio-Visual Scene Classification
    Ming Yin, Tianwei Liang, Xianxian Zeng, Daming Shi · 2026 · IEEE Transactions on Audio Speech and Language Processing
  90. Apprentissage neuronal profond pour l'analyse de contenus multimodaux et temporels
    Valentin Vielzeuf · 2019 · theses.fr (ABES)
  91. Mise en relation de données hétérogènes pour le renforcement des systèmes de sécurité alimentaire - Cas de la production agricole en Afrique de l'Ouest
    Hugo Deléglise, Roberto Interdonato, Maître d'Hôtel, Elodie, Agnès Bégué, Maguelonne Teisseire, Mathieu Roche A · 2019 · Agritrop (Cirad)
  92. Detection and characterization of salient moments for automatic summaries
    Melissa Sanabria · 2021 · theses.fr (ABES)
  93. Leveraging multi-dimensional context & data augmentation for NER
    Tristan Luiggi · 2025 · theses.fr (ABES)
  94. Snapshot ensemble learning with attention-enhanced CNNs for ground-level land cover classification
    Simranjit Singh, Sahil Sharma, Vijay Kumar, Amita Nandal · 2026 · Complex & Intelligent Systems
  95. A Directional Semantic Enhancement Approach with Gated Fusion for Multimodal Arabic Sentiment Analysis
    Ayoub Ben Cheikhi, El Habib Nfaoui, Oumayma Elbiach · 2026 · Machine Learning and Knowledge Extraction
  96. Multimodal Abstractive Summarization of Instructional Videos with Vision-Language Models
    Maham Nazir, Muhammad Aqeel, Richong Zhang, Francesco Setti · 2026 · Lecture notes in computer science
  97. Isolated Sign Language Recognition in Low-Resolution Videos via Privileged Knowledge Distillation
    Zeynep GÖKCE AKER, Yunus Can Bilge, Nazlı İkizler-Cinbiş, Pınar Duygulu · 2026 · Research Square