Attention-Based Multimodal Fusion for Video Description

Hori, Chiori, Hori, Takaaki, Lee, Teng-Yok, Zhang, Ziming, Harsham, Bret, Hershey, John R., Marks, Tim K., Sumi, Kazuhiko·2017·2017 IEEE International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv.2017.450

Cited by (397)

Items 101–200 of 397

  1. Leveraging End-User Data for Enhanced Design Concept Evaluation: A Multimodal Deep Regression Model
    Chenxi Yuan, Tucker James Marion, Mohsen Moghaddam · 2021 · Journal of Mechanical Design
  2. Multimodal Video Summarization via Time-Aware Transformers
    Xindi Shang, Zehuan Yuan, Anran Wang, Changhu Wang · 2021 · ACM International Conference on Multimedia (ACM MM)
  3. Adaptive Multimodal Fusion With Attention Guided Deep Supervision Net for Grading Hepatocellular Carcinoma
    Shangxuan Li, Yanyan Xie, Guangyi Wang, Lijuan Zhang, Wu Zhou · 2022 · IEEE Journal of Biomedical and Health Informatics
  4. Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing
    Haoyue Cheng, Zhaoyang Liu, Hang Zhou, Chen Qian, Wayne Wu, Limin Wang · 2022 · Lecture notes in computer science
  5. RETRACTED ARTICLE: Video captioning: a review of theory, techniques and practices
    Vanita Jain, Fadi M. Al-Turjman, Gopal Chaudhary, Devang Nayar, Varun Gupta, Aayush Kumar · 2022 · Multimedia Tools and Applications · Retracted
  6. Cross-modal text and visual generation: A systematic review. Part 1: Image to text
    Maciej Żelaszczyk, Jacek Mańdziuk · 2023 · Information Fusion
  7. Delving Deeper into the Decoder for Video Captioning
    Haoran Chen, Jianmin Li, Xiaolin Hu · 2020 · Frontiers in artificial intelligence and applications
  8. BAFNet: Bottleneck Attention Based Fusion Network for Sleep Apnea Detection
    Xianhui Chen, Wenjun Ma, Weidong Gao, Xiaomao Fan · 2023 · IEEE Journal of Biomedical and Health Informatics
  9. EgoVQA - An Egocentric Video Question Answering Benchmark Dataset
    Chenyou Fan · 2019 · IEEE International Conference on Computer Vision Workshops
  10. Person Re-Identification Using Spatial and Layer-Wise Attention
    Aske R. Lejbølle, Kamal Nasrollahi, Benjamin Krogh, Thomas Baltzer Moeslund · 2019 · IEEE Transactions on Information Forensics and Security
  11. Attentive Cross-Modal Fusion Network for RGB-D Saliency Detection
    Di Liu, Kao Zhang, Zhenzhong Chen · 2020 · IEEE Transactions on Multimedia
  12. Integrating artificial intelligence with smartphone-based imaging for cancer detection in vivo
    Bofan Song, Rongguang Liang · 2024 · Biosensors and Bioelectronics
  13. Event-centric multi-modal fusion method for dense video captioning
    Zhi Chang, Dexin Zhao, Huilin Chen, Jingdan Li, Pengfei Liu · 2021 · Neural Networks
  14. RPEFlow: Multimodal Fusion of RGB-PointCloud-Event for Joint Optical Flow and Scene Flow Estimation
    Zhexiong Wan, Yuxin Mao, Jing Zhang, Yuchao Dai · 2023 · IEEE/CVF International Conference on Computer Vision (ICCV)
  15. PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition
    Haoxuan You, Yifan Feng, Rongrong Ji, Yue Gao · 2018 · arXiv (Cornell University)
  16. Recurrent convolutional video captioning with global and local attention
    Tao Jin, Yingming Li, Zhongfei Mark Zhang · 2019 · Neurocomputing
  17. Multimodal Learning toward Micro-Video Understanding
    Liqiang Nie, Meng Liu, Xuemeng Song · 2019 · Synthesis lectures on image, video, and multimedia processing
  18. Cross-Modal Dual Learning for Sentence-to-Video Generation
    Yue Liu, Xin Yue Wang, Yitian Yuan, Wenwu Zhu · 2019 · ACM International Conference on Multimedia (ACM MM)
  19. Hierarchical Multimodal Attention for Deep Video Summarization
    Melissa Sanabria, F. Precioso, Thomas Menguy · 2021 · International Conference on Pattern Recognition
  20. Write Like You: Synthesizing Your Cursive Online Chinese Handwriting via Metric‐based Meta Learning
    Shusen Tang, Zhouhui Lian · 2021 · Computer Graphics Forum
  21. Dialog System Technology Challenge 7
    Koichiro Yoshino, Chiori Hori, Julien Pérez, Luis Fernando D’Haro, Lazaros Polymenakos, Chulaka Gunasekara, Walter S. Lasecki, Jonathan K. Kummerfeld, Michel Galley, Chris Brockett et al. · 2019 · arXiv (Cornell University)
  22. Dual residual attention module network for single image super resolution
    Xiumei Wang, Yanan Gu, Xinbo Gao, Hui Zheng · 2019 · Neurocomputing
  23. Multi-attention guided feature fusion network for salient object detection
    Anni Li, Jinqing Qi, Huchuan Lu · 2020 · Neurocomputing
  24. Memory based fusion for multi-modal deep learning
    Darshana Priyasad, Tharindu Fernando, Simon Denman, Sridha Sridharan, Clinton Fookes · 2020 · Information Fusion
  25. Personality Assessment Based on Multimodal Attention Network Learning With Category-Based Mean Square Error
    Xiao Sun, Jie Huang, Shixin Zheng, Xuanheng Rao, Meng Wang · 2022 · IEEE Transactions on Image Processing
  26. I2Transformer: Intra- and Inter-Relation Embedding Transformer for TV Show Captioning
    Yunbin Tu, Liang Li, Li Su, Shengxiang Gao, Chenggang Yan, Zheng-Jun Zha, Zhengtao Yu, Qingming Huang · 2022 · IEEE Transactions on Image Processing
  27. A multi-layer memory sharing network for video captioning
    Tian-Zi Niu, Shan-Shan Dong, Zhen-Duo Chen, Xin Luo, Zi Helen Huang, Shanqing Guo, Xin-Shun Xu · 2022 · Pattern Recognition
  28. Unsupervised Machine Learning Based Scalable Fusion for Active Perception
    Madhura Jayaratne, Daswin De Silva, Damminda Alahakoon · 2019 · IEEE Transactions on Automation Science and Engineering
  29. Hierarchical Vision-Language Alignment for Video Captioning
    Junchao Zhang, Yuxin Peng · 2018 · Lecture notes in computer science
  30. Egocentric video description based on temporally-linked sequences
    Marc Bolaños, Álvaro Peris, Francisco Casacuberta, Sergi Soler, Petia I. Radeva · 2018 · Dipòsit Digital de la Universitat de Barcelona (Universitat de Barcelona)
  31. AppGAN: Generative Adversarial Networks for Generating Robot Approach Behaviors into Small Groups of People
    Fangkai Yang, Christopher Peters · 2019 · IEEE RO-MAN
  32. RNNs Incrementally Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?
    Anil Kag, Ziming Zhang, Venkatesh Saligrama · 2020 · International Conference on Learning Representations
  33. Domain-Specific Semantics Guided Approach to Video Captioning
    M. Hemalatha, Chigurupalli Chandra Sekhar · 2020 · IEEE Winter Conference on Applications of Computer Vision
  34. Gated Recurrent Fusion to Learn Driving Behavior from Temporal Multimodal Data
    Athma Narayanan, Avinash Siravuru, Behzad Dariush · 2020 · IEEE Robotics and Automation Letters
  35. Sentiment Enhanced Multi-Modal Hashtag Recommendation for Micro-Videos
    Chao Yang, Xiaochan Wang, Bin Jiang · 2020 · IEEE Access
  36. Rotation Invariant Point Cloud Classification: Where Local Geometry Meets Global Topology
    Chen Zhao, Jiaqi Yang, Xiong, Xin, Angfan Zhu, Zhiguo Cao, Xin Li · 2019 · arXiv (Cornell University)
  37. A Survey of Video Action Recognition Based on Deep Learning
    Ping Gong, Xudong Luo · 2025 · Knowledge-Based Systems
  38. Stream Attention-based Multi-array End-to-end Speech Recognition
    Xiaofei Wang, Ruizhi Li, Sri Harish Mallidi, Takaaki Hori, Shinji Watanabe, Hynek Hermansky B · 2019 · IEEE International Conference on Acoustics Speech and Signal Processing
  39. Deep Learning–Based Multimedia Analytics
    Wei Zhang, Ting Yao, Shiai Zhu, Abdulmotaleb El Saddik · 2019 · ACM Transactions on Multimedia Computing Communications and Applications
  40. Multi-Stream End-to-End Speech Recognition
    Ruizhi Li, Xiaofei Wang, Sri Harish Mallidi, Shinji Watanabe, Takaaki Hori, Hynek Hermansky B · 2019 · IEEE/ACM Transactions on Audio Speech and Language Processing
  41. Joint Student-Teacher Learning for Audio-Visual Scene-Aware Dialog
    Chiori Hori, Anoop Cherian, Tim K. Marks, Takaaki Hori · 2019 · Interspeech (USB)
  42. Aligning Source Visual and Target Language Domains for Unpaired Video Captioning
    Fenglin Liu, Xian Wu, Chenyu You, Shen Ge, Yuexian Zou, Xu Sun · 2021 · IEEE Transactions on Pattern Analysis and Machine Intelligence
  43. MSB-FCN: Multi-Scale Bidirectional FCN for Object Skeleton Extraction
    Fan Yang, Xin Li, Jianbing Shen · 2020 · IEEE Transactions on Image Processing
  44. Multimodal Imbalance-Aware Gradient Modulation for Weakly-Supervised Audio-Visual Video Parsing
    Jie Fu, Junyu Gao, Bing‐Kun Bao, Changsheng Xu · 2023 · IEEE Transactions on Circuits and Systems for Video Technology
  45. Audio Visual Scene-Aware Dialog (AVSD) Challenge at DSTC7
    Huda Alamri, Vincent Cartillier, Raphael Gontijo Lopes, Abhishek Das, Jue Wang, Irfan A. Essa, Dhruv Batra, Devi Parikh, Anoop Cherian, Tim K. Marks et al. · 2018 · arXiv (Cornell University)
  46. A Case Study on Neural Headline Generation for Editing Support
    Kazuma Murao, Ken Kobayashi, Hayato Kobayashi, Taichi Yatsuka, Takeshi Masuyama, Tatsuru Higurashi, Yoshimune Tabuchi · 2019 · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)
  47. Overview of the seventh Dialog System Technology Challenge: DSTC7
    Luis Fernando D’Haro, Koichiro Yoshino, Chiori Hori, Tim K. Marks, Lazaros Polymenakos, Jonathan K. Kummerfeld, Michel Galley, Xiang Gao · 2020 · Computer Speech & Language
  48. Hierarchical ConViT with Attention-Based Relational Reasoner for Visual Analogical Reasoning
    Wentao He, Jialu Zhang, Jianfeng Ren, Ruibin Bai, Xudong Jiang · 2023 · Proceedings of the AAAI Conference on Artificial Intelligence
  49. Feature Fusion With Predictive Weighting for Spectral Image Classification and Segmentation
    Yu Zhang, Cong Phuoc Huynh, King Ngi Ngan · 2019 · IEEE Transactions on Geoscience and Remote Sensing
  50. Hierarchical LSTMs with Adaptive Attention for Visual Captioning
    Jingkuan Song, Xiangpeng Li, Lianli Gao, Heng Tao Shen · 2018 · arXiv (Cornell University)
  51. Video captioning with boundary-aware hierarchical language decoding and joint video prediction
    Xiangxi Shi, Jianfei Cai, Jiuxiang Gu, Shafiq Joty · 2020 · Neurocomputing
  52. Audio-Visual Scene-Aware Dialog and Reasoning Using Audio-Visual Transformers with Joint Student-Teacher Learning
    Ankit Shah, Shijie Geng, Peng Gao, Anoop Cherian, Takaaki Hori, Tim K. Marks, Jonathan Le Roux, Chiori Hori · 2022 · IEEE International Conference on Acoustics Speech and Signal Processing
  53. Gated Homogeneous Fusion Networks With Jointed Feature Extraction for Defect Prediction
    Dingbang Fang, Shaoying Liu, Liu Ai · 2022 · IEEE Transactions on Reliability
  54. MSMFN: An Ultrasound Based Multi-Step Modality Fusion Network for Identifying the Histologic Subtypes of Metastatic Cervical Lymphadenopathy
    Zheling Meng, Yangyang Zhu, Wenjing Pang, Jie Tian, Fang Nie, Kun Wang · 2022 · IEEE Transactions on Medical Imaging
  55. End-to-End Noisy Speech Recognition Using Fourier and Hilbert Spectrum Features
    Daria Vazhenina, Konstantin Markov · 2020 · Electronics
  56. Automatic Video summarization with Timestamps using natural language processing text fusion
    Ahmed Emad, Fady Bassel, Mark Refaat, Mohamed Abdelhamed, Nada Shorim, Ashraf AbdelRaouf · 2021 · IEEE Annual Computing and Communication Workshop and Conference (CCWC)
  57. Foundation Models for Video Understanding: A Survey
    Neelu Madan, Andreas Møgelmose, Rajat Modi, Yogesh S Rawat, Thomas Baltzer Moeslund · 2024
  58. Self-Supervised Ego-Motion Estimation Based on Multi-Layer Fusion of RGB and Inferred Depth
    Zijie Jiang, Hajime Taira, Naoyuki Miyashita, Masatoshi Okutomi · 2022 · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings
  59. A Multimodal Framework for Video Caption Generation
    Reshmi S. Bhooshan, K. Suresh · 2022 · IEEE Access
  60. Adaptive Curriculum Learning for Video Captioning
    Shanhao Li, Bang Yang, Yuexian Zou · 2022 · IEEE Access
  61. Variational Stacked Local Attention Networks for Diverse Video Captioning
    Tonmoay Deb, Akib Sadmanee, Kishor Kumar Bhaumik, Amin Ahsan Ali, M. Ashraful Amin, A K M Mahbubur Rahman · 2022 · IEEE Winter Conference on Applications of Computer Vision
  62. Multimodal graph neural network for video procedural captioning
    Lei Ji, Rong-Cheng Tu, Kevin Lin, Lijuan Wang, Nan Duan · 2022 · Neurocomputing
  63. Interaction augmented transformer with decoupled decoding for video captioning
    Tao Jin, Zhou Zhao, Peng Wang, Jun Yu, Fei Qing Wu · 2022 · Neurocomputing
  64. Low-Rank HOCA: Efficient High-Order Cross-Modal Attention for Video Captioning
    Tao Jin, Siyu Huang, Yingming Li, Zhongfei Mark Zhang · 2019 · Conference on Empirical Methods in Natural Language Processing (EMNLP)
  65. Network video summarization based on key frame extraction via superpixel segmentation
    Haiyan Jin, Yang Yu, Yumeng Li, Zhaolin Xiao · 2020 · Transactions on Emerging Telecommunications Technologies
  66. Personalized Multimodal Feedback Generation in Education
    Haochen Liu, Zitao Liu, Zhongqin Wu, Jiliang Tang · 2020 · International Conference on Computational Linguistics
  67. An Attempt towards Interpretable Audio-Visual Video Captioning
    Yapeng Tian, Chenxiao Guan, Justin Goodman, Marc Moore, Chenliang Xu · 2018 · arXiv (Cornell University)
  68. Video captioning: A comparative review of where we are and which could be the route
    Daniela Moctezuma, Tania A. Ramírez-delReal, Guillermo Ruíz, Othón González-Chávez · 2023 · Computer Vision and Image Understanding
  69. Bring Event into RGB and LiDAR: Hierarchical Visual-Motion Fusion for Scene Flow
    Hanyu Zhou, Yi Chang, Zhiwei Shi · 2024 · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings
  70. KPQA: A Metric for Generative Question Answering Using Keyphrase Weights
    Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Joongbo Shin, Kyomin Jung · 2021 · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)
  71. Human Action Recognition from Various Data Modalities: A Review
    Hossein Rahmani, Mohammed Bennamoun, Qiuhong Ke · 2021
  72. Wav2NeRF: Audio-driven realistic talking head generation via wavelet-based NeRF
    Ah-Hyung Shin, Jae Ho Lee, Jiwon Hwang, Yoonhyung Kim, Gyeong-Moon Park · 2024 · Image and Vision Computing
  73. Early and late integration of audio features for automatic video description
    Chiori Hori, Takaaki Hori, Tim K. Marks, John R. Hershey · 2017 · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)
  74. Multimodal Semantic Attention Network for Video Captioning
    Liang Sun, Bing Li, Chunfeng Yuan, Zheng-Jun Zha, Weiming Hu · 2019 · IEEE International Conference on Multimedia and Expo (ICME)
  75. Attention-Based Multimodal Fusion for Estimating Human Emotion in Real-World HRI
    Yuanchao Li, Tianyu Zhao, Xun Shen · 2020 · ACM/IEEE International Conference on Human-Robot Interaction (HRI)
  76. Understanding Objects in Video: Object-Oriented Video Captioning via Structured Trajectory and Adversarial Learning
    Fangyi Zhu, Jenq–Neng Hwang, Zhanyu Ma, Guang Chen, Jun Hai Guo · 2020 · IEEE Access
  77. Audio Caption in a Car Setting with a Sentence-Level Loss
    Xuenan Xu, Heinrich Dinkel, Mengyue Wu, Kai Yu · 2021 · International Symposium on Chinese Spoken Language Processing (ISCSLP)
  78. Attention-Based Multi-Hypothesis Fusion for Speech Summarization
    Takatomo Kano, Atsunori Ogawa, Marc Delcroix, Shinji Watanabe · 2021 · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)
  79. Cross-Lingual Cross-Modal Consolidation for Effective Multilingual Video Corpus Moment Retrieval
    Jiaheng Liu, Tan Yu, Hanyu Peng, Mingming Sun, Ping Li · 2022 · Findings of the Association for Computational Linguistics: NAACL
  80. Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition
    Yu‐Chen Hu, Ruizhe Li, Chen Chen, Heqing Zou, Qiushi Zhu, Eng Siong Chng · 2023 · International Joint Conference on Artificial Intelligence
  81. Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning
    Xin Eric Wang, Yuan-Fang Wang, William Yang Wang · 2018 · arXiv (Cornell University)
  82. Multi-encoder multi-resolution framework for end-to-end speech recognition
    Ruizhi Li, Xiaofei Wang, Sri Harish Mallidi, Takaaki Hori, Shinji Watanabe, Hynek Hermansky B · 2018 · arXiv (Cornell University)
  83. DynaFuse: Dynamic Fusion for Resource Efficient Multimodal Machine Learning Inference
    Hamidreza Alikhani, Anil Kanduri, Pasi Liljeberg, Amir M. Rahmani, Nikil D. Dutt · 2023 · IEEE Embedded Systems Letters
  84. Multi-Layer Content Interaction Through Quaternion Product for Visual Question Answering
    Lei Shi, Shijie Geng, Kai Shuang, Chiori Hori, Songxiang Liu, Peng Gao, Sen Su · 2020 · IEEE International Conference on Acoustics Speech and Signal Processing
  85. Video Caption Based Searching Using End-to-End Dense Captioning and Sentence Embeddings
    Akshay Aggarwal, Aniruddha Chauhan, Deepika Kumar, Mamta Mittal, Sudipta Singha Roy, Tai-hoon Kim · 2020 · Symmetry
  86. Learning sequential slice representation with an attention-embedding network for 3D shape recognition and retrieval in MLS point clouds
    Zhipeng Luo, Di Liu, Jonathan Li, Yiping Chen, Zhenlong Xiao, José Marcato, Wesley Nunes Gonçalves, Cheng Wang · 2020 · ISPRS Journal of Photogrammetry and Remote Sensing
  87. Fault diagnosis strategy of industrial process based on multi-source heterogeneous information and deep learning
    Ying Tian · 2023 · Process Safety and Environmental Protection
  88. Role of Statistics in Detecting Misinformation: A Review of the State of the Art, Open Issues, and Future Research Directions
    Zois Boukouvalas, Allison Shafer · 2023 · Annual Review of Statistics and Its Application
  89. Exploiting the local temporal information for video captioning
    Ran Wei, Mi Li, Yaosi Hu, Zhenzhong Chen · 2020 · Journal of Visual Communication and Image Representation
  90. ASCRB: Multi-view based attentional feature selection for CircRNA-binding site prediction
    Lei Li, Zhigang Xue, Xiuquan Du · 2023 · Computers in Biology and Medicine
  91. Mining Semantic Information in Rumor Detection via a Deep Visual Perception Based Recurrent Neural Networks
    Xing Feng, Caili Guo · 2019 · IEEE International Congress on Big Data (BigData Congress)
  92. Multi-modal Automated Speech Scoring using Attention Fusion
    Manraj Singh Grover, Yaman Kumar, Sumit Sarin, Payman Vafaee, Mika Hama, Rajiv Ratn Shah · 2020 · arXiv (Cornell University)
  93. Guiding the Flowing of Semantics: Interpretable Video Captioning via POS Tag
    Xinyu Xiao, Lingfeng Wang, Bin Fan, Shinming Xiang, Chunhong Pan · 2019 · Conference on Empirical Methods in Natural Language Processing (EMNLP)
  94. Dual-Level Decoupled Transformer for Video Captioning
    Yiqi Gao, Xinglin Hou, Wei Suo, Mengyang Sun, Tiezheng Ge, Yuning Jiang, Peng Wang · 2022 · International Conference on Multimedia Retrieval
  95. Multimodal attention-based transformer for video captioning
    M. Hemalatha, Chandra Sekhar C · 2023 · Applied Intelligence
  96. M2P2: Multimodal Persuasion Prediction Using Adaptive Fusion
    Chongyang Bai, Haipeng Chen, Srijan Kumar, Jure Leskovec, V. S. Subrahmanian · 2021 · IEEE Transactions on Multimedia
  97. Active Speaker Detection as a Multi-Objective Optimization with Uncertainty-Based Multimodal Fusion
    Baptiste Pouthier, Laurent Pilati, Leela K. Gudupudi, Charles Bouveyron, Fŕed́eric Precioso · 2021 · Interspeech (USB)
  98. Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attention
    Katsuyuki Nakamura, Hiroki Ohashi, Mitsuhiro Okada · 2021 · ACM International Conference on Multimedia (ACM MM)
  99. Multimodal Data Fusion based on the Global Workspace Theory
    Bao, Cong, Fountas, Zafeirios, Olugbade, Temitayo, Bianchi-Berthouze, Nadia · 2020 · arXiv
  100. Attention-enhanced and trusted multimodal learning for micro-video venue recognition
    Bing Wang, Xianglin Huang, Gang Cao, Lifang Yang, Xiaolong Wei, Zhulin Tao · 2022 · Computers & Electrical Engineering