Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages

Zadeh, Amir, Zellers, Rowan, Pincus, Eli, Morency, Louis-Philippe·2016·IEEE Intelligent Systems
DOI 10.1109/mis.2016.94

Cited by (852)

Items 201–300 of 852

  1. Multimodal Boosting: Addressing Noisy Modalities and Identifying Modality Contribution
    Sijie Mai, Sun Ya, Aolin Xiong, Ying Zeng, Haifeng Hu · 2023 · IEEE Transactions on Multimedia
  2. Strong and Simple Baselines for Multimodal Utterance Embeddings
    Paul Pu Liang, Yao Chong Lim, Yao-Hung Hubert Tsai, Ruslan Salakhutdinov, Louis‐Philippe Morency · 2019 · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)
  3. Effects of Physiological Signals in Different Types of Multimodal Sentiment Estimation
    Shun Katada, Shogo Okada, Kazunori Komatani · 2022 · IEEE Transactions on Affective Computing
  4. A Unimodal Representation Learning and Recurrent Decomposition Fusion Structure for Utterance-Level Multimodal Embedding Learning
    Mai, Sijie, Hu, Haifeng, Xing, Songlong · 2022 · IEEE Transactions on Multimedia
  5. A review and critical analysis of multimodal datasets for emotional AI
    Sadam Al-Azani, El-Sayed M. El-Alfy · 2025 · Artificial Intelligence Review
  6. EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset
    Curtis G. Northcutt, Shengxin Zha, Steven J. Lovegrove, Richard Newcombe · 2020 · IEEE Transactions on Pattern Analysis and Machine Intelligence
  7. Deep-HOSeq: Deep Higher Order Sequence Fusion for Multimodal Sentiment Analysis
    Sunny Verma, Jiwei Wang, Zhefeng Ge, Rujia Shen, Fan Jin, Yang Wang, Fang Chen, Wei Liu · 2020 · Proceedings
  8. Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis
    Ying Zeng, Sijie Mai, Haifeng Hu · 2021 · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP)
  9. Towards Multimodal Sentiment Analysis Debiasing via Bias Purification
    Dingkang Yang, Mingcheng Li, Dongling Xiao, Yang Liu, Kun Yang, Zhaoyu Chen, Yuzheng Wang, Peng Zhai, Ke Jie Li, Lihua Zhang · 2024 · Lecture notes in computer science
  10. Multi‐level feature optimization and multimodal contextual fusion for sentiment analysis and emotion classification
    Mahesh G. Huddar, Sanjeev S. Sannakki, Vijay S. Rajpurohit · 2020 · Computational Intelligence
  11. Convolutional Attention Networks for Multimodal Emotion Recognition from Speech and Text Data
    Chan Woo Lee, Kyu Ye Song, Jeong, Jihoon, Woo Yong Choi · 2018 · arXiv (Cornell University)
  12. A Multimodal Sentiment Analysis Method Based on Fuzzy Attention Fusion
    Yuxing Zhi, Junhuai Li, Huaijun Wang, Jing Chen, Wei Wei · 2024 · IEEE Transactions on Fuzzy Systems
  13. Multiscale 3D-Shift Graph Convolution Network for Emotion Recognition From Human Actions
    Henglin Shi, Wei Peng, Haoyu Chen, Xin Liu, Guoying Zhao · 2022 · IEEE Intelligent Systems
  14. Real time sentiment analysis of natural language using multimedia input
    Rishit Jain, Revant Singh, Sajal Jain, Ruchir Ahluwalia, Jyoti Gupta · 2023 · Multimedia Tools and Applications
  15. M-BERT: Injecting Multimodal Information in the BERT Structure
    Wasifur Rahman, K. M. Azharul Hasan, Amir Zadeh, Louis‐Philippe Morency, Mohammed Ehsan Hoque · 2019 · arXiv (Cornell University)
  16. Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling Fusion
    Sijie Mai, Songlong Xing, Jiaxuan He, Ying Zeng, Haifeng Hu · 2020 · arXiv (Cornell University)
  17. Multimodal Sentiment Analysis Based on Cross-Modal Attention and Gated Cyclic Hierarchical Fusion Networks
    Quan, Zhibang, Sun, Tao, Su, Mengli, Wei, Jishu · 2022 · Computational Intelligence and Neuroscience
  18. Sentiment analysis of social media comments based on multimodal attention fusion network
    Ziyu Liu, Tao Yang, Wen Chen, Jiangchuan Chen, Qinru Li, Jun Zhang · 2024 · Applied Soft Computing
  19. CMJRT: Cross-Modal Joint Representation Transformer for Multimodal Sentiment Analysis
    Meng Yuan Xu, Feifei Liang, Xiangyi Su, Cheng Fang · 2022 · IEEE Access
  20. Computational Study of Primitive Emotional Contagion in Dyadic Interactions
    Giovanna Varni, I. Hupont, C. Clavel, M. Chetouani · 2020 · IEEE Transactions on Affective Computing
  21. Gradient-Guided Modality Decoupling for Missing-Modality Robustness
    Hao Wang, Shengda Luo, Guosheng Hu, Jianguo Zhang · 2024 · Proceedings of the AAAI Conference on Artificial Intelligence
  22. Multimodal sentiment analysis using reliefF feature selection and random forest classifier
    Sujay Angadi, Venkata Siva Reddy · 2019 · International Journal of Computers and Applications
  23. Dual Low-Rank Multimodal Fusion
    Tao Jin, Siyu Huang, Yingming Li, Zhongfei Mark Zhang · 2020 · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP)
  24. Multimodal Behavior Modeling for Socially Interactive Agents
    Catherine Pélachaud, Carlos Busso, Dirk Heylen · 2021 · ACM eBooks
  25. A multimodal time-series method for gifting prediction in live streaming platforms
    Dinghao Xi, Liumin Tang, Runyu Chen, Wei Xiang Xu · 2022 · Information Processing & Management
  26. Emotion and sentiment analysis for intelligent customer service conversation using a multi-task ensemble framework
    Duan Chen, Zhengwei Huang, Tan Yiting, Min Jintao, Ribesh Khanal · 2023 · Cluster Computing
  27. Multimodal Reaction: Information Modulation for Cross-Modal Representation Learning
    Ying Zeng, Sijie Mai, Wenjun Yan, Haifeng Hu · 2023 · IEEE Transactions on Multimedia
  28. A text guided multi-task learning network for multimodal sentiment analysis
    Yuanyi Luo, Rui Wu, Jiafeng Liu, Xianglong Tang · 2023 · Neurocomputing
  29. A depression detection model based on multimodal graph neural network
    Yujing Xia, Lin Liu, Tao Dong, Juan Chen, Yu Cheng, Lin Tang · 2024 · Multimedia Tools and Applications
  30. Incorporating End-to-End Speech Recognition Models for Sentiment Analysis
    Egor Lakomkin, Mohammad Ali Zamani, Cornelius Weber, Sven Magg, Stefan Wermter · 2019 · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings
  31. X-AWARE: ConteXt-AWARE Human-Environment Attention Fusion for Driver Gaze Prediction in the Wild
    Lukas Stappen, Georgios Rizos, Björn Wolfgang Schuller · 2020 · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION
  32. Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML)
    2018
  33. Multimodal Sentiment Analysis With Mutual Information-Based Disentangled Representation Learning
    Hao Sun, Ziwei Niu, Hongyi Wang, Xin‐Yao Yu, Jiaqing Liu, Yen-Wei Chen, Lanfen Lin · 2025 · IEEE Transactions on Affective Computing
  34. $$\text {H}^2\text {CAN}$$: heterogeneous hypergraph attention network with counterfactual learning for multimodal sentiment analysis
    Changqin Huang, Zhenheng Lin, Qionghao Huang, Xiaodi Huang, Fan Jiang, Jili Chen · 2025 · Complex & Intelligent Systems
  35. Actual Cause-Guided Adaptive Gradient Scaling for Balanced Multimodal Sentiment Analysis
    Jili Chen, Qionghao Huang, Changqin Huang, Xiaodi Huang · 2025 · ACM Transactions on Multimedia Computing Communications and Applications
  36. A Personalized Affective Memory Model for Improving Emotion Recognition
    Pablo V. A. Barros, German Ignacio Parisi, Stefan Wermter · 2019 · International Conference on Machine Learning
  37. Robust-MSA: Understanding the Impact of Modality Noise on Multimodal Sentiment Analysis
    Huisheng Mao, Baozheng Zhang, Hua Xu, Ziqi Yuan, Yihe Liu · 2023 · Proceedings of the AAAI Conference on Artificial Intelligence
  38. ABAW5 Challenge: A Facial Affect Recognition Approach Utilizing Transformer Encoder and Audiovisual Fusion
    Ziyang Zhang, Liuwei An, Zishun Cui, Ao Xu, Tengteng Dong, Yueqi Jiang, Jingyi Shi, Xin Liu, Xiao Sun, Meng Wang · 2023 · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
  39. A graph convolution-based heterogeneous fusion network for multimodal sentiment analysis
    Tong Zhao, Junjie Peng, Yansong Huang, Lan Wang, Huiran Zhang, Zesu Cai · 2023 · Applied Intelligence
  40. Towards robust sentiment analysis with multimodal interaction graph and hybrid contrastive learning
    Peizhu Gong, Jin Liu, Xiliang Zhang, Xingye Li, Lai Wei, Huihua He · 2025 · Pattern Recognition
  41. M$^{3}$SA: Multimodal Sentiment Analysis Based on Multi-Scale Feature Extraction and Multi-Task Learning
    Changkai Lin, Hongju Cheng, Qiang Rao, Yang Yang · 2024 · IEEE/ACM Transactions on Audio Speech and Language Processing
  42. Multi-Grained Multimodal Interaction Network for Sentiment Analysis
    Lingyong Fang, Gongshen Liu, Ru Zhang · 2024 · IEEE International Conference on Acoustics Speech and Signal Processing
  43. Multimodal sentiment analysis using deep learning and fuzzy logic: A comprehensive survey
    Hoang Nam Do, Huyen Trang Phan, Ngoc Thanh Nguyên · 2024 · Applied Soft Computing
  44. CCIN-SA:Composite cross modal interaction network with attention enhancement for multimodal sentiment analysis
    Li Yang, Junhong Zhong, Teng Wen, Yuan Jiang Liao · 2025 · Information Fusion
  45. TCTR: Text-Guided Contrastive Learning with Token-Level Reconstruction Network for missing modalities in multimodal sentiment analysis
    Zhihao Yang, Qing He, MingHao Yu, Nisuo Du, Yijie Lu · 2025 · Information Fusion
  46. Cross-Modal Sentiment Sensing with Visual-Augmented Representation and Diverse Decision Fusion
    Sun Zhang, Bo Li, Chunyong Yin · 2021 · Sensors
  47. Elder emotion classification through multimodal fusion of intermediate layers and cross-modal transfer learning
    P. Sreevidya, S. Veni, O.V. Ramana Murthy · 2022 · Signal Image and Video Processing
  48. DJMF: A discriminative joint multi-task framework for multimodal sentiment analysis based on intra- and inter-task dynamics
    Yao Zheng, Junpeng Gong, Yujun Wen, Pengzhou Zhang · 2023 · Expert Systems with Applications
  49. Multichannel Cross-Modal Fusion Network for Multimodal Sentiment Analysis Considering Language Information Enhancement
    Ronglong Hu, Jizheng Yi, Aibin Chen, Lijiang Chen · 2024 · IEEE Transactions on Industrial Informatics
  50. Label distribution for multimodal machine learning
    Yi Cheng Ren, Ning Xu, Miaogen Ling, Xin Geng · 2021 · Frontiers of Computer Science
  51. Enhanced Experts with Uncertainty-Aware Routing for Multimodal Sentiment Analysis
    Zixian Gao, Disen Hu, Xun Jiang, Huimin Lu, Heng Tao Shen, Xing Xu · 2024 · ACM International Conference on Multimedia (ACM MM)
  52. PAMoE-MSA: polarity-aware mixture of experts network for multimodal sentiment analysis
    Changqin Huang, Zhenheng Lin, Zhongmei Han, Qionghao Huang, Fan Jiang, Xiaodi Huang · 2025 · International Journal of Multimedia Information Retrieval
  53. Sentiment Analysis on User-generated Video, Audio and Text
    Ashwini Rao, Akriti Ahuja, Shyam Kansara, Vrunda Patel · 2021 · International Conference on Computing, Communication, and Intelligent Systems (ICCCIS)
  54. StyleBERT: Text-audio sentiment analysis with Bi-directional Style Enhancement
    Fei Lin, Shengqiang Liu, Cong Zhang, Fan Jin, Zizhao Wu · 2022 · Information Systems
  55. Speech Emotion Recognition With ASR Transcripts: a Comprehensive Study on Word Error Rate and Fusion Techniques
    Yuanchao Li, Peter Bell, Catherine Lai · 2024 · IEEE Spoken Language Technology Workshop (SLT)
  56. Unimodal Training-Multimodal Prediction: Cross-Modal Federated Learning With Hierarchical Aggregation
    Rongyu Zhang, Xiao-Wei Chi, Guiliang Liu, W. Zhang, Yuan Du, Fang Wang · 2023 · IEEE Transactions on Mobile Computing
  57. Combating Human Trafficking with Deep Multimodal Models
    Edmund Tong, Amir Zadeh, Cara Jones, Louis‐Philippe Morency · 2017 · arXiv (Cornell University)
  58. Dominant SIngle-Modal SUpplementary Fusion (SIMSUF) for Multimodal Sentiment Analysis
    Jian Huang, Yanli Ji, Zhen Qin, Yang Yang, Heng Tao Shen · 2023 · IEEE Transactions on Multimedia
  59. A dual-channel multimodal sentiment analysis framework based on three-way decision
    Xingang Wang, Mengyi Wang, Hai Cui, Yijia Zhang · 2024 · Engineering Applications of Artificial Intelligence
  60. Injecting Multimodal Information Into Pre-Trained Language Model for Multimodal Sentiment Analysis
    Sijie Mai, Ying Zeng, Aolin Xiong, Haifeng Hu · 2025 · IEEE Transactions on Affective Computing
  61. M2H2: A Multimodal Multiparty Hindi Dataset For Humor Recognition in Conversations
    Dushyant Singh Chauhan, Gopendra Vikram Singh, Navonil Majumder, Amir Zadeh, Asif Ekbal, Pushpak Bhattacharyya, Louis‐Philippe Morency, Soujanya Poria · 2021 · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION
  62. THFN: Emotional health recognition of elderly people using a Two-Step Hybrid feature fusion network along with Monte-Carlo dropout
    S. Jothimani, K. Premalatha · 2023 · Biomedical Signal Processing and Control
  63. MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences
    Wei Han, Hui Chen, Min‐Yen Kan, Soujanya Poria · 2022 · Conference on Empirical Methods in Natural Language Processing (EMNLP)
  64. Multi-View Interactive Representations for Multimodal Sentiment Analysis
    Zemin Tang, Qi Xiao, Yunchuan Qin, Xu Zhou, Joey Tianyi Zhou, Kenli Li · 2024 · IEEE Transactions on Consumer Electronics
  65. Customising General Large Language Models for Specialised Emotion Recognition Tasks
    Liyizhe Peng, Zixing Zhang, Tao Pang, Jing Han, Huan Zhao, Hao Chen, Björn Wolfgang Schuller · 2024 · IEEE International Conference on Acoustics Speech and Signal Processing
  66. Getting the subtext without the text: Scalable multimodal sentiment classification from visual and acoustic modalities
    Nathaniel Blanchard, Daniel Moreira, Aparna Bharati, Walter J. Scheirer · 2018
  67. From the Token to the Review: A Hierarchical Multimodal approach to Opinion Mining
    Alexandre Garcia, Pierre Colombo, Florence d’Alché–Buc, Slim Essid, Chloé Clavel · 2019 · Conference on Empirical Methods in Natural Language Processing (EMNLP)
  68. Simple and Effective Multimodal Learning Based on Pre-Trained Transformer Models
    Kazuki Miyazawa, Yuta Kyuragi, Takayuki Nagai · 2022 · IEEE Access
  69. SCANET: Improving multimodal representation and fusion with sparse‐ and cross‐attention for multimodal sentiment analysis
    Hao Wang, Ming-Chuan Yang, Zheng Li, Zhenhua Liu, Jie Hu, Zi-Wang Fu, Feng Liu · 2022 · Comput. Animat. Virtual Worlds
  70. Emotion Quantification Using Variational Quantum State Fidelity Estimation
    Jaiteg Singh, Farman Ali, Babar Shah, Kamalpreet Singh Bhangu, Daehan Kwak · 2022 · IEEE Access
  71. An Ensemble Approach to Utterance Level Multimodal Sentiment Analysis
    Mahesh G. Huddar, Sanjeev S. Sannakki, Vijay S. Rajpurohit · 2018 · International Conference on Computational Techniques, Electronics and Mechanical Systems (CTEMS)
  72. Multimodal Neural Network Acceleration on a Hybrid CPU-FPGA Architecture: A Case Study
    Mehdi Trabelsi Ajili, Yuko Hara–Azumi · 2022 · IEEE Access
  73. Instance-sequence reasoning for video question answering
    Rui Liu, Yahong Han · 2022 · Frontiers of Computer Science
  74. Cross-modality Representation Interactive Learning for Multimodal Sentiment Analysis
    Jian Huang, Yanli Ji, Yang Yang, Heng Tao Shen · 2023 · ACM International Conference on Multimedia (ACM MM)
  75. Modality-collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition
    Chengxin Chen, Pengyuan Zhang · 2024 · ACM Transactions on Multimedia Computing Communications and Applications
  76. Cross-Modal Dynamic Transfer Learning for Multimodal Emotion Recognition
    Soyeon Hong, Hyeoungguk Kang, Hyunsouk Cho · 2024 · IEEE Access
  77. Meta Noise Adaption Framework for Multimodal Sentiment Analysis With Feature Noise
    Ziqi Yuan, Baozheng Zhang, Hua Xu, Kai Gao · 2024 · IEEE Transactions on Multimedia
  78. Deep Sentiment Features of Context and Faces for Affective Video Analysis
    Claudio Baecchi, Tiberio Uricchio, Marco Bertini, Alberto Del Bimbo · 2017 · International Conference on Multimedia Retrieval
  79. A Review of Sentiment Semantic Analysis Technology and Progress
    Yiming Wang, Yuan Rao, Lianwei Wu · 2017 · International Conference on Computational Intelligence and Security (CIS)
  80. Deep Cross-Attention Network for Crowdfunding Success Prediction
    Zhe Tang, Yi Yang, Wen Li, Defu Lian, Lixin Duan · 2022 · IEEE Transactions on Multimedia
  81. Mixture of disentangled experts with missing modalities for robust multimodal sentiment analysis
    Xiang Li, Xiaoming Zhang, Dezhuang Miao, Xianfu Cheng, D S Li, Honggui Han, Zhoujun Li · 2026 · Information Fusion
  82. MMT: Multi-way Multi-modal Transformer for Multimodal Learning
    Jiajia Tang, Kang Li, Ming Hou, Xuanyu Jin, Wanzeng Kong, Yu Ding, Qibin Zhao · 2022 · International Joint Conference on Artificial Intelligence
  83. Sense-aware BERT and Multi-task Fine-tuning for Multimodal Sentiment Analysis
    Lingyong Fang, Gongshen Liu, Ru Zhang · 2022 · International Joint Conference on Neural Networks (IJCNN)
  84. ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognition
    Yuanchao Li, Zeyu Zhao, Ondřej Klejch, Peter Bell, Catherine Lai · 2023 · Interspeech (USB)
  85. Knowledge enhancement for speech emotion recognition via multi-level acoustic feature
    Huan Zhao, Nianxin Huang, Haijiao Chen · 2024 · Connection Science
  86. Affective Video Content Analysis: Decade Review and New Perspectives
    Junxiao Xue, Jie Wang, Xiaozhen Liu, Qian Zhang, Xuecheng Wu · 2024 · Big Data Mining and Analytics
  87. Analysis of different affective state multimodal recognition approaches with missing data-oriented to virtual learning environments
    Camilo Salazar, Edwin Montoya, José Aguilar · 2021 · Heliyon
  88. Multimodal sentiment analysis with asymmetric window multi-attentions
    Helang Lai, Xueming Yan · 2021 · Multimedia Tools and Applications
  89. Affect-LM: A Neural Language Model for Customizable Affective Text Generation
    Sayan Ghosh, Mathieu Chollet, Eugene Laksana, Louis‐Philippe Morency, Stefan J. Scherer · 2017 · arXiv (Cornell University)
  90. A Multi-Focus-Driven Multi-Branch Network for Robust Multimodal Sentiment Analysis
    Chuanqi Tao, Jiaming Li, Tianzi Zang, Peng Gao · 2025 · Proceedings of the AAAI Conference on Artificial Intelligence
  91. TPFN: Applying Outer Product Along Time to Multimodal Sentiment Analysis Fusion on Incomplete Data
    Binghua Li, Chao Li, Feng Duan, Ning Zheng, Qibin Zhao · 2020 · Lecture notes in computer science
  92. Multi-Tensor Fusion Network with Hybrid Attention for Multimodal Sentiment Analysis
    Haiwei Xue, Xueming Yan, Shengyi Jiang, Helang Lai · 2020 · International Conference on Machine Learning and Cybernetics
  93. Deep Modular Co-Attention Shifting Network for Multimodal Sentiment Analysis
    Piao Shi, Min Qiang Hu, Xuefeng Shi, Fuji Ren · 2023 · ACM Transactions on Multimedia Computing Communications and Applications
  94. MACSA: A multimodal aspect-category sentiment analysis dataset with multimodal fine-grained aligned annotations
    Hao Yang, SI Zheng-ming, Yanyan Zhao, Jianwei Liu, Yang Wu, Bing Qin · 2024 · Multimedia Tools and Applications
  95. Multimodal GRU with directed pairwise cross-modal attention for sentiment analysis
    Zhenkai Qin, Qining Luo, Zhidong Zang, Hongpeng Fu · 2025 · Scientific Reports
  96. Utilizing BERT Intermediate Layers for Multimodal Sentiment Analysis
    Wenwen Zou, Jundi Ding, Chao Wang · 2022 · IEEE International Conference on Multimedia and Expo (ICME)
  97. CLAP: Contrastive Language-Audio Pre-training Model for Multi-modal Sentiment Analysis
    Tingzhen Zhao, Ming Chai Kong, Tian Liang, Qiang Zhu, Kun Kuang, Fei Qing Wu · 2023 · International Conference on Multimedia Retrieval
  98. Multimodal consistency-specificity fusion based on information bottleneck for sentiment analysis
    Wei Liu, Shenchao Cao, Sun Zhang · 2024 · Journal of King Saud University - Computer and Information Sciences
  99. MIT-FRNet: Modality-invariant temporal representation learning-based feature reconstruction network for missing modalities
    Jiayao Li, Saihua Cai, Li Li, Ruizhi Sun, Gang Yuan, Rui Zhu · 2024 · Expert Systems with Applications
  100. SIA-Net: Sparse Interactive Attention Network for Multimodal Emotion Recognition
    Shuzhen Li, Tong Zhang, C. L. Philip Chen · 2024 · IEEE Transactions on Computational Social Systems