Multimodal sentiment analysis with word-level fusion and reinforcement learning
Open accessAbstract
With the increasing popularity of video sharing websites such as YouTube and Facebook, multimodal sentiment analysis has received increasing attention from the scientific community. Contrary to previous works in multimodal sentiment analysis which focus on holistic information in speech segments such as bag of words representations and average facial expression intensity, we propose a novel deep architecture for multimodal sentiment analysis that is able to perform modality fusion at the word level. In this paper, we propose the Gated Multimodal Embedding LSTM with Temporal Attention (GME-LSTM(A)) model that is composed of 2 modules. The Gated Multimodal Embedding allows us to alleviate the difficulties of fusion when there are noisy modalities. The LSTM with Temporal Attention can perform word level fusion at a finer fusion resolution between the input modalities and attends to the most important time steps. As a result, the GME-LSTM(A) is able to better model the multimodal structure of speech through time and perform better sentiment comprehension. We demonstrate the effectiveness of this approach on the publicly-available Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis (CMU-MOSI) dataset by achieving state-of-the-art sentiment classification and regression results. Qualitative analysis on our model emphasizes the importance of the Temporal Attention Layer in sentiment prediction because the additional acoustic and visual modalities are noisy. We also demonstrate the effectiveness of the Gated Multimodal Embedding in selectively filtering these noisy modalities out. These results and analysis open new areas in the study of sentiment analysis in human communication and provide new models for multimodal fusion.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Performing temporal alignment has been shown to help tackle the multimodal alignment problem in the case of time-series data.
“As a preprocessing step, performing temporal alignment [26] has been shown to help tackle the multimodal alignment problem in the case of time-series data.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,989 · see citing works
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 7,936 · see citing works
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · see citing works
Opinion mining and sentiment analysis
Pang, B., & Lee, L. (2008) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000011 · Cited by 6,372 · see citing works
Learning to Forget: Continual Prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 5,869 · see citing works
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · see citing works
Neural Architecture Search with Reinforcement LearningPreprint
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 5,091 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
A Convolutional Neural Network for Modelling Sentences
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1062 · Cited by 3,601 · see citing works
Lexicon-based methods for sentiment analysis
Taboada, M., Brooke, J., Tofiloski, M., Voll, K., & Stede, M. (2011) · Computational Linguistics · doi:10.1162/coli_a_00049 · Cited by 3,301 · see citing works
Attention-based LSTM for Aspect-level Sentiment Classification
Wang, Y., Huang, M., zhu, X., & Zhao, L. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1058 · Cited by 2,450 · see citing works
Image Captioning with Semantic Attention
You, Q., Jin, H., Wang, Z., Chen, F., & Luo, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.503 · Cited by 1,859 · see citing works
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644 · see citing works
Fast Human Detection Using a Cascade of Histograms of Oriented Gradients
Zhu, Q., Yeh, M.-C., Cheng, K.-T., & Avidan, S. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.119 · Cited by 1,399 · see citing works
Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.94 · Cited by 852 · see citing works
Self-citationDeep Unordered Composition Rivals Syntactic Methods for Text Classification
Iyyer, M., Manjunatha, V., Boyd-Graber, J. L., & Daumé, H. (2015) · doi:10.3115/v1/p15-1162 · Cited by 729 · see citing works
COVAREP — A collaborative voice analysis repository for speech technologies
Degottex, G., Kane, J., Drugman, T., Raitio, T., & Scherer, S. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6853739 · Cited by 695 · see citing works
Deep Learning-Based Document Modeling for Personality Detection from Text
Majumder, N., Poria, S., Gelbukh, A. F., & Cambria, E. (2017) · IEEE Intelligent Systems · doi:10.1109/mis.2017.23 · Cited by 671 · see citing works
Poria, S., Cambria, E., & Gelbukh, A. F. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1303 · Cited by 553 · see citing works
Valstar, M., Gratch, J. S., Schuller, B. W., Ringeval, F., Lalanne, D., Torres, M. T., Scherer, S. J., Stratou, G., Cowie, R., & Pantić, M. (2016) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2988257.2988258 · Cited by 516 · see citing works
Towards multimodal sentiment analysis: harvesting opinions from the web
Morency, L.-P., Mihalcea, R., & Doshi, P. (2011) · Proceedings of the 13th international conference on multimodal interfaces · doi:10.1145/2070481.2070509 · Cited by 470 · see citing works
Self-citationYoutube movie reviews: Sentiment analysis in an audio-visual context
Wollmer, M., Weninger, F., Knaup, T., Schuller, B., Sun, C., Sagae, K., & Morency, L.-P. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.34 · Cited by 403 · see citing works
Self-citationMOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · arXiv · doi:10.48550/arxiv.1606.06259 · Cited by 350 · see citing works
Self-citationDeep Compositional Captioning: Describing Novel Object Categories without Paired Training Data
Hendricks, L. A., Venugopalan, S., Rohrbach, M., Mooney, R. J., Saenko, K., & Darrell, T. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.8 · Cited by 280 · see citing works
(2009) · Springer eBooks · doi:10.1007/978-0-387-73003-5_2171 · Cited by 269 · see citing works
Deep multimodal fusion for persuasiveness prediction
Nojavanasghari, B., Gopinath, D., Koushik, J., Baltrušaitis, T., & Morency, L. (2016) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2993148.2993176 · Cited by 247 · see citing works
Self-citation(2013) · Cited by 236 · see citing works
A Shared Task on Multimodal Machine Translation and Crosslingual Image Description
Specia, L., Frank, S., Sima’an, K., & Elliott, D. (2016) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w16-2346 · Cited by 228 · see citing works
Select-additive learning: Improving generalization in multimodal sentiment analysis
Wang, H., Meghawat, A., Morency, L., & Xing, E. P. (2017) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2017.8019301 · Cited by 224 · see citing works
Self-citationPoria, S., Peng, H., Hussain, A., Howard, N., & Cambria, E. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.09.117 · Cited by 214 · see citing works
Sentic LDA: Improving on LDA with semantic similarity for aspect-based sentiment analysis
Poria, S., Chaturvedi, I., Cambria, E., & Bisio, F. (2016) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2016.7727784 · Cited by 163 · see citing works
Extracting Opinion Expressions with semi-Markov Conditional Random Fields
Yang, B., & Cardie, C. (2012) · Cited by 139 · see citing works
Concept-Level Sentiment Analysis with Dependency-Based Semantic Parsing: A Novel Approach
Agarwal, B., Poria, S., Mittal, N., Gelbukh, A. F., & Hussain, A. (2015) · Cognitive Computation · doi:10.1007/s12559-014-9316-6 · Cited by 137 · see citing works
Miller, G. (2004) · Science · doi:10.1126/science.304.5667.34 · Cited by 130 · see citing works
Computational Analysis of Persuasiveness in Social Multimedia
Park, S., Shim, H. S., Chatterjee, M., Sagae, K., & Morency, L. (2014) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2663204.2663260 · Cited by 127 · see citing works
Self-citationScherer, S. J., Lucas, G., Gratch, J. S., Rizzo, A., & Morency, L. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2440264 · Cited by 122 · see citing works
Self-citationConvolutional Experts Constrained Local Model for Facial Landmark Detection
Zadeh, A., Baltrušaitis, T., & Morency, L. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.256 · Cited by 72 · see citing works
Self-citationAdolescent Suicidal Risk Assessment in Clinician-Patient Interaction
Venek, V., Scherer, S. J., Morency, L., Rizzo, A., & Pestian, J. P. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2518665 · Cited by 68 · see citing works
Self-citationDependency-Based Semantic Parsing for Concept-Level Text Analysis
Poria, S., Agarwal, B., Gelbukh, A. F., Hussain, A., & Howard, N. (2014) · Lecture notes in computer science · doi:10.1007/978-3-642-54906-9_10 · Cited by 63 · see citing works
Fuzzy Clustering for Semi-supervised Learning – Case Study: Construction of an Emotion Lexicon
Poria, S., Gelbukh, A. F., Das, D., & Bandyopadhyay, S. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-37807-2_7 · Cited by 35 · see citing works
Wang, H., Meghawat, A., Morency, L., & Xing, E. P. (2016) · arXiv (Cornell University) · Cited by 29 · see citing works
Self-citationYu, Z., Scherer, S., DeVault, D., Gratch, J. S., Stratou, G., Morency, L., & Cassell, J. (2013) · Cited by 19 · see citing works
Self-citationVerbal Behaviors and Persuasiveness in Online Multimedia Content
Chatterjee, M., Park, S., Shim, H. S., Sagae, K., & Morency, L. (2014) · doi:10.3115/v1/w14-5908 · Cited by 12 · see citing works
Self-citationCombining Two Perspectives on Classifying Multimodal Data for Recognizing Speaker Traits
Chatterjee, M., Park, S., Morency, L., & Scherer, S. J. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2820747 · Cited by 12 · see citing works
Self-citationThe Trio Goo Catechist : Studying Jesus, the True Catechist
Goo, M. (2024) · Jurnal Pastoral Kateketik. · doi:10.70343/d9mm9s19 · Cited by 1 · see citing works
Flagged first, then by citations
1 of the 50 references couldn’t be matched to a known paper.
Research works printed but not linked: 7
- Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2017. Multimodal Machine Learning: A Survey and Taxonomy. arXiv preprint arXiv:1705.09406 (2017). · arXiv
- Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: an open source facial behavior analysis toolkit. In Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on. IEEE, 1-10.
- Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. COVAREP-A collaborative voice analysis repository for speech technolo- gies. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 960-964.
- Verónica Pérez-Rosas, Rada Mihalcea, and Louis-Philippe Morency. 2013. Utterance-Level Multimodal Sentiment Analysis.. In ACL (1). 973-982.
- Soujanya Poria, Erik Cambria, and Alexander F. Gelbukh. 2015. Deep Convo- lutional Neural Network Textual Features and Multiple Kernel Learning for Utterance-level Multimodal Sentiment Analysis. In Proceedings of the 2015 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015. 2539-2544. http://aclweb.org/anthology/D/D15/ D15-1303.pdf
- Michel Valstar, Jonathan Gratch, Björn Schuller, Fabien Ringeval, Dennis Lalanne, Mercedes Torres Torres, Stefan Scherer, Giota Stratou, Roddy Cowie, and Maja Pantic. 2016. AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge. In Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge. ACM, 3-10.
- Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2017. Recent Trends in Deep Learning Based Natural Language Processing. arXiv preprint arXiv:1708.02709 (2017). · arXiv
Cited by
Sorted by citations
Deep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 2,043 · see citing works
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433 · see citing works
Self-citationEfficient Low-rank Multimodal Fusion With Modality-Specific Factors
Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1209 · Cited by 1,170 · see citing works
Self-citationMISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Hazarika, D., Zimmermann, R., & Poria, S. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413678 · Cited by 1,073 · see citing works
Sentiment analysis using deep learning architectures: a review
Yadav, A., & Vishwakarma, D. K. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-019-09794-5 · Cited by 841 · see citing works
Integrating Multimodal Information in Large Pretrained Transformers
Rahman, W., Hasan, M. K., Lee, S., Zadeh, A. B., Mao, C., Morency, L., & Hoque, M. E. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.214 · Cited by 743 · see citing works
Self-citationSentiment Analysis Based on Deep Learning: A Comparative Study
Dang, N. C., Moreno-García, M. N., & la Prieta, F. D. (2020) · Electronics · doi:10.3390/electronics9030483 · Cited by 713 · see citing works
Deep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535 · see citing works
Self-citationSentiment Analysis and Opinion Mining
Zhang, L., & Liu, B. (2023) · Springer eBooks · doi:10.1007/978-1-4899-7502-7_907-2 · Cited by 536 · see citing works
ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection
Hazarika, D., Poria, S., Mihalcea, R. F., Cambria, E., & Zimmermann, R. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1280 · Cited by 465 · see citing works
Sun, Z., Sarma, P., Sethares, W., & Liang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6431 · Cited by 410 · see citing works
Multimodal sentiment analysis based on fusion methods: A survey
Zhu, L., Zhu, Z., Zhang, C., Xu, Y., & Kong, X. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.02.028 · Cited by 368 · see citing works
MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation
Hu, J., Liu, Y., Zhao, J., & Jin, Q. (2021) · doi:10.18653/v1/2021.acl-long.440 · Cited by 340 · see citing works
Image–text sentiment analysis via deep multimodal attentive fusion
Huang, F., Zhang, X., Zhao, Z., Xu, J. J., & Li, Z. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.01.019 · Cited by 301 · see citing works
CTNet: Conversational Transformer Network for Emotion Recognition
Lian, Z., Liu, B., & Tao, J. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3049898 · Cited by 300 · see citing works
Disentangled Representation Learning for Multimodal Emotion Recognition
Yang, D., Huang, S., Kuang, H., Du, Y., & Zhang, L. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547754 · Cited by 273 · see citing works
Sun, L., Lian, Z., Liu, B., & Tao, J. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3274829 · Cited by 283 · see citing works
Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis
Han, W., Chen, H., Gelbukh, A. F., Zadeh, A., Morency, L., & Poria, S. (2021) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3462244.3479919 · Cited by 261 · see citing works
Self-citationEntity-Sensitive Attention and Fusion Network for Entity-Level Multimodal Sentiment Classification
Yu, J., Jiang, J., & Xia, R. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2957872 · Cited by 235 · see citing works
Multimodal Language Analysis with Recurrent Multistage Fusion
Liang, P. P., Liu, Z., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1014 · Cited by 230 · see citing works
Self-citationMachine Learning Approaches to Facial and Text Analysis: Discovering CEO Oral Communication Styles
Choudhury, P., Wang, D. J., Carlson, N. A., & Khanna, T. (2019) · Strategic Management Journal · doi:10.1002/smj.3067 · Cited by 210 · see citing works
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224 · see citing works
“To the best of our knowledge, Chen et al. [109] first based the idea of word-level modality fusion to propose a gated multimodal embedding LSTM with temporal attention (GME-LSTM(A)) model, which consists of two modules, i.e., gated multimodal embedding and LSTM with temporal attention.”
BackgroundFoundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Liang, P. P., Zadeh, A., & Morency, L. (2024) · ACM Computing Surveys · doi:10.1145/3656580 · Cited by 306 · see citing works
Self-citationContextual Inter-modal Attention for Multi-modal Sentiment Analysis
Ghosal, D., Akhtar, M. S., Chauhan, D. S., Poria, S., Ekbal, A., & Bhattacharyya, P. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1382 · Cited by 196 · see citing works
Sentiment analysis using deep learning approaches: an overview
Habimana, O., Li, Y., Li, R., Gu, X., & Yu, G. (2019) · Science China Information Sciences · doi:10.1007/s11432-018-9941-6 · Cited by 191 · see citing works
Sentiment Analysis in Social Networks
Gonçalves, J. P. M. (2016) · Elsevier eBooks · doi:10.1016/c2015-0-01864-0 · Cited by 182 · see citing works
UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
Hasan, M. K., Rahman, W., Bagher Zadeh, A., Zhong, J., Tanveer, M. I., Morency, L.-P., & Hoque, M. (. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1211 · Cited by 174 · see citing works
Self-citationSurvey of Deep Representation Learning for Speech Emotion Recognition
Latif, S., Rana, R. K., Khalifa, S., Jurdak, R., Qadir, J., & Schuller, B. W. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3114365 · Cited by 178 · see citing works
CentralNet: A Multilayer Approach for Multimodal Fusion
Vielzeuf, V., Lechervy, A., Pateux, S., & Jurie, F. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-11024-6_44 · Cited by 166 · see citing works
Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment
Gu, Y., Yang, K., Fu, S., Chen, S., Li, X., & Marsic, I. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1207 · Cited by 160 · see citing works
TMBL: Transformer-based multimodal binding learning model for multimodal sentiment analysis
Huang, J., Zhou, J. W., Tang, Z., Lin, J., & Chen, C. Y. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111346 · Cited by 162 · see citing works
Multimodal sentimental analysis for social media applications: A comprehensive review
Chandrasekaran, G., Nguyen, T. N., & Hemanth, D. J. (2021) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1415 · Cited by 168 · see citing works
Multitask-Learning-Based Deep Neural Network for Automatic Modulation Classification
Chang, S., Huang, S., Zhang, R., Feng, Z., & Liu, L. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3091523 · Cited by 159 · see citing works
TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis
Wang, Z., Wan, Z., & Wan, X. (2020) · Proceedings of The Web Conference 2020 · doi:10.1145/3366423.3380000 · Cited by 156 · see citing works
Deep Neural Network for Robust Modulation Classification Under Uncertain Noise Conditions
Hu, S., Pei, Y., Liang, P. P., & Liang, Y. (2019) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2019.2951594 · Cited by 155 · see citing works
Self-citationMISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment AnalysisPreprint
Hazarika, D., Zimmermann, R., & Poria, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.03545 · Cited by 154 · see citing works
Mai, S., Hu, H., & Xing, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1046 · Cited by 153 · see citing works
Video sentiment analysis with bimodal information-augmented multi-head attention
Wu, T., Peng, J., Zhang, W., Zhang, H., Tan, S., Yi, F., Ma, C., & Huang, Y. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107676 · Cited by 137 · see citing works
Yang, K., Xu, H., & Gao, K. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413690 · Cited by 132 · see citing works
A Review of Key Technologies for Emotion Analysis Using Multimodal Information
Zhu, X., Guo, C., Feng, H., Huang, Y., Feng, Y., Wang, X., & Wang, R. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10287-z · Cited by 137 · see citing works
Shenoy, A., & Sardana, A. (2020) · doi:10.18653/v1/2020.challengehml-1.3 · Cited by 124 · see citing works
Gated Mechanism for Attention Based Multi Modal Sentiment Analysis
Kumar, A., & Vepa, J. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053012 · Cited by 123 · see citing works
GraphMFT: A graph network based multimodal fusion technique for emotion recognition in conversation
Li, J., Wang, X., Lv, G., & Zeng, Z. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126427 · Cited by 120 · see citing works
Wu, Y., Lin, Z., Zhao, Y., Qin, B., & Zhu, L.-N. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.417 · Cited by 129 · see citing works
Zhang, Y., Song, D., Li, X., Zhang, P., Wang, P., Rong, L., Yu, G., & Wang, B. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.04.003 · Cited by 114 · see citing works
Li, J., Wang, X., Lv, G., & Zeng, Z. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3261279 · Cited by 115 · see citing works
A survey on deep reinforcement learning for audio-based applications
Latif, S., Cuayáhuitl, H., Pervez, F., Shamshad, F., Ali, H. S., & Cambria, E. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10224-2 · Cited by 116 · see citing works
Attention-Based Modality-Gated Networks for Image-Text Sentiment Analysis
Huang, F., Wei, K., Weng, J., & Li, Z. (2020) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3388861 · Cited by 98 · see citing works
MASAD: A large-scale dataset for multimodal aspect-based sentiment analysis
Zhou, J., Zhao, J., Huang, J. X., Hu, Q., & He, L. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.05.040 · Cited by 96 · see citing works
A Survey of Cutting-edge Multimodal Sentiment Analysis
Singh, U., Abhishek, K., & Azad, H. K. (2024) · ACM Computing Surveys · doi:10.1145/3652149 · Cited by 99 · see citing works
M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis
Wang, X., He, J., Jin, Z., Yang, M., Wang, Y., & Qu, H. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114794 · Cited by 6 · see citing works
Mai, S., Xing, S., & Hu, H. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2925966 · Cited by 89 · see citing works
Gkoumas, D., Li, Q., Lioma, C., Yu, Y., & Song, D. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.09.005 · Cited by 94 · see citing works
SMIN: Semi-Supervised Multi-Modal Interaction Network for Conversational Emotion Recognition
Lian, Z., Liu, B., & Tao, J. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3141237 · Cited by 89 · see citing works
Mahajan, P., & Baths, V. (2021) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2021.623607 · Cited by 89 · see citing works
Mai, S., Xing, S., & Hu, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3068598 · Cited by 89 · see citing works
DialogueRNN: An Attentive RNN for Emotion Detection in Conversations
Majumder, N., Poria, S., Hazarika, D., Mihalcea, R. F., Gelbukh, A. F., & Cambria, E. (2019) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v33i01.33016818 · Cited by 901 · see citing works
Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities
Zeng, J., Liu, T., & Zhou, J. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3532064 · Cited by 94 · see citing works
Multimodal sentiment analysis based on multi-head attention mechanism
Chen, X., Lu, G., & Yan, J. (2020) · International Conference on Machine Learning and Soft Computing (ICMLSC) · doi:10.1145/3380688.3380693 · Cited by 78 · see citing works
Seq2Seq2Sentiment: Multimodal Sequence to Sequence Models for Sentiment Analysis
Pham, H., Manzini, T., Liang, P. P., & Póczos, B. (2018) · doi:10.18653/v1/w18-3308 · Cited by 76 · see citing works
Self-citationMulti-modal emotion expression and online charity crowdfunding success
Zhao, K., Zhou, L., & Zhao, X. (2022) · Decision Support Systems · doi:10.1016/j.dss.2022.113842 · Cited by 76 · see citing works
Sun, H., Chen, Y., & Lin, L. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3233070 · Cited by 77 · see citing works
Humor Knowledge Enriched Transformer for Understanding Multimodal Humor
Hasan, M. K., Lee, S., Rahman, W., Zadeh, A., Mihalcea, R. F., Morency, L., & Hoque, M. E. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17534 · Cited by 76 · see citing works
Self-citationDeep learning for aspect-based sentiment analysis: a review
Zhu, L., Xu, M., Bao, Y., Xu, Y., & Kong, X. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1044 · Cited by 85 · see citing works
Multimodal Emotion Recognition With Temporal and Semantic Consistency
Chen, B., Cao, Q., Hou, M., Zhang, Z., Lu, G., & Zhang, D. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3129331 · Cited by 71 · see citing works
Multi-Fusion Residual Memory Network for Multimodal Human Sentiment Comprehension
Mai, S., Hu, H., Jia, X., & Xing, S. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3000510 · Cited by 66 · see citing works
Dynamic attention-based explainable recommendation with textual and visual fusion
Liu, P., Zhang, L., & Gulla, J. A. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2019.102099 · Cited by 68 · see citing works
Lu, Q., Sun, X., Gao, Z., Long, Y., Feng, J., & Zhang, H. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103538 · Cited by 68 · see citing works
Multimodal Sentiment Analysis via RNN variants
Agarwal, A., Yadav, A., & Vishwakarma, D. (2019) · International Conference on Big Data, Cloud Computing, Data Science & Engineering · doi:10.1109/bcd.2019.8885108 · Cited by 61 · see citing works
Sentiment Analysis: Comprehensive Reviews, Recent Advances, and Open Challenges
Lu, Q., Sun, X., Long, Y., Gao, Z., Feng, J. H., & Sun, T. S. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3294810 · Cited by 72 · see citing works
Dynamically Adjust Word Representations Using Unaligned Multimodal Information
Guo, J., Tang, J., Dai, W., Ding, Y., & Kong, W. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548137 · Cited by 63 · see citing works
Attention-based multimodal sentiment analysis and emotion recognition using deep neural networks
Aslam, A., Sargano, A. B., & Habib, Z. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110494 · Cited by 60 · see citing works
Masked Graph Learning With Recurrent Alignment for Multimodal Emotion Recognition in Conversation
Meng, T., Zhang, F., Shou, Y., Shao, H., Ai, W., & Li, K. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3434495 · Cited by 60 · see citing works
Detecting Depression with Word-Level Multimodal Fusion
Rohanian, M., Hough, J., & Purver, M. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2283 · Cited by 58 · see citing works
Semantic Alignment Network for Multi-Modal Emotion Recognition
Hou, M.-X., Zhang, Z., Liu, C., & Lu, G. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3247822 · Cited by 58 · see citing works
DNN Multimodal Fusion Techniques for Predicting Video Sentiment
Williams, J. A., Comanescu, R., Radu, O., & Tian, L. (2018) · doi:10.18653/v1/w18-3309 · Cited by 56 · see citing works
Large language models meet text-centric multimodal sentiment analysis: a survey
Yang, H., Zhao, Y., Wu, Y., Wang, S., Zheng, T., Zhang, H., Ma, Z., Che, W., Wang, S., Wei, S., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4593-8 · Cited by 88 · see citing works
Self-citationRobust Modulation Classification under Uncertain Noise Condition Using Recurrent Neural Network
Hu, S., Pei, Y., Liang, P. P., & Liang, Y. (2018) · Globecom · doi:10.1109/glocom.2018.8647582 · Cited by 56 · see citing works
Self-citationA deep learning architecture of RA-DLNet for visual sentiment analysis
Yadav, A., & Vishwakarma, D. K. (2020) · Multimedia Systems · doi:10.1007/s00530-020-00656-7 · Cited by 56 · see citing works
Lin, R., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3303711 · Cited by 56 · see citing works
SWAFN: Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis
Chen, M., & Li, X. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.93 · Cited by 54 · see citing works
Self-citationModality-invariant temporal representation learning for multimodal sentiment classification
Sun, H., Liu, J., Chen, Y., & Lin, L. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.10.031 · Cited by 57 · see citing works
TCHFN: Multimodal sentiment analysis based on Text-Centric Hierarchical Fusion Network
Hou, J., Omar, N., Tiun, S. B., Saad, S., & He, Q. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112220 · Cited by 55 · see citing works
Modeling Incongruity between Modalities for Multimodal Sarcasm Detection
Wu, Y., Zhao, Y., Lu, X., Qin, B., Wu, Y., Sheng, J., & Li, J. (2021) · IEEE Multimedia · doi:10.1109/mmul.2021.3069097 · Cited by 55 · see citing works
Huan, R., Zhong, G., Chen, P., & Liang, R. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3338769 · Cited by 53 · see citing works
ConKI: Contrastive Knowledge Injection for Multimodal Sentiment Analysis
Yu, Y., Zhao, M. M., Qi, S.-A., Sun, F., Wang, B., Guo, W., Wang, X. L., Yang, L., & Niu, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.860 · Cited by 52 · see citing works
A comprehensive survey on deep learning-based approaches for multimodal sentiment analysis
Ghorbanali, A., & Sohrabi, M. K. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10555-8 · Cited by 51 · see citing works
Boosting crop classification by hierarchically fusing satellite, rotational, and contextual data
Barrière, V., Claverie, M., Schneider, M., Lemoine, G., & d’Andrimont, R. (2024) · Remote Sensing of Environment · doi:10.1016/j.rse.2024.114110 · Cited by 51 · see citing works
Tu, G., Liang, B., Jiang, D., & Xu, R. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3223517 · Cited by 51 · see citing works
TMMF: Temporal Multi-Modal Fusion for Single-Stage Continuous Gesture Recognition
Gammulle, H., Denman, S., Sridharan, S., & Fookes, C. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3108349 · Cited by 50 · see citing works
Sentiment-influenced trading system based on multimodal deep reinforcement learning
Chen, Y., & Huang, S.-H. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107788 · Cited by 48 · see citing works
Yang, D., Liu, Y., Huang, C., Li, M., Zhao, X. X., Wang, Y., Yang, K., Wang, Y., Zhai, P., & Zhang, L. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110370 · Cited by 48 · see citing works
Cross-modal complementary network with hierarchical fusion for multimodal sentiment classification
Peng, C., Zhang, C., Xue, X., Gao, J., Liang, H., & Niu, Z. (2021) · Tsinghua Science & Technology · doi:10.26599/tst.2021.9010055 · Cited by 48 · see citing works
Recent advances in deep learning based sentiment analysis
Yuan, J., Wu, Y., Lu, X., Zhao, Y., Qin, B., & Liu, T. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1634-3 · Cited by 48 · see citing works
Hybrid cross-modal interaction learning for multimodal sentiment analysis
Fu, Y., Zhang, Z., Yang, R., & Yao, C. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127201 · Cited by 46 · see citing works
CMU-MOSEAS: A Multimodal Language Dataset for Spanish, Portuguese, German and French
Zadeh, A. B., Cao, Y., Hessner, S., Liang, P. P., Poria, S., & Morency, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.141 · Cited by 46 · see citing works
Self-citationMultimodal Local-Global Ranking Fusion for Emotion Recognition
Liang, P. P., Zadeh, A., & Morency, L. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3243019 · Cited by 45 · see citing works
Self-citationA review on sentiment discovery and analysis of educational big‐data
Han, Z., Wu, J., Huang, C., Huang, Q., & Zhao, M. (2019) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1328 · Cited by 44 · see citing works
Multimodal Sentiment Analysis using Audio and Text for Crime Detection
Boukabous, M., & Azizi, M. (2022) · International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET) · doi:10.1109/iraset52964.2022.9738175 · Cited by 45 · see citing works
A Deep Multi-task Contextual Attention Framework for Multi-modal Affect Analysis
Akhtar, M. S., Chauhan, D. S., & Ekbal, A. (2020) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3380744 · Cited by 44 · see citing works
- Authors
- Chen, M., Wang, S., Liang, P. P., Baltrušaitis, T., Zadeh, A., & Morency, L.-P.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 19th ACM International Conference on Multimodal Interaction
- Publisher
- ACM
- Published
- 2017-11-03
- Pages
- 163-171
- OpenAlex
- W2767249564
- Cited by
- 334
- References
- 50
- Citations by year
- 2026: 15 · 2025: 32 · 2024: 59 · 2023: 51 · 2022: 34 · 2021: 44 · 2020: 33 · 2019: 41 · 2018: 23 · 2017: 1 · 2016: 1
- FWCI
- 12.22
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal sentiment analysis, temporal attention, LSTM, reinforcement learning, sentiment intensity, CMU-MOSI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
