The DOI matches its Crossref record; 62 of the 63 references it cites were found in a registry or an index, and 1 are not checked yet.
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Open accessAbstract
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, Louis-Philippe Morency. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This dataset encompasses 3837 videos collected from over 1000 distinctive YouTube speakers, with a reasonably balanced gender distribution of 57% male and 43% female speakers, highlighting the dataset’s focus on inclusivity.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The thematic diversity of the CMU-MOSEI dataset is notable, with a spread across 250 different themes. The three predominant themes are comments (16.2%), debates (2.9%), and consultations (1.8%), while the rest of the themes are nearly evenly distributed.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One of CMU-MOSEI’s unique characteristics is its annotations at the utterance level, with 23,259 samples annotated in this manner.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset [30] is a crucial resource in multimodal sentiment analysis and emotion recognition, distinguished by its scale, which is unmatched in the field.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [30]
The cited source says what the claim attributes to it.
In the citing paper
“We generate open-ended AQAs about speech using the four commonly used datasets IEMOCAP [23], MOSEI [24], LibriTTS [25], and Vox-Celeb2 [26,27].”
From Joint Audio and Speech Understanding (2023) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Cortes, C., & Vapnik, V. N. (1995) · Machine Learning · doi:10.1023/a:1022627411411 · Cited by 34,565
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252
Thumbs up?: sentiment classification using machine learning techniques
Pang, B., Lee, L., & Vaithyanathan, S. (2002) · Proceedings of the ACL-02 conference on Empirical methods in natural language processing - EMNLP '02 · doi:10.3115/1118693.1118704 · Cited by 7,024
Research works printed but not linked: 3
- Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarep-a col- laborative voice analysis repository for speech tech- nologies. In Acoustics, Speech and Signal Process- ing (ICASSP), 2014 IEEE International Conference on. IEEE, pages 960-964.
- Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmerman. 2018. Memn: Multimodal emotional memory network for emotion recognition in dyadic conversational videos. In NAACL.
- Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. 2016. Re- current Highway Networks. arXiv preprint arXiv:1607.03474 . · arXiv
Cited by
Sorted by citations
Items 201–300 of 1,003
Video-Based Cross-Modal Auxiliary Network for Multimodal Sentiment Analysis
Chen, R., Zhou, W., Li, Y., & Zhou, H. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3197420 · Cited by 46
“From the experimental results reported in Table XV. we evaluate the VCAN on the CMU-MOSEI dataset and obtain 69.4% (VCAN-A), 71.3% (VCAN-G),72.5% (VCAN-S), and 73.9% (VCAN-R) accuracies with the bi-modal inputs.”
MethodInfluentialRobust Multimodal Sentiment Analysis via Tag Encoding of Uncertain Missing Modalities
Zeng, J., Zhou, J., & Liu, T. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3207572 · Cited by 49
“Following [43], [44], we also adopt Open-Face2.”
CMU-MOSEAS: A Multimodal Language Dataset for Spanish, Portuguese, German and French
Zadeh, A. B., Cao, Y., Hessner, S., Liang, P. P., Poria, S., & Morency, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.141 · Cited by 46
“…for computational analysis of multimodal language (focusing on analysis of sentiment, subjectivity, or emotions (Yu et al., 2020; Poria et al., 2019; Zadeh et al., 2018b; Park et al., 2014; W¨ollmer et al., 2013; Poria et al., 2020)), other commonly spoken languages across the globe lag behind.”
BackgroundMethodInfluentialSelf-citationEvaluating Self-Supervised Speech Representations for Speech Emotion Recognition
Atmaja, B. T., & Sasou, A. (2022) · IEEE Access · doi:10.1109/access.2022.3225198 · Cited by 47
“CMU-MOSEI is both sentiment analysis and emotion recognition corpus with multimodal data (audio, video, text) [16].”
BackgroundMethodInfluentialA fine-grained modal label-based multi-stage network for multimodal sentiment analysis
Peng, J., Wu, T., Zhang, W., Cheng, F., Tan, S., Yi, F., & Huang, Y. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119721 · Cited by 47
Multimodal Mutual Attention-Based Sentiment Analysis Framework Adapted to Complicated Contexts
He, L., Wang, Z., Wang, L., & Li, F. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3276075 · Cited by 45
“Fortunately, with the development of social media and mobile terminals, a large amount of multimodal data emerged over the internet, which laid a solid data foundation for multimodal sentiment analysis [2], [3], [4], [5], [6], [7], [8], [9], [10].”
BackgroundMethodInfluentialWeakening the Dominant Role of Text: CMOSI Dataset and Multimodal Semantic Enhancement Network
Jin, C., Luo, C., Yan, M. G., Zhao, G., Zhang, G., & Zhang, S. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3282953 · Cited by 44
Graph Capsule Aggregation for Unaligned Multimodal Sequences
Wu, J., Mai, S., & Hu, H. (2021) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3462244.3479931 · Cited by 45
“In this section, we evaluate our proposed model GraphCAGE on two frequently-used datasets: CMU-MOSI[38] and CMU-MOSEI[36].”
Siddiqui, M. F. H., Dhakal, P., Yang, X., & Javaid, A. Y. (2022) · Multimodal Technologies and Interaction · doi:10.3390/mti6060047 · Cited by 46
Detecting Hate Speech in Multi-modal MemesPreprint
Das, A., Wahi, J. S., & Li, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.14891 · Cited by 88
“…to improve the prediction accuracy by developing more sophisticated attention mechanism to better capture the interaction between two modalities, while some (Zadeh et al., 2018) (Majumder et al., 2018) introduce very innovative ways of fusion, which utilizes graph or hierarchical architecture.”
BackgroundCo-space Representation Interaction Network for multimodal sentiment analysis
Shi, H., Pu, Y., Zhao, Z., Huang, J., Zhou, D., Xu, D., & Cao, J. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111149 · Cited by 46
Deep learning methods improve genomic prediction of wheat breeding
Montesinos‐López, A., Crespo‐Herrera, L., Dreisigacker, S., Gerard, G., Vitale, P., Pierre, C. S., Govindan, V., Tadesse, Z., Flores, M. C., Pérez‐Rodríguez, P., et al. (2024) · Frontiers in Plant Science · doi:10.3389/fpls.2024.1324090 · Cited by 43
A Deep Multi-task Contextual Attention Framework for Multi-modal Affect Analysis
Akhtar, M. S., Chauhan, D. S., & Ekbal, A. (2020) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3380744 · Cited by 44
“Recently, a dynamic fusion graph (DFG) for the fusion of tri-modal inputs has been proposed in [56].”
BackgroundMethodResultInfluentialZhang, Q., Shi, L., Liu, P., Zhu, Z., & Xu, L. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03343-4 · Cited by 43
“We use two public multimodal sentiment analysis datasets, CMU-MOSI [10] and CMU-MOSEI [11].”
BackgroundMethodRetractedMSAF: Multimodal Split Attention FusionPreprint
Su, L., Hu, C., Li, G., & Cao, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.07175 · Cited by 72
“ontains 22852 annotated video segments (utterances) from 1000 distinct speakers and 250 topics gathered from online video sharing websites. Each utterance is annotated with a sentiment intensity from [-3, 3]. The train, validation and test set contain 16322, 1871, and 4659 samples respectively. Following recent works,…”
MethodInfluentialA feature-based restoration dynamic interaction network for multimodal sentiment analysis
Zeng, Y., Li, Z., Chen, Z., & Ma, H. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107335 · Cited by 42
Refashioning Emotion Recognition Modeling: The Advent of Generalized Large Models
Zhang, Z., Peng, L., Pang, T., Han, J., Zhao, H., & Schuller, B. W. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3396345 · Cited by 52
“…to integrating information from different modalities [8], [9], developing large-scale emotion datasets for model training and validation [10], [11], transferring knowledge of other relevant tasks by leveraging pretrained models [12], [13], and increasing the interpretability and providing…”
EMOE: Modality-Specific Enhanced Dynamic Emotion Experts
Fang, Y., Huang, W., Wan, G., Su, K., & Ye, M. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01335 · Cited by 52
“Considering the generality of our approach for both MER and MIR tasks, we evaluate EMOE on the following three datasets: CMU-MOSI [62], CMU-MOSEI [65] and MIntRec [66]. by using the metrics: 7-class Accuracy (ACC-7), Binary Accuracy (ACC-2), F1-score (F1), and Mean-absolute Error (MAE).”
Multi-Attention Fusion Network for Video-based Emotion Recognition
Wang, Y., Wu, J., & Hoashi, K. (2019) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3340555.3355720 · Cited by 41
MEmoR: A Dataset for Multimodal Emotion Reasoning in Videos
Shen, G., Wang, X. Y., Duan, X., Li, H., & Zhu, W. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413909 · Cited by 44
“For example, OMG-Emotion dataset [2] and CMU-MOSEI dataset [51] collect user-generated monologue videos from YouTube and annotate Ekman’s basic emotions in utterance level.”
BackgroundMethodSignNet II: A Transformer-Based Two-Way Sign Language Translation Model
Chaudhary, L., Ananthanarayana, T., Hoq, E., & Nwogu, I. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3232389 · Cited by 47
“…improvements in using multi-scale input features for SLT, as well as the recent successes in multimodal fusion-based learning models ([37] and [5], [36]), [1] introduced a set of fusion-transformers to jointly encode three different scales of the input sign sequences and decode with a…”
BackgroundRate-Adaptive Coding Mechanism for Semantic Communications With Multi-Modal Data
He, Y., Yu, G., & Cai, Y. (2023) · IEEE Transactions on Communications · doi:10.1109/tcomm.2023.3335977 · Cited by 46
“• CMU-MOSEI [30] is the largest MSA dataset to date and is an improvement over MOSI.”
BackgroundDefending Multimodal Fusion Models against Single-Source Adversaries
Yang, K., Lin, W.-Y., Barman, M., Condessa, F., & Kolter, J. Z. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00335 · Cited by 48
“Multimodal neural networks have demonstrated remarkable performance across a variety of vision tasks, such as scene understanding [14], object detection [35, 12], sentiment analysis [40, 3, 39, 19], speech recognition [1], and medical imaging [11].”
BackgroundFactorized Multimodal Transformer for Multimodal Sequential LearningPreprint
Zadeh, A., Mao, C., Shi, K., Zhang, Y., Liang, P. P., Poria, S., & Morency, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09826 · Cited by 51
“Multimodal models based on LSTMs include Multi-view LSTMs Rajagopalan et al. (2016), Memory Fusion Network (Zadeh et al., 2018a) with its recurrent and graph variants (Liang et al., 2018; Zadeh et al., 2018c), as well as Multi-attention Recurrent Networks (Zadeh et al., 2018b).”
MethodSelf-citationKim, D., & Kang, P. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.07.035 · Cited by 39
Learning speaker-independent multimodal representation for sentiment analysis
Wang, J., Wang, S., Lin, M., Xu, Z., & Guo, W. (2023) · Information Sciences · doi:10.1016/j.ins.2023.01.116 · Cited by 40
Guo, L., Song, Y., & Ding, S. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111969 · Cited by 39
Triple disentangled representation learning for multimodal affective analysis
Zhou, Y., Liang, X., Chen, H., Zhao, Y., Chen, X., & Yu, L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102663 · Cited by 41
TF-BERT: Tensor-based fusion BERT for multimodal sentiment analysis
Hou, J., Omar, N., Tiun, S. B., Saad, S., & He, Q. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107222 · Cited by 43
Hsu, J.-H., & Wu, C. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3258900 · Cited by 39
Fcdnet: Fuzzy Cognition-Based Dynamic Fusion Network for Multimodal Sentiment Analysis
Liu, S., Luo, Z., & Fu, W. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3407739 · Cited by 40
The FaceChannel: A Fast and Furious Deep Neural Network for Facial Expression Recognition
Barros, P. V. A., Churamani, N., & Sciutti, A. (2020) · SN Computer Science · doi:10.1007/s42979-020-00325-6 · Cited by 38
“heir generalization of expression representations enabling a universal and automatic FER machine. The development of such models was supported by the collection of several \in-the-wild" datasets [15,16,17,18] that provided large amounts of well-labelled data. These datasets usually contain emotion expressions fro…”
BackgroundNHFNET: A Non-Homogeneous Fusion Network for Multimodal Sentiment Analysis
Fu, Z., Liu, F., Xu, Q., Qi, J., Fu, X., Zhou, A., & Li, Z. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859836 · Cited by 37
“The experimental results show that our proposed method achieves the state-of-the-art on the CMU-MOSEI [17] dataset.”
MethodWang, H., Li, X., Ren, Z., Wang, M., & Ma, C. (2023) · Sensors · doi:10.3390/s23052679 · Cited by 37
“At present, the primary multimodal datasets used are the Flickr30k dataset [15], the VQA dataset [16], and the CMU-MOSEI dataset [17].”
MethodEvaluating significant features in context‐aware multimodal emotion recognition with XAI methods
Khalane, A., Makwana, R., Shaikh, T., & Ullah, A. (2023) · Expert Systems · doi:10.1111/exsy.13403 · Cited by 48
“Furthermore, we were required to use a model-agnostic method so that it was versatile enough to explain the behavior of any model, which has been previously used or might be used in the future to train our dataset, regardless of its architecture or learning algorithm (Speith, 2022).”
Token-disentangling Mutual Transformer for multimodal emotion recognition
Yin, G., Liu, Y., Liu, T., Zhang, H., Fang, F., Tang, C., & Jiang, L. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108348 · Cited by 43
A Unimodal Reinforced Transformer With Time Squeeze Fusion for Multimodal Sentiment Analysis
He, J., Mai, S., & Hu, H. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3078074 · Cited by 37
“We apply our model to MOSEI [5] dataset, which contains bothalignedandunalignedunimodalsequences.”
MethodDIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations
Lyu, Y., Liang, P. P., Deng, Z., Salakhutdinov, R., & Morency, L. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534148 · Cited by 46
“These approaches individually focus on building interpretable components for either modality importance [48], cross-modal interactions [41, 59, 66], or the reasoning process on top of cross-modal interactions [3, 60].”
BackgroundInfluentialSelf-citationLi, Z., Huang, Z., Pan, Y., Yu, J., Liu, W., Chen, H., Luo, Y., Wu, D., & Wang, H. H. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124236 · Cited by 36
Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition?
Shukla, A., Petridis, S., & Pantić, M. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3062406 · Cited by 42
“A lot of contemporary affect analysis approaches are based on deep neural networks that study both the visual and audio modalities [45], [46], [47].”
BackgroundMultimodal and Context-Aware Emotion Perception Model With Multiplicative Fusion
Mittal, T., Bera, A., & Manocha, D. (2021) · IEEE Multimedia · doi:10.1109/mmul.2021.3068387 · Cited by 36
Emotion Recognition in Conversations Using Brain and Physiological Signals
Saffaryazdi, N., Goonesekera, Y., Saffaryazdi, N., Hailemariam, N. D., Temesgen, E. G., Nanayakkara, S. C., Broadbent, E., & Billinghurst, M. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490099.3511148 · Cited by 37
“MELD [61] and CMU-MOSEI [96] are large scale datasets which has been widely used in deep learning-based studies [50, 53].”
BackgroundFlexible Vertical Federated Learning With Heterogeneous Parties
Castiglia, T., Wang, S., & Patterson, S. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3309701 · Cited by 42
“We utilize three datasets for our experiments: the MOSEI dataset [53], the ImageNet dataset [54], and the ModelNet40 dataset [55].”
Survey of deep emotion recognition in dynamic data using facial, speech and textual cues
Zhang, T., & Tan, Z. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17944-9 · Cited by 36
A review on deepfake generation and detection: bibliometric analysis
Kaushal, A., Kumar, S., & Kumar, R. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18706-x · Cited by 36
FDR-MSA: Enhancing multimodal sentiment analysis through feature disentanglement and reconstruction
Fu, Y., Huang, B., Wen, Y., & Zhang, P. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111965 · Cited by 35
MDKAT: Multimodal Decoupling With Knowledge Aggregation and Transfer for Video Emotion Recognition
Wang, J., Wang, C., Guo, L., Zhao, S., Wang, D., Zhang, S., Zhao, X., Yu, J., Wang, Y., Yang, Y., et al. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3571534 · Cited by 38
“We evaluate the performance of the proposed MDKAT on two typical emotional video datasets such as CMU-MOSI [31] and CMU-MOSEI [32].”
General Debiasing for Multimodal Sentiment Analysis
Sun, T., Ni, J., Wang, W., Jing, L., Wei, Y., & Nie, L. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612051 · Cited by 37
“In this section, we conducted extensive experiments on two widely-used benchmark datasets, ( i.e., MOSI and MOSEI), to answer the following research questions.”
BackgroundInfluentialAdaMoW: Multimodal Sentiment Analysis Based on Adaptive Modality-Specific Weight Fusion Network
Zhang, J., Wu, X., & Huang, C. (2023) · IEEE Access · doi:10.1109/access.2023.3276932 · Cited by 36
“For the raw data, we directly obtained the preliminary processed features in the dataset publisher’s CMU-Multimodal SDK [40], and then input the feature vectors of different modalities into the unimodal feature extraction module of our proposed model.”
MethodAdapt and explore: Multimodal mixup for representation learning
Lin, R., & Hu, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102216 · Cited by 34
Enhancing multimodal depression diagnosis through representation learning and knowledge transfer
Yang, S., Cui, L., Wang, L., Wang, T., & You, J. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e25959 · Cited by 37
“In practice, we commence the transfer learning process by training our framework on datasets like CMU-MOSI [41] and CMU-MOSEI [42], which contain diverse multi-modal emotion recognition data.”
MethodQuantum Cognitively Motivated Decision Fusion for Video Sentiment Analysis
Gkoumas, D., Li, Q., Dehdashti, S., Melucci, M., Yu, Y., & Song, D. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i1.16165 · Cited by 34
A Cross-Corpus Study on Speech Emotion Recognition
Milner, R., Jalal, M. A., Ng, R. W. M., & Hain, T. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003838 · Cited by 35
“But now with the introduction of larger datasets, such as the MOSEI database which has around 65 hours of natural emotion data [20], more effort can be put on cross-corpus research focussing on a single language and a single age group.”
BackgroundInfluentialFeng, J., Wang, G., Zheng, C., Cai, Y., Fu, Z., Wang, Y., Wei, X.-Y., & Li, Q. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3284474 · Cited by 35
“To incorporate more modalities into the process of training and testing, CMU-MOSEI [14] is proposed with visual, textual and acoustic data to analyze the opinion sentiment and emotion intensity.”
MethodInferring Sentiments from Supervised Classification of Text and Speech cues using Fuzzy Rules
Vashishtha, S., & Susan, S. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.03.348 · Cited by 33
Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition
Dai, W., Liu, Z., Yu, T., & Fung, P. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.30 · Cited by 33
i-Code: An Integrative and Composable Multimodal Learning Framework
Yang, Z., Fang, Y., Zhu, C., Pryzant, R., Chen, D., Shi, Y., Yi‐chong, X., Qian, Y., Gao, M., Chen, Y., et al. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26290 · Cited by 45
Towards Robust Multimodal Sentiment Analysis Under Uncertain Signal Missing
Li, M., Yang, D., & Zhang, L. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3324552 · Cited by 33
“MOSEI [36] is a sentiment analysis dataset made up of 23,454 movie video clips, which has 16,326, 1,871, and 4,659 samples in train, valid, and test data.”
BackgroundEmotions Don't Lie: An Audio-Visual Deepfake Detection Method Using Affective CuesPreprint
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.06711 · Cited by 33
“There are many other applications of video processing that use and combine multiple modalities for audio-visual speech recognition [28], emotion recognition [40, 63], and language and vision tasks [17, 29].”
BackgroundMethodEnsemble Pretrained Models for Multimodal Sentiment Analysis using Textual and Video Data Fusion
Liu, Z., Braytee, A., Anaissi, A., Zhang, G., Qin, L., & Akram, J. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589335.3651971 · Cited by 37
“Notewor-thy datasets such as IEMOCAP, DEAP, CMU-MOSI, CMU-MOSEI, and MELD have become instrumental in offering unparalleled insights into human emotional dynamics through different modalities [3, 10, 18, 30, 33].”
Wang, X., Lyu, J., Kim, B., Parameshachari, B. D., Li, K., & Li, Q. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3419140 · Cited by 38
“…to verify and analyze the performance of the model in the laboratory, including the Multimodal Corpus of Sentiment Intensity (MOSI) dataset [41], the Multimodal Opinion Sentiment and Emotion Intensity (MOSEI) dataset [42], and the Chinese Single and Multimodal Sentiment (SIMS) dataset [43].”
Li, M., Zhu, Z., Li, K., & Pei, H. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3430045 · Cited by 33
“The dataset we used is CMU-MOSEI [50] raw data, which includes sentences, video clips, and audio segments.”
Tian, F., Zhang, L., Zhu, L., Zhao, M., Liu, J., Dong, Q., & Zhao, Q. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3420445 · Cited by 33
Multimodal Sentiment Analysis with Preferential Fusion and Distance-aware Contrastive Learning
Ma, F., Zhang, Y., & Sun, X. (2023) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme55011.2023.00237 · Cited by 31
Cross-modal contrastive learning for multimodal sentiment recognition
Yang, S., Cui, L., Wang, L., & Wang, T. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05355-8 · Cited by 31
“In this study, we evaluate the proposed framework using three benchmark datasets, namely CMU-MOSI [34], CMU-MOSEI [35], and CH-SIMS [36].”
Noise-Resistant Multimodal Transformer for Emotion Recognition
Liu, Y., Zhang, H., Zhan, Y., Chen, Z., Yin, G., Wei, L., & Chen, Z. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02304-3 · Cited by 35
“, 2016), MOSEI (Zadeh et al., 2018), and IEMOCAP (Busso et al.”
InfluentialSSLMM: Semi-Supervised Learning with Missing Modalities for Multimodal Sentiment Analysis
Wang, Y., Jian, H., Zhuang, J., Guo, H., & Leng, Y. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103058 · Cited by 34
Speech Emotion Recognition ‘in the Wild’ Using an Autoencoder
Dissanayake, V., Zhang, H., Billinghurst, M., & Nanayakkara, S. C. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1356 · Cited by 31
“Researchers have been exploring possibilities using crowdsourcing technologies and scraping available data from the internet [7, 20].”
BackgroundSentiment Word Aware Multimodal Refinement for Multimodal Sentiment Analysis with ASR Errors
Wu, Y., Zhao, Y., Yang, H., Chen, S., Qin, B., Cao, X., & Zhao, W. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.109 · Cited by 33
Modality-specific Learning Rates for Effective Multimodal Additive Late-fusion
Yao, Y., & Mihalcea, R. F. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.143 · Cited by 36
“…sentiment analysis (Zadeh et al., 2016; Yao et al., 2020; Poria et al., 2019), emotion recognition (Busso et al., 2008; Jaiswal et al., 2020; Zadeh et al., 2018), and other real-world applications such as disaster classification (Tian et al., 2018) and movie genre classification (Ovalle et…”
BackgroundDeng, H., Yang, Z., Hao, T., Li, Q., & Liu, W. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3211197 · Cited by 32
“To this end, quite a few works [20], [22], [30] use a multi-stage strategy for multimodal fusion.”
BackgroundMethodMultimodal Sentiment Recognition With Multi-Task Learning
Zhang, S., Yin, C., & Yin, Z. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3224929 · Cited by 31
“This paper aims at achieving sentiment recognition for multimodal sequences and all the studies are conducted on CMUMOSI (CMU Multimodal Opinion Sentiment Intensity) [32] and CMU-MOSEI (CMU Multimodal Opinion Sentiment and Emotion Intensity) [33] datasets proposed by Zadeh et al.”
MethodOn the Utility of Self-Supervised Models for Prosody-Related Tasks
Lin, G.-T., Feng, C.-L., Huang, W., Tseng, Y., Lin, T.-H., Li, C.-A., Lee, H.-Y., & Ward, N. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10023234 · Cited by 34
“The large SSL models, wav2vec 2.0, HuBERT, and WavLM, even improve on the state-of-the-art (SOTA) performance [26] in the 2-label setup.”
InfluentialZhang, X., Cui, W., Hu, B., & Li, Y. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3354382 · Cited by 30
“Restrictions apply. score from -3 (strongly negative) to 3 (strongly positive) thus is associated with one sentiment value.”
BackgroundMethodUEFN: Efficient uncertainty estimation fusion network for reliable multimodal sentiment analysis
Wang, S., Ratnavelu, K., & Shibghatullah, A. S. (2024) · Applied Intelligence · doi:10.1007/s10489-024-06113-6 · Cited by 30
“CMU-MOSEI [43] represents an advancement over MOSI, featuring a larger volume of utterances, increased sample diversity, an expanded speaker pool, and a broader range and 4,659 samples for testing (see Fig.”
Improving Multimodal fusion via Mutual Dependency Maximisation
Colombo, P., Chapuis, É., Labeau, M., & Clavel, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.21 · Cited by 36
“They are then gathered and a final representation is computed by a gated multi-view memory (Zadeh et al., 2018a).”
BackgroundMethodInfluentialTransformer-Based Interactive Multi-Modal Attention Network for Video Sentiment Detection
Zhuang, X., Liu, F., Hou, J., Hao, J., & Cai, X. (2022) · Neural Processing Letters · doi:10.1007/s11063-021-10713-5 · Cited by 30
“Zadeh et al. [38] proposed a dynamic fusion graph network (DFG) to study the nature of cross-modal dynamics in multi-modal language.”
BackgroundMethodMAG+: An Extended Multimodal Adaptation Gate for Multimodal Sentiment Analysis
Zhao, X., Chen, Y., Li, W., Gao, L., & Tang, B. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746536 · Cited by 30
PIRNet: Personality-Enhanced Iterative Refinement Network for Emotion Recognition in Conversation
Lian, Z., Liu, B., & Tao, J. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3192469 · Cited by 33
“Due to the inherent imbalance across various emotions [47]–[49], we choose weighted average F1-score (WAF) as the primary evaluation metric and weighted average accuracy (WAA) as the secondary evaluation metric, in line with previous works [51], [52].”
MethodResultHwang, Y., & Kim, J.-H. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.2 · Cited by 31
A Survey of Cross-Modal Visual Content Generation
Nazarieh, F., Feng, Z., Awais, M., Wang, W., & Kittler, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3351601 · Cited by 32
Wang, R., Yang, Q., Tian, S., Yu, L., He, X., & Wang, B. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129163 · Cited by 30
Multimodal Sentiment Analysis—A Comprehensive Survey From a Fusion Methods Perspective
Zhao, K., Zheng, M., Li, Q., & Liu, J. (2025) · IEEE Access · doi:10.1109/access.2025.3554665 · Cited by 39
Multi-Model Fusion Framework Using Deep Learning for Visual-Textual Sentiment Classification
Al-Tameemi, I. K. S., Feizi‐Derakhshi, M., Pashazadeh, S., & Asadpour, M. (2023) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.040997 · Cited by 31
“Two types of recent research on interpretable multimodal learning have been conducted: (1) attempts to construct interpretable multimodal models through careful model design [38] and (2) post hoc explanations of black-box multimodal models [39].”
Zhang, D., Li, S., Zhu, Q., & Zhou, G. (2020) · IEEE Access · doi:10.1109/access.2020.2969205 · Cited by 28
“• EvaluationMetrics and Significance Test: The performance is evaluated using accuracy (Acc) and F1measure as [9].”
BackgroundMethodMultimodal Boosting: Addressing Noisy Modalities and Identifying Modality Contribution
Mai, S., Ya, S., Xiong, A., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3306489 · Cited by 34
“We evaluate the CMU-MOSI [26] and CMU-MOSEI [78] datasets in our experiment.”
BackgroundMethodIntra- and Inter-Modal Curriculum for Multimodal Learning
Zhou, Y., Wang, X. Y., Chen, H., Duan, X., & Zhu, W. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612468 · Cited by 28
“Typical tasks include multimodal classification [5, 8, 11, 13, 14, 17, 29, 35–37, 43, 69, 79, 80] and regression [40–42 , 58].”
BackgroundModulated Fusion using Transformer for Linguistic-Acoustic Emotion Recognition
Delbrouck, J.-B., Tits, N., & Dupont, S. (2020) · doi:10.18653/v1/2020.nlpbt-1.1 · Cited by 30
Multistage linguistic conditioning of convolutional layers for speech emotion recognition
Triantafyllopoulos, A., Reichel, U. D., Liu, S., Huber, S., Eyben, F., & Schuller, B. W. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1072479 · Cited by 29
Effects of Physiological Signals in Different Types of Multimodal Sentiment Estimation
Katada, S., Okada, S., & Komatani, K. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3155604 · Cited by 29
“…ideally, sentiment labels annotated by users themselves (hereafter referred to as self-sentiment (SS) labels) should be used for sentiment analysis, but many previous works use sentiment labels annotated by a third party (hereafter referred to as third-party sentiment (TS) labels) [10], [11], [12].”
MethodInfluentialM5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining
Dong, X. M., Zhan, X., Wu, Y., Wei, Y., Kampffmeyer, M., Wei, X.-Y., Lu, M., Wang, Y., & Liang, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02057 · Cited by 36
“Other datasets include CMU-MOSEI [54] and XMedia [36], where CMUMOSEI mainly focuses on the emotional analysis and XMedia is utilized for cross-modal retrieval.”
MethodMai, S., Hu, H., & Xing, S. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3082398 · Cited by 26
Speech-Text Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment
Yu, T., Gao, H., Lin, T.-E., Yang, M., Wu, Y., Ma, W., Wang, C., Huang, F., & Li, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.438 · Cited by 33
GLoMo: Global-Local Modal Fusion for Multimodal Sentiment Analysis
Yan, Z., Zhang, Y., Hu, Z., Zhang, X., Deng, J., & Ren, F. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681527 · Cited by 33
“For instance, GME-LSTM [7] and DFG [59] perform fusion across tokens from each modality, whereas RAVEB [24] employs an RNN-based approach to capture representations between modalities.”
InfluentialA review and critical analysis of multimodal datasets for emotional AI
Al-Azani, S., & El-Alfy, E.-S. M. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11271-1 · Cited by 37
Deep-HOSeq: Deep Higher Order Sequence Fusion for Multimodal Sentiment Analysis
Verma, S., Wang, J., Ge, Z., Shen, R., Jin, F., Wang, Y., Chen, F., & Liu, W. (2020) · Proceedings · doi:10.1109/icdm50108.2020.00065 · Cited by 26
“We perform experiments on the CMU-MOSI [20] and CMU-MOSEI datasets [21] where both the datasets consist of opinion videos collected from YouTube with only a single person in front of the camera expressing his opinion.”
MethodMulti-modal Sarcasm Detection Based on Contrastive Attention Mechanism
Zhang, X., Chen, Y., & Li, G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88480-2_66 · Cited by 30
Zeng, Y., Mai, S., & Hu, H. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.109 · Cited by 31
Lin, R., & Hu, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.36 · Cited by 34
“As shown in Table 5, to further explore the relationship of different losses, we conduct ablation experiments of MMCL on CMU-MOSEI dataset under different loss hyperparameters setting.”
BackgroundMethodInfluentialA Multimodal Sentiment Analysis Approach Based on a Joint Chained Interactive Attention Mechanism
Qiu, K., Zhang, Y., Zhao, J., Zhang, S., Wang, Q., & Chen, F. (2024) · Electronics · doi:10.3390/electronics13101922 · Cited by 26
Metadata
- Authors
- Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P.
- Affiliations
- California Miramar University (US)
- Nanyang Technological University (SG)
- Agency for Science, Technology and Research (SG)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2018
- Pages
- 2236-2246
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2883409523
- Cited by
- 1,433
- References
- 62
- Citations by year
- 2026: 281 · 2025: 300 · 2024: 266 · 2023: 202 · 2022: 153 · 2021: 113 · 2020: 73 · 2019: 39 · 2018: 6
- FWCI
- 17.41
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- CMU-MOSEI, multimodal fusion, emotion recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
