The DOI matches its Crossref record; 62 of the 63 references it cites were found in a registry or an index, and 1 are not checked yet.
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Open accessAbstract
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, Louis-Philippe Morency. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This dataset encompasses 3837 videos collected from over 1000 distinctive YouTube speakers, with a reasonably balanced gender distribution of 57% male and 43% female speakers, highlighting the dataset’s focus on inclusivity.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The thematic diversity of the CMU-MOSEI dataset is notable, with a spread across 250 different themes. The three predominant themes are comments (16.2%), debates (2.9%), and consultations (1.8%), while the rest of the themes are nearly evenly distributed.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One of CMU-MOSEI’s unique characteristics is its annotations at the utterance level, with 23,259 samples annotated in this manner.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset [30] is a crucial resource in multimodal sentiment analysis and emotion recognition, distinguished by its scale, which is unmatched in the field.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [30]
The cited source says what the claim attributes to it.
In the citing paper
“We generate open-ended AQAs about speech using the four commonly used datasets IEMOCAP [23], MOSEI [24], LibriTTS [25], and Vox-Celeb2 [26,27].”
From Joint Audio and Speech Understanding (2023) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Cortes, C., & Vapnik, V. N. (1995) · Machine Learning · doi:10.1023/a:1022627411411 · Cited by 34,565
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252
Thumbs up?: sentiment classification using machine learning techniques
Pang, B., Lee, L., & Vaithyanathan, S. (2002) · Proceedings of the ACL-02 conference on Empirical methods in natural language processing - EMNLP '02 · doi:10.3115/1118693.1118704 · Cited by 7,024
Research works printed but not linked: 3
- Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarep-a col- laborative voice analysis repository for speech tech- nologies. In Acoustics, Speech and Signal Process- ing (ICASSP), 2014 IEEE International Conference on. IEEE, pages 960-964.
- Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmerman. 2018. Memn: Multimodal emotional memory network for emotion recognition in dyadic conversational videos. In NAACL.
- Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. 2016. Re- current Highway Networks. arXiv preprint arXiv:1607.03474 . · arXiv
Cited by
Sorted by citations
Items 101–200 of 1,003
MissModal: Increasing Robustness to Missing Modality in Multimodal Sentiment Analysis
Lin, R., & Hu, H. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00628 · Cited by 98
“CMU-MOSEI (Zadeh et al., 2018b) expands the multimodal data into 20k video clips from 3,228 videos in 250 diverse topics collected by 1,000 distinct YouTube speakers.”
BackgroundInfluentialLiang, T., Lin, G., Feng, L., Zhang, Y., & Lv, F. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00804 · Cited by 93
PS-Mixer: A Polar-Vector and Strength-Vector Mixer Model for Multimodal Sentiment Analysis
Lin, H., Zhang, P., Ling, J., Yang, Z., Lee, L., & Liu, W. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103229 · Cited by 85
Deep Emotional Arousal Network for Multimodal Sentiment Analysis and Emotion Recognition
Zhang, F., Li, X., Lim, C. P., Hua, Q., Dong, C.-R., & Zhai, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.07.006 · Cited by 81
Sentiment-aware multimodal pre-training for multimodal sentiment analysis
Ye, J., Zhou, J., Tian, J., Wang, R., Zhou, J., Gui, T., Zhang, Q., & Huang, X. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.110021 · Cited by 87
Al-Saadawi, H. F. T., Daş, B., & Daş, R. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124852 · Cited by 81
Disentanglement Translation Network for multimodal sentiment analysis
Zeng, Y., Yan, W., Mai, S., & Hu, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102031 · Cited by 85
Qiao, C., Li, M., Liu, Y., & Tian, Z. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3399612 · Cited by 92
Low Rank Fusion based Transformers for Multimodal Sequences
Sahay, S., Okur, E., Kumar, S. H., & Nachman, L. (2020) · doi:10.18653/v1/2020.challengehml-1.4 · Cited by 93
Tailor Versatile Multi-Modal Learning for Multi-Label Emotion Recognition
Zhang, Y., Chen, M., Shen, J., & Wang, C. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20895 · Cited by 85
Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation
Reza, M. K., Prater-Bennette, A., & Asif, M. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3476487 · Cited by 78
“We tested our adaptation method for multimodal sentiment analysis on CMU-MOSI [69] and CMU-MOSEI [3] datasets, and report the results in Table 5.”
BackgroundMethodNoise Imitation Based Adversarial Training for Robust Multimodal Sentiment Analysis
Yuan, Z., Liu, Y., Xu, H., & Gao, K. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3267882 · Cited by 87
“MOSEI [45] expands the MOSI dataset by enlarging the number of utterances and enriching the variety of samples, speakers, and topics.”
BackgroundMulti-Modal Sequence Fusion via Recursive Attention for Emotion Recognition
Beard, R., Das, R., Ng, R. W. M., Gopalakrishnan, P. G. K., Eerens, L., Świętojański, P., & Mikšík, O. (2018) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k18-1025 · Cited by 77
“The first two datasets provide audio and visual modalities while CMU-MOSEI adds also text transcriptions.”
BackgroundMethodResultInfluentialContrastive Multimodal Fusion with TupleInfoNCE
Liu, Y., Fan, Q., Zhang, S., Dong, H., Funkhouser, T. A., & Yi, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00079 · Cited by 77
“We present results for three popular multi-modality tasks: semantic segmentation on NYUv2 [26], 3D object detection on SUN RGB-D [27], and sentiment analysis on MOSEI [36] and MOSI [35] in Section 5.”
MethodMultimodal Sentiment Analysis With Two-Phase Multi-Task Learning
Yang, B., Wu, L., Zhu, J., Shao, B., Lin, X., & Liu, T. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3178204 · Cited by 82
Multi-level Attention Map Network for Multimodal Sentiment Analysis
Xue, X., Zhang, C., Niu, Z., & Wu, X. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3155290 · Cited by 78
“In the sub-class of hybrid fusion methods, many technical tricks have been employed, such as those via graph fusion [24], [25], gate-based fusion [6], attention-based fusion [7], [10], quantum-based fusion [11], [26], and fusions with multi-task learning [8], [27], and so on.”
BackgroundRodis, N., Sardianos, C., Grammatikis, P. I. R., Sarigiannidis, P. G., Varlamis, I., & Papadopoulos, G. T. (2024) · IEEE Access · doi:10.1109/access.2024.3467062 · Cited by 105
“In particular, an explainable fusion scheme is proposed in [85], which is applied in the context of sentiment analysis and provides a numerical effectiveness metric calculated from the fusion parameters, indicating how modality interactions contribute to the final prediction.”
MethodInfluentialSeq2Seq2Sentiment: Multimodal Sequence to Sequence Models for Sentiment Analysis
Pham, H., Manzini, T., Liang, P. P., & Póczos, B. (2018) · doi:10.18653/v1/w18-3308 · Cited by 82
“As a result, several multimodal datasets, such as CMU-MOSI (Zadeh et al., 2016) and later CMU-MOSEI (Zadeh et al., 2018c), ICT-MMMO (W¨ollmer et al., 2013) and YouTube (Morency et al., 2011), take advantage of the abundance of multimodal data on the Internet.”
InfluentialSelf-citationMultimodal Learning using Optimal Transport for Sarcasm and Humor Detection
Pramanick, S., Roy, A., & Johns, V. M. P. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00062 · Cited by 84
Sun, H., Chen, Y., & Lin, L. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3233070 · Cited by 81
“The CMU-MOSEI [20] dataset is an enlarged version of the CMU-MOSI [19] dataset.”
BackgroundMethodFan, C., Zhu, K., Tao, J., Yi, G., Xue, J., & Lv, Z. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3423671 · Cited by 81
“The CMU-MOSEI [62] dataset is a large-scale MSA and emotion recognition dataset consisting of 23,454 YouTube monolog video clips covering 250 different topics from 1,000 different speakers.”
Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations
Meng, T., Shou, Y., Ai, W., Yin, N., & Li, K. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3445325 · Cited by 74
“Index Terms —Data augmentation, data imbalance, feature fusion, graph neural network, multimodal emotion recognition in conversations (MERC).”
Zeng, Y., Li, Z., Tang, Z., Chen, Z., & Ma, H. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119240 · Cited by 74
Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimoda Emotion Recognition
Li, D., Wang, Y., Funakoshi, K., & Okumura, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.996 · Cited by 92
Towards Emotion-aided Multi-modal Dialogue Act Classification
Saha, T., Patra, A., Saha, S., & Bhattacharyya, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.402 · Cited by 73
“During our initial search, we obtained several multi-modal ER datasets which include Youtube (Morency et al., 2011), MOUD (P´erez-Rosas et al., 2013), IEMO-CAP (Busso et al., 2008), ICT-MMMO (W¨ollmer et al., 2013), CMU-MOSI (Zadeh et al., 2016), CMU-MOSEI (Zadeh et al., 2018) and MELD (Poria et al., 2019) etc.”
BackgroundLatif, S., Rana, R. K., Khalifa, S., Jurdak, R., & Schuller, B. W. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3167013 · Cited by 75
“They fine-tune a transformer trained on a masked language modelling task and can improve emotion recognition performance by 3 % on the CMU-MOSE dataset [58].”
Multi-Task Momentum Distillation for Multimodal Sentiment Analysis
Lin, R., & Hu, H. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3282410 · Cited by 75
“In this section, we compare the proposed MTMD with other baselines on CMU-MOSI [68] and CMU-MOSEI [69], and conduct the ablation study to demonstrate the effectiveness of the proposed method.”
BackgroundMethodMIntRec: A New Dataset for Multimodal Intent Recognition
Zhang, H., Xu, H., Wang, X. Y., Zhou, Q., Zhao, S., & Teng, J. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547906 · Cited by 77
“For example, many datasets have been proposed in multimodal sentiment analysis [6, 54, 57, 58] and emotion recognition [34].”
BackgroundMethodTellamekala, M. K., Amiriparian, S., Schuller, B. W., André, E., Giesbrecht, T., & Valstar, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3325770 · Cited by 75
“We evaluate the proposed COLD fusion approach on: (a) dimensional emotion recognition from face and voice modalities in the AVEC 2019 CES [4] and IEMOCAP [20] datasets, and (b) categorical emotion recognition from face, voice and text modalities in the CMU-MOSEI [21] and IEMOCAP datasets.”
BackgroundMethodMER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning
Lian, Z., Sun, H., Sun, L., Chen, K., Xu, M., Wang, K., Xu, K., He, Y., Li, Y., Zhao, J., et al. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612836 · Cited by 70
“Annotating with both discrete and dimensional emotions is common in current datasets [4, 5].”
BackgroundMethodSelf-citationM3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database
Zhao, J., Zhang, T., Hu, J., Liu, Y., Jin, Q., Wang, X., & Li, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.391 · Cited by 72
Multi-Fusion Residual Memory Network for Multimodal Human Sentiment Comprehension
Mai, S., Hu, H., Jia, X., & Xing, S. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3000510 · Cited by 66
Toward Explainable Affective Computing: A Review
Cortiñas-Lorenzo, K., & Lacey, G. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3270027 · Cited by 87
“For instance, the authors of SEWA [68] indicated the age range, gender, and culture of participants, while, in CMU-MOSEI [69], only the gender is informed.”
BackgroundMultimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications
Manzoor, M. A., Albarri, S., Xian, Z., Meng, Z., Nakov, P., & Liang, S. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3617833 · Cited by 86
“In multimodal information retrieval, the system searches documents with different modalities such as text, images, videos, or physiological signals and images.”
BackgroundLu, Q., Sun, X., Gao, Z., Long, Y., Feng, J., & Zhang, H. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103538 · Cited by 69
Yang, D.-K., Kuang, H., Huang, S., & Zhang, L.-H. (2022) · ACM Multimedia · doi:10.1145/3503161.3547755 · Cited by 97
“We conduct experiments on three standard datasets of multimodal video understanding, including CMU-MOSI [48], CMU-MOSEI [47], and IEMOCAP [3].”
BackgroundMethodLiu, J., Capurro, D., Nguyen, A., & Verspoor, K. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104466 · Cited by 66
Li, M., Yang, D., Lei, Y., Wang, S., Wang, S., Su, L., Yang, K., Wang, Y., Sun, M., & Zhang, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28871 · Cited by 75
“Each sample of MOSI and MOSEI is labeled by human annotators with a sentiment score of -3 (strongly negative) to +3 (strongly positive).”
BackgroundMethodInfluentialMultimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition
Guo, Z., Jin, T., & Zhao, Z. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.94 · Cited by 77
Learning to Balance the Learning Rates Between Various Modalities via Adaptive Tracking Factor
Ya, S., Mai, S., & Hu, H. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3101421 · Cited by 65
Sentiment Analysis: Comprehensive Reviews, Recent Advances, and Open Challenges
Lu, Q., Sun, X., Long, Y., Gao, Z., Feng, J. H., & Sun, T. S. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3294810 · Cited by 77
Li, M., Yang, D., Zhao, X., Wang, S., Wang, Y., Yang, K., Sun, M., Kou, D., Qian, Z., & Zhang, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01184 · Cited by 78
“We conduct extensive experiments on three MSA datasets with word-aligned data, including MOSI [64], MOSEI [65], and IEMOCAP [2].”
MethodDynamically Adjust Word Representations Using Unaligned Multimodal Information
Guo, J., Tang, J., Dai, W., Ding, Y., & Kong, W. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548137 · Cited by 65
“The proposed algorithm is tested using two public benchmark multimodal sentiment analysis and emotion recognition datasets: CMU-MOSI [39] and CMU-MOSEI [40].”
MethodInfluentialSurvey on multimodal approaches to emotion recognition
Aruna Gladys, A., & Vetriselvi, V. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126693 · Cited by 62
A cognitive brain model for multimodal sentiment analysis based on attention neural networks
Li, Y., Zhang, K., Wang, J., & Gao, X. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.10.021 · Cited by 64
Boosting Multi-modal Model Performance with Adaptive Gradient Modulation
Li, H., Li, X., Hu, P., Lei, Y., Li, C., & Zhou, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02030 · Cited by 64
“Therefore, the strength of modality competition tends to be unrelated to the modality type.”
BackgroundNatural Language Supervision For General-Purpose Audio Representations
Elizalde, B., Deshmukh, S., & Wang, H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448504 · Cited by 61
“The training datasets for the 4.6M collection are: WavCaps [6], Au-dioSet [2], FSD50K [11], Clotho [12], AudioCaps [13], MACS [14], WavText5k [5], SoundDesc [15], NSynth [16], FMA [17], Mosi [18], Meld [19], Iemocap [20], Mosei [21], MSP-Podcast [22], CochlScene [23], LJspeech [24], EpicK-itchen [25], Kinectics700 [26…”
Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module
Liu, Y., Yuan, Z., Mao, H., Liang, Z., Yang, W., Qiu, Y., Cheng, T., Li, X., Xu, H., & Gao, K. (2022) · Proceedings of the 2022 International Conference on Multimodal Interaction · doi:10.1145/3536221.3556630 · Cited by 64
“Moreover, the obvious label bias in previous MSA corpus (MOSEI [46] contains 69% positive instances and CH-SIMS contains 69% negative instances) results in trivial unimodal solutions, which further prevents acoustic and visual representation learning.”
BackgroundMethodInfluentialQuantum-inspired Neural Network for Conversational Emotion Recognition
Li, Q., Gkoumas, D., Sordoni, A., Nie, J., & Melucci, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17567 · Cited by 60
“…structures, hybrid memories have been constructed by introducing an additional cell that aggregates the hidden units of unimodal recurrent structures at a time stamp, and is fed to the next time stamp as an additional input (Liang et al. 2018; Zadeh et al. 2018c,a; Bagher Zadeh et al. 2018).”
BackgroundMEDT: Using Multimodal Encoding-Decoding Network as in Transformer for Multimodal Sentiment Analysis
Qi, Q., Lin, L., Zhang, R., & Xue, C. (2022) · IEEE Access · doi:10.1109/access.2022.3157712 · Cited by 66
“The CMU-MOSEI [15] dataset expands its data with a higher number of utterances and greater variety in samples, speakers, and topics than CMU-MOSI.”
BackgroundMethodKim, H., Kang, H., Choi, H., Jung, D., & Hong, T. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.104938 · Cited by 64
Complementary Fusion of Multi-Features and Multi-Modalities in Sentiment AnalysisPreprint
Chen, F., Luo, Z., Xu, Y., & Ke, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.08138 · Cited by 88
“(Ghosal et al. 2018) proposes a recurrent neural network-based multimodal attention framework that leverages contextual information for utterance-level sentiment prediction and shows a state-of-theart model on the CMU-MOSI and CMU-MOSEI datasets.”
BackgroundMethodInfluentialAttention-based Multi-modal Sentiment Analysis and Emotion Detection in Conversation using RNN.
Huddar, M. G., Sannakki, S. S., & Rajpurohit, V. S. (2020) · International Journal of Interactive Multimedia and Artificial Intelligence · doi:10.9781/ijimai.2020.07.004 · Cited by 59
“Other notable works include [24] [25] [26] where multimodal sentiment and emotion detection is addressed using deep learning-based models.”
BackgroundMethodMmlatch: Bottom-Up Top-Down Fusion For Multimodal Sentiment Analysis
Paraskevopoulos, G., Georgiou, E., & Potamianos, A. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746418 · Cited by 62
A survey of transformer-based multimodal pre-trained modals
Han, X., Wang, Y.-T., Feng, J.-L., Deng, C., Chen, Z.-H., Huang, Y.-A., Su, H., Hu, L., & Hu, P.-W. (2023) · Neurocomputing · doi:10.1016/j.neucom.2022.09.136 · Cited by 59
Multimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70
Learning to Learn Better Unimodal Representations via Adaptive Multimodal Meta-Learning
Ya, S., Mai, S., & Hu, H. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3178231 · Cited by 61
“(iii) Graph-based fusion methods have emerged recently, which can effectively extract interaction across time series or modalities [25], [26], [27].”
BackgroundMethodInfluentialChen, Q., Huang, G., & Wang, Y. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3192728 · Cited by 58
“We employ two public datasets, the CMU-MOSI dataset [28] and the CMU-MOSEI dataset [29].”
MethodTEDT: Transformer-Based Encoding–Decoding Translation Network for Multimodal Sentiment Analysis
Wang, F., Tian, S.-W., Yu, L., Liu, J., Wang, J., Li, K., & Wang, Y. (2022) · Cognitive Computation · doi:10.1007/s12559-022-10073-9 · Cited by 55
“In this work, we used two public multimodal sentiment analysis datasets (CMU-MOSI [41] and CMU-MOSEI [42]) to evaluate our proposed model TEDT.”
MethodInfluentialSemantic Alignment Network for Multi-Modal Emotion Recognition
Hou, M.-X., Zhang, Z., Liu, C., & Lu, G. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3247822 · Cited by 63
Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Space
Xie, Z., Yang, Y. Z., Wang, J., Liu, X., & Li, X. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3376564 · Cited by 65
“In addition, there are some multimodal fusion methods, such as graph-based fusion meth-ods [23], [24], [25], [26], tensor-based fusion methods [1], [27], and attention-based fusion methods [2], [28], [29].”
BackgroundMethodSentiment analysis based on text information enhancement and multimodal feature fusion
Liu, Z., Cai, L., Yang, W., & Liu, J. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110847 · Cited by 59
Deep Multimodal Emotion Recognition on Human Speech: A Review
Koromilas, P., & Giannakopoulos, T. (2021) · Applied Sciences · doi:10.3390/app11177962 · Cited by 58
“For this reason, it is mostly used for sentiment analysis architectures (e.g., [63,66,67,70,83–85]).”
InfluentialFirdaus, M., Chauhan, H., Ekbal, A., & Bhattacharyya, P. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.393 · Cited by 58
Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling
Yan, X., Xue, H., Jiang, S., & Liu, Z. (2021) · Applied Artificial Intelligence · doi:10.1080/08839514.2021.2000688 · Cited by 57
“To evaluate the effectiveness of our proposed approach, we compared MTFNCMM with other baseline models on CMU-MOSI Zadeh et al. (2016) and CMU-MOSEI Zadeh et al.”
MethodResultInfluentialEmbracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion
Gao, Z., Jiang, X., Xu, X., Shen, F., Li, Y., & Shen, H. T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02538 · Cited by 79
“The CMU-MOSI [41] and CMU-MOSEI [42] datasets are widely employed video datasets for Multimodal Sentiment Analysis (MSA) task.”
MethodSpeaker-Invariant Affective Representation Learning via Adversarial Training
Li, H., Tu, M., Huang, J., Narayanan, S. S., & Georgiou, P. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054580 · Cited by 57
“CMU-MOSEI dataset [27] contains 23,453 single-speaker video segments carefully chosen from YouTube.”
Improving Depression Level Estimation by Concurrently Learning Emotion Intensity
Qureshi, S. A., Dias, G. H., Hasanuzzaman, M., & Saha, S. (2020) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2020.2998234 · Cited by 58
“In order to estimate the level of depression and the intensity of emotions concurrently, we propose three different multi-task architectures that take as input the transcript files from the DAIC-WOZ [16] and the CMU-MOSEI datasets [17].”
BackgroundMethodInfluentialA deep learning architecture of RA-DLNet for visual sentiment analysis
Yadav, A., & Vishwakarma, D. K. (2020) · Multimedia Systems · doi:10.1007/s00530-020-00656-7 · Cited by 56
“• Chen et al. [63] introduced Sentibank 2.0 called Deep-SentiBank which used CNN-based deep networks for visual sentiment classification.”
MethodInfluentialSPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding
Chung, Y.-A., Zhu, C., & Zeng, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.152 · Cited by 66
Multi-modal Multi-label Emotion Recognition with Heterogeneous Hierarchical Message Passing
Zhang, D., Ju, X., Zhang, W., Li, J., Li, S., Zhu, Q., & Zhou, G. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17686 · Cited by 55
“For MOSEI, we refer to the original paper 3 (Zadeh et al. 2018).”
MethodLin, R., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3303711 · Cited by 59
Dynamic hypergraph convolutional network for multimodal sentiment analysis
Huang, J., Pu, Y., Zhou, D., Cao, J., Gu, J., Zhao, Z., & Xu, D. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126992 · Cited by 57
SWAFN: Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis
Chen, M., & Li, X. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.93 · Cited by 54
“Firstly, we use use MV-LSTM (Rajagopalan et al., 2016), BC-LSTM (Poria et al., 2017a), CAT-LSTM (Poria et al., 2017b), GMELSTM (Chen et al., 2017) , TFN (Zadeh et al., 2017), CHFusion (Majumder et al., 2018), LMF (Liu et al., 2018), MFN (Zadeh et al., 2018a), RMFN (Liang et al., 2018) and MARN (Zadeh et al., 2018c) as…”
BackgroundMethodInfluentialJoint multimodal sentiment analysis based on information relevance
Chen, D., Su, W., Wu, P., & Hua, B. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103193 · Cited by 60
Learning fine-grained representation with token-level alignment for multimodal sentiment analysis
Li, X., Zhang, H., Dong, Z., Cheng, X., Liu, Y., & Zhang, X. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126274 · Cited by 57
CIME: Contextual Interaction-Based Multimodal Emotion Analysis With Enhanced Semantic Information
Wang, R., Guo, C., Cambria, E., Rida, I., Yuan, H., Piran, M. J., Feng, Y., & Zhu, X. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3572495 · Cited by 63
“We evaluate our proposed framework on two widely used benchmarks for emotion recognition: IEMOCAP [43] and MO-SEI [44].”
Self-citationCombining cross-modal knowledge transfer and semi-supervised learning for speech emotion recognition
Zhang, S., Chen, M., Chen, J., Li, Y.-F., Wu, Y., Li, M., & Zhu, C. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107340 · Cited by 54
Deep CNN with late fusion for real time multimodal emotion recognition
Dixit, C., & Satapathy, S. M. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122579 · Cited by 58
Huan, R., Zhong, G., Chen, P., & Liang, R. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3338769 · Cited by 60
“CMU-MOSEI [48] is a larger dataset than CMU-MOSI and contains 22,586 video clips from 1,000 different speakers, covering 250 topic types.”
BackgroundMethodPolarity and Intensity: the Two Aspects of Sentiment Analysis
Tian, L., Lai, C., & Moore, J. D. (2018) · doi:10.18653/v1/w18-3306 · Cited by 58
“This is because in previous studies (e.g., Zadeh et al. (2018a)) the verbal modality was shown to be the most effective for unimodal sentiment analysis, while the vocal modality was shown to be the least effective.”
BackgroundMethodResultInfluentialSLUE: New Benchmark Tasks For Spoken Language Understanding Evaluation on Natural Speech
Shon, S., Pasad, A., Wu, F., Brusco, P., Artzi, Y., Livescu, K., & Han, K. J. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746137 · Cited by 60
ConKI: Contrastive Knowledge Injection for Multimodal Sentiment Analysis
Yu, Y., Zhao, M. M., Qi, S.-A., Sun, F., Wang, B., Guo, W., Wang, X. L., Yang, L., & Niu, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.860 · Cited by 58
Kakuba, S., Poulose, A., & Han, D. S. (2023) · IEEE Access · doi:10.1109/access.2023.3325037 · Cited by 55
“3) CMU-MOSEI DATASET Zadeh et al. [24] proposed the CMU multimodal opinion sentiment and emotion intensity (CMU-MOSEI) dataset that is used in emotion recognition and sentiment analysis.”
BackgroundVideo multimodal sentiment analysis using cross-modal feature translation and dynamical propagation
Gan, C., Tang, Y., Fu, X., Zhu, Q., Jain, D. K., & García, S. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111982 · Cited by 52
Proceedings of the Third Workshop on Abusive Language Online
(2019) · doi:10.18653/v1/w19-35 · Cited by 52
Shared-Private Memory Networks For Multimodal Sentiment Analysis
Zhao, X., Chen, Y., Liu, S., & Tang, B. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3222023 · Cited by 50
“We follow the prior works [1], [10], [15], and conduct experiments on two datasets: CMU-MOSEI [13] and CMU-MOSI [43].”
BackgroundMethodXiao, L., Wu, X., Wu, W., Yang, J., & He, L. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747542 · Cited by 49
Li, Z., Guo, Q., Pan, Y., Ding, W., Yu, J., Zhang, Y., Liu, W., Chen, H., Wang, H., & Xie, Y. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101891 · Cited by 49
Li, X., Liu, J., Xie, Y., Gong, P., Zhang, X., & He, H. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111126 · Cited by 49
Enhancing Multimodal Emotion Recognition through Attention Mechanisms in BERT and CNN Architectures
Makhmudov, F., Kultimuratov, A., & Cho, Y. I. (2024) · Applied Sciences · doi:10.3390/app14104199 · Cited by 57
“However, as we progressed from convolutional layers 1 and 4, the number of input channels started at 16 and doubled, subsequently increasing in increments of 16, ultimately reaching 80 output channels in convolutional layer 4.”
BackgroundMulti-modal Multi-label Emotion Detection with Modality and Label Dependence
Zhang, D., Ju, X., Li, J., Li, S., Zhu, Q., & Zhou, G. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.291 · Cited by 48
“Following (Zadeh et al., 2018b), the textual input dimension dT is set to 300, the visual input dimension dV is set to 35 and the acoustic input dimension dA is set to 74.”
BackgroundMethodInfluentialRecent advances in deep learning based sentiment analysis
Yuan, J., Wu, Y., Lu, X., Zhao, Y., Qin, B., & Liu, T. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1634-3 · Cited by 50
MF-BERT: Multimodal Fusion in Pre-Trained BERT for Sentiment Analysis
He, J., & Hu, H. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3139856 · Cited by 48
Sentiment Analysis and Emotion Recognition from Speech Using Universal Speech Representations
Atmaja, B. T., & Sasou, A. (2022) · Sensors · doi:10.3390/s22176369 · Cited by 58
“[11] listed nine speech sentiment analysis datasets and 13 SER datasets.”
BackgroundMethodMIA-Net: Multi-Modal Interactive Attention Network for Multi-Modal Affective Analysis
Li, S., Zhang, T., Chen, B., & Chen, C. L. P. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3259010 · Cited by 48
“More recently, some multi-modal sentiment analysis and multi-modal emotion recognition datasets have been published and significantlypromotedthedevelopmentofmulti-modalaffective analysis, such as CMU-MOSI [33], CMU-MOSEI [34] and MELD [35].”
BackgroundMethodLearning discriminative multi-relation representations for multimodal sentiment analysis
Tang, Z., Xiao, Q., Zhou, X., Li, Y., Chen, C., & Li, K. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119125 · Cited by 50
Nguyen, C.-V. T., Mai, T., The, S., Kieu, H.-D., & Le, D.-T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.937 · Cited by 55
“In addition, there have been advancements in multimodal models that leverage the dependencies and complementarities of multiple modalities to improve the ERC performance (Poria et al., 2017; Hazarika et al., 2018b; Zadeh et al., 2018).”
BackgroundMethodFMSA-SC: A Fine-Grained Multimodal Sentiment Analysis Dataset Based on Stock Comment Videos
Song, L., Chen, S., Meng, Z., Sun, M., & Shang, X. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3363641 · Cited by 49
“Most of the existing MSA datasets are mainly constructed based on the videos collected from movies, TV series, or video blogs posted on video sharing websites, such as the MOSEI [24] and MOSI [15] datasets.”
BackgroundMethodRajendran, S., Pan, W., Sabuncu, M. R., Chen, Y., Zhou, J., & Wang, F. (2024) · Patterns · doi:10.1016/j.patter.2023.100913 · Cited by 47
Metadata
- Authors
- Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P.
- Affiliations
- California Miramar University (US)
- Nanyang Technological University (SG)
- Agency for Science, Technology and Research (SG)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2018
- Pages
- 2236-2246
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2883409523
- Cited by
- 1,433
- References
- 62
- Citations by year
- 2026: 281 · 2025: 300 · 2024: 266 · 2023: 202 · 2022: 153 · 2021: 113 · 2020: 73 · 2019: 39 · 2018: 6
- FWCI
- 17.41
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- CMU-MOSEI, multimodal fusion, emotion recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
