The DOI matches its Crossref record; 62 of the 63 references it cites were found in a registry or an index, and 1 are not checked yet.
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Open accessAbstract
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, Louis-Philippe Morency. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This dataset encompasses 3837 videos collected from over 1000 distinctive YouTube speakers, with a reasonably balanced gender distribution of 57% male and 43% female speakers, highlighting the dataset’s focus on inclusivity.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The thematic diversity of the CMU-MOSEI dataset is notable, with a spread across 250 different themes. The three predominant themes are comments (16.2%), debates (2.9%), and consultations (1.8%), while the rest of the themes are nearly evenly distributed.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One of CMU-MOSEI’s unique characteristics is its annotations at the utterance level, with 23,259 samples annotated in this manner.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset [30] is a crucial resource in multimodal sentiment analysis and emotion recognition, distinguished by its scale, which is unmatched in the field.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [30]
The cited source says what the claim attributes to it.
In the citing paper
“We generate open-ended AQAs about speech using the four commonly used datasets IEMOCAP [23], MOSEI [24], LibriTTS [25], and Vox-Celeb2 [26,27].”
From Joint Audio and Speech Understanding (2023) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Cortes, C., & Vapnik, V. N. (1995) · Machine Learning · doi:10.1023/a:1022627411411 · Cited by 34,565
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252
Thumbs up?: sentiment classification using machine learning techniques
Pang, B., Lee, L., & Vaithyanathan, S. (2002) · Proceedings of the ACL-02 conference on Empirical methods in natural language processing - EMNLP '02 · doi:10.3115/1118693.1118704 · Cited by 7,024
Research works printed but not linked: 3
- Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarep-a col- laborative voice analysis repository for speech tech- nologies. In Acoustics, Speech and Signal Process- ing (ICASSP), 2014 IEEE International Conference on. IEEE, pages 960-964.
- Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmerman. 2018. Memn: Multimodal emotional memory network for emotion recognition in dyadic conversational videos. In NAACL.
- Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. 2016. Re- current Highway Networks. arXiv preprint arXiv:1607.03474 . · arXiv
Cited by
Sorted by citations
Items 401–500 of 1,003
Beyond Emotion: A Multi-Modal Dataset for Human Desire Understanding
Jia, A., He, Y. L., Zhang, Y., Uprety, S., Song, D., & Lioma, C. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.108 · Cited by 19
“In addition, Zadeh et al. (2018) proposed an extended version of MOSI, which consists of textual, acoustic and visual clues.”
BackgroundCross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignment
Liang, P. P., Wu, P., Ziyin, L., Morency, L.-P., & Salakhutdinov, R. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475247 · Cited by 22
“[34, 48, 67]) for alignment and enables accurate cross-modal generalization.”
BackgroundSelf-citationSDRS: Sentiment-Aware Disentangled Representation Shifting for Multimodal Sentiment Analysis
Zhao, S., Yang, Z., Shi, H., Feng, X., Meng, L., Qin, B., Yan, C. C., Tao, J., & Ding, G. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3539225 · Cited by 18
“CMU-MOSEI [24] is collected from YouTube monologue videos and those containing too many characters are removed.”
Modeling the Clause-Level Structure to Multimodal Sentiment Analysis via Reinforcement Learning
Zhang, D., Li, S., Zhu, Q., & Zhou, G. (2019) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2019.00131 · Cited by 17
“, CMU-MOSI [21] and CMU-MOSEI [22].”
Employing Co-Learning to Evaluate the Explainability of Multimodal Sentiment Analysis
Jain, D. K., Rahate, A., Joshi, G., Walambe, R. A., & Kotecha, K. V. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3176403 · Cited by 17
“Extensive multimodal sentiment analysis [20] and emotion recognition [21] datasets are introduced.”
BackgroundMethodLifelong Text-Audio Sentiment Analysis learning
Lin, Y., Ji, P., Chen, X., & He, Z. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.02.008 · Cited by 18
VideoPro: A Visual Analytics Approach for Interactive Video Programming
He, J., Wang, X., Kam-Kwai, W., Huang, X., Chen, C., Chen, Z., Wang, F., Zhu, M., & Qu, H. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3326586 · Cited by 17
“This classification scheme is established according to the experts’ requirements and previous work practices [3, 62].”
MethodCross-Modal Dynamic Transfer Learning for Multimodal Emotion Recognition
Hong, S., Kang, H., & Cho, H. (2024) · IEEE Access · doi:10.1109/access.2024.3356185 · Cited by 20
“Multimodal Emotion Recognition (MER) aims to understand human emotions by integrating multiple sources such as language, facial expression, and speech information [4], [5], [6], [7], [8].”
BackgroundMethodYin, B., Chen, Z., & Tao, M. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3364673 · Cited by 17
“The training dataset is CREMA-D [18] and MOSEI [45].”
MethodExploring Inner Speech Recognition via Cross-Perception Approach in EEG and fMRI
Qin, J., Zong, L., & Liu, F. (2024) · Applied Sciences · doi:10.3390/app14177720 · Cited by 19
Enhancing Cross-Language Multimodal Emotion Recognition With Dual Attention Transformers
Zaidi, S. A. M., Latif, S., & Qadir, J. (2024) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2024.3486904 · Cited by 39
“However, the potential of integrating audio and text remains underexplored [9], [10].”
MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition
Lian, Z., Sun, L., Ren, Y., Gu, H., Sun, H., Chen, L., Liu, B., & Tao, J. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3653457 · Cited by 71
“Among existing datasets, some provide multi-label samples (such as IEMOCAP [10] and CMU-MOSEI [17]), some provide unlabeled samples (such as CH-SIMS v2 [20]), and none of themprovidesanoise-corruptedtestset.”
InfluentialCross-modal context-gated convolution for multi-modal sentiment analysis
Wen, H., You, S., & Fu, Y. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.03.025 · Cited by 19
MUSER: MUltimodal Stress detection using Emotion Recognition as an Auxiliary Task
Yao, Y., Papakostas, M., Burzo, M. G., Abouelenien, M., & Mihalcea, R. F. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.216 · Cited by 25
Multimodal Sentiment Analysis Based on a Cross-Modal Multihead Attention Mechanism
Deng, L., Liu, B., & Li, Z. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.042150 · Cited by 18
Context-based emotion recognition: A survey
Abbas, R., Ni, B., Ma, R., Li, T., Lu, Y., & Lu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129073 · Cited by 20
Interpretable Multimodal Capsule Fusion
Wu, J., Mai, S., & Hu, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3178236 · Cited by 16
Estimating the Uncertainty in Emotion Class Labels With Utterance-Specific Dirichlet Priors
Wu, W., Zhang, C., Wu, X., & Woodland, P. C. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3221801 · Cited by 20
“…definition of AER is to classify each utter-ance (i.e., a data sample of emotions with verbal evidence) into a set of pre-defined discrete emotional states (referred to as emotion classes in this paper, such as happy, sad, and frus-trated etc. ) [4], [5], [6], [7], [8], [9], [10], [11].”
BackgroundSense-aware BERT and Multi-task Fine-tuning for Multimodal Sentiment Analysis
Fang, L., Liu, G., & Zhang, R. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892116 · Cited by 16
“To demonstrate the validity of the method, we evaluated it on the public multimodal sentiment analysis datasets CMU-MOSI [8] and CMU-MOSEI [9].”
Method3MASSIV: Multilingual, Multimodal and Multi-Aspect dataset of Social Media Short Videos
Gupta, V., Mittal, T., Mathur, P., Mishra, V., Maheshwari, M., Bera, A., Mukherjee, D., & Manocha, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02039 · Cited by 16
“One 2 https://cloud.google.com/vision, https://sensifai.com/ such application is for understanding the projected [80] and evoked emotions [33,44] from multimedia content like ad-vertisements and movies.”
BackgroundDBT: multimodal emotion recognition based on dual-branch transformer
Yi, Y., Tian, Y., He, C., Fan, Y., Hu, X., & Xu, Y. (2022) · The Journal of Supercomputing · doi:10.1007/s11227-022-05001-5 · Cited by 20
“The feasibility of the proposed 1 3 DBT: multimodal emotion recognition based on dual‑branch… method was verified on publicly available datasets (IEMOCAP [4], CMU-MOSEI [44], and CMU-MOSI [43]).”
MethodContext-Based Adaptive Multimodal Fusion Network for Continuous Frame-Level Sentiment Prediction
Huang, M., Qing, C., Tan, J., & Xu, X. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3321971 · Cited by 18
“Thedatasetswith anutteranceastheannotationlevelarerepresentedbyIEMO-CAP[22]andCMU-Mosei[23].”
Knowledge enhancement for speech emotion recognition via multi-level acoustic feature
Zhao, H., Huang, N., & Chen, H. (2024) · Connection Science · doi:10.1080/09540091.2024.2312103 · Cited by 16
“The final results were presented in CMU-MOSI (Zadeh et al., 2016), CMU-MOSEI (Zadeh et al., 2018), and IEMOCAP (Busso et al., 2008), all three datasets, have achieved significant breakthroughs.”
Sun, J. F., Zhang, X., Han, S., Ruan, Y.-P., & Li, T. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i13.29440 · Cited by 16
Lai, S., Li, J., Guo, G., Hu, X., Li, Y., Tan, Y., Song, Z., Liu, Y., Ren, Z., Wang, C., et al. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651442 · Cited by 16
“CMU-MOSEI [60] stands as the most extensive and comprehensive dataset available for sentiment analysis and emotion recognition.”
BackgroundInfluentialAffective Video Content Analysis: Decade Review and New Perspectives
Xue, J., Wang, J., Liu, X., Zhang, Q., & Wu, X. (2024) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020048 · Cited by 16
Salazar, C., Montoya, E., & Aguilar, J. (2021) · Heliyon · doi:10.1016/j.heliyon.2021.e07253 · Cited by 16
“No dataset with the four characteristics was found, some of the explored datasets were: IEMOCAP, Enterface [42], Mosei [43], SEMAINE [44], MOSI [45], MOUD [46], MELD [47], Humaine [48, 49], CK [50], CK+ [51], DEAP [52], AMIGOS [53, 54], AFEW-VA [13].”
BackgroundMultimodal fusion via cortical network inspired losses
Shankar, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.83 · Cited by 16
“CMU-MOSEI (Zadeh et al., 2018b) is a similar dataset consisting of around 23k review videos taken from YouTube.”
BackgroundMethodInfluentialZhao, X., Li, X., Jiang, R., & Tang, B. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102745 · Cited by 18
Exploring Interpretability in Deep Learning for Affective Computing: A Comprehensive Review
Zhang, X., Zhang, T., Sun, L., Zhao, J., & Jin, Q. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3723005 · Cited by 18
“The contribution of different facial regions to emotion expression is uneven.”
InfluentialA Multi-Focus-Driven Multi-Branch Network for Robust Multimodal Sentiment Analysis
Tao, C., Li, J., Zang, T., & Gao, P. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i2.32146 · Cited by 18
“We evaluate the mean absolute error (MAE) and pearson correlation coefficient (Corr) for regression, seven-class accuracy (Acc-7), five-class accuracy (Acc-5), binary accuracy (Acc-2), and F1 score on CMU-MOSI and CMU-MOSEI.”
InfluentialTPFN: Applying Outer Product Along Time to Multimodal Sentiment Analysis Fusion on Incomplete Data
Li, B., Li, C., Duan, F., Zheng, N., & Zhao, Q. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58586-0_26 · Cited by 15
“We evaluate our method on two datasets: CMU-MOSI [53] and CMU-MOSEI [54].”
MethodMulti-Tensor Fusion Network with Hybrid Attention for Multimodal Sentiment Analysis
Xue, H., Yan, X., Jiang, S., & Lai, H. (2020) · International Conference on Machine Learning and Cybernetics · doi:10.1109/icmlc51923.2020.9469572 · Cited by 15
Affective Interaction: Attentive Representation Learning for Multi-Modal Sentiment Classification
Zhang, Y., Tiwari, P., Rong, L., Chen, R., Alnajem, N. A., & Hossain, M. S. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3527175 · Cited by 15
“The CMU-MOSEI [50], the extended version of MUStARD (MUStARDext for short) 1 [4] and Memotion(2) [37] datasets meet the criteria.”
BackgroundInfluentialA Multi-Level Circulant Cross-Modal Transformer for Multimodal Speech Emotion Recognition
Gong, P., Liu, J., Wu, Z., Han, B., Wang, Y. K., & He, H. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.028291 · Cited by 15
Pace-Adaptive and Noise-Resistant Contrastive Learning for Multimodal Feature Fusion
Guo, X., Kot, A. C., & Kong, A. W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3252270 · Cited by 17
TMMDA: A New Token Mixup Multimodal Data Augmentation for Multimodal Sentiment Analysis
Zhao, X., Chen, Y., Liu, S., Zang, X., Xiang, Y., & Tang, B. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583406 · Cited by 15
“The raw feature of visual and acoustic modalities are pre-extracted from CMU-MOSI [43] and CMU-MOSEI [44] datasets, respectively.”
BackgroundMethodInfluentialExtract-Transform-Load for Video Streams
Kossmann, F., Wu, Z., Lai, E. Y., Tatbul, N., Cao, L., Kraska, T., & Madden, S. R. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3598581.3598600 · Cited by 15
Enhanced Multimodal Representation Learning with Cross-modal KD
Chen, M., Xing, L., Wang, Y., & Zhang, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01132 · Cited by 17
“Multimodal learning has shown much promise in a wide spectrum of applications, such as video understanding [20, 39, 44], sentiment analysis [1, 19, 47] and medical image segmentation [7, 49].”
BackgroundOSAN: A One-Stage Alignment Network to Unify Multimodal Alignment and Unsupervised Domain Adaptation
Liu, Y., Qiao, L., Lu, C., Yin, D., Chen, L., Peng, H., & Ren, B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00346 · Cited by 15
Deep Modular Co-Attention Shifting Network for Multimodal Sentiment Analysis
Shi, P., Hu, M. Q., Shi, X., & Ren, F. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3634706 · Cited by 15
“1 This dataset [64] was collected from YouTube monologue videos and we removed those with too many characters.”
MethodSchnitzer, B. L., Vural, U. C., Schnitzer, B., Sardar, M. U., Fuerst, O., & Korn, O. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3660244 · Cited by 18
“For instance, Bagher Zadeh et al. combine audio, video, and text-based emotion recognition into one feature set [5].”
BackgroundMethodAVES: An Audio-Visual Emotion Stream Dataset for Temporal Emotion Detection
Li, Y., Gan, W., Lü, K., Jiang, D., & Jain, R. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3440924 · Cited by 16
“The AVES dataset, similar to the other in-the-wild emotion datasets [18], [23], [24], is collected from online video sites and contains personal information such as facial images or speech.”
Exploiting Unsupervised Data for Emotion Recognition in Conversations
Jiao, W., Lyu, M. R.-T., & King, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.435 · Cited by 15
An Entanglement-driven Fusion Neural Network for Video Sentiment Analysis
Gkoumas, D., Li, Q., Yu, Y., & Song, D. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/239 · Cited by 15
Utilizing BERT Intermediate Layers for Multimodal Sentiment Analysis
Zou, W., Ding, J., & Wang, C. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9860014 · Cited by 17
“We evaluate our proposed approach on two high-quality multimodal data, CMU-MOSI [13] and CMU-MOSEI [14].”
MethodMultimodal Sentiment Analysis on Unaligned Sequences Via Holographic Embedding
Ma, Y., & Ma, B. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747646 · Cited by 14
CLAP: Contrastive Language-Audio Pre-training Model for Multi-modal Sentiment Analysis
Zhao, T., Kong, M. C., Liang, T., Zhu, Q., Kuang, K., & Wu, F. Q. (2023) · International Conference on Multimedia Retrieval · doi:10.1145/3591106.3592296 · Cited by 16
“In this study, we validate our model on two widely-used public MSA benchmarks: CMU-MOSI [44] and CMU-MOSEI [43].”
MethodMultimodal consistency-specificity fusion based on information bottleneck for sentiment analysis
Liu, W., Cao, S., & Zhang, S. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.101943 · Cited by 14
“Its variants, Recurrent MFN (RMFN) (Liang et al., 2018) and Graph-MFN (Zadeh et al., 2018b), divided the fusion process into several stages and aggregated the information from multi-stages with a hierarchical manner or a probability graph model.”
InfluentialSIA-Net: Sparse Interactive Attention Network for Multimodal Emotion Recognition
Li, S., Zhang, T., & Chen, C. L. P. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3409715 · Cited by 14
“CMU-MOSEI dataset [72] is the largest multimodal sentiment analysis dataset, containing over 22 000 video segments.”
BackgroundFu, J., Fu, Y., Xue, H., & Xu, Z. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01724-5 · Cited by 16
“We evaluated the effectiveness of our model on two benchmark datasets, CMU-MOSI [38] and CMU-MOSEI [39].”
Multi-task Gated Contextual Cross-Modal Attention Framework for Sentiment and Emotion Analysis
Sangwan, S., Chauhan, D. S., Akhtar, M. S., Ekbal, A., & Bhattacharyya, P. (2019) · Communications in computer and information science · doi:10.1007/978-3-030-36808-1_72 · Cited by 14
Barros, P. V. A., Barakova, E. I., & Wermter, S. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3002657 · Cited by 14
Leveraging Pre-Trained Language Model for Speech Sentiment Analysis
Shon, S., Brusco, P., Pan, J. S., Han, K. J., & Watanabe, S. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1723 · Cited by 19
AFR-BERT: Attention-based mechanism feature relevance fusion multimodal sentiment analysis model
Mingyu, J., Zhou, J., & Ning, W. (2022) · PLoS ONE · doi:10.1371/journal.pone.0273936 · Cited by 16
“CMU-MOSEI (Multimodal Opinion Sentiment and Emotion Intensity) [14] dataset comes from over 1,000 online YouTube speakers and contains 3229 videos with a total of 22676 conversations.”
BackgroundMethodA Large Scale Speech Sentiment Corpus
Chen, E., Lu, Z., Xu, H., Cao, L., Zhang, Y., & Fan, J. (2020) · Language Resources and Evaluation · Cited by 14
Multimodal Deep Learning for Mental Disorders Prediction from Audio Speech SamplesPreprint
Naderi, H., Soleimani, B. H., & Matwin, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01067 · Cited by 29
“Zadeh et al. (Zadeh et al. 2018b) proposed a Graph Memory Fusion Network(Graph-MFN) model that considers every combination of modalities as vertices inside a graph and calculates the efficacies of the connections between different nodes to learn the best fusion mechanism for modalities in multimodal language.”
BackgroundMethodInfluentialImproving Multimodal Accuracy Through Modality Pre-training and AttentionPreprint
Ismail, A. A., Hasan, M. M., & Ishtiaq, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.06102 · Cited by 28
“Multimodal Sentiment Analysis CMU MOSI dataset [8], Multimodal Emotion Recognition CMU MOSEI dataset [9] and Multimodal Speaker Trait Recognition Persuasion Opinion Multimodal POM dataset [10].”
BackgroundMethodTriagedMSA: Triaging Sentimental Disagreement in Multimodal Sentiment Analysis
Luo, Y., Liu, W., Sun, Q., Li, S., Li, J., Wu, R., & Tang, X. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3524789 · Cited by 15
“CMU-MOSEI [31]: This dataset is an enhancement of CMU-MOSI .”
Multi-Level Contrastive Learning for Multimodal Sentiment Analysis
Yan, Z., Bai, W., Zhang, Y., Deng, J., Hu, Z., Zhang, X., & Ren, F. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3613116 · Cited by 16
“For MSA, which predicts the intensity of emotions in spoken utterances, we evaluate on three datasets: CMU-MOSI [3], CMU-MOSEI [67], and CH-SIMS [52].”
Stappen, L., Baird, A., Lienhart, M., Bätz, A., & Schuller, B. W. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2022.773154 · Cited by 22
“Similarly, Zadeh et al. (2018) annotated YouTube videos with labels for subjectivity and sentiment intensity (Wöllmer et al., 2013) was the first to transfer the dimensional concept to YouTube videos.”
BackgroundMethodLiu, P., Zheng, X. W., Li, H., Liu, J., Ren, Y., Zhu, H., & Sun, L. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096470 · Cited by 13
“We conduct experiments on three datasets, CMU-MOSI [17], CMU-MOSEI [18] and CH-SIMS [1], respectively.”
MethodLearning Cross-Modal Audiovisual Representations with Ladder Networks for Emotion Recognition
Goncalves, L., & Busso, C. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096138 · Cited by 13
“In forward methods, studies explore the use of cross-modal architectures, which focus on building interactions between different modalities [4, 9–11].”
Low-rank tensor fusion and self-supervised multi-task multimodal sentiment analysis
Miao, X., Zhang, X., & Zhang, H. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-18032-8 · Cited by 13
“We utilized two publicly available sentiment analysis datasets to evaluate our model: the CMU Multi-modal Opinion-level Sentiment Intensity (CMU-MOSI) [42] and the CMU Multi-modal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) [43].”
Explainable Multimodal Learning in Remote Sensing: Challenges and Future Directions
Günther, A., Najjar, H., & Dengel, A. R. (2024) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2024.3404596 · Cited by 13
“Conversely, other researchers have taken a distinct path by conceiving model-specific ante-hoc methods, to effectively accommodate the inherent characteristics of the fusion pipeline [24], [25].”
BackgroundMethodYang, D., Li, M., Qu, L., Yang, K., Zhai, P., Wang, S., & Zhang, L. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3435561 · Cited by 16
“MOSEI [63] is a large-scale sentiment analysis benchmark containing 22,856 samples.”
Graph Reconstruction Attention Fusion Network for Multimodal Sentiment Analysis
Hu, R., Yi, J., Chen, L., & Jin, Z. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3452204 · Cited by 13
“MOSEI [22]: The MOSEI dataset contains 23 453 annotated video segments and 250 topics from 1000 different speakers, and is the next generation of MOSI consisting of online social comment videos, which are emotionally annotated at the video level.”
Li, X., Zhang, F., Hua, Q., & Dong, C.-R. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121515 · Cited by 13
Domain-Separated Bottleneck Attention Fusion Framework for Multimodal Emotion Recognition
He, P., Yu, J., Ge, C., Yu, Y., Xu, W., Wang, L., Liu, T., & Kan, Z. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3711865 · Cited by 16
“The details are outlined in Table 1 as follows: — CMU-MOSI [72] dataset comprises 2,199 samples of short monologue video clips.”
InfluentialTemporal text-guided feedback-based progressive fusion network for multimodal sentiment analysis
Yang, Z., He, Q., Du, N., & He, Q. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2024.12.117 · Cited by 13
SHAPE: An Unified Approach to Evaluate the Contribution and Cooperation of Individual Modalities
Hu, P., Li, X., & Zhou, Y. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/425 · Cited by 23
Weakly-supervised Multi-task Learning for Multimodal Affect RecognitionPreprint
Dai, W., Cahyawijaya, S., Bang, Y., & Fung, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.11560 · Cited by 15
“For example, quite a few works focus on the fusion of modalities, such as the Tensor Fusion Network (Zadeh et al., 2017), Memory Fusion Network (Zadeh et al., 2018a), Multimodal Adaptation Gate (Rahman et al., 2020).”
BackgroundMethodInfluentialDynamic Cross-Feature Fusion for American Sign Language Translation
Ananthanarayana, T., Kotecha, N., Srivastava, P., Chaudhary, L., Wilkins, N., & Nwogu, I. (2021) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg52635.2021.9667027 · Cited by 13
DeToxy: A Large-Scale Multimodal Dataset for Toxicity Classification in Spoken Utterances
Ghosh, S., Lepcha, S., Sakshi, S., Shah, R. R., & Umesh, S. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10752 · Cited by 12
“CMU-MOSEI [21] 44,977 217 44,760 1,000 3 3 3 65:53:36 CMU-MOSI [22] 2,199 67 2,132 98 3 7 3 02:36:17 Common Voice [23] 1,584,219 2,888 1,581,331 66,173 3 7 7 ≈ 2181 : 00 : 00 IEMOCAP [24] 10,087 274 9,813 10 3 3 7 11:28:12 LJ Speech [25] 13,100 40 13,060 1 3 7 7 23:55:17 MELD [26] 13,708 142 13,566 304 3 3 3 12:02:44…”
Gu, R., Wang, T., Deng, J., & Cheng, L. (2023) · Applied Sciences · doi:10.3390/app13074299 · Cited by 12
“successively proposed a tensor fusion network [13], memory fusion network [14] and dynamic fusion graph [15].”
BackgroundDenoising Bottleneck with Mutual Information Maximization for Video Multimodal Fusion
Wu, S., Dai, D., Qin, Z., Liu, T., Lin, B., Cao, Y., & Sui, Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.124 · Cited by 18
ACF: An Adaptive Compression Framework for Multimodal Network in Embedded Devices
Cai, Q., Liu, X., Zhang, K., Xie, X. N., Tong, X., & Li, K. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3303350 · Cited by 15
“Multimodal Deep Learning: Deep multimodal fusion networks (DMFNs) have been successful in many fields, such as emotion recognition [23], [24], action recognition [40], and so on [41], [42].”
BackgroundMethodInfluentialBrain-Inspired Recognition System Based on Multimodal In-Memory Computing Framework for Edge AI
Xiao, H., Xie, D., Hu, X., Zhou, Y., Zhou, G., & Duan, S. (2023) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2023.3337532 · Cited by 12
“We evaluate the effectiveness of our method on three benchmark datasets, namely MOSI [42], MOSEI [43], and MELD [44].”
MethodMixture of Attention Variants for Modal Fusion in Multi-Modal Sentiment Analysis
He, C., Zhang, X., Song, D., Shen, Y., Mao, C., Wen, H., Zhu, D., & Cai, L. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8020014 · Cited by 12
“(2) Graph-MFN [62]: This model’s concept is similar to that of the MFN model, except that Graph-MFN uses a dynamic fusion graph to replace the fusion block in the MFN model.”
MethodInfluentialMulti-Modal Hierarchical Empathetic Framework for Social Robots With Affective Body Control
Gao, Y., Fu, Y., Sun, M., & Gao, F. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3356511 · Cited by 12
“Three datasets (CMU-MOSI [63], CMU-MOSEI [64], IEMOCAP [65]) are described as follows: CMU-MOSI is a multi-modal sentiment analysis dataset with a collection of 2,199 opinion video clips.”
MethodText-Centric Multimodal Contrastive Learning for Sentiment Analysis
Peng, H., Gu, X. F., Li, J., Wang, Z., & Xu, H. (2024) · Electronics · doi:10.3390/electronics13061149 · Cited by 15
CCMA: CapsNet for audio–video sentiment analysis using cross-modal attention
Li, H., Guo, A., & Li, Y. (2024) · The Visual Computer · doi:10.1007/s00371-024-03453-9 · Cited by 12
“CMU-MOSEI [44] is a larger sentiment analysis dataset consisting of video clips taken from YouTube with both sentiment and emotional labels, but in this study, we focus on the sentiment labels.”
An autoencoder-based self-supervised learning for multimodal sentiment analysis
Feng, W., Wang, X., Cao, D., & Lin, D. (2024) · Information Sciences · doi:10.1016/j.ins.2024.120682 · Cited by 13
Liu, W., Xu, H., Hua, Y., Chi, Y., & Gao, K. (2024) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2024.200410 · Cited by 13
Zhang, H., Yan, Y., Cai, Z., Zhan, P., Chen, B. Y., Jiang, B., & Xie, B. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112346 · Cited by 14
Reinforcement learning in sentiment analysis: a review and future directions
Eyu, J. M., Yau, K. A., Liu, L., & Chong, Y.-W. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10967-0 · Cited by 15
“The model is trained using multimodal datasets, namely CMU-MOSI (Zadeh et al. 2016) (D.1) and CMU-MOSEI (Zadeh et al. 2018) (D.2).”
Infusing Affective Computing Models into Advertising Research on Emotions
Wen, T. J., Chuan, C., Anghelcev, G., Sar, S., Yun, J. T., & Xu, Y. (2024) · Journal of Advertising · doi:10.1080/00913367.2024.2409254 · Cited by 13
Li, Z., Liu, P., Pan, Y., Yu, J., Liu, W., Chen, H., Luo, Y., & Wang, H. (2024) · Applied Intelligence · doi:10.1007/s10489-024-06150-1 · Cited by 14
“We validate our approach on two classic datasets, namely, CMU-MOSI [25] and CMU-MOSEI [26].”
Model Collaboration at Network Edge: Feature-Large Models for Real-Time IoT Communications
Yu, X., Zhang, S., Zhang, H., & Song, L. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3523953 · Cited by 15
“…integrated model framework can be applied to various tasks, such as classification or semantic segmentation, as well as different modal datasets, including single modal datasets like ImageNet [40], CIFAR-10 [41], or PASCAL VOC [42], and multiple modal datasets like CMU-MOSEI [43] and How2 [44].”
A review of the emotion recognition model of robots
Zhao, M., Gong, L., & Din, A. S. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06245-3 · Cited by 13
“The DFG trained by the Memory Fusion Network pipeline shows superior emotion recognition and quantification functions [16].”
Conv-Enhanced Transformer and Robust Optimization Network for robust multimodal sentiment analysis
Sun, B., Jia, L., Cui, Y., Wang, N., & Jiang, T. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129842 · Cited by 12
TEMSA:Text enhanced modal representation learning for multimodal sentiment analysis
Chen, J., Song, S., Tan, Y., & Xia, H. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104391 · Cited by 13
Uncertain Multimodal Intention and Emotion Understanding in the Wild
Yang, Q., Shi, Q., Wang, T., & Ye, M. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02300 · Cited by 13
Sentiment Analysis of Fashion Related Posts in Social Media
Yuan, Y., & Lam, W. P. (2022) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3488560.3498423 · Cited by 12
“For the lack of large-scaled dataset, datasets such as MOUD [30] MOSI [43], and MOSEI [44] are constructed based on product review and recommendation videos from YouTube, each associated with a sentiment label.”
BackgroundMethodMultimodal interaction enhanced representation learning for video emotion recognition
Xia, X., Zhao, Y. H., & Jiang, D. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.1086380 · Cited by 12
“To validate the effectiveness of our proposed method, we conduct experiments on two popular video emotion recognition benchmarks, including the Interactive Emotional Dyadic Motion Capture dataset (IEMOCAP) (Busso et al., 2008) and the CMU Multimodal Opinion Sentiment and Emotion Intensity dataset (CMU-MOSEI) (Zadeh an…”
BackgroundMethodInfluentialAdapted Multimodal Bert with Layer-Wise Fusion for Sentiment Analysis
Chlapanis, O. S., Paraskevopoulos, G., & Potamianos, A. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094923 · Cited by 13
“The contributions of our work: • AMB is evaluated on multimodal sentiment analysis with the CMU-MOSEI database to achieve new state-of-the-art results, regardless of being lightweight and data-efficient due to a low trainable parameter budget.”
MethodInfluentialVideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning
Wang, Y., Zeng, D., Wada, S., & Kurihara, S. (2023) · IEEE Access · doi:10.1109/access.2023.3280187 · Cited by 13
“VideoAdviser significantly improves UniMSE’s MAE score by 12.3% for MOSI, and outperforms a strong baseline method VAE-AMDT’s MAE score by 2.4% for MOSEI.”
BackgroundMethodInfluentialTemporal Graph Convolutional Network for Multimodal Sentiment Analysis
Huang, J., Lin, Z., Yang, Z., & Liu, W. (2021) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3462244.3479939 · Cited by 12
“• CMU-MOSEI [39]: The CMU-MOSEI dataset contains 22,856 annotated video segments from 5,000 videos, 1,000 distinct speakers, and 250 different topics.”
BackgroundMethodInfluentialUnbiased Missing-Modality Multimodal Learning
Dai, R., Li, C., Yan, Y., Mo, L., Qin, K., & He, T. (2025) · 2025 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02272 · Cited by 11
“To verify the effectiveness of MD 2 M, we conduct experiments on two multimodal sentiment analysis datasets: CMU-MOSI [59] and CMU-MOSEI [60].”
Bimodal fusion network with multi-head attention for multimodal sentiment analysis
Zhang, R., Xue, C., Qi, Q., Lin, L., Zhang, J., & Zhang, L. (2023) · Applied Sciences · doi:10.3390/app13031915 · Cited by 11
Metadata
- Authors
- Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P.
- Affiliations
- California Miramar University (US)
- Nanyang Technological University (SG)
- Agency for Science, Technology and Research (SG)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2018
- Pages
- 2236-2246
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2883409523
- Cited by
- 1,433
- References
- 62
- Citations by year
- 2026: 281 · 2025: 300 · 2024: 266 · 2023: 202 · 2022: 153 · 2021: 113 · 2020: 73 · 2019: 39 · 2018: 6
- FWCI
- 17.41
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- CMU-MOSEI, multimodal fusion, emotion recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
