The DOI matches its Crossref record; 62 of the 63 references it cites were found in a registry or an index, and 1 are not checked yet.
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Open accessAbstract
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, Louis-Philippe Morency. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This dataset encompasses 3837 videos collected from over 1000 distinctive YouTube speakers, with a reasonably balanced gender distribution of 57% male and 43% female speakers, highlighting the dataset’s focus on inclusivity.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The thematic diversity of the CMU-MOSEI dataset is notable, with a spread across 250 different themes. The three predominant themes are comments (16.2%), debates (2.9%), and consultations (1.8%), while the rest of the themes are nearly evenly distributed.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One of CMU-MOSEI’s unique characteristics is its annotations at the utterance level, with 23,259 samples annotated in this manner.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset [30] is a crucial resource in multimodal sentiment analysis and emotion recognition, distinguished by its scale, which is unmatched in the field.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [30]
The cited source says what the claim attributes to it.
In the citing paper
“We generate open-ended AQAs about speech using the four commonly used datasets IEMOCAP [23], MOSEI [24], LibriTTS [25], and Vox-Celeb2 [26,27].”
From Joint Audio and Speech Understanding (2023) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Cortes, C., & Vapnik, V. N. (1995) · Machine Learning · doi:10.1023/a:1022627411411 · Cited by 34,565
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252
Thumbs up?: sentiment classification using machine learning techniques
Pang, B., Lee, L., & Vaithyanathan, S. (2002) · Proceedings of the ACL-02 conference on Empirical methods in natural language processing - EMNLP '02 · doi:10.3115/1118693.1118704 · Cited by 7,024
Research works printed but not linked: 3
- Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarep-a col- laborative voice analysis repository for speech tech- nologies. In Acoustics, Speech and Signal Process- ing (ICASSP), 2014 IEEE International Conference on. IEEE, pages 960-964.
- Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmerman. 2018. Memn: Multimodal emotional memory network for emotion recognition in dyadic conversational videos. In NAACL.
- Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. 2016. Re- current Highway Networks. arXiv preprint arXiv:1607.03474 . · arXiv
Cited by
Sorted by citations
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259
“Both tasks are frequently used to benchmark models’ ability to fuse multimodal (sentiment) information (Poria et al., 2017; Zadeh et al., 2018a; Liang et al., 2018; Tsai et al., 2019; Pham et al., 2019; Wang et al., 2019).”
BackgroundMethodInfluentialSelf-citationMELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations
Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., & Mihalcea, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1050 · Cited by 1,416
“The other publicly available multimodal emotion and sentiment recognition datasets are MOSEI (Zadeh et al., 2018b), MOSI (Zadeh et al., 2016b), and MOUD (P´erez-Rosas et al., 2013), however none of these datasets is conversational.”
BackgroundSelf-citationMISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Hazarika, D., Zimmermann, R., & Poria, S. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413678 · Cited by 1,073
“com/platform/ 4Please refer to [19, 60] and their respective SDKs (https://github.”
BackgroundMethodInfluentialSelf-citationDynamic Neural Networks: A Survey
Han, Y., Huang, G., Song, S., Yang, L., Wang, H., & Wang, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3117837 · Cited by 848
Gandhi, A., Adhvaryu, K., Poria, S., Cambria, E., & Hussain, A. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.09.025 · Cited by 783
“[47] DFG 2018 Word-level Fusion MFN+DFG CMU-MOSEI Multi-view sequential learning that consists of three main components System of LSTMs consists of multiple Longshort Term Memory (LSTM) networks, Deltamemory Attention Network 85.”
BackgroundSelf-citationYu, W., Xu, H., Yuan, Z., & Wu, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17289 · Cited by 816
Integrating Multimodal Information in Large Pretrained Transformers
Rahman, W., Hasan, M. K., Lee, S., Zadeh, A. B., Mao, C., Morency, L., & Hoque, M. E. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.214 · Cited by 778
“We use two evaluation metrics for BA and F1, one used in (Zadeh et al., 2018d) and one used in (Tsai et al.”
BackgroundMethodInfluentialSelf-citationAn introduction to Deep Learning in Natural Language Processing: Models, techniques, and tools
Lauriola, I., Lavelli, A., & Aiolli, F. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.05.103 · Cited by 755
Harnessing multimodal data integration to advance precision oncology
Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J., & Shah, S. P. (2022) · Nature Reviews Cancer · doi:10.1038/s41568-021-00408-3 · Cited by 662
A systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 713
CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality
Yu, W., Xu, H., Meng, F., Zhu, Y., Ma, Y., Wu, J., Zou, J., & Yang, K. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.343 · Cited by 482
“MFN needs word-level alignment in three modalities.”
BackgroundMethodInfluentialSelf-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372
“Researchers also evaluate representations for sentiment analysis with the INTERFACE [233] and MOSEI (CMU Multimodal Opinion Sentiment and Emotion Intensity) [234] datasets.”
MethodMultimodal sentiment analysis based on fusion methods: A survey
Zhu, L., Zhu, Z., Zhang, C., Xu, Y., & Kong, X. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.02.028 · Cited by 375
Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
Liu, A. T., Yang, S.-W., Chi, P.-H., Hsu, P.-C., & Lee, H.-Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054458 · Cited by 352
“To demonstrate domain invariant transferability of the proposed representation across different datasets, the Mocking-jay model is pre-trained on LibriSpeech and applied on the MOSEI [25] dataset.”
MethodTETFN: A text enhanced transformer fusion network for multimodal sentiment analysis
Wang, D., Guo, X., Tian, Y., Liu, J., He, L., & Luo, X. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.109259 · Cited by 349
Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research
Poria, S., Hazarika, D., Majumder, N., & Mihalcea, R. F. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3038167 · Cited by 362
Self-citationSun, Z., Sarma, P., Sethares, W., & Liang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6431 · Cited by 410
Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413570 · Cited by 319
“There are many other applications of video processing that use and combine multiple modalities for audio-visual speech recognition [28], emotion recognition [40, 63], and language and vision tasks [17, 29].”
BackgroundMethodHybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis
Mai, S., Zeng, Y., Zheng, S., & Hu, H. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3172360 · Cited by 307
Disentangled Representation Learning for Multimodal Emotion Recognition
Yang, D., Huang, S., Kuang, H., Du, Y., & Zhang, L. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547754 · Cited by 294
UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition
Hu, G., Lin, T.-E., Zhao, Y., Lu, G., Wu, Y., & Li, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.534 · Cited by 304
Sun, L., Lian, Z., Liu, B., & Tao, J. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3274829 · Cited by 288
“Note that, there are two distinct methods for the binary formulation, i.e., negative/non-negative [25], and negative/positive [12].”
MethodInfluentialM3ER: Multiplicative Multimodal Emotion Recognition using Facial, Textual, and Speech Cues
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i02.5492 · Cited by 23
Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis
Han, W., Chen, H., Gelbukh, A. F., Zadeh, A., Morency, L., & Poria, S. (2021) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3462244.3479919 · Cited by 265
“The CMU-MOSEI dataset [46] is an upgraded version of CMU-MOSI concerning the number of samples.”
MethodSelf-citationZhang, S., Yang, Y., Chen, C., Zhang, X., Leng, Q., & Zhao, X. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121692 · Cited by 290
Decoupled Multimodal Distilling for Emotion Recognition
Li, Y., Wang, Y., & Cui, Z. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00641 · Cited by 249
“We compare DMD with the current state-of-the-art MER methods under the same dataset settings (unaligned or aligned), including EF-LSTM, LF-LSTM, TFN [33], LMF [14], MFM [29], RAVEN [30], Graph-MFN [36], MCTN [26], MulT [28], PMR [17], MICA [13], MISA [7], and FDMER [32].”
InfluentialMai, S., Hu, H., & Xing, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5347 · Cited by 237
Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis
Akhtar, M. S., Chauhan, D. S., Ghosal, D., Poria, S., Ekbal, A., & Bhattacharyya, P. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1034 · Cited by 238
“Weighted accuracy as a metric is chosen due to unbalanced samples across various emotions and it is also in line with the other existing works (Zadeh et al., 2018c).”
InfluentialSelf-citationA Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
“Zadeh et al. [30] introduced a novel multimodal fusion technique known as Dynamic Fusion Graph (DFG) to investigate the interplay between modalities in human multimodal language.”
BackgroundLv, F., Chen, X., Huang, Y., Duan, L., & Lin, G. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00258 · Cited by 231
“Table 5 displays the ablation study on the CMU-MOSEI benchmark.”
BackgroundJiang, Y., Li, W., Hossain, M. S., Chen, M., Alelaiwi, A. A., & Al-Hammadi, M. H. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.06.019 · Cited by 224
Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis
Zhang, H., Wang, Y., Yin, G., Liu, K., Liu, Y., & Yu, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.49 · Cited by 264
“MOSEI .”
MethodInfluentialEmotion recognition from unimodal to multimodal analysis: A review
Ezzameli, K., & Mahersia, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101847 · Cited by 206
ConFEDE: Contrastive Feature Decomposition for Multimodal Sentiment Analysis
Yang, J., Yu, Y., Niu, D., Guo, W., & Xu, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.421 · Cited by 201
“We compare our methods with the state-of-the-art baselines in Table 1 and 2: LF-DNN (Yu et al., 2020), MFN (Zadeh et al., 2018a), LMF (Liu et al., 2018), TFN (Zadeh et al., 2017), MulT (Tsai et al., 2019), MISA (Hazarika et al., 2020), MAG-BERT (Rahman et al., 2020), HyCon (Mai et al., 2021) and…”
BackgroundMethodInfluentialEmotiCon: Context-Aware Multimodal Emotion Recognition Using Frege’s Principle
Mittal, T., Guhan, P., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01424 · Cited by 191
Sentiment analysis using deep learning approaches: an overview
Habimana, O., Li, Y., Li, R., Gu, X., & Yu, G. (2019) · Science China Information Sciences · doi:10.1007/s11432-018-9941-6 · Cited by 198
“First of all, we acknowledge Zahed et al. [127] for creating a new large dataset for multi-modal sentiment analysis.”
BackgroundInfluentialCross-Modal Enhancement Network for Multimodal Sentiment Analysis
Wang, D., Liu, S., Wang, Q., Tian, Y., He, L., & Gao, X. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3183830 · Cited by 196
“H UMANS naturally communicate with each other through multimodal signals such as language, audio and vision [1].”
BackgroundMethodMai, S., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3171679 · Cited by 182
“2) CMU-MOSEI: CMU-MOSEI [26] is a large multimodal sentiment analysis and emotion recognition dataset that contains a total number of 2928 videos.”
BackgroundMethodCubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimation
Sun, H., Wang, H., Liu, J., Chen, Y., & Lin, L. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548025 · Cited by 191
“As is shown in the table, CubeMLP achieves a MAE of 0.770 on CMU-MOSI and 0.529 on CMU-MOSEI, which is competitive with other state-of-the-art approaches.”
MethodInfluentialAutomated emotion recognition: Current trends and future perspectives
Maithri, M., Raghavendra, U., Gudigar, A., Samanth, J., Barua, P. D., Murugappan, M., Chakole, Y., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106646 · Cited by 172
UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
Hasan, M. K., Rahman, W., Bagher Zadeh, A., Zhong, J., Tanveer, M. I., Morency, L.-P., & Hoque, M. (. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1211 · Cited by 174
“…et al., 2017; Zadeh et al., 2017), multimodal fusion approaches (Liang et al., 2018; Tsai et al., 2018; Liu et al., 2018; Zadeh et al., 2018a; Barezi et al., 2018) as well as resources (Poria et al., 2018a; Zadeh et al., 2018c, 2016; Park et al., 2014; Rosas et al., 2013; W¨ollmer et al., 2013).”
BackgroundMethodInfluentialSelf-citationSurvey of Deep Representation Learning for Speech Emotion Recognition
Latif, S., Rana, R. K., Khalifa, S., Jurdak, R., Qadir, J., & Schuller, B. W. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3114365 · Cited by 193
“CMU-MOSEI [85] English single speaker multimodal natural anger, anxious, disgust, happiness, neutral, sadness, surprise and fear 65 hours Yes”
BackgroundGCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation
Lian, Z., Chen, L., Sun, L., Liu, B., & Tao, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3234553 · Cited by 165
“We also observe a similar phenomenon on the CMU-MOSI and CMU-MOSEI datasets.”
BackgroundInfluentialMultimodal sentimental analysis for social media applications: A comprehensive review
Chandrasekaran, G., Nguyen, T. N., & Hemanth, D. J. (2021) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1415 · Cited by 170
“Another research work (Zadeh et al., 2018) on multimodal sentiment analysis was performed with a new multimodal fusion technique called dynamic fusion graph (DFG).”
BackgroundCOGMEN: COntextualized GNN based Multimodal Emotion recognitioN
Joshi, A., Bhat, A., Jain, A., Singh, A., & Modi, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.306 · Cited by 199
Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion
Siriwardhana, S., Kaluarachchi, T. I., Billinghurst, M., & Nanayakkara, S. C. (2020) · IEEE Access · doi:10.1109/access.2020.3026823 · Cited by 171
“Since the dataset is not uniformly distributed the classes, we followed prior work [4], [40], and only used the four most frequent labels, which are Happy, Sad, Anger, and Excitement.”
BackgroundMethodInfluentialA Comprehensive Survey on Deep Learning Multi-Modal Fusion: Methods, Technologies and Applications
Jiao, T., Guo, C., Feng, X., Chen, Y., & Song, J. (2024) · Computers, Materials & Continua · doi:10.32604/cmc.2024.053204 · Cited by 158
Mai, S., Hu, H., & Xing, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1046 · Cited by 153
“The results on IEMOCAP and CMU-MOSEI datasets are shown in Table 2 and Table 3 core with slight margin.”
BackgroundMethodInfluentialAOBERT: All-modalities-in-One BERT for multimodal sentiment analysis
Kim, K.-H., & Park, S. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.022 · Cited by 149
Multimodal Video Sentiment Analysis Using Deep Learning Approaches, a Survey
Abdu, S. A., Yousef, A. H., & Salem, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.06.003 · Cited by 154
Quantum-inspired multimodal fusion for video sentiment analysis
Li, Q., Gkoumas, D., Lioma, C., & Melucci, M. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.08.006 · Cited by 143
“We evaluate our model by comparing it against state-of-the-art (SOTA) baselines on two benchmarking multimodal sentiment analysis datasets, namely CMU-MOSI [25] and CMU-MOSEI [6].”
BackgroundMethodInfluentialKalateh, S., Estrada-Jimenez, L. A., Nikghadam-Hojjati, S., & Barata, J. (2024) · IEEE Access · doi:10.1109/access.2024.3430850 · Cited by 170
“For instance, Zadeh et al. [111] introduced a Tensor Fusion Network that combines text, audio, and video features into a single framework, outperforming unimodal approaches by effectively fusing multimodal data.”
BackgroundAdvances in Sentiment Analysis - Techniques, Applications, and Challenges
Li, J. (2023) · Artificial intelligence · doi:10.5772/intechopen.111293 · Cited by 140
Cai, Y., Li, X., & Li, J. (2023) · Sensors · doi:10.3390/s23052455 · Cited by 149
“CMU-MOSEI [214] Natural 23,453 annotated video segments; 1000 speaker; 250 topics 6 Textual signals visual signals; audio signals;”
BackgroundVideo sentiment analysis with bimodal information-augmented multi-head attention
Wu, T., Peng, J., Zhang, W., Zhang, H., Tan, S., Yi, F., Ma, C., & Huang, Y. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107676 · Cited by 138
“Some popular methods such as Tensor Fusion Network (TFN) [10], Memory Fusion Network (MFN) [13], Dynamic Fusion Network (DFN) [14] try to capture the interaction relations between modalities dynamically.”
MethodRecognizing Emotions in Video Using Multimodal DNN Feature Fusion
Williams, J., Kleinegesse, S., Comanescu, R., & Radu, O. (2018) · Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML) · doi:10.18653/v1/w18-3302 · Cited by 132
“First, we created our own custom data split from the CMU-MOSEI challenge data so that we could utilize a held-out test set.”
MethodInfluentialSentiment Analysis and Topic Recognition in Video Transcriptions
Stappen, L., Baird, A., Cambria, E., & Schuller, B. W. (2021) · IEEE Intelligent Systems · doi:10.1109/mis.2021.3062200 · Cited by 132
Self-citationXue, Z., & Marculescu, R. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00256 · Cited by 161
“Two examples in CMU-MOSEI [51] for emotion recognition.”
BackgroundMethodYang, K., Xu, H., & Gao, K. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413690 · Cited by 132
“To prove the effectiveness of our method, we evaluate it on the public multimodal sentiment analysis datasets CMU-MOSI [35] and CMU-MOSEI [36].”
MethodShenoy, A., & Sardana, A. (2020) · doi:10.18653/v1/2020.challengehml-1.3 · Cited by 124
Gated Mechanism for Attention Based Multi Modal Sentiment Analysis
Kumar, A., & Vepa, J. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053012 · Cited by 123
“Accuracy F1-Score Approach Accuracy F1-Score Zadeh et al [3] 77.1 79.1 Zadeh et al [8] * 76.0 76.0 Chen et al [5] 76.5 73.4 Zadeh et al [7] 76.9 77.0 Georgiou et al [9] 76.9 76.9 Poria et al [2] 77.64 - Ghosal et al [10] 82.31 80.69 Ghosal et al [10] 79.80 - Sun et al [4] 80 S T u = 0.91, which…”
BackgroundMethodInfluentialMultimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features
Mamieva, D., Abdusalomov, A. B., Kutlimuratov, A., Muminov, B. B., & Whangbo, T. K. (2023) · Sensors · doi:10.3390/s23125475 · Cited by 119
Siriwardhana, S., Reis, A., Weerasekera, R., & Nanayakkara, S. C. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1212 · Cited by 137
Survey on bimodal speech emotion recognition from acoustic and linguistic information fusion
Atmaja, B. T., Sasou, A., & Akagi, M. (2022) · Speech Communication · doi:10.1016/j.specom.2022.03.002 · Cited by 124
Wu, Y., Lin, Z., Zhao, Y., Qin, B., & Zhu, L.-N. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.417 · Cited by 129
“MFN (Zadeh et al., 2018a) captures the interactions across both the different modalities and time.”
BackgroundMethodResultInfluentialMARLIN: Masked Autoencoder for facial video Representation LearnINg
Cai, Z., Ghosh, S., Stefanov, K., Dhall, A., Cai, J., Rezatofighi, H., Haffari, R., & Hayat, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00150 · Cited by 122
“Following the prior works [7, 25], we use overall accuracy( ↑ ) as metrics.”
BackgroundMethodInfluentialA cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning
Wang, L., Peng, J., Zheng, C., Zhao, T., & Zhu, L. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103675 · Cited by 111
Distribution-Consistent Modal Recovering for Incomplete Multimodal Learning
Wang, Y., Cui, Z., & Li, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02013 · Cited by 117
“To verify the effectiveness of DiCMoR, we conduct experiments on two standard multimodal video understanding datasets, including CMU-MOSI [30] and CMU-MOSEI [31].”
MethodMultimodal research in vision and language: A review of current and emerging trends
Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermann, R., & Zadeh, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.009 · Cited by 118
Self-citationMultimodal sentiment analysis with unidirectional modality translation
Yang, B., Shao, B., Wu, L., & Lin, X. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.09.041 · Cited by 101
Context-aware Interactive Attention for Multi-modal Sentiment and Emotion Analysis
Chauhan, D. S., Akhtar, M. S., Ekbal, A., & Bhattacharyya, P. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1566 · Cited by 97
“Weighted accuracy as a metric is chosen due to unbalanced samples across various emotions and it is also in line with the other existing works (Zadeh et al., 2018c).”
BackgroundResultInfluentialSpeech Driven Talking Face Generation From a Single Image and an Emotion Condition
Eskimez, Ş. E., Zhang, Y., & Duan, Z. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3099900 · Cited by 108
A Practical Tutorial on Explainable AI Techniques
Bennetot, A., Donadello, I., El-Qadi, A., Dragoni, M., Frossard, T., Wagner, B., Saranti, A., Tulli, S., Trocan, M., Chatila, R., et al. (2024) · ACM Computing Surveys · doi:10.1145/3670685 · Cited by 111
“Their taxonomy identified five main explanations techniques, namely: feature importance[38], surrogate model[28], example-driven[9], provenance-based[42], and declarative induction[26].”
Zhang, Y., Wang, J., Liu, Y., Rong, L., Zheng, Q., Song, D., Tiwari, P., & Qin, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.01.005 · Cited by 99
Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence
Zadeh, A., Chan, M., Liang, P. P., Tong, E., & Morency, L. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00901 · Cited by 107
“The following baselines are among the state of the art for TVQA [26], MovieQA [43] and CMU-MOSEI [58] dataset.”
BackgroundMethodInfluentialSelf-citationAF: An Association-Based Fusion Method for Multi-Modal Classification
Liang, X., Qian, Y., Guo, Q. J., Cheng, H., & Liang, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3125995 · Cited by 98
Multimodal sentiment analysis: A survey
Lai, S., Hu, X., Xu, H., Ren, Z., & Liu, Z. (2023) · Displays · doi:10.1016/j.displa.2023.102563 · Cited by 106
Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis
Tsai, Y.-H. H., Ma, M., Yang, M., Salakhutdinov, R., & Morency, L.-P. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.143 · Cited by 98
“Two recent methods, Graph-MFN (Zadeh et al., 2018) and Multimodal Factorized Model (MFM) (Tsai et al., 2019b), attempted to interpret relationships between modality interactions and learning for human language.”
BackgroundMethodInfluentialSelf-citationZheng, C., Wu, Z., Feng, J., Fu, Z., & Cai, Y. (2021) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme51207.2021.9428274 · Cited by 101
“CMU-MOSI [16] is a multimodal sentiment analysis dataset which consists of 2199 opinion video clips.”
BackgroundM-SENA: An Integrated Platform for Multimodal Sentiment Analysis
Mao, H., Yuan, Z., Xu, H., Yu, W., Liu, Y., & Gao, K. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.20 · Cited by 116
Liu, Y., Dai, W., Feng, C., Wang, W., Yin, G., Zeng, J., & Shan, S. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548190 · Cited by 114
“[2] presented a dynamic database, CMU-MOSEI, supporting multiple labels consisting of six basic expressions.”
BackgroundLe, H.-D., Lee, G.-S., Kim, S.-H., Kim, S., & Yang, H.-J. (2023) · IEEE Access · doi:10.1109/access.2023.3244390 · Cited by 97
“Weevaluate our proposed method on two standard benchmark multimodal emotion recognition datasets: IEMOCAP [11] and CMU-MOSEI [12].”
BackgroundMethodSKEAFN: Sentiment Knowledge Enhanced Attention Fusion Network for multimodal sentiment analysis
Zhu, C., Chen, M., Zhang, S., Sun, C., Liang, H., Liu, Y., & Chen, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101958 · Cited by 95
A Survey of Cutting-edge Multimodal Sentiment Analysis
Singh, U., Abhishek, K., & Azad, H. K. (2024) · ACM Computing Surveys · doi:10.1145/3652149 · Cited by 103
“YouTube[215] TAV English FL MultimodalOpinionUtteranceDataset(MOUD)[263] TAV Spanish FL InstituteforCreativeTechnologiesMulti-ModalMovieOpinion(ICT-MMMO)[342] TAV English FL/DeL MultimodalOpinionSentimentIntensity(MOSI)[355] TAV English FL 1,646TEDtalkvideosfor14differenttags[346] Text,Audio English DeL Spokendialogte…”
BackgroundM2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis
Wang, X., He, J., Jin, Z., Yang, M., Wang, Y., & Qu, H. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114794 · Cited by 7
“To better reveal the characteristics of multimodal interactions captured by the model, the system should further summarize the instances according to the interaction types [69,70,79].”
BackgroundMethodMai, S., Xing, S., & Hu, H. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2925966 · Cited by 89
“As presented in Table IV, in terms of CMU-MOSEI dataset, LMFN’s accuracy (not include the end2end version) ranks third and is lower than that of BC-LSTM and CAT-LSTM.”
BackgroundMethodInfluentialGkoumas, D., Li, Q., Lioma, C., Yu, Y., & Song, D. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.09.005 · Cited by 96
“For MOSI and MOSEI, we use the CMU-Multi-modal Data SDK1 [22] for feature extraction.”
BackgroundMethodInfluentialStappen, L., Baird, A., Schumann, L., & Schuller, B. W. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3097002 · Cited by 92
“MOSEI [33] contains videos of 250 topics for sentiment analysis (7 classes) and emotion recognition (6”
InfluentialMultimodal End-to-End Sparse Model for Emotion Recognition
Dai, W., Cahyawijaya, S., Liu, Z., & Fung, P. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.417 · Cited by 91
Bhattacharya, U., Childs, E., Rewkowski, N., & Manocha, D. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475223 · Cited by 91
“These include the log-energy spectrum and 13 coefficients containing sufficient information on the speaker’s pitch, intonation, prosody, and other relevant parameters [53], [43].”
BackgroundSMIN: Semi-Supervised Multi-Modal Interaction Network for Conversational Emotion Recognition
Lian, Z., Liu, B., & Tao, J. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3141237 · Cited by 92
“Different from MFN [23] and its extended version [16], this method decomposed the fusion problem into multiple stages and learned cross-view interactions using this multistage fusion method.”
BackgroundMethodInfluentialMultimodal Emotion Recognition Fusion Analysis Adapting BERT With Heterogeneous Feature Unification
Lee, S., Han, D. K., & Ko, H. (2021) · IEEE Access · doi:10.1109/access.2021.3092735 · Cited by 99
“Pu Liang et al. [59] proposed a Recurrent Multistage Fusion Network (RMFN) that decomposes a multimodal fusion problem into multiple stages using LSTM to capture synchronous and asynchronous multimodal interactions.”
BackgroundMethodInfluentialBAFN: Bi-Direction Attention Based Fusion Network for Multimodal Sentiment Analysis
Tang, J., Liu, D., Jin, X., Peng, Y. K., Zhao, Q., Ding, Y., & Kong, W. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3218018 · Cited by 90
“The CMU-MOSEI dataset [58] is an extension of CMU-MOSI associated with many more utterance segments.”
MethodWang, J. Z., Zhao, S., Wu, C., Adams, R. B., Newman, M. G., Shafir, T., & Tsachor, R. P. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3273517 · Cited by 109
Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification
Yang, F., Peng, X., Ghosh, G., Shilon, R., Ma, H., Moore, E., & Predović, G. (2019) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w19-3502 · Cited by 94
“Attention mechanism was initially proposed in neural machine translation to dynamically adjust the focus on the source sentence (Bahdanau et al., 2014), but its application has been extended to many areas including multimodal fusion (Lu et al., 2018; Ghosal et al., 2018; Bagher Zadeh et al., 2018).”
BackgroundMethodEmotional Speech-Driven Animation with Content-Emotion Disentanglement
Daněček, R., Chhatre, K., Tripathi, S., Wen, Y., Black, M. J., & Bolkart, T. (2023) · SIGGRAPH Asia Conference Papers · doi:10.1145/3610548.3618183 · Cited by 103
“Works such as [22,57,74] automatically animate artist-controllable FACS rigs but also lack explicit speech-driven emotion control and some require additional inputs, i.e., a transcript [22,23].”
BackgroundConvolutional Attention Networks for Multimodal Emotion Recognition from Speech and Text Data
Choi, W. Y., Song, K. Y., & Lee, C. W. (2018) · doi:10.18653/v1/w18-3304 · Cited by 88
Mai, S., Xing, S., & Hu, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3068598 · Cited by 89
“In this section, we conduct extensive experiments to evaluate TCM-LSTM on multimodal sentiment analysis and emotion recognition on CMU-MOSI [22], IEMOCAP [57], and CMU-MOSEI [58] datasets.”
BackgroundMethodTag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities
Zeng, J., Liu, T., & Zhou, J. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3532064 · Cited by 94
“Following [42, 46], we also adopt OpenFace2.”
Multi-Modal Sarcasm Detection and Humor Classification in Code-Mixed Conversations
Bedi, M., Kumar, S., Akhtar, M. S., & Chakraborty, T. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3083522 · Cited by 96
“Surveys on multi-modal analysis [15], [16], [17], [18], [19], [20] reveal two prime objectives while handling multi-modal contents:”
Background
Metadata
- Authors
- Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P.
- Affiliations
- California Miramar University (US)
- Nanyang Technological University (SG)
- Agency for Science, Technology and Research (SG)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2018
- Pages
- 2236-2246
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2883409523
- Cited by
- 1,433
- References
- 62
- Citations by year
- 2026: 281 · 2025: 300 · 2024: 266 · 2023: 202 · 2022: 153 · 2021: 113 · 2020: 73 · 2019: 39 · 2018: 6
- FWCI
- 17.41
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- CMU-MOSEI, multimodal fusion, emotion recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
