A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Open accessThe DOI matches its Crossref record; 134 of the 142 references it cites were found in a registry or an index; 73 of the 129 sources its claims cite were checked for support.
Abstract
Multimodal emotion recognition (MER) refers to the identification and understanding of human emotional states by combining different signals, including-but not limited to-text, speech, and face cues. MER plays a crucial role in the human-computer interaction (HCI) domain. With the recent progression of deep learning technologies and the increasing availability of multimodal datasets, the MER domain has witnessed considerable development, resulting in numerous significant research breakthroughs. However, a conspicuous absence of thorough and focused reviews on these deep learning-based MER achievements is observed. This survey aims to bridge this gap by providing a comprehensive overview of the recent advancements in MER based on deep learning. For an orderly exposition, this paper first outlines a meticulous analysis of the current multimodal datasets, emphasizing their advantages and constraints. Subsequently, we thoroughly scrutinize diverse methods for multimodal emotional feature extraction, highlighting the merits and demerits of each method. Moreover, we perform an exhaustive analysis of various MER algorithms, with particular focus on the model-agnostic fusion methods (including early fusion, late fusion, and hybrid fusion) and fusion based on intermediate layers of deep models (encompassing simple concatenation fusion, utterance-level interaction fusion, and fine-grained interaction fusion). We assess the strengths and weaknesses of these fusion strategies, providing guidance to researchers to help them select the most suitable techniques for their studies. In summary, this survey aims to provide a thorough and insightful review of the field of deep learning-based MER. It is intended as a valuable guide to aid researchers in furthering the evolution of this dynamic and impactful field.
Overview
CoChat Summary
## TL;DR
This paper is a survey of deep learning-based multimodal emotion recognition (MER) using speech, text, and face cues. It reviews popular datasets, feature extraction methods, fusion strategies, and evaluation metrics, and concludes with open challenges for the field.
## Key findings
- The survey identifies 17 advanced multimodal emotion recognition techniques in its research-results analysis table, covering features, fusion strategies, datasets, and metrics.
- For speech features, it notes a shift from hand-crafted tools such as OpenSMILE, COVAREP, OpenEAR, and librosa toward deep/self-supervised methods such as wav2vec, wav2vec 2.0, HuBERT, and WavLM; it states that WavLM outperforms all preceding models on speaker recognition, speech recognition, and emotion recognition tasks.
- For text features, the paper describes a progression from bag-of-words and early embeddings like word2vec/GloVe to contextual models such as ELMo, BERT, SSWE, and Emo2Vec.
- In its fusion analysis, the paper says fine-grained interaction fusion remains a popular choice, and that IEMOCAP emerges as the predominant dataset choice, while MELD and YouTube are gaining traction.
## Methods
- This is a narrative survey of deep learning-based MER, organized around datasets, feature extraction, fusion methods, evaluation metrics, and research-result analysis.
- It synthesizes information from popular MER datasets such as IEMOCAP, YouTube, MOUD, ICT-MMMO, CMU-MOSI, CMU-MOSEI, OMG, MELD, SEWA, CH-SIMS, and CH-SIMS v2.0, reporting their sample sizes and annotation schemes where stated.
- The paper reviews and categorizes methods, especially model-agnostic fusion (early, late, hybrid) and intermediate layer fusion (simple concatenation, utterance-level interaction, fine-grained interaction), and summarizes common classification/regression metrics such as ACC, UACC, F1, UF1, MSE, RMSE, PCC, and CCC.
## Limitations
- The paper states that many MER datasets are limited by small scale, language restriction, class imbalance, cultural bias, or scripted/non-natural settings; these are limitations of the datasets reviewed, not a limitation of the survey itself.
- Evident from the text: the survey is descriptive rather than an empirical study, so it does not report new experiments or a unified benchmark comparison beyond summarizing prior work.
- Not stated in the paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
63 Valid · 4 Not valid · 40 Not checked · 83 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“but failed to mention early fusion and late fusion techniques.”
§ A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face > 1. Introduction
What the sources say
- Disputedcites [22] Koromilas, P., and Giannakopoulos, T. (2021). Deep multimodal emotion recognition on human speech: A review. Appl. Sci., 11. sourceExperimental — accuracy not yet measured
The Introduction (p. 1) explicitly names and explains both early and late fusion. Section 3.3 (p. 7) also describes an approach that combines early and late fusion, contrary to the claim that the survey failed to mention them. (LLM, conf=0.95)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“In each video clip, only the speaker’s face is visible, thereby eliminating potential distractions or influences from other facial expressions.”
§ A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face > 2. Popular Datasets in Multimodal Emotion Recognition > 2.11. CH-SIMS
What the sources say
- Supportedcites [34] Yu, W., Xu, H., Meng, F., Zhu, Y., Ma, Y., Wu, J., Zou, J., and Yang, K. (2020, January 5–10). Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online. source
Section 3.1 (Data Collection) states that, for each video clip, no faces appear except the speaker’s. That constraint supports the claim that other people’s facial expressions are absent as potential distractions. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The inclusion of emotional dramas, interviews, modern television shows, talk shows, video blogs, movies, period dramas, variety shows, and other scenarios has ensured that the dataset mirrors the wide-ranging and dynamic environments in which emotions are expressed in real life.”
§ A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face > 2. Popular Datasets in Multimodal Emotion Recognition > 2.12. CH-SIMS V2.0
What the sources say
- Supportedcites [35] Liu, Y., Yuan, Z., Mao, H., Liang, Z., Yang, W., Qiu, Y., Cheng, T., Li, X., Xu, H., and Gao, K. (2022, January 7–11). Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module. Proceedings of the 2022 International Conference on Multimodal Interaction, Bengaluru, India. source
In §3.1, “Diverse video scenarios,” the paper lists melodrama, interviews, modern TV, talk shows, vlogs, films, costume TV, variety shows, and other scenes as sources for CH-SIMS v2.0. It also says the collection varies filming conditions to simulate complex real-world scenarios. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Most recently, Chen et al. introduced WavLM [57], the first universal speech pre-training model capable of being applied to a broad range of tasks, including speaker recognition, speech recognition, and emotion recognition.”
§ A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face > 3. Feature Extraction > 3.1. Speech Feature Extraction
What the sources say
- Partly supportedcites [57] Chen, 2022, Wavlm: Large-scale self-supervised pre-training for full stack speech processing, IEEE J. Sel. Top. Signal Process. sourceExperimental — accuracy not yet measured
The Introduction (pp. 1–2) presents WavLM as a general pre-trained model for a broad range of speech tasks. The Universal Representation Evaluation (p. 5) includes speaker identification, automatic speech recognition, and emotion recognition. The paper does not establish that WavLM was the first universal speech pre-training model; its Introduction (p. 1) discusses earlier pre-trained models evaluated on full-stack speech tasks. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Empirical evidence underscored the superior performance and generalization of wav2vec 2.0 over its predecessor.”
§ A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face > 3. Feature Extraction > 3.1. Speech Feature Extraction
What the sources say
- Partly supportedcites [55] Baevski, 2020, wav2vec 2.0: A framework for self-supervised learning of speech representations, Adv. Neural Inf. Process. Syst. sourceExperimental — accuracy not yet measured
The Conclusion (p. 8) reports that wav2vec 2.0 outperformed the previous best result on a LibriSpeech setup using 100 times less labeled data, supporting superior performance over earlier methods. The cited passages do not establish superior generalization specifically over its wav2vec predecessor; the comparison in Model (p. 2) describes an architectural difference, not a generalization result. (LLM, conf=0.78)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
31 more claims · 4 with their cited sources checked
Create a free account to see all 190 claims and to verify the 2 references still open.
- Probably not a substantial claim. 123
- It may state two separate claims as one. 81
- The quote only partly states this claim. 1
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Earlier word embedding models, e.g., word2vec [60] and GloVe [61], predominantly relied on syntactic contexts for training.”
MethodDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 1×
“To mitigate the highlighted limitations, Peters et al. introduced the ELMo (Embeddings from Language Models) [62] method.”
MethodLearning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · cited 1×
“For instance, Tran et al. [72] proposed a simple and efficient three-dimensional convolutional neural network (3D-CNN) for extracting spatiotemporal features.”
BackgroundActive Shape Models-Their Training and Application
Cootes, T. F., Taylor, C. J., Cooper, D. H., & Graham, J. (1995) · Computer Vision and Image Understanding · doi:10.1006/cviu.1995.1004 · Cited by 7,190 · cited 1×
“Traditional facial feature extraction methods include Local Binary Pattern (LBP) [66], Active Appearance Model (AAM) [67], Active Shape Model (ASM) [68], Scale Invariant Feature Transform (SIFT) [69], Histograms of Oriented Gradient (HOG) [70], and Gabor Wavelet Transform [71].”
MethodJoint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · cited 1×
“The Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset, assembled by the Signal Analysis and Interpretation Lab (SAIL) at the University of Southern California, serves as a prominent multimodal resource for emotion recognition research [24].”
Method
Research works printed but not linked: 1
- Tripathi, S., Tripathi, S., and Beigi, H. (2018). Multi-Modal Emotion Recognition on Iemocap with Neural Networks. arXiv.
Cited by
Wu, Y., Mi, Q., & Gao, T. (2025) · Biomimetics · doi:10.3390/biomimetics10070418 · Cited by 182
“Multimodal information fusion is crucial for integrating data from various sources in order to capture the complex nature of emotions [13].”
Al-Saadawi, H. F. T., Daş, B., & Daş, R. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124852 · Cited by 103
Multimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70
TF-BERT: Tensor-based fusion BERT for multimodal sentiment analysis
Hou, J., Omar, N., Tiun, S. B., Saad, S., & He, Q. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107222 · Cited by 51
SSLMM: Semi-Supervised Learning with Missing Modalities for Multimodal Sentiment Analysis
Wang, Y., Jian, H., Zhuang, J., Guo, H., & Leng, Y. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103058 · Cited by 40
A multimodal emotion recognition system using deep convolution neural networks
Almulla, M. (2024) · Journal of Engineering Research · doi:10.1016/j.jer.2024.03.021 · Cited by 38
Enhancement of Virtual Assistants Through Multimodal AI for Emotion Recognition
Rajesh, S. G., Madangarli, S. V., Pisharady, G. S., & Subrahmanyam, R. (2025) · IEEE Access · doi:10.1109/access.2025.3577664 · Cited by 28
“Our framework takes a step in this direction by utilizing datasets that better reflect spontaneous emotional expressions, improving real-world applicability while maintaining computational efficiency [46].”
Multi-Scale Temporal Fusion Network for Real-Time Multimodal Emotion Recognition in IoT Environments
Yoon, S., & Kim, B. (2025) · Sensors · doi:10.3390/s25165066 · Cited by 17
“The primary challenge involves the temporal complexity of human emotions, which manifest across multiple time scales simultaneously [6].”
State-of-the-art Multimodal Emotion Recognition: A comprehensive survey and taxonomy
Yazici, A., Kucukyilmaz, T., Dokeroglu, T., Sharipbay, A., Lee, M.-H., & Tyler, B. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200642 · Cited by 11
Dutt, A., Ismail, M. A., Herawan, T., & Hashem, M. (2024) · IEEE Access · doi:10.1109/access.2024.3496929 · Cited by 13
“• Hybrid fusion : Integrating modalities at intermediate levels of feature extraction [98] 172934 VOLUME 12, 2024 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
Predicting affective engagement and mental strain from prosodic speech features
Yache, V. P., Moradbakhti, L., Neuner, I., & Veselinović, T. (2025) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2025.1656292 · Cited by 6
“These frameworks often employ deep learning architectures such as transformers or recurrent networks to model temporal dependencies and contextual cues, enhancing emotion inference over isolated prosodic features alone (36).”
Cháves-Villota, A., Jiménez-Martín, A., Jojoa, M., Bahillo, A., & García-Domínguez, J. J. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2025.101873 · Cited by 15
Hussain, M., Chen, C., Albouq, S. S., Shinan, K., Alanazi, F., Iqbal, M. W., & Ashraf, M. U. (2025) · PLoS ONE · doi:10.1371/journal.pone.0312867 · Cited by 13
Sun, Y., & Zhou, T. (2025) · IEEE Access · doi:10.1109/access.2025.3591447 · Cited by 21
“…tasks, the objec-tive is typically to classify the overall emotional content of an isolated segment, focusing primarily on how to effectively extract emotional cues from different modalities–such as text, audio, and vision–and how to fuse these modalities for unified prediction [45], [50], [51].”
Thilakavathy, P., Manikandan, G., Deepa, R., et al. (2024) · doi:10.1109/icscna63714.2024.10864102 · Cited by 7
“Deep learning-based multimodal emotion recognition (DL-MER) is presented in a comprehensive analysis by Lian, H., et al., [19] which analyses multimodal datasets, feature extraction approaches, and fusion methodologies.”
Huang, P. F. (2024) · Entertainment Computing · doi:10.1016/j.entcom.2024.100696 · Cited by 7
Lightweight Spatio-Temporal Convolutional Neural Network for Audio-Visual Emotion Recognition
Ding, S. Y., Tang, T. B., & Lu, C. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3566773 · Cited by 8
“Thus, there is a growing need for compact and lightweight CNN solutions to improve AVER system robustness while meeting deployment constraints [26], [33], [34].”
Self-citationAi, Y., Zhang, Y., Wang, J., & Xu, N. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00280-2 · Cited by 10
“In facial expression recognition, combining fine-grained facial action units (Aus) analysis methods can effectively capture subtle changes in facial expressions of the elderly, especially the feature differences produced during the aging process [44, 45].”
Liu, Y., Chen, X., Peng, Z., Li, Y., Li, X., Song, P., Unoki, M., & Zhao, Z. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3561606 · Cited by 7
“Speechemotionrecognition(SER)dealingwithemotional featureconfusionhasbeenextensivelyexploredthroughvarious featureextraction[1],[2]andclassifierlearningmethods[3],[4].”
Facial Expression Recognition Using Deep Convolution Neural Networks
Almulla, M. (2024) · IEEE Annual Congress on Artificial Intelligence of Things (AIoT) · doi:10.1109/aiot63253.2024.00022 · Cited by 8
Govindaraju, V., & Thangam, D. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-6806-0.ch018 · Cited by 8
Multimodal EEG–EMG and FEM-Based Adaptive Control of Passive Upper-Limb Exoskeletons
Bibbò, L., Laganà, F., Pullano, S. A., & Angiulli, G. (2026) · Sensors · doi:10.3390/s26123924 · Cited by 11
Zhang, N., & Leong, W. Y. (2025) · Eurasia Journal of Mathematics Science and Technology Education · doi:10.29333/ejmste/16661 · Cited by 7
Xiu, T., Sun, Y., Zhang, X., Gao, Y., Wu, J., Zhang, A. Y., & Li, H. (2025) · Journal of Organizational and End User Computing · doi:10.4018/joeuc.368008 · Cited by 6
“This multimodal approach fully captures users’ emotional states (Lian et al., 2023).”
Facial and Speech-Based Emotion Recognition Using Sequential Pattern Mining
Song, Y., & Chung, K. (2025) · Electronics · doi:10.3390/electronics14204015 · Cited by 4
Fu, J., Deng, S., Zheng, W., Zheng, C., Liu, J., Li, W., Zeng, Y., Xie, H., Mai, Y., Li, C., et al. (2025) · BMC Cancer · doi:10.1186/s12885-025-14650-7 · Cited by 5
“By mining patients’ daily expressions—including texts, speech, and social media behaviors—algorithms can detect subtle emotional fluctuations, offering objective early-warning indicators [25].”
Meng, J., Zhu, Z., Qi, J., & Zhang, H. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-06524-9 · Cited by 4
Modeling Continuous Emotions in Text Data using IEMOCAP Database
Messaoudi, A., Boughrara, H., & Lachiri, Z. (2024) · doi:10.1109/atsip62566.2024.10638843 · Cited by 4
“Word embedding techniques have emerged as a powerful alternative that addresses the limitations of BoW [15].”
BackgroundLeveraging Emotional AI for Improved Human-Computer Interactions: An Interdisciplinary Perspective
Govindaraju, V., & Thangam, D. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-2794-4.ch005 · Cited by 4
Karthikeyan, V., & Priyadharsini, S. S. (2026) · Evolving Systems · doi:10.1007/s12530-026-09797-y · Cited by 5
Kejriwal, J., & Beňuš, Š. (2025) · Speech Communication · doi:10.1016/j.specom.2025.103240 · Cited by 6
Pan, Y. (2025) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2025.0160613 · Cited by 6
“The traditional self-report and facial expression-based strategy is susceptible to individual differences and subjectivity [3] [4].”
Jiang, G.-F., Zhou, N., Wang, S.-M., & Ni, Y. (2026) · Results in Engineering · doi:10.1016/j.rineng.2026.111903 · Cited by 2
IPA 2.0: Validation of an Interpretable Emotion-Attention Index for Neuro-Adaptive Learning with AI
Arranz-Romero, J., Roig-Vila, R., & Cazorla, M. (2026) · Applied Sciences · doi:10.3390/app16052515 · Cited by 2
Seamless monitoring of stress levels leveraging a foundational model for time sequences
Gabrielli, D., Prenkaj, B., & Velardi, P. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103336 · Cited by 6
Chen, Y., Han, J. H., Lv, J., Qin, X., & Xu, X. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3617788 · Cited by 4
“This process entails the extraction of features from many data sources, including speech [2], [3], [4], facial expressions [5], [6], [7], physiological [8], [9], and text [10], [11], followed by the application of machine learning or deep learning algorithms to identify or predict an individual’s…”
Bridging Discrete and Continuous: A Multimodal Strategy for Complex Emotion Detection
Jia, J., Zhang, H., & Liang, J. (2025) · Machine learning for signal processing ... · doi:10.1109/mlsp62443.2025.11204253 · Cited by 18
“Multimodal methods have been widely chosen in research because they can extract features from various modalities [21, 22, 23, 24].”
Lu, L., Yuan, L., & Chen, L. (2025) · Entertainment Computing · doi:10.1016/j.entcom.2025.101005 · Cited by 2
Morán-Mirabal, L. F., Güemes-Frese, L. E., Favarony-Avila, M., et al. (2025) · Data · doi:10.3390/data10070103 · Cited by 2
“In this context, biometric and behavioral indicators have been effectively used to analyze learners’ cognitive load and emotional states during task execution [9,10].”
Harmonizing the past: EEG-based brain network unveil modality-specific mechanisms of nostalgia
Shuxiang, H., Liu, Y., Yue, Q., Huan, Z., & Maoping, Z. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1517449 · Cited by 3
A Comprehensive Review of Machine Learning Approaches for Speech Emotion Recognition
Kalokhe, T. D., & Kulkarni, R. (2024) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-22308 · Cited by 3
Sinap, V. (2024) · Kahramanmaraş Sütçü İmam Üniversitesi Mühendislik Bilimleri Dergisi · doi:10.17780/ksujes.1420530 · Cited by 5
A Deep GRU-BiLSTM Network for Multi-modal Emotion Recognition from Text
Yacoubi, I., Ferjaoui, R., & Khalifa, A. B. (2024) · International Conference on Advanced Technologies for Signal and Image Processing · doi:10.1109/atsip62566.2024.10638944 · Cited by 2
“In fact, multimodality can take two different forms: first, the integration of different modalities, including image, audio, text, video, and others; using different multimodal fusion techniques, encompassing early fusion, late fusion, or hybrid fusion strategies [11], [12].”
BackgroundSarkar, A., Chatterjee, I., Dhar, A., Das, J., Roy, P., Ghosh, P., & Das, S. (2024) · International Conference on Computer, Electrical & Communication Engineering (ICCECE) · doi:10.1109/iccece58645.2024.10497244 · Cited by 3
“The quality and variety of the data matter for making the emotion detection system work well for different situations [7].”
BackgroundConstruction of music emotion recognition and classification model supported by neural networks
Wu, M. (2026) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-026-00453-6 · Cited by 1
Wang, Y., Shou, Y., Tan, Y., Ai, W., Meng, T., & Li, K. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132002 · Cited by 2
“Meanwhile, the importance of different modalities fluctuates dynamically during the dialogue, affecting the model’s ability to capture emotional changes [18], [19], [20].”
KOTIAN, S. (2026) · Interdisciplinary Journal of AI Machine Learning & Data Science · doi:10.66261/fetdj998
Harnessing Multimodal Unlabeled Data for Enhanced Speech Emotion Recognition
Goncalves, L., & Busso, C. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3647876 · Cited by 1
“These in-cludeaudio-textcombinations[44],[45],[46],[47],audio-visual combinations[48],[49],[50],[51],[52],[53],[54],andtheinte-grationofallthreemodalities[55],[56],[57].”
Green Reinforcement Learning for Adaptive Mental Health Support Systems using multimodal recognition
Soniya, P., & -, A. S. (2025) · International Conference on Artificial Intelligence for Innovations in Healthcare Industries (ICAIIHI) · doi:10.1109/icaiihi67124.2025.11403592 · Cited by 1
Santhiya, P., Shanmugavadivel, K., Rajalakshmi, R., & Krishnamoorthy, N. (2025) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-025-00811-w · Cited by 2
Alhawiti, K. M. (2025) · Sensors · doi:10.3390/s25226959 · Cited by 5
“In parallel, contrastive learning meth-ods have emerged as powerful tools for extracting deep representations from speech, enabling semantic, emotional, and conversational cues to be modeled effectively [6–9].”
A Complementary Fusion Framework for Robust Multimodal Emotion Recognition
Yi, M.-H., Kwak, K.-C., & Shin, J. (2025) · Electronics · doi:10.3390/electronics14224444 · Cited by 2
IoT-powered wearable assist device for visually impaired
Lokesh, S., Sukesh, P., Tejaswini, S., et al. (2025) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2025.2582033 · Cited by 1
“Multimodal emotion recognition is one of the key research areas in assistive technology. a deep learning framework to detect emotions through text, images, and audio information [1].”
A Virtual Music-Therapist Dialogue System Based on Natural Language Processing
Zhao, Y., Meng, N., Yuan, S., Mu, P., Zhao, L., & Yu, Z. (2025) · International Conference on Artificial Intelligence, Virtual Reality and Interaction Design · doi:10.1145/3777730.3777763 · Cited by 1
“By integrating voice, text, facial expression and other modes, it can obtain higher accuracy and robustness than the single-mode method [11].”
Self-citationMultiMoodIfy a Lego-Like Emotion-Aware Music Recommender for Spotify
Marsico, M. D., & Rotiroti, M. (2025) · Biannual Conference of the Italian SIGCHI Chapter (CHItaly) · doi:10.1145/3750069.3750080 · Cited by 1
RBi-LSTM Based CNN Model for Speech Emotion Recognition
Padhira, K. R., Gundla, N. R., Kumar, K., Sarangi, P. K., & Jabbar, M. A. (2025) · International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO) · doi:10.1109/icrito66076.2025.11241506 · Cited by 1
Vietnamese Emotion Recognition from Voice and Text: A Confidence-Based Approach
Thi, N. T. H., Pham, M. D., Le, S. T., Pham, D. D., Huynh, K. T., Tuyen, N. T. K., & Le, T. D. (2025) · International Conference on Multimedia Analysis and Pattern Recognition (MAPR) · doi:10.1109/mapr67746.2025.11133879 · Cited by 1
“…combines features from different modalities at the input level; Late Fusion, which processes each modality separately and merges the outputs; and Intermediate Fusion, which integrates features at intermediate layers, often using architectures like Transformers to model complex interactions [17].”
Automated Personality Analysis from Handwriting: A Systematic Survey
Mukherjee, S., & Ghosh, I. (2025) · SN Computer Science · doi:10.1007/s42979-025-04150-7 · Cited by 1
“It would be interesting to explore multi-modal personality analysis using handwriting, text, speech, facial expression, or gait [118].”
Chaugule, I., & Sankaye, S. R. (2025) · doi:10.38124/ijisrt/25jun755 · Cited by 1
A neural approach to the Turing Test: The role of emotions
Pizzi, R., Quan, H., Matteucci, M., Mentasti, S., & Sassi, R. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107362 · Cited by 1
Attention-Enhanced Convolutional Neural Network for Emotion Recognition in Facial Expressions
Desai, D., Sharma, M., Sri, G. A., et al. (2025) · doi:10.1109/icitiit64777.2025.11040780 · Cited by 1
Lin, C., & Liang, M. (2025) · Journal of Circuits Systems and Computers · doi:10.1142/s0218126625502305 · Cited by 1
Audio-Visual Emotion Recognition Using Deep Learning Methods
Tolegenov, M., Saheer, L. B., & Oghaz, M. M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77915-2_24 · Cited by 1
Personalized Emotion Detection Adapting Models to Individual Emotional Expressions
Mainali, D., Shrestha, S., Thapa, U., et al. (2024) · International Journal of Innovative Science and Research Technology (IJISRT) · doi:10.38124/ijisrt/ijisrt24oct1478 · Cited by 1
Multimodal Emotion Recognition Using Computer Vision: A Comprehensive Approach
Mohan, G. B., Sarrvesh, N., Hrishikeasan, N. P., Boopalamani, J., & Doss, S. (2024) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt61001.2024.10725638 · Cited by 2
“[8] The paper adopts a holistic approach to multimedia content analysis, combining advanced video (ResNet CNNs) and audio (MLP, custom CNN, SVM) techniques.”
MethodEmotion Detection via Voice and Speech Recognition
Chethan, C. V., S, G. K., & C, K. Y. (2024) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-15385 · Cited by 1
Towards Unified Affective AI: A Cross-Modal Survey of Emotion Recognition, Generation, and Control
Mobbs, R., Makris, D., & Argyriou, V. (2026) · Applied Sciences · doi:10.3390/app16189333
Adaptive Learning Analytics Architecture for Inclusive and Sustainable Universities
Shrivastava, A. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-8352-1.ch014
Emotune: Exploring Context-Aware LLM Support for Driver Emotion Regulation
Radetzky, L., Shedge, K., Patel, K., Önbaş, T., Decaro, G., Neufeld, L., Jangir, D., Chandrayan, V., & Álvarez, I. (2026) · International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI) · doi:10.1145/3828158.3838237
Dual-path fusion network for multimodal sentiment analysis under uncertain missing modalities
Li, J., & Liu, W. (2026) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-026-03312-0
Zhao, J., & Liu, F. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1867034 · Cited by 1
Hassan, M., Liying, C. A. O., Rai, H. M., & Ghani, A. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104779
Psychological Foundations of Human Emotion and its Relevance to AI
MALLELA, R. R., KUMAR, B. N., SUDHAMINI, Y., & Reddy, G. S. (2026) · Wiley eBooks · doi:10.1002/9781394479771.ch5
Zhang, D., Ma, Q., Shen, Z., Ma, W., Yonggang, Y., & Kong, S. (2026) · Multimedia Systems · doi:10.1007/s00530-026-02640-z
“In contrast, simple decision-level fusion strategies [14 ](e.g., voting or linear weighting) offer lower computational overhead but struggle to address modality conflicts and fail to capture the dominance differ - ences of distinct emotion categories across modalities.”
Bastidas-Benalcazar, N., Santos-Solís, D., Cueva-Argudo, D., Lechón-Santacruz, E., Almeida-Galárraga, D. A., Navas, P., Mora, H. C., Orozco-Garzón, N., & Tirado-Espín, A. (2026) · Technologies · doi:10.3390/technologies14090523
MS²FL: Modality-shared and Modality-specific Feature Learning for Multimodal Emotion Recognition
Li, X., Chen, H., Wu, M., Song, Q., Li, F., Huang, J., Cui, F.-Q., & Lv, Z. (2026) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3836771 · Cited by 1
“Although this method is simple to implement, it may not fully exploit inter-modal interactions, which can limit the model performance [34].”
Investigating squeeze-and-excitation placement in residual networks for facial emotion recognition
Raju, A. P., Sudhakar, K., & Rubini, P. E. (2026) · Array · doi:10.1016/j.array.2026.101163 · Cited by 2
Rishu, & Kukreja, V. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.116241
Lu, Y., He, Q., Du, N., Wang, Y., & Yang, Z. (2026) · Ain Shams Engineering Journal · doi:10.1016/j.asej.2026.104377
HAFT: Hierarchical Audio-Enhanced Fusion Transformer for Efficient Multimodal Sentiment Analysis
Dong, Q., Lu, T., Shi, X., Xiao, L., & Liu, G. (2026) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194026500579
Rajeswari, A., Madasamy, K. S., Jerin Austin Dhas., J., R.Jenifer, Ebinezer, D., & Kumar, M. V. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-28100-5_62
Trends and Advances in Learning-Based Emotion Recognition Across Heterogeneous Data Modalities
George, J., & Menaka, D. (2026) · doi:10.1109/iccpct70290.2026.11654824
“Its applications span across diverse domains, including healthcare monitoring, human-computer interaction, affective computing and assistive technologies [2].”
Rigs2Emotion: Automatic emotion classification from facial rigs
Bernier, M., & Arcelin, B. (2026) · doi:10.1145/3799818.3812096
Mishra, P., Parab, S., Saha, T., Gu, W., & Kato, S. (2026) · Information Systems Frontiers · doi:10.1007/s10796-026-10789-1 · Cited by 1
Tong, M., Liu, J.-L., Zeng, L., Gu, J., Yue, Y., Yu, Y., Qiu, Y.-F., & Yang, F. (2026) · Psychiatric Quarterly · doi:10.1007/s11126-026-10298-z
Bansiwal, B., Agnihotri, N., Gupta, V., Govil, Y., & Vijay, R. K. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2026.110984
Enhancing Emotion Recognition in Autistic Children through GAN-Based Synthetic Data Augmentation
Ramzan, H. A., Ahmad, W., & Adnan, S. M. (2026) · Iranian Journal of Science and Technology Transactions of Electrical Engineering · doi:10.1007/s40998-026-01162-8
A multilayer transformers fusion model with gating network for multimodal emotion recognition
Lv, J., Liang, Y., Chen, H., Gong, X., Li, J., Lin, K., Wang, L., Yuan, J., Zeng, X., & Chen, R. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-026-01209-3
“Most methods rely on single-modal analysis in the early stages of emotion recognition [2].”
Adapting the Self-Distillation Transformer for Vietnamese Multimodal Emotion Recognition
Nguyen, H. L., & Pham, T. S. (2026) · 2026 11th International Conference on Applying New Technology in Green Buildings (ATiGB) · doi:10.1109/atigb70203.2026.11628157
“The system is engineered to resolve the "Modality Gap" [8], [9] through three critical stages [16], [17].”
Voice Evaluation System Based on Sensor Data Acquisition and Acoustic Feature Extraction
Hu, C., Lü, J., & Sun, Y. (2026) · doi:10.1109/ccssta69471.2026.11635217
“Consequently, they possess insufficient diagnostic capability for pronunciation errors [4], [5].”
EVA: Emotion-Aware Conversational AI with Personalized Voice for Adolescents
Ramesh, P., & Paramanandam, K. (2026) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet65726.2026.11632547
“On the other hand, the combination of speech and text increases emotional intelligence due to the analysis of prosodic features of speech together with semantics in text, leading to better emotional recognition [5].”
Multimodal Emotion Recognition under Class Imbalance with Focus on Minority Emotions
Szabóová, M., Pylypiv, R., & Magyar, J. (2026) · International Conference on Intelligent Engineering Systems · doi:10.1109/ines69513.2026.11661270
“Prior studies report mixed findings regarding their relative effectiveness, suggesting that performance depends on dataset characteristics, modality reliability, and training conditions [10], [12].”
Sujatha, T., Prakash, J., V, Vijay, S. A. A., & Devadoss, M. P. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133306
Hong, C., Li, X., Huang, D., Du, Y., Lee, Y.-L., Liu, J., Chen, X., & Fan, Y. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-026-08609-z
“With the rapid development of deep neural networks, sentiment analysis has gradually evolved from single-modality modeling [6, 7] to multimodal collaborative analysis, where more accurate sentiment understanding is supported by the complementary information provided by different modalities.”
Bai, X., & Bai, M. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1759927
“Emotion Labeling Systems (ELS) automatically record and classify students’ emotional states, which greatly improves recognition and analysis of emotions (Lian et al., 2023).”
Lai, R. S. T., Theng, L. B., Tee, M. K. T., Tan, C. L., & Chua, C. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12186-w · Cited by 1
A Review of Multimodal Emotion Recognition: Deep Learning-Based Methods and Applications
Cofino, C. L., Fulmaran, J. P., Gutib, V. R. B., & Escorial, R. B. (2026) · International Conference on Innovations in Cybersecurity and Data Science (ICICDS) · doi:10.1109/icicds70526.2026.11604820
“Subsequently, some later studies hint at intermediate, attention-focused fusion methods in which the relative significance of one modality relative to the other is dynamically considered, leading to stronger, more adaptable emotion recognition systems [28].”
Wu, J. (2026) · doi:10.1109/isades69945.2026.11608097
“These restricted bi-modal methods are prone to background noise interference and fail to capture the multisensory feature redundancy and complementary mechanisms exhibited by the human brain [6].”
Affective Speech-to-Speech Chatbot for Context-Aware Human-Machine Interaction
Kumar, C. A., Prajwal, A., Satvika, A., Rithvik, B., & Thrishal, S. S. (2026) · International Conference on Computer Networks, Big Data and IoT (ICCBI) · doi:10.1109/iccbi68589.2026.11619693
“Lian et al. [5] surveyed a broad family of deep-learning approaches that combine audio, text, and visual streams for emotion detection.”
Metadata
- Authors
- Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y.
- Affiliations
- Southeast University (CN)
- Corresponding author
- Zong, Yuan
- Venue
- Entropy
- Publisher
- MDPI AG
- Published
- 2023-10-12
- Volume
- 25
- Issue
- 10
- Pages
- 1440
- ISSN
- 1099-4300 (ISSN-L)
- PMC
- PMC10606253
- OpenAlex
- W4387573449
- Cited by
- 224
- References
- 142
- Citations by year
- 2026: 124 · 2025: 71 · 2024: 29
- FWCI
- 9.44
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Entropy (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Key R & D Project (2022YFC2405600, 4007032320)
- Zhishan Young Scholarship of Southeast University (2022YFC2405600, 4007032320)
- Postdoctoral Scientific Research Foundation of Southeast University (2022YFC2405600, 4007032320)
- Jiangsu Province Excellent Postdoctoral Program (2022YFC2405600, 4007032320)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- multimodal emotion recognition, speech emotion recognition, text-based emotion recognition, facial emotion recognition, multimodal dataset, emotional feature extraction, feature-level fusion, decision fusion, hybrid fusion, human-computer interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.