The DOI matches its Crossref record; 134 of the 142 references it cites were found in a registry or an index; 73 of the 129 sources its claims cite were checked for support.
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Open accessAbstract
Multimodal emotion recognition (MER) refers to the identification and understanding of human emotional states by combining different signals, including-but not limited to-text, speech, and face cues. MER plays a crucial role in the human-computer interaction (HCI) domain. With the recent progression of deep learning technologies and the increasing availability of multimodal datasets, the MER domain has witnessed considerable development, resulting in numerous significant research breakthroughs. However, a conspicuous absence of thorough and focused reviews on these deep learning-based MER achievements is observed. This survey aims to bridge this gap by providing a comprehensive overview of the recent advancements in MER based on deep learning. For an orderly exposition, this paper first outlines a meticulous analysis of the current multimodal datasets, emphasizing their advantages and constraints. Subsequently, we thoroughly scrutinize diverse methods for multimodal emotional feature extraction, highlighting the merits and demerits of each method. Moreover, we perform an exhaustive analysis of various MER algorithms, with particular focus on the model-agnostic fusion methods (including early fusion, late fusion, and hybrid fusion) and fusion based on intermediate layers of deep models (encompassing simple concatenation fusion, utterance-level interaction fusion, and fine-grained interaction fusion). We assess the strengths and weaknesses of these fusion strategies, providing guidance to researchers to help them select the most suitable techniques for their studies. In summary, this survey aims to provide a thorough and insightful review of the field of deep learning-based MER. It is intended as a valuable guide to aid researchers in furthering the evolution of this dynamic and impactful field.
Overview
CoChat Summary
## TL;DR
This paper is a survey of deep learning-based multimodal emotion recognition (MER) using speech, text, and face cues. It reviews popular datasets, feature extraction methods, fusion strategies, and evaluation metrics, and concludes with open challenges for the field.
## Key findings
- The survey identifies 17 advanced multimodal emotion recognition techniques in its research-results analysis table, covering features, fusion strategies, datasets, and metrics.
- For speech features, it notes a shift from hand-crafted tools such as OpenSMILE, COVAREP, OpenEAR, and librosa toward deep/self-supervised methods such as wav2vec, wav2vec 2.0, HuBERT, and WavLM; it states that WavLM outperforms all preceding models on speaker recognition, speech recognition, and emotion recognition tasks.
- For text features, the paper describes a progression from bag-of-words and early embeddings like word2vec/GloVe to contextual models such as ELMo, BERT, SSWE, and Emo2Vec.
- In its fusion analysis, the paper says fine-grained interaction fusion remains a popular choice, and that IEMOCAP emerges as the predominant dataset choice, while MELD and YouTube are gaining traction.
## Methods
- This is a narrative survey of deep learning-based MER, organized around datasets, feature extraction, fusion methods, evaluation metrics, and research-result analysis.
- It synthesizes information from popular MER datasets such as IEMOCAP, YouTube, MOUD, ICT-MMMO, CMU-MOSI, CMU-MOSEI, OMG, MELD, SEWA, CH-SIMS, and CH-SIMS v2.0, reporting their sample sizes and annotation schemes where stated.
- The paper reviews and categorizes methods, especially model-agnostic fusion (early, late, hybrid) and intermediate layer fusion (simple concatenation, utterance-level interaction, fine-grained interaction), and summarizes common classification/regression metrics such as ACC, UACC, F1, UF1, MSE, RMSE, PCC, and CCC.
## Limitations
- The paper states that many MER datasets are limited by small scale, language restriction, class imbalance, cultural bias, or scripted/non-natural settings; these are limitations of the datasets reviewed, not a limitation of the survey itself.
- Evident from the text: the survey is descriptive rather than an empirical study, so it does not report new experiments or a unified benchmark comparison beyond summarizing prior work.
- Not stated in the paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
63 Valid · 4 Not valid · 40 Not checked · 83 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“but failed to mention early fusion and late fusion techniques.”
§ A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face > 1. Introduction
What the sources say
- Disputedcites [22] Koromilas, P., and Giannakopoulos, T. (2021). Deep multimodal emotion recognition on human speech: A review. Appl. Sci., 11. sourceExperimental — accuracy not yet measured
The Introduction (p. 1) explicitly names and explains both early and late fusion. Section 3.3 (p. 7) also describes an approach that combines early and late fusion, contrary to the claim that the survey failed to mention them. (LLM, conf=0.95)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“In each video clip, only the speaker’s face is visible, thereby eliminating potential distractions or influences from other facial expressions.”
§ A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face > 2. Popular Datasets in Multimodal Emotion Recognition > 2.11. CH-SIMS
What the sources say
- Supportedcites [34] Yu, W., Xu, H., Meng, F., Zhu, Y., Ma, Y., Wu, J., Zou, J., and Yang, K. (2020, January 5–10). Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online. source
Section 3.1 (Data Collection) states that, for each video clip, no faces appear except the speaker’s. That constraint supports the claim that other people’s facial expressions are absent as potential distractions. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The inclusion of emotional dramas, interviews, modern television shows, talk shows, video blogs, movies, period dramas, variety shows, and other scenarios has ensured that the dataset mirrors the wide-ranging and dynamic environments in which emotions are expressed in real life.”
§ A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face > 2. Popular Datasets in Multimodal Emotion Recognition > 2.12. CH-SIMS V2.0
What the sources say
- Supportedcites [35] Liu, Y., Yuan, Z., Mao, H., Liang, Z., Yang, W., Qiu, Y., Cheng, T., Li, X., Xu, H., and Gao, K. (2022, January 7–11). Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module. Proceedings of the 2022 International Conference on Multimodal Interaction, Bengaluru, India. source
In §3.1, “Diverse video scenarios,” the paper lists melodrama, interviews, modern TV, talk shows, vlogs, films, costume TV, variety shows, and other scenes as sources for CH-SIMS v2.0. It also says the collection varies filming conditions to simulate complex real-world scenarios. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Most recently, Chen et al. introduced WavLM [57], the first universal speech pre-training model capable of being applied to a broad range of tasks, including speaker recognition, speech recognition, and emotion recognition.”
§ A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face > 3. Feature Extraction > 3.1. Speech Feature Extraction
What the sources say
- Partly supportedcites [57] Chen, 2022, Wavlm: Large-scale self-supervised pre-training for full stack speech processing, IEEE J. Sel. Top. Signal Process. sourceExperimental — accuracy not yet measured
The Introduction (pp. 1–2) presents WavLM as a general pre-trained model for a broad range of speech tasks. The Universal Representation Evaluation (p. 5) includes speaker identification, automatic speech recognition, and emotion recognition. The paper does not establish that WavLM was the first universal speech pre-training model; its Introduction (p. 1) discusses earlier pre-trained models evaluated on full-stack speech tasks. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Empirical evidence underscored the superior performance and generalization of wav2vec 2.0 over its predecessor.”
§ A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face > 3. Feature Extraction > 3.1. Speech Feature Extraction
What the sources say
- Partly supportedcites [55] Baevski, 2020, wav2vec 2.0: A framework for self-supervised learning of speech representations, Adv. Neural Inf. Process. Syst. sourceExperimental — accuracy not yet measured
The Conclusion (p. 8) reports that wav2vec 2.0 outperformed the previous best result on a LibriSpeech setup using 100 times less labeled data, supporting superior performance over earlier methods. The cited passages do not establish superior generalization specifically over its wav2vec predecessor; the comparison in Model (p. 2) describes an architectural difference, not a generalization result. (LLM, conf=0.78)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
31 more claims · 4 with their cited sources checked
Create a free account to see all 190 claims and to verify the 2 references still open.
- Probably not a substantial claim. 123
- It may state two separate claims as one. 81
- The quote only partly states this claim. 1
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Earlier word embedding models, e.g., word2vec [60] and GloVe [61], predominantly relied on syntactic contexts for training.”
MethodDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 1×
“To mitigate the highlighted limitations, Peters et al. introduced the ELMo (Embeddings from Language Models) [62] method.”
MethodLearning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · cited 1×
“For instance, Tran et al. [72] proposed a simple and efficient three-dimensional convolutional neural network (3D-CNN) for extracting spatiotemporal features.”
BackgroundActive Shape Models-Their Training and Application
Cootes, T. F., Taylor, C. J., Cooper, D. H., & Graham, J. (1995) · Computer Vision and Image Understanding · doi:10.1006/cviu.1995.1004 · Cited by 7,190 · cited 1×
“Traditional facial feature extraction methods include Local Binary Pattern (LBP) [66], Active Appearance Model (AAM) [67], Active Shape Model (ASM) [68], Scale Invariant Feature Transform (SIFT) [69], Histograms of Oriented Gradient (HOG) [70], and Gabor Wavelet Transform [71].”
MethodJoint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · cited 1×
“The Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset, assembled by the Signal Analysis and Interpretation Lab (SAIL) at the University of Southern California, serves as a prominent multimodal resource for emotion recognition research [24].”
Method
Research works printed but not linked: 1
- Tripathi, S., Tripathi, S., and Beigi, H. (2018). Multi-Modal Emotion Recognition on Iemocap with Neural Networks. arXiv.
Cited by
Wu, Y., Mi, Q., & Gao, T. (2025) · Biomimetics · doi:10.3390/biomimetics10070418 · Cited by 154
“Multimodal information fusion is crucial for integrating data from various sources in order to capture the complex nature of emotions [13].”
Al-Saadawi, H. F. T., Daş, B., & Daş, R. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124852 · Cited by 103
Multimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70
TF-BERT: Tensor-based fusion BERT for multimodal sentiment analysis
Hou, J., Omar, N., Tiun, S. B., Saad, S., & He, Q. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107222 · Cited by 43
SSLMM: Semi-Supervised Learning with Missing Modalities for Multimodal Sentiment Analysis
Wang, Y., Jian, H., Zhuang, J., Guo, H., & Leng, Y. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103058 · Cited by 40
Metadata
- Authors
- Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y.
- Affiliations
- Southeast University (CN)
- Corresponding author
- Zong, Yuan
- Venue
- Entropy
- Publisher
- MDPI AG
- Published
- 2023-10-12
- Volume
- 25
- Issue
- 10
- Pages
- 1440
- ISSN
- 1099-4300 (ISSN-L)
- PMC
- PMC10606253
- OpenAlex
- W4387573449
- Cited by
- 224
- References
- 142
- Citations by year
- 2026: 124 · 2025: 71 · 2024: 29
- FWCI
- 9.44
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Entropy (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Key R & D Project (2022YFC2405600, 4007032320)
- Zhishan Young Scholarship of Southeast University (2022YFC2405600, 4007032320)
- Postdoctoral Scientific Research Foundation of Southeast University (2022YFC2405600, 4007032320)
- Jiangsu Province Excellent Postdoctoral Program (2022YFC2405600, 4007032320)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- multimodal emotion recognition, speech emotion recognition, text-based emotion recognition, facial emotion recognition, multimodal dataset, emotional feature extraction, feature-level fusion, decision fusion, hybrid fusion, human-computer interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face. https://app.cochat.ai/papers/10.3390/e25101440
- The paper
- Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023). A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face. Entropy. https://doi.org/10.3390/e25101440
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.