The DOI matches its Crossref record; 134 of the 142 references it cites were found in a registry or an index; 73 of the 129 sources its claims cite were checked for support.
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Open accessAbstract
Multimodal emotion recognition (MER) refers to the identification and understanding of human emotional states by combining different signals, including-but not limited to-text, speech, and face cues. MER plays a crucial role in the human-computer interaction (HCI) domain. With the recent progression of deep learning technologies and the increasing availability of multimodal datasets, the MER domain has witnessed considerable development, resulting in numerous significant research breakthroughs. However, a conspicuous absence of thorough and focused reviews on these deep learning-based MER achievements is observed. This survey aims to bridge this gap by providing a comprehensive overview of the recent advancements in MER based on deep learning. For an orderly exposition, this paper first outlines a meticulous analysis of the current multimodal datasets, emphasizing their advantages and constraints. Subsequently, we thoroughly scrutinize diverse methods for multimodal emotional feature extraction, highlighting the merits and demerits of each method. Moreover, we perform an exhaustive analysis of various MER algorithms, with particular focus on the model-agnostic fusion methods (including early fusion, late fusion, and hybrid fusion) and fusion based on intermediate layers of deep models (encompassing simple concatenation fusion, utterance-level interaction fusion, and fine-grained interaction fusion). We assess the strengths and weaknesses of these fusion strategies, providing guidance to researchers to help them select the most suitable techniques for their studies. In summary, this survey aims to provide a thorough and insightful review of the field of deep learning-based MER. It is intended as a valuable guide to aid researchers in furthering the evolution of this dynamic and impactful field.
Overview
CoChat Summary
## TL;DR
This paper is a survey of deep learning-based multimodal emotion recognition (MER) using speech, text, and face cues. It reviews popular datasets, feature extraction methods, fusion strategies, and evaluation metrics, and concludes with open challenges for the field.
## Key findings
- The survey identifies 17 advanced multimodal emotion recognition techniques in its research-results analysis table, covering features, fusion strategies, datasets, and metrics.
- For speech features, it notes a shift from hand-crafted tools such as OpenSMILE, COVAREP, OpenEAR, and librosa toward deep/self-supervised methods such as wav2vec, wav2vec 2.0, HuBERT, and WavLM; it states that WavLM outperforms all preceding models on speaker recognition, speech recognition, and emotion recognition tasks.
- For text features, the paper describes a progression from bag-of-words and early embeddings like word2vec/GloVe to contextual models such as ELMo, BERT, SSWE, and Emo2Vec.
- In its fusion analysis, the paper says fine-grained interaction fusion remains a popular choice, and that IEMOCAP emerges as the predominant dataset choice, while MELD and YouTube are gaining traction.
## Methods
- This is a narrative survey of deep learning-based MER, organized around datasets, feature extraction, fusion methods, evaluation metrics, and research-result analysis.
- It synthesizes information from popular MER datasets such as IEMOCAP, YouTube, MOUD, ICT-MMMO, CMU-MOSI, CMU-MOSEI, OMG, MELD, SEWA, CH-SIMS, and CH-SIMS v2.0, reporting their sample sizes and annotation schemes where stated.
- The paper reviews and categorizes methods, especially model-agnostic fusion (early, late, hybrid) and intermediate layer fusion (simple concatenation, utterance-level interaction, fine-grained interaction), and summarizes common classification/regression metrics such as ACC, UACC, F1, UF1, MSE, RMSE, PCC, and CCC.
## Limitations
- The paper states that many MER datasets are limited by small scale, language restriction, class imbalance, cultural bias, or scripted/non-natural settings; these are limitations of the datasets reviewed, not a limitation of the survey itself.
- Evident from the text: the survey is descriptive rather than an empirical study, so it does not report new experiments or a unified benchmark comparison beyond summarizing prior work.
- Not stated in the paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
63 Valid · 4 Not valid · 40 Not checked · 83 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“but failed to mention early fusion and late fusion techniques.”
§ A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face > 1. Introduction
What the sources say
- Disputedcites [22] Koromilas, P., and Giannakopoulos, T. (2021). Deep multimodal emotion recognition on human speech: A review. Appl. Sci., 11. sourceExperimental — accuracy not yet measured
The Introduction (p. 1) explicitly names and explains both early and late fusion. Section 3.3 (p. 7) also describes an approach that combines early and late fusion, contrary to the claim that the survey failed to mention them. (LLM, conf=0.95)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“In each video clip, only the speaker’s face is visible, thereby eliminating potential distractions or influences from other facial expressions.”
§ A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face > 2. Popular Datasets in Multimodal Emotion Recognition > 2.11. CH-SIMS
What the sources say
- Supportedcites [34] Yu, W., Xu, H., Meng, F., Zhu, Y., Ma, Y., Wu, J., Zou, J., and Yang, K. (2020, January 5–10). Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online. source
Section 3.1 (Data Collection) states that, for each video clip, no faces appear except the speaker’s. That constraint supports the claim that other people’s facial expressions are absent as potential distractions. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The inclusion of emotional dramas, interviews, modern television shows, talk shows, video blogs, movies, period dramas, variety shows, and other scenarios has ensured that the dataset mirrors the wide-ranging and dynamic environments in which emotions are expressed in real life.”
§ A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face > 2. Popular Datasets in Multimodal Emotion Recognition > 2.12. CH-SIMS V2.0
What the sources say
- Supportedcites [35] Liu, Y., Yuan, Z., Mao, H., Liang, Z., Yang, W., Qiu, Y., Cheng, T., Li, X., Xu, H., and Gao, K. (2022, January 7–11). Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module. Proceedings of the 2022 International Conference on Multimodal Interaction, Bengaluru, India. source
In §3.1, “Diverse video scenarios,” the paper lists melodrama, interviews, modern TV, talk shows, vlogs, films, costume TV, variety shows, and other scenes as sources for CH-SIMS v2.0. It also says the collection varies filming conditions to simulate complex real-world scenarios. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Most recently, Chen et al. introduced WavLM [57], the first universal speech pre-training model capable of being applied to a broad range of tasks, including speaker recognition, speech recognition, and emotion recognition.”
§ A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face > 3. Feature Extraction > 3.1. Speech Feature Extraction
What the sources say
- Partly supportedcites [57] Chen, 2022, Wavlm: Large-scale self-supervised pre-training for full stack speech processing, IEEE J. Sel. Top. Signal Process. sourceExperimental — accuracy not yet measured
The Introduction (pp. 1–2) presents WavLM as a general pre-trained model for a broad range of speech tasks. The Universal Representation Evaluation (p. 5) includes speaker identification, automatic speech recognition, and emotion recognition. The paper does not establish that WavLM was the first universal speech pre-training model; its Introduction (p. 1) discusses earlier pre-trained models evaluated on full-stack speech tasks. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Empirical evidence underscored the superior performance and generalization of wav2vec 2.0 over its predecessor.”
§ A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face > 3. Feature Extraction > 3.1. Speech Feature Extraction
What the sources say
- Partly supportedcites [55] Baevski, 2020, wav2vec 2.0: A framework for self-supervised learning of speech representations, Adv. Neural Inf. Process. Syst. sourceExperimental — accuracy not yet measured
The Conclusion (p. 8) reports that wav2vec 2.0 outperformed the previous best result on a LibriSpeech setup using 100 times less labeled data, supporting superior performance over earlier methods. The cited passages do not establish superior generalization specifically over its wav2vec predecessor; the comparison in Model (p. 2) describes an architectural difference, not a generalization result. (LLM, conf=0.78)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
31 more claims · 4 with their cited sources checked
Create a free account to see all 190 claims and to verify the 2 references still open.
- Probably not a substantial claim. 123
- It may state two separate claims as one. 81
- The quote only partly states this claim. 1
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Earlier word embedding models, e.g., word2vec [60] and GloVe [61], predominantly relied on syntactic contexts for training.”
MethodDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 1×
“To mitigate the highlighted limitations, Peters et al. introduced the ELMo (Embeddings from Language Models) [62] method.”
MethodLearning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · cited 1×
“For instance, Tran et al. [72] proposed a simple and efficient three-dimensional convolutional neural network (3D-CNN) for extracting spatiotemporal features.”
BackgroundActive Shape Models-Their Training and Application
Cootes, T. F., Taylor, C. J., Cooper, D. H., & Graham, J. (1995) · Computer Vision and Image Understanding · doi:10.1006/cviu.1995.1004 · Cited by 7,190 · cited 1×
“Traditional facial feature extraction methods include Local Binary Pattern (LBP) [66], Active Appearance Model (AAM) [67], Active Shape Model (ASM) [68], Scale Invariant Feature Transform (SIFT) [69], Histograms of Oriented Gradient (HOG) [70], and Gabor Wavelet Transform [71].”
MethodJoint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · cited 1×
“The Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset, assembled by the Signal Analysis and Interpretation Lab (SAIL) at the University of Southern California, serves as a prominent multimodal resource for emotion recognition research [24].”
Methodlibrosa: Audio and Music Signal Analysis in Python
McFee, B., Raffel, C., Liang, D., Ellis, D., McVicar, M., Battenberg, E., & Nieto, O. (2015) · Proceedings of the Python in Science Conference · doi:10.25080/majora-7b98e3ed-003 · Cited by 3,033 · cited 1×
“In current MER, it is common practice to use OpenSMILE [45], COVAREP [46], Open-EAR [47], and librosa [48] to extract the aforementioned acoustic features that represent the emotional characteristics of the speech modality.”
MethodHuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3122291 · Cited by 2,983
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259 · cited 1×
“For example, Tsai et al. [114] proposed a multimodal Transformer (MULT) for unaligned multimodal language sequences.”
BackgroundTensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 1×
“One of the pioneers in this area is Zadeh et al. [99], who proposed the Tensor Fusion Network (TFN) for MER.”
BackgroundSurvey on speech emotion recognition: Features, classification schemes, and databases
El Ayadi, M., Kamel, M. S., & Karray, F. (2011) · Pattern Recognition · doi:10.1016/j.patcog.2010.09.020 · Cited by 2,073
Vocal communication of emotion: A review of research paradigms
Scherer, K. (2003) · Speech Communication · doi:10.1016/s0167-6393(02)00084-5 · Cited by 2,044 · cited 2×
“As a result, these features have become indispensable in the field of emotion recognition and are widely acknowledged by researchers [37,38].”
BackgroundWavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3188113 · Cited by 1,926 · cited 1×
“Most recently, Chen et al. introduced WavLM [57], the first universal speech pre-training model capable of being applied to a broad range of tasks, including speaker recognition, speech recognition, and emotion recognition.”
BackgroundOpenface 2.0: facial behavior analysis toolkit
Baltrusaitis, T., Zadeh, A., Lim, Y. C., & Morency, L.-P. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00019 · Cited by 1,861 · cited 1×
“At present, commonly utilized libraries consist of OKAO, Computer Expression Recognition Toolbox (CERT) [74], OpenFace [75], and FACET.”
MethodDeep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 1,753
Self-citationA review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433 · cited 2×
“Zadeh et al. [30] introduced a novel multimodal fusion technique known as Dynamic Fusion Graph (DFG) to investigate the interplay between modalities in human multimodal language.”
BackgroundMELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations
Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., & Mihalcea, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1050 · Cited by 1,416 · cited 1×
“In the domain of MER, the Multimodal EmotionLines Dataset (MELD) [32] offers a unique perspective by focusing on the emotional intricacies present in multi-participant dialogue.”
Backgroundwav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290
Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification
Tang, D., Wei, F., Yang, N., Zhou, M., Liu, T., & Qin, B. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1146 · Cited by 1,181
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors
Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1209 · Cited by 1,170
Murray, I. R., & Arnott, J. L. (1993) · The Journal of the Acoustical Society of America · doi:10.1121/1.405558 · Cited by 1,098 · cited 1×
“As a result, these features have become indispensable in the field of emotion recognition and are widely acknowledged by researchers [37,38].”
BackgroundSpeech emotion recognition using hidden Markov models
Nwe, T. L., Foo, S. W., & De Silva, L. C. (2003) · Speech Communication · doi:10.1016/s0167-6393(03)00099-2 · Cited by 920 · cited 1×
“Research indicates that the speech rate tends to be faster for fear, disgust, anger, and happiness; normal for surprise; and slower for sadness [44].”
BackgroundMemory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855
Yu, W., Xu, H., Yuan, Z., & Wu, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17289 · Cited by 816
COVAREP — A collaborative voice analysis repository for speech technologies
Degottex, G., Kane, J., Drugman, T., Raitio, T., & Scherer, S. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6853739 · Cited by 695 · cited 1×
“In current MER, it is common practice to use OpenSMILE [45], COVAREP [46], Open-EAR [47], and librosa [48] to extract the aforementioned acoustic features that represent the emotional characteristics of the speech modality.”
MethodA Review and Meta-Analysis of Multimodal Affect Detection Systems
D'mello, S. K., & Kory, J. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 628
Han, W., Chen, H., & Poria, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.723 · Cited by 570 · cited 2×
“Furthermore, Han et al. [96] pointed out that the aforementioned methods lack control in the information flow from raw input to fused embedding, which may lead to information loss and an increased risk of unintended noise.”
BackgroundMethodThe computer expression recognition toolbox (CERT)
Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., & Bartlett, M. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771414 · Cited by 539 · cited 1×
“At present, commonly utilized libraries consist of OKAO, Computer Expression Recognition Toolbox (CERT) [74], OpenFace [75], and FACET.”
MethodFusing audio, visual and textual clues for sentiment analysis from multimodal content
Poria, S., Cambria, E., Howard, N., Huang, G.-B., & Hussain, A. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.01.095 · Cited by 534 · cited 1×
“Poria et al. [85], for example, developed separate unimodal affect recognition models for speech, text, and visual modalities in 2016.”
BackgroundPham, H., Liang, P. P., Manzini, T., Morency, L.-P., & Póczos, B. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016892 · Cited by 512
Multi-attention Recurrent Network for Human Communication Comprehension
Zadeh, A., Liang, P. P., Poria, S., Vij, P., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12024 · Cited by 505 · cited 1×
“Zadeh et al. [110] proposed a novel network for understanding human communication based on word-level features called Multi-Attention Recurrent Network (MARN).”
BackgroundCH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality
Yu, W., Xu, H., Meng, F., Zhu, Y., Ma, Y., Wu, J., Zou, J., & Yang, K. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.343 · Cited by 482
Towards multimodal sentiment analysis: harvesting opinions from the web
Morency, L.-P., Mihalcea, R., & Doshi, P. (2011) · Proceedings of the 13th international conference on multimodal interfaces · doi:10.1145/2070481.2070509 · Cited by 470
Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
Wang, Y., Shen, Y., Liu, Z., Liang, P. P., Zadeh, A., & Morency, L.-P. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017216 · Cited by 466
Nicolaou, M. A., Gunes, H., & Pantic, M. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.9 · Cited by 450 · cited 3×
“For continuous emotion prediction tasks, a lower MSE signifies that the model’s predictions align more closely with the true emotion values [119].”
BackgroundMethodInfluentialOpenEAR — Introducing the munich open-source emotion and affect recognition toolkit
Eyben, F., Wollmer, M., & Schuller, B. (2009) · 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops · doi:10.1109/acii.2009.5349350 · Cited by 418 · cited 1×
“In current MER, it is common practice to use OpenSMILE [45], COVAREP [46], Open-EAR [47], and librosa [48] to extract the aforementioned acoustic features that represent the emotional characteristics of the speech modality.”
MethodCross-dataset learning and person-specific normalisation for automatic Action Unit detection
Baltrusaitis, T., Mahmoud, M., & Robinson, P. (2015) · 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2015.7284869 · Cited by 412
Sun, Z., Sarma, P., Sethares, W., & Liang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6431 · Cited by 410
Youtube movie reviews: Sentiment analysis in an audio-visual context
Wollmer, M., Weninger, F., Knaup, T., Schuller, B., Sun, C., Sagae, K., & Morency, L.-P. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.34 · Cited by 403
Automatic speech emotion recognition using modulation spectral features
Wu, S., Falk, T. H., & Chan, W.-Y. (2011) · Speech Communication · doi:10.1016/j.specom.2010.08.013 · Cited by 389 · cited 1×
“The array of prosodic features contributing to the depiction of emotions in speech encompasses, among others, fundamental frequency (F0), energy, and speech rate [39–42].”
Backgroundvq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
Baevski, A., Schneider, S., & Auli, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05453 · Cited by 388
The first facial expression recognition and analysis challenge
Valstar, M. F., Jiang, B., Mehu, M., Pantic, M., & Scherer, K. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771374 · Cited by 385
Multimodal sentiment analysis using hierarchical fusion with context modeling
Majumder, N., Hazarika, D., Gelbukh, A., Cambria, E., & Poria, S. (2018) · Knowledge-Based Systems · doi:10.1016/j.knosys.2018.07.041 · Cited by 383
Emotion recognition by speech signals
Kwon, O.-W., Chan, K., Hao, J., & Lee, T.-W. (2003) · 8th European Conference on Speech Communication and Technology (Eurospeech 2003) · doi:10.21437/eurospeech.2003-80 · Cited by 366
Databases, features and classifiers for speech emotion recognition: a review
Swain, M., Routray, A., & Kabisatpathy, P. (2018) · International Journal of Speech Technology · doi:10.1007/s10772-018-9491-z · Cited by 356
MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · arXiv · doi:10.48550/arxiv.1606.06259 · Cited by 350
Multimodal Emotion Recognition using Deep Learning
Abdullah, S. M. S. A., Ameen, S. Y. A., M. Sadeeq, M. A., & Zeebaree, S. (2021) · Journal of Applied Science and Technology Trends · doi:10.38094/jastt20291 · Cited by 347 · cited 1×
“Various experimental studies have attested to the superior performance of MER models over unimodal ones, placing MER at the vanguard of research within affective computing [77–80].”
BackgroundMultimodal sentiment analysis based on fusion methods: A survey
Zhu, L., Zhu, Z., Zhang, C., Xu, Y., & Kong, X. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.02.028 · Cited by 347 · cited 1×
“Finally, the FACET library extracts a multitude of visual features including facial action units, facial landmarks, head poses, gaze tracking, and HOG features [76].”
BackgroundMultimodal sentiment analysis with word-level fusion and reinforcement learning
Chen, M., Wang, S., Liang, P. P., Baltrušaitis, T., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3136801 · Cited by 334 · cited 1×
“To the best of our knowledge, Chen et al. [109] first based the idea of word-level modality fusion to propose a gated multimodal embedding LSTM with temporal attention (GME-LSTM(A)) model, which consists of two modules, i.e., gated multimodal embedding and LSTM with temporal attention.”
BackgroundAutomatic face identification system using flexible appearance models
Lanitis, A., Taylor, C., & Cootes, T. (1995) · Image and Vision Computing · doi:10.1016/0262-8856(95)99726-h · Cited by 328
Learning Factorized Multimodal Representations
Tsai, Y.-H. H., Liang, P. P., Zadeh, A., Morency, L.-P., & Salakhutdinov, R. (2018) · arXiv · doi:10.48550/arxiv.1806.06176 · Cited by 323
Ensemble methods for spoken emotion recognition in call-centres
Morrison, D., Wang, R., & De Silva, L. C. (2007) · Speech Communication · doi:10.1016/j.specom.2006.11.004 · Cited by 320
CTNet: Conversational Transformer Network for Emotion Recognition
Lian, Z., Liu, B., & Tao, J. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3049898 · Cited by 296 · cited 4×
“It is calculated by combining precision and recall according to the formula provided below [118]: where Precision i = TP i TP i + FP i is the precision of class i , and Recall i = TP i TP i + FN i is the recall of class i .”
BackgroundMethodInfluentialM3ER: Multiplicative Multimodal Emotion Recognition using Facial, Textual, and Speech Cues
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i02.5492 · Cited by 253
Facial Expression Recognition Using Facial Movement Features
Zhang, L., & Tjondronegoro, D. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.13 · Cited by 253 · cited 1×
“During the initial phases of emotion recognition research, researchers primarily concentrated on the recognition of single modalities, e.g., speech emotion recognition [1–5] text emotion recognition [6,7], and facial expression recognition [8–11].”
BackgroundFuzzy bag-of-words model for document representation
Zhao, R., & Mao, K. (2018) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2017.2690222 · Cited by 252 · cited 1×
“Traditional methods for extracting emotional features use the bag-of-words (BoW) model [58,59].”
MethodDecoupled Multimodal Distilling for Emotion Recognition
Li, Y., Wang, Y., & Cui, Z. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00641 · Cited by 249
Multimodal Sentiment Analysis: Addressing Key Issues and Setting Up the Baselines
Poria, S., Majumder, N., Hazarika, D., Cambria, E., Gelbukh, A., & Hussain, A. (2018) · IEEE Intelligent Systems · doi:10.1109/mis.2018.2882362 · Cited by 246
Sewa db: A rich database for audio-visual emotion and sentiment research in the wild
Kossaifi, J., Walecki, R., Panagakis, Y., Shen, J., Schmitt, M., Ringeval, F., Han, J., Pandit, V., Toisoul, A., Schuller, B., et al. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2944808 · Cited by 243
Emotion recognition from text using semantic labels and separable mixture models
Wu, C.-H., Chuang, Z.-J., & Lin, Y.-C. (2006) · ACM Transactions on Asian Language Information Processing · doi:10.1145/1165255.1165259 · Cited by 242 · cited 1×
“During the initial phases of emotion recognition research, researchers primarily concentrated on the recognition of single modalities, e.g., speech emotion recognition [1–5] text emotion recognition [6,7], and facial expression recognition [8–11].”
BackgroundMai, S., Hu, H., & Xing, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5347 · Cited by 237 · cited 1×
“Notably, Mai et al. [105] introduced a pioneering adversarial encoder–decoder classifier framework aimed at acquiring a modality-invariant embedding space.”
Background(2013) · Cited by 236
Towards an intelligent framework for multimodal affective data analysis
Poria, S., Cambria, E., Hussain, A., & Huang, G.-B. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.10.005 · Cited by 236 · cited 1×
“Poria et al. [82] were pioneers in the implementation of this fusion method.”
MethodMultimodal Language Analysis with Recurrent Multistage Fusion
Liang, P. P., Liu, Z., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1014 · Cited by 230 · cited 1×
“Liang et al. [113] presented Recurrent Multistage Fusion Network (RMFN), which decomposes the fusion problem into multiple stages.”
BackgroundLv, F., Chen, X., Huang, Y., Duan, L., & Lin, G. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00258 · Cited by 210 · cited 1×
“Additionally, to tackle the problem of cross-modal asynchrony in multimodal sequences, Lv et al. [116] introduced the Progressive Modality Reinforcement (PMR) method, which relies on cross-modal Transformers.”
MethodMulti-level Multiple Attentions for Contextual Multimodal Sentiment Analysis
Poria, S., Cambria, E., Hazarika, D., Mazumder, N., Zadeh, A., & Morency, L.-P. (2017) · 2017 IEEE International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2017.134 · Cited by 193
Selective Transfer Machine for Personalized Facial Expression Analysis
Chu, W.-S., De la Torre, F., & Cohn, J. F. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2547397 · Cited by 192 · cited 1×
“Traditional facial feature extraction methods include Local Binary Pattern (LBP) [66], Active Appearance Model (AAM) [67], Active Shape Model (ASM) [68], Scale Invariant Feature Transform (SIFT) [69], Histograms of Oriented Gradient (HOG) [70], and Gabor Wavelet Transform [71].”
MethodMai, S., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3171679 · Cited by 182 · cited 1×
“Subsequently, Mai et al. [98] aimed to minimize redundancy in the fusion process by introducing the Multimodal Information Bottleneck (MIB) for MER.”
Method(2015) · doi:10.1145/2808196.2811642 · Cited by 178
Shoumy, N. J., Ang, L.-M., Seng, K. P., Rahaman, D. M. M., & Zia, T. (2020) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2019.102447 · Cited by 171
Multi-modal fusion network with complementarity and importance for emotion recognition
Liu, S., Gao, P., Li, Y., Fu, W., & Ding, W. (2023) · Information Sciences · doi:10.1016/j.ins.2022.11.076 · Cited by 164 · cited 1×
“Recently, Liu et al. [107] devised a multimodal fusion network with complementarity and importance for emotion recognition.”
BackgroundSpatio-temporal convolutional features with nested LSTM for facial expression recognition
Yu, Z., Liu, G., Liu, Q., & Deng, J. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.07.028 · Cited by 162 · cited 1×
“Yu et al. [73] proposed a new architecture called spatio-temporal convolutional features with nested LSTM (STC-LSTM), which is based on three deep learning subnetworks: 3DCNN, T-LSTM, and C-LSTM.”
BackgroundMultimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion
Siriwardhana, S., Kaluarachchi, T., Billinghurst, M., & Nanayakkara, S. (2020) · IEEE Access · doi:10.1109/access.2020.3026823 · Cited by 158
TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis
Wang, Z., Wan, Z., & Wan, X. (2020) · Proceedings of The Web Conference 2020 · doi:10.1145/3366423.3380000 · Cited by 156
Mai, S., Hu, H., & Xing, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1046 · Cited by 153 · cited 1×
“These include Temporal Tensor Fusion Network (T2FN) [101], Hierarchical Feature Fusion Network (HFFN) [102], Modality-based Redundancy Reduction Fusion (MRRF) [103], and Semi-Tensor Product (STP) [104].”
MethodVideo sentiment analysis with bimodal information-augmented multi-head attention
Wu, T., Peng, J., Zhang, W., Zhang, H., Tan, S., Yi, F., Ma, C., & Huang, Y. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107676 · Cited by 145 · cited 1×
“Additionally, Wu et al. [106] developed a fusion network named Bimodal Information-augmented Multi-Head Attention (BIMHA), which relies on multi-head attention and comprises a total of four layers.”
BackgroundA survey of affect recognition methods: audio, visual and spontaneous expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2007) · Proceedings of the 9th international conference on Multimodal interfaces · doi:10.1145/1322192.1322216 · Cited by 144
Shou, Y., Meng, T., Ai, W., Yang, S., & Li, K. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.06.072 · Cited by 132
Recognizing Emotions in Video Using Multimodal DNN Feature Fusion
Williams, J., Kleinegesse, S., Comanescu, R., & Radu, O. (2018) · Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML) · doi:10.18653/v1/w18-3302 · Cited by 132 · cited 1×
“Further progress in early fusion was seen in the work of Williams et al. [84].”
MethodWu, Y., Lin, Z., Zhao, Y., Qin, B., & Zhu, L.-N. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.417 · Cited by 129 · cited 1×
“Although multimodal feature fusion methods based on utterance-level features can use global features for prediction, they ignore fine-grained interactions between modalities in the modeling process, which may fail to capture local information [108].”
BackgroundSelf-citationError weighted semi-coupled hidden Markov model for audio-visual emotion recognition
Lin, J.-C., Wu, C.-H., & Wei, W.-L. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2011.2171334 · Cited by 121 · cited 2×
“(1) Prosodic features : Emotions precipitate physiological changes that directly impact speech production and, thus, significantly affect speech intonation [36].”
BackgroundThe OMG-Emotion Behavior Dataset
Barros, P., Churamani, N., Lakomkin, E., Siqueira, H., Sutherland, A., & Wermter, S. (2018) · 2018 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489099 · Cited by 121 · cited 1×
“The One-Minute Gradual-Emotion Recognition (OMG) dataset [31], which incorporates videos from a multitude of YouTube channels, provides a comprehensive view of various emotional behaviors within different contexts.”
BackgroundSelf-supervised learning of a facial attribute embedding from video
Wiles, O., Koepke, A. S., & Zisserman, A. (2018) · arXiv · doi:10.48550/arxiv.1808.06882 · Cited by 111
Emotion Recognition from Text Based on Automatically Generated Rules
Shaheen, S. I., El-Hajj, W., Hajj, H. M., & Elbassuoni, S. (2014) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2014.80 · Cited by 110 · cited 1×
“During the initial phases of emotion recognition research, researchers primarily concentrated on the recognition of single modalities, e.g., speech emotion recognition [1–5] text emotion recognition [6,7], and facial expression recognition [8–11].”
BackgroundEnd-to-end multimodal affect recognition in real-world environments
Tzirakis, P., Chen, J., Zafeiriou, S., & Schuller, B. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.10.011 · Cited by 105 · cited 1×
“Tzirakis et al. [117] presented a textual architecture based on Transformers and an attention-based fusion strategy to effectively integrate diverse modal features and enhance sentiment recognition performance.”
BackgroundChang, H., Liu, B., Zong, Y., Lu, C., & Wang, X. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3292452 · Cited by 100 · cited 1×
“Essentially, a vast array of deep learning architectures optimize their models by either maximizing or minimizing certain entropy-associated loss functions [17–20].”
BackgroundSelf-citationA hybrid latent space data fusion method for multimodal emotion recognition
Nemati, S., Rohani, R., Basiri, M. E., Abdar, M., Yen, N. Y., & Makarenkov, V. (2019) · IEEE Access · doi:10.1109/access.2019.2955637 · Cited by 98 · cited 1×
“Within the field of Music Emotion Recognition (MER), Nemati et al. [90] introduced a comparable hybrid fusion methodology.”
MethodEmotion Recognition from Skeletal Movements
Sapiński, T., Kamińska, D., Pelikant, A., & Anbarjafari, G. (2019) · Entropy · doi:10.3390/e21070646 · Cited by 98
Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization
Liang, P. P., Liu, Z., Tsai, Y.-H. H., Zhao, Q., Salakhutdinov, R., & Morency, L.-P. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1152 · Cited by 95 · cited 1×
“These include Temporal Tensor Fusion Network (T2FN) [101], Hierarchical Feature Fusion Network (HFFN) [102], Modality-based Redundancy Reduction Fusion (MRRF) [103], and Semi-Tensor Product (STP) [104].”
MethodText classification by augmenting bag of words (BOW) representation with co-occurrence feature
Department of Computer Science and Engineering, G. E. C., George K, S., & Joseph, S. (2014) · IOSR Journal of Computer Engineering · doi:10.9790/0661-16153438 · Cited by 94
Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities
Zeng, J., Liu, T., & Zhou, J. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3532064 · Cited by 94 · cited 1×
“…the attention of many researchers, aiming to combine information from various modalities, which can corroborate and complement each other, thus providing more comprehensive and accurate information for emotional judgment and significantly improving the performance of emotional judgments [12–15].”
BackgroundLiang, T., Lin, G., Feng, L., Zhang, Y., & Lv, F. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00804 · Cited by 93 · cited 1×
“Based on this observation, Liang et al. [115] introduced the Modality-Invariant Cross-Modal Attention (MICA) method, which learns cross-modal interactions in a modality-invariant space and effectively solves the sequence matching problem in unaligned features.”
MethodFacial Expression Recognition Using Local Transitional Pattern on Gabor Filtered Facial Images
Ahsan, T., Jabid, T., & Chong, U. (2013) · IETE Technical Review · doi:10.4103/0256-4602.107339 · Cited by 93
Learning person-specific models for facial expression and action unit recognition
Chen, J.-X., Liu, X., Tu, P., & Aragones, A. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2013.02.002 · Cited by 88
Seq2Seq2Sentiment: Multimodal Sequence to Sequence Models for Sentiment Analysis
Pham, H., Manzini, T., Liang, P. P., & Póczos, B. (2018) · doi:10.18653/v1/w18-3308 · Cited by 87
A multimodal convolutional neuro-fuzzy network for emotion understanding of movie clips
Nguyen, T.-L., Kavuri, S., & Lee, M. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.06.010 · Cited by 84 · cited 1×
“Nguyen et al. [91] proposed a novel multimodal sentiment recognition model that utilizes three convolutional neural networks to separately process the low-level features of speech, text, and visual modalities.”
BackgroundMultimodal Fusion with Deep Neural Networks for Audio-Video Emotion RecognitionPreprint
Ortega, J. D. S., Senoussaoui, M. E. A., Granger, É., Pedersoli, M., Cardinal, P., & Koerich, A. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.03196 · Cited by 84
Emo2Vec: Learning Generalized Emotion Representation by Multi-task Training
Xu, P., Madotto, A., Wu, C.-S., Park, J. H., & Fung, P. (2018) · Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis · doi:10.18653/v1/w18-6243 · Cited by 83 · cited 1×
“Following this, Xu et al. [65] unveiled Emo2Vec, embedding emotional semantics into fixed-size vector representations at the word level.”
MethodSun, L., Lian, Z., Tao, J., Liu, B., & Niu, M. (2020) · Proceedings of the 1st International on Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop · doi:10.1145/3423327.3423672 · Cited by 81 · cited 1×
“Recently, Sun et al. [88,89] employed LSTM and a self-attention mechanism to capture complex temporal dependencies in feature sequences.”
Semi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching
Liang, J., Li, R., & Jin, Q. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413579 · Cited by 73 · cited 1×
“Liang et al. [94] postulate that despite the potential misalignment of emotional expressions at the frame level across different modalities, the overall emotional states should exhibit similarity at a coarse-grained discourse level.”
BackgroundMake Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module
Liu, Y., Yuan, Z., Mao, H., Liang, Z., Yang, W., Qiu, Y., Cheng, T., Li, X., Xu, H., & Gao, K. (2022) · Proceedings of the 2022 International Conference on Multimodal Interaction · doi:10.1145/3536221.3556630 · Cited by 64 · cited 1×
“In this study, Liu et al. [35] constructed the CH-SIMS v2.”
MethodDeep Multimodal Emotion Recognition on Human Speech: A Review
Koromilas, P., & Giannakopoulos, T. (2021) · Applied Sciences · doi:10.3390/app11177962 · Cited by 58 · cited 1×
“Koromilas et al. [22] exhaustively introduced an intermediate fusion method based on a deep multimodal learning architecture, but failed to mention early fusion and late fusion techniques.”
MethodNNIME: The NTHU-NTUA Chinese interactive multimodal emotion corpus
Chou, H.-C., Lin, W.-C., Chang, L.-C., Li, C.-C., Ma, H.-P., & Lee, C.-C. (2017) · 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII) · doi:10.1109/acii.2017.8273615 · Cited by 57 · cited 1×
“The NTHU-NTUA Chinese Interactive Emotion Corpus (NNIME) [29] is a ground-breaking initiative aimed at creating an interactive multimodal corpus for Chinese emotional expressions.”
BackgroundContinuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network
Huang, J., Li, Y., Tao, J., Lian, Z., Wen, Z., Yang, M., & Yi, J. (2017) · Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge · doi:10.1145/3133944.3133946 · Cited by 54 · cited 1×
“Furthermore, Huang et al. [86] adopted Long Short-Term Memory (LSTM) networks to create affect prediction models that fully leverage affective information from each modality.”
BackgroundSpeech Emotion Recognition via an Attentive Time–Frequency Neural Network
Lu, C., Zheng, W., Lian, H., Zong, Y., Tang, C., Li, S., & Zhao, Y. (2023) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3219825 · Cited by 51
Self-citationWords Can Shift: Dynamically Adjusting Word Representations Using Nonverbal BehaviorsPreprint
Wang, Y., Shen, Y., Liu, Z., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.09362 · Cited by 44
Modality-based Factorization for Multimodal Fusion
Barezi, E. J., & Fung, P. (2019) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/w19-4331 · Cited by 43 · cited 1×
“These include Temporal Tensor Fusion Network (T2FN) [101], Hierarchical Feature Fusion Network (HFFN) [102], Modality-based Redundancy Reduction Fusion (MRRF) [103], and Semi-Tensor Product (STP) [104].”
MethodMultimodal Continuous Emotion Recognition with Data Augmentation Using Recurrent Neural Networks
Huang, J., Li, Y., Tao, J., Lian, Z., Niu, M., & Yang, M. (2018) · Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop · doi:10.1145/3266302.3266304 · Cited by 40 · cited 1×
“In a following study, Huang et al. [83] enhanced data richness to evaluate the efficacy of early fusion.”
BackgroundPham, H., Liang, P. P., Manzini, T., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.07809 · Cited by 38
Chang, H., Zong, Y., Zheng, W., Xiao, Y., Wang, X., Zhu, J. P., Shi, M., Cheng, L., & Yang, H. J. (2023) · Journal of Neural Engineering · doi:10.1088/1741-2552/acbe20 · Cited by 38
Self-citationMultimodal Emotion Recognition and Sentiment Analysis via Attention Enhanced Recurrent Model
Sun, L., Xu, M., Lian, Z., Liu, B., Tao, J., Wang, M., & Cheng, Y. (2021) · Proceedings of the 2nd on Multimodal Sentiment Analysis Challenge · doi:10.1145/3475957.3484456 · Cited by 36 · cited 1×
“Recently, Sun et al. [88,89] employed LSTM and a self-attention mechanism to capture complex temporal dependencies in feature sequences.”
Multimodal Emotion Recognition in Deep Learning:a Survey
Gu, X., Shen, Y., & Xu, J. (2021) · 2021 International Conference on Culture-oriented Science & Technology (ICCST) · doi:10.1109/iccst53801.2021.00027 · Cited by 36 · cited 1×
“Gu et al. [21] provided a comprehensive analysis of the framework and research methods applicable to video and music emotion recognition, with particular emphasis on deep learning techniques.”
BackgroundLiu, F., Shen, S.-Y., Fu, Z.-W., Wang, H.-Y., Zhou, A.-M., & Qi, J.-Y. (2022) · Entropy · doi:10.3390/e24071010 · Cited by 21 · cited 1×
“With the rapid development of deep learning techniques, MER based on deep learning has emerged as one of the most prominent research topics in the field of emotion recognition [16].”
BackgroundCross-Corpus Speech Emotion Recognition Based on Multi-Task Learning and Subdomain Adaptation
Fu, H., Zhuang, Z., Wang, Y., Huang, C., & Duan, W. (2023) · Entropy · doi:10.3390/e25010124 · Cited by 20
Learning Local to Global Feature Aggregation for Speech Emotion Recognition
Lu, C., Lian, H., Zheng, W., Zong, Y., Zhao, Y., & Li, S. (2023) · INTERSPEECH 2023 · doi:10.21437/interspeech.2023-543 · Cited by 19
Self-citationDeep Implicit Distribution Alignment Networks for cross-Corpus Speech Emotion Recognition
Zhao, Y., Wang, J., Zong, Y., Zheng, W., Lian, H., & Zhao, L. (2023) · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49357.2023.10095388 · Cited by 19
Self-citationFen, L., Chen, J., Li, K., Tan, W., Cai, C., & Ayub, M. S. (2022) · Entropy · doi:10.3390/e24121836 · Cited by 18 · cited 1×
“These include Temporal Tensor Fusion Network (T2FN) [101], Hierarchical Feature Fusion Network (HFFN) [102], Modality-based Redundancy Reduction Fusion (MRRF) [103], and Semi-Tensor Product (STP) [104].”
MethodDeep Transductive Transfer Regression Network for Cross-Corpus Speech Emotion Recognition
Zhao, Y., Wang, J., Ye, R., Zong, Y., Zheng, W., & Zhao, L. (2022) · Interspeech 2022 · doi:10.21437/interspeech.2022-679 · Cited by 16 · cited 1×
“This is a research hotspot in the field of speech emotion [49–53].”
BackgroundSelf-citationMultimodal Attention Dynamic Fusion Network for Facial Micro-Expression Recognition
Yang, H., Xie, L., Pan, H., Li, C., Wang, Z., & Zhong, J. (2023) · Entropy · doi:10.3390/e25091246 · Cited by 14 · cited 1×
“During the initial phases of emotion recognition research, researchers primarily concentrated on the recognition of single modalities, e.g., speech emotion recognition [1–5] text emotion recognition [6,7], and facial expression recognition [8–11].”
BackgroundA Multi-Modal Fusion Method Based on Higher-Order Orthogonal Iteration Decomposition
Liu, F., Chen, J., Tan, W., & Cai, C. (2021) · Entropy · doi:10.3390/e23101349 · Cited by 13
Progressively Discriminative Transfer Network for Cross-Corpus Speech Emotion Recognition
Lu, C., Tang, C., Zhang, J., & Zong, Y. (2022) · Entropy · doi:10.3390/e24081046 · Cited by 12 · cited 1×
“During the initial phases of emotion recognition research, researchers primarily concentrated on the recognition of single modalities, e.g., speech emotion recognition [1–5] text emotion recognition [6,7], and facial expression recognition [8–11].”
BackgroundSelf-citationAn Improved Multimodal Dimension Emotion Recognition Based on Different Fusion Methods
Su, H., Liu, B., Tao, J., Dong, Y., Huang, J., Lian, Z., & Song, L. (2020) · 2020 15th IEEE International Conference on Signal Processing (ICSP) · doi:10.1109/icsp48669.2020.9321008 · Cited by 10 · cited 1×
“In an attempt to augment performance, Su et al. [87] proposed a multi-level segmented decision-level fusion model for affect recognition.”
BackgroundMultidimensional Feature in Emotion Recognition Based on Multi-Channel EEG Signals
Li, Q., Liu, Y., Liu, Q., Zhang, Q., Yan, F., Ma, Y., & Zhang, X. (2022) · Entropy · doi:10.3390/e24121830 · Cited by 10 · cited 1×
“Essentially, a vast array of deep learning architectures optimize their models by either maximizing or minimizing certain entropy-associated loss functions [17–20].”
BackgroundAdapting Multiple Distributions for Bridging Emotions from Different Speech Corpora
Zong, Y., Lian, H., Chang, H., Lu, C., & Tang, C. (2022) · Entropy · doi:10.3390/e24091250 · Cited by 10
Self-citationDeep Emotion Recognition using Facial, Speech and Textual Cues: A SurveyPreprint
Zhang, T., & Tan, Z. (2021) · doi:10.36227/techrxiv.15184302.v2 · Cited by 9 · cited 1×
“Zhang et al. [23] thoroughly depicted single-modal emotion recognition technologies, such as facial expression recognition, speech emotion recognition, and text emotion recognition.”
BackgroundZheng, J., Zhang, S., Wang, X., & Zeng, Z. (2022) · arXiv · doi:10.48550/arxiv.2201.03969 · Cited by 6
Analysis and detection of human emotion and stress from speech signals
Nwe, T. L. (2004) · National University of Singapore · Cited by 2
Research works printed but not linked: 1
- Tripathi, S., Tripathi, S., and Beigi, H. (2018). Multi-Modal Emotion Recognition on Iemocap with Neural Networks. arXiv.
Cited by
Wu, Y., Mi, Q., & Gao, T. (2025) · Biomimetics · doi:10.3390/biomimetics10070418 · Cited by 182
“Multimodal information fusion is crucial for integrating data from various sources in order to capture the complex nature of emotions [13].”
Al-Saadawi, H. F. T., Daş, B., & Daş, R. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124852 · Cited by 103
Multimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70
TF-BERT: Tensor-based fusion BERT for multimodal sentiment analysis
Hou, J., Omar, N., Tiun, S. B., Saad, S., & He, Q. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107222 · Cited by 51
SSLMM: Semi-Supervised Learning with Missing Modalities for Multimodal Sentiment Analysis
Wang, Y., Jian, H., Zhuang, J., Guo, H., & Leng, Y. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103058 · Cited by 40
Metadata
- Authors
- Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y.
- Affiliations
- Southeast University (CN)
- Corresponding author
- Zong, Yuan
- Venue
- Entropy
- Publisher
- MDPI AG
- Published
- 2023-10-12
- Volume
- 25
- Issue
- 10
- Pages
- 1440
- ISSN
- 1099-4300 (ISSN-L)
- PMC
- PMC10606253
- OpenAlex
- W4387573449
- Cited by
- 224
- References
- 142
- Citations by year
- 2026: 124 · 2025: 71 · 2024: 29
- FWCI
- 9.44
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Entropy (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Key R & D Project (2022YFC2405600, 4007032320)
- Zhishan Young Scholarship of Southeast University (2022YFC2405600, 4007032320)
- Postdoctoral Scientific Research Foundation of Southeast University (2022YFC2405600, 4007032320)
- Jiangsu Province Excellent Postdoctoral Program (2022YFC2405600, 4007032320)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- multimodal emotion recognition, speech emotion recognition, text-based emotion recognition, facial emotion recognition, multimodal dataset, emotional feature extraction, feature-level fusion, decision fusion, hybrid fusion, human-computer interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.