Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 3 of the 10 sources its claims cite were checked for support.
Abstract
Goal: As an essential human-machine interactive task, emotion recognition has become an emerging area over the decades. Although previous attempts to classify emotions have achieved high performance, several challenges remain open: 1) How to effectively recognize emotions using different modalities remains challenging. 2) Due to the increasing amount of computing power required for deep learning, how to provide real-time detection and improve the robustness of deep neural networks is important. Method: In this paper, we propose a deep learning-based multimodal emotion recognition (MER) called Deep-Emotion, which can adaptively integrate the most discriminating features from facial expressions, speech, and electroencephalogram (EEG) to improve the performance of the MER. Specifically, the proposed Deep-Emotion framework consists of three branches, i.e., the facial branch, speech branch, and EEG branch. Correspondingly, the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network. For work on the speech branch, this paper proposes a lightweight fully convolutional neural network (LFCNN) for the efficient extraction of speech emotion features. Regarding the study of EEG branches, we proposed a tree-like LSTM (tLSTM) model capable of fusing multi-stage features for EEG emotion feature extraction. Finally, we adopted the strategy of decision-level fusion to integrate the recognition results of the above three modes, resulting in more comprehensive and accurate performance. Result and Conclusions: Extensive experiments on the CK+, EMO-DB, and MAHNOB-HCI datasets have demonstrated the advanced nature of the Deep-Emotion method proposed in this paper, as well as the feasibility and superiority of the MER approach.
Overview
CoChat Summary
TL;DR
The paper proposes Deep-Emotion, a multimodal emotion recognition framework that combines facial expressions, speech, and EEG through decision-level fusion. It reports strong results on CK+, EMO-DB, and MAHNOB-HCI, while noting that fusion did not significantly outperform facial-expression or EEG modes on MAHNOB-HCI.
Key findings
- The improved GhostNet achieved 98.27% average accuracy on CK+ under tenfold cross-validation, outperforming traditional GhostNet at 90.21%; fear expression accuracy was only 95%, possibly because fear and contempt have similar features.
- The LFCNN for speech achieved 94.36% average accuracy and 94.38% F1-score on EMO-DB, with a model size of only 2.28 MB.
- On MAHNOB-HCI, the tLSTM obtained advanced results in EEG emotion recognition, and the fusion method improved both arousal and valence dimensions; fusion was significantly higher than the speech mode (\(p<0.05\), paired \(t\) test) but not significantly better than facial expressions or EEG (\(p>0.05\)).
- The paper claims this is the first attempt to combine facial expressions, speech, and EEG for multimodal emotion recognition.
Methods
- Deep-Emotion has three branches: an improved GhostNet for facial expression recognition, a lightweight fully convolutional neural network (LFCNN) for speech emotion recognition, and a tree-like LSTM (tLSTM) for EEG emotion recognition.
- Decision-level fusion uses an optimal weight distribution search algorithm: weights are enumerated from 0.00 to 1.00 with step 0.01, and RMSE is minimized to select the best weight distribution.
- Experiments use CK+, EMO-DB, and MAHNOB-HCI; CK+ used tenfold cross-validation. Data preprocessing, full LFCNN structure, experiment setup, and algorithm pseudo-code are stated to be in the Supplementary Materials.
Limitations
- The paper states that on MAHNOB-HCI the fusion method was only significantly higher than the speech mode, not facial expressions or EEG; it suggests this may be because subjects behaved normally rather than mimicking facial expressions.
- The paper states that SER on MAHNOB-HCI is challenging because the speech signal includes both the subject’s voice and the stimulus material’s voice, possibly explaining the relatively average SER recognition rate.
- The paper states that the work is based on open-source datasets and has not conducted independent data collection; it also notes future work should explore dynamic weight allocation. The text makes evident that many methodological details are relegated to Supplementary Materials, limiting full standalone reproducibility from the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
2 Valid · 6 Not valid · 4 Not checked · 34 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“EEG signals extracted from the central nervous system can more accurately and objectively reflect changes in people's emotions than other signals [2], [11].”
§ Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG > Introduction
What the sources say
- Not checked — no full text of the sourcecites [2] source
- Partly supportedcites [11] sourceExperimental — accuracy not yet measured
Section 2.2 says EEG records central nervous system activity and is an effective measure for studying emotion. The introduction (p. 1:2) notes that facial and speech cues can be affected when people conceal their feelings, but it does not establish that EEG reflects emotional changes more accurately and objectively than other signals generally. (LLM, conf=0.84)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“emotions are considered a complex representation that cannot be reliably captured with unimodal signals, since genuine emotion can be hidden by different facial expressions or tones [11].”
§ Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG > Introduction
What the sources say
- Partly supportedcites [11] sourceExperimental — accuracy not yet measured
Section 5.3 says people may conceal feelings behind social masks and that a single modality may be insufficient for accurate recognition; the introduction (1:2) says deliberate concealment can affect facial- or speech-based methods. The claim overstates this as a categorical inability to capture emotion reliably with any unimodal signal, whereas the review discusses limitations that may arise. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The size of our proposed model is only 2.28 MB”
§ Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG > Results and Discussion > Results on EMO-DB
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network.”
§ Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“outperforming the traditional GhostNet (90.21”
§ Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG > Results and Discussion > Results on CK+
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
25 more claims
Create a free account to see all 46 claims and every source verdict.
- It may state two separate claims as one. 6
- The quote only partly states this claim. 2
- Probably not a substantial claim. 9
Citations
Cites
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380
GhostNet: More Features From Cheap Operations
Han, K., Wang, Y., Tian, Q., Guo, J., Xu, C., & Xu, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00165 · Cited by 5,222
GhostNet: More Features From Cheap OperationsPreprint
Han, K., Wang, Y.-H., Tian, Q., Guo, J.-Y., Xu, C.-J., & Xu, C. (2019) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1911.11907 · Cited by 4,540
Mehrabian, A. (2017) · communication theory · doi:10.4324/9781315080918-15 · Cited by 879
Russell, J. A. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.3.345 · Cited by 857 · cited 1×
“In contrast, the most commonly used multi-dimensional paradigm is the arousal-valence 2D model proposed by Russell [7], arousal and valence are levels of excitement and positivity, respectively, and this definition method is conducive to our quantitative research on emotions.”
MethodEmotion recognition using multi-modal data and machine learning techniques: A tutorial and review
Zhang, J., Yin, Z., Chen, P., & Nichele, S. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.01.011 · Cited by 806
EEG Based Emotion Recognition: A Tutorial and Review
Li, X., Zhang, Y., Tiwari, P., Song, D., Hu, B., Yang, M., Zhao, Z., Kumar, N., & Marttinen, P. (2023) · ACM Computing Surveys · doi:10.1145/3524499 · Cited by 508
3-D Convolutional Recurrent Neural Networks With Attention Model for Speech Emotion Recognition
Chen, M., He, X., Yang, J., & Zhang, H. (2018) · IEEE Signal Processing Letters · doi:10.1109/lsp.2018.2860246 · Cited by 503
Deep learning-based facial emotion recognition for human–computer interaction applications
Chowdary, M. K., Nguyen, T. N., & Hemanth, D. J. (2023) · Neural Computing and Applications · doi:10.1007/s00521-021-06012-8 · Cited by 416
Clustering-Based Speech Emotion Recognition by Incorporating Learned Features and Deep BiLSTM
Mustaqeem, Sajjad, M., & Kwon, S. (2020) · IEEE Access · doi:10.1109/access.2020.2990405 · Cited by 402
Multimodal Emotion Recognition using Deep Learning
Abdullah, S. M. S. A., Ameen, S. Y. A., M. Sadeeq, M. A., & Zeebaree, S. (2021) · Journal of Applied Science and Technology Trends · doi:10.38094/jastt20291 · Cited by 347 · cited 2×
“The ways of expressing emotions can be broadly classified into two categories: external representations, such as facial expressions and speech, and internal representations, such as electroencephalography (EEG) and heart rate [1].”
BackgroundMiddya, A. I., Nag, B., & Roy, S. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108580 · Cited by 235
Real-Time Video Emotion Recognition Based on Reinforcement Learning and Domain Knowledge
Zhang, K., Li, Y., Wang, J., Cambria, E., & Li, X. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3072412 · Cited by 184
Hybrid LSTM-Transformer Model for Emotion Recognition From Speech Audio Files
Andayani, F., Theng, L. B., Tsun, M. T., & Chua, C. (2022) · IEEE Access · doi:10.1109/access.2022.3163856 · Cited by 178
Automated emotion recognition: Current trends and future perspectives
Maithri, M., Raghavendra, U., Gudigar, A., Samanth, J., Barua, P. D., Murugappan, M., Chakole, Y., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106646 · Cited by 172 · cited 1×
“In a previous study [17], it was proposed that fusing facial expressions, speech, and EEG could be a promising direction for future research in emotion recognition.”
BackgroundAtt-Net: Enhanced emotion recognition system using lightweight self-attention module
Mustaqeem, M., & Kwon, S. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107101 · Cited by 160
Cross-Subject EEG Emotion Recognition With Self-Organized Graph Neural Network
Li, J., Li, S., Pan, J., & Wang, F. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.611653 · Cited by 157
Self-citationAudio-visual emotion fusion (AVEF): A deep efficient weighted approach
Ma, Y., Hao, Y., Chen, M., Chen, J., Lu, P., & Košir, A. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.06.003 · Cited by 155 · cited 1×
“Some recent studies [14], [15], [16] have attempted to leverage multiple modalities to boost the performance of emotion recognition, which can demonstrate the complementarity of emotion among multiple modes.”
BackgroundYao, Z., Wang, Z., Liu, W., Liu, Y., & Pan, J. (2020) · Speech Communication · doi:10.1016/j.specom.2020.03.005 · Cited by 152 · cited 1×
“Over the past decade, a majority of emotion recognition studies have been focused on unimodal emotion recognition (UER) using only one mode [8], [9], [10].”
BackgroundSelf-citationAdvances in Multimodal Emotion Recognition Based on Brain–Computer Interfaces
He, Z., Li, Z., Yang, F., Wang, L., Li, J., Zhou, C., & Pan, J. (2020) · Brain Sciences · doi:10.3390/brainsci10100687 · Cited by 151
Self-citationLIGHT-SERNET: A Lightweight Fully Convolutional Neural Network for Speech Emotion Recognition
Aftab, A., Morsali, A., Ghaemmaghami, S., & Champagne, B. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9746679 · Cited by 130
Fusion of Facial Expressions and EEG for Multimodal Emotion Recognition
Huang, Y., Yang, J., Liao, P., & Pan, J. (2017) · Computational Intelligence and Neuroscience · doi:10.1155/2017/2107451 · Cited by 115 · cited 1×
“Some recent studies [14], [15], [16] have attempted to leverage multiple modalities to boost the performance of emotion recognition, which can demonstrate the complementarity of emotion among multiple modes.”
BackgroundSelf-citationA multimodal emotion recognition method based on facial expressions and electroencephalography
Tan, Y., Sun, Z., Duan, F., Solé-Casals, J., & Caiafa, C. F. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.103029 · Cited by 88 · cited 1×
“We found that in previous studies [19], [20], [21], decision-level fusion methods are not only easy to implement but also exhibit better performance than feature-level fusion.”
ResultZhou, H., Du, J., Zhang, Y., Wang, Q., Liu, Q.-F., & Lee, C.-H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3096037 · Cited by 85
Speech Emotion Recognition Using Quaternion Convolutional Neural Networks
Muppidi, A., & Radfar, M. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414248 · Cited by 85
Multimodal Emotion Recognition With Temporal and Semantic Consistency
Chen, B., Cao, Q., Hou, M., Zhang, Z., Lu, G., & Zhang, D. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3129331 · Cited by 71
Self-citationHybrid Emotion-Aware Monitoring System Based on Brainwaves for Internet of Medical Things
Meng, W., Cai, Y. H., Yang, L. T., & Chiu, W.-Y. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3079461 · Cited by 58
A Hybrid Model for Driver Emotion Detection Using Feature Fusion Approach
Sukhavasi, S. B., Sukhavasi, S. B., Elleithy, K. M., El-Sayed, A., & Elleithy, A. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19053085 · Cited by 58
Two-Stage Fuzzy Fusion Based-Convolution Neural Network for Dynamic Emotion Recognition
Wu, M., Su, W., Chen, L., Pedrycz, W., & Hirota, K. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2966440 · Cited by 55
Speech Expression Multimodal Emotion Recognition Based on Deep Belief Network
Liu, D., Chen, L., Wang, Z., & Diao, G. (2021) · Journal of Grid Computing · doi:10.1007/s10723-021-09564-0 · Cited by 38
MindLink-Eumpy: An Open-Source Python Toolbox for Multimodal Emotion Recognition
Li, R., Liang, Y., Liu, X., Wang, B., Huang, W., Cai, Z., Ye, Y., Qiu, L., & Pan, J. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.621493 · Cited by 36
Self-citationFace Emotion Recognition From Static Image Based on Convolution Neural Networks
NASRI, M. A., Hmani, M. A., Mtibaa, A., Petrovska‐Delacrétaz, D., Slima, M. B., & Hamida, A. B. (2020) · International Conference on Advanced Technologies for Signal and Image Processing · doi:10.1109/atsip49331.2020.9231537 · Cited by 25
Visual attention based composite dense neural network for facial expression recognition
Shaik, N. S., & Cherukuri, T. K. (2022) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-022-03843-8 · Cited by 19
Hierarchical fusion of visual and physiological signals for emotion recognition
Fang, Y., Rong, R., & Huang, J. (2021) · Multidimensional Systems and Signal Processing · doi:10.1007/s11045-021-00774-z · Cited by 17 · cited 1×
“We found that in previous studies [19], [20], [21], decision-level fusion methods are not only easy to implement but also exhibit better performance than feature-level fusion.”
ResultDeep Residual Learning for Facial Emotion Recognition
Mishra, S., Joshi, B., Paudyal, R., et al. (2021) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-16-1866-6_22 · Cited by 14
Recognition of Emotion in Speech-related Audio Files with LSTM-Transformer
Andayani, F., Theng, L. B., Tsun, M. T. K., & Chua, C. (2022) · International Conference on Computing and Informatics (ICCI) · doi:10.1109/icci54321.2022.9756100 · Cited by 12
Emotion Recognition Using Deep Learning
Priya, R., Hanmandlu, M., & Vasikarla, S. (2021) · IEEE Applied Imagery Pattern Recognition Workshop · doi:10.1109/aipr52630.2021.9762207 · Cited by 5
Cited by
Emotion recognition in EEG signals using deep learning methods: A review
Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D. L., Gorriz, J. M., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107450 · Cited by 240
A Review of Key Technologies for Emotion Analysis Using Multimodal Information
Zhu, X., Guo, C., Feng, H., Huang, Y., Feng, Y., Wang, X., & Wang, R. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10287-z · Cited by 193
A review on EEG-based multimodal learning for emotion recognition
Pillalamarri, R., & Shanmugam, U. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11126-9 · Cited by 183
Using transformers for multimodal emotion recognition: Taxonomies and state of the art review
Hazmoune, S., & Bougamouza, F. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108339 · Cited by 167
Multimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features
Mamieva, D., Abdusalomov, A. B., Kutlimuratov, A., Muminov, B. B., & Whangbo, T. K. (2023) · Sensors · doi:10.3390/s23125475 · Cited by 160
Metadata
- Authors
- Pan, J., Fang, W., Zhang, Z., Chen, B., Zhang, Z., & Wang, S.
- Affiliations
- South China Normal University (CN)
- University of Leicester (GB)
- Corresponding author
- Chen, Bingzhi
- Venue
- IEEE Open Journal of Engineering in Medicine and Biology
- Published
- 2024
- Volume
- 5
- Pages
- 396-403
- ISSN
- 2644-1276 (ISSN-L)
- PMC
- PMC11186647
- OpenAlex
- W4320002631
- Semantic Scholar
- cbdc02c9cfa1d14a88447691f7b511712350b010
- Cited by
- 121
- References
- 35
- Citations by year
- 2026: 35 · 2025: 48 · 2024: 25
- FWCI
- 31.22
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- IEEE Open Journal of Engineering in Medicine and Biology (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- STI 2030-Major (2022ZD0208900)
- National Natural Science Foundation of China (62076103, 62271217)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- multimodal emotion recognition, facial expression recognition, speech emotion recognition, EEG-based emotion recognition, GhostNet, lightweight convolutional neural network, decision fusion, real-time emotion recognition, human-machine interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.