The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 3 of the 10 sources its claims cite were checked for support.
Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG
Open accessAbstract
Goal: As an essential human-machine interactive task, emotion recognition has become an emerging area over the decades. Although previous attempts to classify emotions have achieved high performance, several challenges remain open: 1) How to effectively recognize emotions using different modalities remains challenging. 2) Due to the increasing amount of computing power required for deep learning, how to provide real-time detection and improve the robustness of deep neural networks is important. Method: In this paper, we propose a deep learning-based multimodal emotion recognition (MER) called Deep-Emotion, which can adaptively integrate the most discriminating features from facial expressions, speech, and electroencephalogram (EEG) to improve the performance of the MER. Specifically, the proposed Deep-Emotion framework consists of three branches, i.e., the facial branch, speech branch, and EEG branch. Correspondingly, the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network. For work on the speech branch, this paper proposes a lightweight fully convolutional neural network (LFCNN) for the efficient extraction of speech emotion features. Regarding the study of EEG branches, we proposed a tree-like LSTM (tLSTM) model capable of fusing multi-stage features for EEG emotion feature extraction. Finally, we adopted the strategy of decision-level fusion to integrate the recognition results of the above three modes, resulting in more comprehensive and accurate performance. Result and Conclusions: Extensive experiments on the CK+, EMO-DB, and MAHNOB-HCI datasets have demonstrated the advanced nature of the Deep-Emotion method proposed in this paper, as well as the feasibility and superiority of the MER approach.
Overview
CoChat Summary
TL;DR
The paper proposes Deep-Emotion, a multimodal emotion recognition framework that combines facial expressions, speech, and EEG through decision-level fusion. It reports strong results on CK+, EMO-DB, and MAHNOB-HCI, while noting that fusion did not significantly outperform facial-expression or EEG modes on MAHNOB-HCI.
Key findings
- The improved GhostNet achieved 98.27% average accuracy on CK+ under tenfold cross-validation, outperforming traditional GhostNet at 90.21%; fear expression accuracy was only 95%, possibly because fear and contempt have similar features.
- The LFCNN for speech achieved 94.36% average accuracy and 94.38% F1-score on EMO-DB, with a model size of only 2.28 MB.
- On MAHNOB-HCI, the tLSTM obtained advanced results in EEG emotion recognition, and the fusion method improved both arousal and valence dimensions; fusion was significantly higher than the speech mode (\(p<0.05\), paired \(t\) test) but not significantly better than facial expressions or EEG (\(p>0.05\)).
- The paper claims this is the first attempt to combine facial expressions, speech, and EEG for multimodal emotion recognition.
Methods
- Deep-Emotion has three branches: an improved GhostNet for facial expression recognition, a lightweight fully convolutional neural network (LFCNN) for speech emotion recognition, and a tree-like LSTM (tLSTM) for EEG emotion recognition.
- Decision-level fusion uses an optimal weight distribution search algorithm: weights are enumerated from 0.00 to 1.00 with step 0.01, and RMSE is minimized to select the best weight distribution.
- Experiments use CK+, EMO-DB, and MAHNOB-HCI; CK+ used tenfold cross-validation. Data preprocessing, full LFCNN structure, experiment setup, and algorithm pseudo-code are stated to be in the Supplementary Materials.
Limitations
- The paper states that on MAHNOB-HCI the fusion method was only significantly higher than the speech mode, not facial expressions or EEG; it suggests this may be because subjects behaved normally rather than mimicking facial expressions.
- The paper states that SER on MAHNOB-HCI is challenging because the speech signal includes both the subject’s voice and the stimulus material’s voice, possibly explaining the relatively average SER recognition rate.
- The paper states that the work is based on open-source datasets and has not conducted independent data collection; it also notes future work should explore dynamic weight allocation. The text makes evident that many methodological details are relegated to Supplementary Materials, limiting full standalone reproducibility from the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
2 Valid · 6 Not valid · 4 Not checked · 34 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“EEG signals extracted from the central nervous system can more accurately and objectively reflect changes in people's emotions than other signals [2], [11].”
§ Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG > Introduction
What the sources say
- Not checked — no full text of the sourcecites [2] source
- Partly supportedcites [11] sourceExperimental — accuracy not yet measured
Section 2.2 says EEG records central nervous system activity and is an effective measure for studying emotion. The introduction (p. 1:2) notes that facial and speech cues can be affected when people conceal their feelings, but it does not establish that EEG reflects emotional changes more accurately and objectively than other signals generally. (LLM, conf=0.84)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“emotions are considered a complex representation that cannot be reliably captured with unimodal signals, since genuine emotion can be hidden by different facial expressions or tones [11].”
§ Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG > Introduction
What the sources say
- Partly supportedcites [11] sourceExperimental — accuracy not yet measured
Section 5.3 says people may conceal feelings behind social masks and that a single modality may be insufficient for accurate recognition; the introduction (1:2) says deliberate concealment can affect facial- or speech-based methods. The claim overstates this as a categorical inability to capture emotion reliably with any unimodal signal, whereas the review discusses limitations that may arise. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The size of our proposed model is only 2.28 MB”
§ Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG > Results and Discussion > Results on EMO-DB
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network.”
§ Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“outperforming the traditional GhostNet (90.21”
§ Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG > Results and Discussion > Results on CK+
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
25 more claims
Create a free account to see all 46 claims and every source verdict.
- It may state two separate claims as one. 6
- The quote only partly states this claim. 2
- Probably not a substantial claim. 9
Citations
Cites
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380
GhostNet: More Features From Cheap Operations
Han, K., Wang, Y., Tian, Q., Guo, J., Xu, C., & Xu, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00165 · Cited by 5,222
GhostNet: More Features From Cheap OperationsPreprint
Han, K., Wang, Y.-H., Tian, Q., Guo, J.-Y., Xu, C.-J., & Xu, C. (2019) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1911.11907 · Cited by 4,540
Mehrabian, A. (2017) · communication theory · doi:10.4324/9781315080918-15 · Cited by 879
Russell, J. A. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.3.345 · Cited by 857
Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review
Zhang, J., Yin, Z., Chen, P., & Nichele, S. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.01.011 · Cited by 806
EEG Based Emotion Recognition: A Tutorial and Review
Li, X., Zhang, Y., Tiwari, P., Song, D., Hu, B., Yang, M., Zhao, Z., Kumar, N., & Marttinen, P. (2023) · ACM Computing Surveys · doi:10.1145/3524499 · Cited by 508
3-D Convolutional Recurrent Neural Networks With Attention Model for Speech Emotion Recognition
Chen, M., He, X., Yang, J., & Zhang, H. (2018) · IEEE Signal Processing Letters · doi:10.1109/lsp.2018.2860246 · Cited by 503
Deep learning-based facial emotion recognition for human–computer interaction applications
Chowdary, M. K., Nguyen, T. N., & Hemanth, D. J. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-06012-8 · Cited by 452
Clustering-Based Speech Emotion Recognition by Incorporating Learned Features and Deep BiLSTM
Mustaqeem, Sajjad, M., & Kwon, S. (2020) · IEEE Access · doi:10.1109/access.2020.2990405 · Cited by 402
Cited by
Emotion recognition in EEG signals using deep learning methods: A review
Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D. L., Gorriz, J. M., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107450 · Cited by 240
A Review of Key Technologies for Emotion Analysis Using Multimodal Information
Zhu, X., Guo, C., Feng, H., Huang, Y., Feng, Y., Wang, X., & Wang, R. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10287-z · Cited by 151
A review on EEG-based multimodal learning for emotion recognition
Pillalamarri, R., & Shanmugam, U. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11126-9 · Cited by 155
Using transformers for multimodal emotion recognition: Taxonomies and state of the art review
Hazmoune, S., & Bougamouza, F. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108339 · Cited by 140
Multimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features
Mamieva, D., Abdusalomov, A. B., Kutlimuratov, A., Muminov, B. B., & Whangbo, T. K. (2023) · Sensors · doi:10.3390/s23125475 · Cited by 119
Metadata
- Authors
- Pan, J., Fang, W., Zhang, Z., Chen, B., Zhang, Z., & Wang, S.
- Affiliations
- South China Normal University (CN)
- University of Leicester (GB)
- Corresponding author
- Chen, Bingzhi
- Venue
- IEEE Open Journal of Engineering in Medicine and Biology
- Published
- 2024
- Volume
- 5
- Pages
- 396-403
- ISSN
- 2644-1276 (ISSN-L)
- PMC
- PMC11186647
- OpenAlex
- W4320002631
- Semantic Scholar
- cbdc02c9cfa1d14a88447691f7b511712350b010
- Cited by
- 121
- References
- 35
- Citations by year
- 2026: 35 · 2025: 48 · 2024: 25
- FWCI
- 31.22
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- IEEE Open Journal of Engineering in Medicine and Biology (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- STI 2030-Major (2022ZD0208900)
- National Natural Science Foundation of China (62076103, 62271217)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- multimodal emotion recognition, facial expression recognition, speech emotion recognition, EEG-based emotion recognition, GhostNet, lightweight convolutional neural network, decision fusion, real-time emotion recognition, human-machine interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG. https://app.cochat.ai/papers/10.1109/ojemb.2023.3240280
- The paper
- Pan, J., Fang, W., Zhang, Z., Chen, B., Zhang, Z., & Wang, S. (2024). Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG. IEEE Open Journal of Engineering in Medicine and Biology. https://doi.org/10.1109/ojemb.2023.3240280
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.