Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 121Emotion and Mood RecognitionSpeech and Audio ProcessingSentiment Analysis and Opinion Mining

The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 3 of the 10 sources its claims cite were checked for support.

Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG

Open access
Pan, Jiahui, Fang, Weijie, Zhang, Zhihang, Chen, Bingzhi, Zhang, Zheng, Wang, Shuihua·2024·IEEE Open Journal of Engineering in Medicine and Biology

Abstract

Goal: As an essential human-machine interactive task, emotion recognition has become an emerging area over the decades. Although previous attempts to classify emotions have achieved high performance, several challenges remain open: 1) How to effectively recognize emotions using different modalities remains challenging. 2) Due to the increasing amount of computing power required for deep learning, how to provide real-time detection and improve the robustness of deep neural networks is important. Method: In this paper, we propose a deep learning-based multimodal emotion recognition (MER) called Deep-Emotion, which can adaptively integrate the most discriminating features from facial expressions, speech, and electroencephalogram (EEG) to improve the performance of the MER. Specifically, the proposed Deep-Emotion framework consists of three branches, i.e., the facial branch, speech branch, and EEG branch. Correspondingly, the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network. For work on the speech branch, this paper proposes a lightweight fully convolutional neural network (LFCNN) for the efficient extraction of speech emotion features. Regarding the study of EEG branches, we proposed a tree-like LSTM (tLSTM) model capable of fusing multi-stage features for EEG emotion feature extraction. Finally, we adopted the strategy of decision-level fusion to integrate the recognition results of the above three modes, resulting in more comprehensive and accurate performance. Result and Conclusions: Extensive experiments on the CK+, EMO-DB, and MAHNOB-HCI datasets have demonstrated the advanced nature of the Deep-Emotion method proposed in this paper, as well as the feasibility and superiority of the MER approach.

Overview

CoChat Summary

TL;DR

The paper proposes Deep-Emotion, a multimodal emotion recognition framework that combines facial expressions, speech, and EEG through decision-level fusion. It reports strong results on CK+, EMO-DB, and MAHNOB-HCI, while noting that fusion did not significantly outperform facial-expression or EEG modes on MAHNOB-HCI.

Key findings

  • The improved GhostNet achieved 98.27% average accuracy on CK+ under tenfold cross-validation, outperforming traditional GhostNet at 90.21%; fear expression accuracy was only 95%, possibly because fear and contempt have similar features.
  • The LFCNN for speech achieved 94.36% average accuracy and 94.38% F1-score on EMO-DB, with a model size of only 2.28 MB.
  • On MAHNOB-HCI, the tLSTM obtained advanced results in EEG emotion recognition, and the fusion method improved both arousal and valence dimensions; fusion was significantly higher than the speech mode (\(p<0.05\), paired \(t\) test) but not significantly better than facial expressions or EEG (\(p>0.05\)).
  • The paper claims this is the first attempt to combine facial expressions, speech, and EEG for multimodal emotion recognition.

Methods

  • Deep-Emotion has three branches: an improved GhostNet for facial expression recognition, a lightweight fully convolutional neural network (LFCNN) for speech emotion recognition, and a tree-like LSTM (tLSTM) for EEG emotion recognition.
  • Decision-level fusion uses an optimal weight distribution search algorithm: weights are enumerated from 0.00 to 1.00 with step 0.01, and RMSE is minimized to select the best weight distribution.
  • Experiments use CK+, EMO-DB, and MAHNOB-HCI; CK+ used tenfold cross-validation. Data preprocessing, full LFCNN structure, experiment setup, and algorithm pseudo-code are stated to be in the Supplementary Materials.

Limitations

  • The paper states that on MAHNOB-HCI the fusion method was only significantly higher than the speech mode, not facial expressions or EEG; it suggests this may be because subjects behaved normally rather than mimicking facial expressions.
  • The paper states that SER on MAHNOB-HCI is challenging because the speech signal includes both the subject’s voice and the stimulus material’s voice, possibly explaining the relatively average SER recognition rate.
  • The paper states that the work is based on open-source datasets and has not conducted independent data collection; it also notes future work should explore dynamic weight allocation. The text makes evident that many methodological details are relegated to Supplementary Materials, limiting full standalone reproducibility from the main text.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/ojemb.2023.3240280

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1. Structure of Ghost bottleneck.
Fig. 1. Structure of Ghost bottleneck.CC BY-NC-ND · PubMed Central
Fig. 2. Structure of traditional convolution and Ghost module.
Fig. 2. Structure of traditional convolution and Ghost module.CC BY-NC-ND · PubMed Central
Fig. 3. Structure of the proposed LFCNN model for SER. (A): When the task is expressed with a discrete paradigm. (B): When the task is expressed with a arousal-valence 2D model.
Fig. 3. Structure of the proposed LFCNN model for SER. (A): When the task is expressed with a discrete paradigm. (B): When the task is expressed with a arousal-valence 2D model.CC BY-NC-ND · PubMed Central
Fig. 4. Structure of the proposed tLSTM model for EER.
Fig. 4. Structure of the proposed tLSTM model for EER.CC BY-NC-ND · PubMed Central
Fig. 5. The confusion matrix obtained by Ten-fold cross-validation on the CK+ dataset.
Fig. 5. The confusion matrix obtained by Ten-fold cross-validation on the CK+ dataset.CC BY-NC-ND · PubMed Central
Fig. 6. The accuracy of UER and MER for each subject in the MAHNOB-HCI dataset. The horizontal axis represents the subject ID, and the vertical axis represents the accuracy rate (\%). FE: facial expressions.
Fig. 6. The accuracy of UER and MER for each subject in the MAHNOB-HCI dataset. The horizontal axis represents the subject ID, and the vertical axis represents the accuracy rate (\%). FE: facial expressions.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

2 Valid · 6 Not valid · 4 Not checked · 34 Own results and interpretations

25 more claims

Create a free account to see all 46 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 6
  • The quote only partly states this claim. 2
  • Probably not a substantial claim. 9

Create a free account to see these claims.

Citations

Cites

Show all 35 references

Cited by

See all 121

Metadata

Authors
Pan, J., Fang, W., Zhang, Z., Chen, B., Zhang, Z., & Wang, S.
Affiliations
  • South China Normal University (CN)
  • University of Leicester (GB)
Corresponding author
Chen, Bingzhi
Venue
IEEE Open Journal of Engineering in Medicine and Biology
Published
2024
Volume
5
Pages
396-403
ISSN
2644-1276 (ISSN-L)
OpenAlex
W4320002631
Cited by
121
References
35
Citations by year
2026: 35 · 2025: 48 · 2024: 25
FWCI
31.22
Type
journal article
Language
en
Open access
Locations
  • IEEE Open Journal of Engineering in Medicine and Biology (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • STI 2030-Major (2022ZD0208900)
  • National Natural Science Foundation of China (62076103, 62271217)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
multimodal emotion recognition, facial expression recognition, speech emotion recognition, EEG-based emotion recognition, GhostNet, lightweight convolutional neural network, decision fusion, real-time emotion recognition, human-machine interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG. https://app.cochat.ai/papers/10.1109/ojemb.2023.3240280
The paper
Pan, J., Fang, W., Zhang, Z., Chen, B., Zhang, Z., & Wang, S. (2024). Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG. IEEE Open Journal of Engineering in Medicine and Biology. https://doi.org/10.1109/ojemb.2023.3240280

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.