Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 124Emotion and Mood RecognitionEEG and Brain-Computer InterfacesFace and Expression Recognition

The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index; 5 of the 12 sources its claims cite were checked for support.

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

Open access

Abstract

Emotion recognition has attracted a lot of attention in recent years. It is widely used in health care, teaching, human-computer interaction, and other fields. Various human emotional features are often used to recognize different emotions. Currently, there is more and more research on multimodal emotion recognition based on the fusion of multiple features. This paper proposes a deep learning model of multimodal emotion recognition based on the fusion of electroencephalogram (EEG) signals and facial expressions to achieve an excellent classification effect. First, the pre-trained convolution neural network (CNN) is used to extract the facial features from the facial expressions. Next, the attention mechanism is introduced to extract more critical facial frame features. Then, we apply CNNs to extract spatial features from original EEG signals, which use a local convolution kernel and a global convolution kernel to learn the features of left and right hemispheres channels and all EEG channels. After feature-level fusion, the fusion features of the facial expression features and EEG features are fed into the classifier for emotion recognition. This paper conducted experiments on the DEAP and MAHNOB-HCI datasets to evaluate the performance of the proposed model. The accuracy of valence dimension classification is 96.63%, and arousal dimension classification is 97.15% on the DEAP dataset, while 96.69% and 96.26% on the MAHNOB-HCI dataset. The experimental results show that the proposed model can effectively carry out emotion recognition.

Overview

CoChat Summary

TL;DR

This paper proposes a deep learning model that fuses EEG signals and facial expressions for multimodal emotion recognition, using a pre-trained CNN with attention for facial features and CNNs with local/global kernels for EEG spatial features. It reports high binary valence/arousal classification accuracy on DEAP and MAHNOB-HCI, with fusion outperforming either modality alone.

Key findings

  • On DEAP, the proposed model achieved 96.63% accuracy for valence and 97.15% for arousal in binary classification.
  • On MAHNOB-HCI, it achieved 96.69% accuracy for valence and 96.26% for arousal.
  • Ablation results show multimodal fusion outperformed single modalities: DEAP valence was 93.59% for facial expressions, 85.72% for EEG, and 96.63% for fusion; DEAP arousal was 94.03%, 87.97%, and 97.15%, respectively. MAHNOB-HCI valence was 93.34%, 85.98%, and 96.69%; arousal was 93.06%, 85.23%, and 96.26%.
  • The paper reports that its model outperformed several existing approaches on DEAP and MAHNOB-HCI, such as Huang et al. (DEAP valence 80.30%, arousal 74.23%), Zhu et al. (70.01%, 77.08%), Li et al. (71.00%, 58.75%), and Koelstra et al. (74.00%, 70.00%).

Methods

  • The model extracts facial features using a pre-trained CNN (DeepVANet, pre-trained on AFEW-VA) as an encoder, then applies an attention mechanism to weight important facial frames; input samples are T × 3 × 64 × 64 with T = 5 frames per second.
  • EEG spatial features are extracted with CNNs using a local convolution kernel for left/right brain channels and a global convolution kernel for all EEG channels; in DEAP and MAHNOB-HCI, c = 32 channels, h = 64, and K = 20 kernels.
  • Facial and EEG features are fused at the feature level and fed into a fully connected classifier. Experiments use DEAP (32 subjects, 40 trials each, 2400 samples per subject) and MAHNOB-HCI (25 participants used, 20 trials each, 1611 samples per subject), with 5-fold cross-validation; training used PyTorch, batch size 128, learning rate 0.00001, Adam optimizer, and binary cross entropy.

Limitations

  • The paper does not include an explicit limitations section; its conclusion states future work to explore a more reliable pre-training model to reduce resources and time, and to introduce more modalities such as non-physiological signals.
  • Evident from the text: MAHNOB-HCI experiments used only 25 of 30 participants because three recording files were corrupted and two participants’ data were incomplete.
  • Evident from the text: the tasks are binary high/low valence and arousal classifications, not continuous or multi-class emotion recognition; also, DEAP facial expressions were recorded for the first 22 subjects, while the experiments state both datasets use data of all subjects, suggesting a possible sample mismatch.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/access.2023.3263670

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

3 Valid · 0 Not valid · 4 Not checked · 16 Own results and interpretations

9 more claims

Create a free account to see all 23 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 2
  • Probably not a substantial claim. 8

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

1 of 1

Citations

Cites

Show all 36 references

Cited by

See all 124

Metadata

Authors
Wang, S., Qu, J., Zhang, Y., & Zhang, Y.
Affiliations
Liaoning Normal University (CN)
Venue
IEEE Access
Published
2023
Volume
11
Pages
33061-33068
ISSN
2169-3536 (ISSN-L)
OpenAlex
W4361987556
Cited by
124
References
36
Citations by year
2026: 29 · 2025: 56 · 2024: 32 · 2023: 7
FWCI
29.96
Type
journal article
Language
en
Open access
Locations
  • IEEE Access (publishedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Natural Science Foundation of China (61772252)
  • Natural Science Foundation of Liaoning Province of China (2019-MS-216)
  • Scientific Research Foundation of the Education Department of Liaoning Province (LJKZ0965)
  • Huzhou Science and Technology Plan Project (2022GZ08)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
Keywords
multimodal emotion recognition, EEG signals, facial expressions, feature-level fusion, attention mechanism, convolutional neural networks, valence and arousal classification, DEAP dataset

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Multimodal Emotion Recognition From EEG Signals and Facial Expressions. https://app.cochat.ai/papers/10.1109/access.2023.3263670
The paper
Wang, S., Qu, J., Zhang, Y., & Zhang, Y. (2023). Multimodal Emotion Recognition From EEG Signals and Facial Expressions. IEEE Access. https://doi.org/10.1109/access.2023.3263670

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.