CoChat

Multimodal Emotion Recognition: Emotion Classification Through the Integration of EEG and Facial Expressions

Open access
Erdem Güler, Songül, Patlar Akbulut, Fatma · 2025 · IEEE Access

The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index; 10 of the 21 sources its claims cite were checked for support.

Abstract

Despite advances in the field of emotion recognition, the research field still faces two main limitations: the use of deep models for increasingly complex calculations and the identification of emotions through various data types. This study aims to advance the knowledge on multimodal emotion recognition by combining electroencephalography (EEG) signals with facial expressions, using advanced models such as Transformer, Long Short-Term Memory (LSTM), and Gated Recurrent Unit (GRU). The results validate the effectiveness of this approach, demonstrating the high accuracy of the Gated Recurrent Unit (GRU) model, which achieved an average of 91.8% classification accuracy on unimodal (EEG-only) data and an average of 97.8% classification accuracy on multimodal (EEG and facial expressions) datasets in the multi-class emotion categories. The findings emphasize that by applying a multi-class classification framework, multimodal approaches offer significant improvements over traditional unimodal techniques. This work presents a framework that captures complex neural dynamics and visible emotional cues, enhancing the robustness and accuracy of emotion recognition systems. These results have important practical implications, showing how integrating various data sources with advanced models can overcome the limitations of single-modality systems.

Overview

CoChat Summary

TL;DR

This paper combines EEG signals and facial expressions from the DEAP dataset to classify emotions into four valence–arousal categories, comparing GRU, LSTM, and Transformer models. The GRU model performed best, reaching 91.8% average accuracy on EEG-only data and 97.8% on multimodal EEG plus facial expression data.

Key findings

  • The GRU model achieved the highest accuracy in both settings: 91.8% for unimodal EEG-only classification and 97.8% for multimodal EEG plus facial expression classification.
  • Multimodal data improved every model: the LSTM reached 96% overall accuracy with multimodal data versus 85% unimodal, and the Transformer reached 84% multimodal versus 69% unimodal.
  • The GRU also had the lowest test loss: 0.278 for unimodal EEG and 0.104 for multimodal data, compared with LSTM at 0.45 and 0.179, and Transformer at 0.791 and 0.465.
  • The study used four classes—HVHA, HVLA, LVHA, and LVLA—rather than binary valence/arousal classification, and reports that facial expressions plus EEG usually achieve higher emotion recognition performance.

Methods

  • The DEAP dataset was used, containing 32-channel EEG recordings and facial expression videos for the first 22 participants; the paper reports 17,360 processed EEG signal samples and 3,000 images per 60-second video at 50 FPS.
  • EEG data were segmented into one-second windows, labeled into four classes using valence and arousal thresholds of 5, and split into 70% training, 15% validation, and 15% testing.
  • Facial expression analysis used FER with MTCNN and Haar Cascade face detection; missing facial data were filled with synthetic samples, and excess data were downsampled to match EEG sample counts.
  • GRU, LSTM, and Transformer models were trained for 330 epochs with a learning rate of 0.0001, using accuracy, precision, recall, F1-score, confusion matrices, and loss curves for evaluation.

Limitations

  • The paper states that future research should prioritize larger and more diverse datasets, explore novel deep learning architectures and modality integrations, and use feature extraction to distinguish contrasting emotional states within the same category.
  • Evident from the text: facial video data were limited to only 22 participants who consented to recording, and technical issues during recording meant some participants had fewer than the expected 40 videos.
  • Evident from the text: synthetic data were generated to fill gaps in facial expression data, and the paper notes fewer HVLA samples in the test data, which may affect class balance and model evaluation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/access.2025.3538642

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

7 Valid · 1 Not valid · 6 Not checked · 102 Own results and interpretations

47 more claims

Create a free account to see all 116 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 35
  • It may state two separate claims as one. 37
  • The quote only partly states this claim. 8

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 6 · the rest aren’t loaded

1 more finding

Create a free account to see every finding of this check.

See all findings — free account

Citations

Cites

Research works printed but not linked: 1

  1. 10.5555/3045118.3045167 · doi:10.5555/3045118.3045167

Cited by

See all 21

Metadata

Authors
Erdem Güler, S., & Patlar Akbulut, F.
Affiliations
Department of Computer Engineering, Istanbul Kültür University, İstanbul, Türkiye
Venue
IEEE Access
Published
2025
Volume
13
Pages
24587-24603
ISSN
2169-3536 (ISSN-L)
OpenAlex
W4407128417
Cited by
21
References
40
Citations by year
2026: 11 · 2025: 10
FWCI
21.87
Type
journal article
Language
en
Open access
Locations
  • IEEE Access (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
multimodal emotion recognition, EEG, facial expressions, emotion classification, gated recurrent unit, LSTM, transformer models, neural dynamics, emotional cues

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.