CoChat
Sign Up
Cited by 36Emotion and Mood RecognitionEEG and Brain-Computer InterfacesFace and Expression Recognition

A Bimodal Emotion Recognition Approach through the Fusion of Electroencephalography and Facial Sequences

Open access
Muhammad, Farah, Hussain, Muhammad, Aboalsamh, Hatim·2023·Diagnostics
DOI 10.3390/diagnostics13050977PMC PMC10000366
Open this paper in CoChat — free accountOpen PDF

Abstract

In recent years, human–computer interaction (HCI) systems have become increasingly popular. Some of these systems demand particular approaches for discriminating actual emotions through the use of better multimodal methods. In this work, a deep canonical correlation analysis (DCCA) based multimodal emotion recognition method is presented through the fusion of electroencephalography (EEG) and facial video clips. A two-stage framework is implemented, where the first stage extracts relevant features for emotion recognition using a single modality, while the second stage merges the highly correlated features from the two modalities and performs classification. Convolutional neural network (CNN) based Resnet50 and 1D-CNN (1-Dimensional CNN) have been utilized to extract features from facial video clips and EEG modalities, respectively. A DCCA-based approach was used to fuse highly correlated features, and three basic human emotion categories (happy, neutral, and sad) were classified using the SoftMax classifier. The proposed approach was investigated based on the publicly available datasets called MAHNOB-HCI and DEAP. Experimental results revealed an average accuracy of 93.86% and 91.54% on the MAHNOB-HCI and DEAP datasets, respectively. The competitiveness of the proposed framework and the justification for exclusivity in achieving this accuracy were evaluated by comparison with existing work.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/diagnostics13050977

Cites

75 of 76 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Studies combining EEG and facial expressions have demonstrated that multimodal fusion achieves higher classification accuracy than unimodal EEG or facial expression methods.

    “Huang et al. achieved a prediction accuracy of 74.38% using only EEG data and 66.38% using only facial expression data, both of which are unimodal methods, for the four basic emotional states in the their custom dataset. However, by implementing two different data fusion techniques, they improved these accuracy rates to 81.25% and 82.75%.”

    From Multimodal Emotion Recognition: Emotion Classification Through the Integration of EEG and Facial Expressions (2025) · cites this paper as [16]

    The Experimental Results (sections 4.2–4.5; page not provided) support the broad point: this paper reports higher accuracy for its EEG–facial fusion model than for either unimodal model. But its comparison with Huang et al. in section 4.5 reports fusion accuracies of 69.75% for valence and 70.00% for arousal; it does not substantiate the citing sentence’s four accuracy figures for Huang. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Multimodal approaches combining EEG with other data types have been shown to improve emotion recognition performance compared with unimodal methods.

    “Using fusion techniques at the feature and decision levels, Zheng et al. [12] showed, for example, that combining EEG and eye movement data could improve emotion recognition performance.”

    From Multimodal Emotion Recognition: Emotion Classification Through the Integration of EEG and Facial Expressions (2025) · cites this paper as [16]

    The Related Work section (2.3; page unavailable) reports that Zheng et al. used EEG and eye movements for emotion recognition, and the Introduction says multimodal methods can outperform unimodal ones. But the cited paper is not Zheng et al.’s study, and our copy does not say that Zheng et al. used both feature-level and decision-level fusion or establish that their method improved performance. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Chaparro et al. and Muhammad et al. achieved accuracies of 97.00% and 93.86%, respectively, using the MAHNOB-HCI dataset combined with EEG and facial expressions.

    “Chaparro et al. [19] and Muhammad et al. [16] achieved accuracies of 97.00% and 93.86%, respectively, using the MAHNOB-HCI dataset combined with EEG and facial expressions.”

    From Multimodal Emotion Recognition: Emotion Classification Through the Integration of EEG and Facial Expressions (2025) · cites this paper as [16]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 77 references couldn’t be matched to a known paper.

Research works printed but not linked: 11

  1. [7]Ghafurian M. Lakatos G. Tao Z. Dautenhahn K. Design and Evaluation of Affective Expressions of a Zoomorphic Robot Proceedings of the Lecture Notes in Computer Science Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics Springer Berlin/Heidelberg, Germany 2020 12483
  2. [8]Ekman P. Expression and the Nature of Emotion Approaches to Emotion Psychology Press London, UK 1984
  3. [14]Zhang J. Zhao S. Huang W. Hu S. Brain Effective Connectivity Analysis from EEG for Positive and Negative Emotion Proceedings of the Lecture Notes in Computer Science Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics Springer Berlin/Heidelberg, Germany 2017 Volume 10637
  4. [23]Liu W. Qiu J. Zheng W. Lu B.L. Multimodal Emotion Recognition Using Deep Canonical Correlation Analysis arxiv 2019 1908.05349
  5. [38]Iandola F.N. Han S. Moskewicz M.W. Ashraf K. Dally W.J. Keutzer K. SqueezeNet: AlexNet-Level Accuracy with 50x Fewer Parameters and <0.5MB Model Size arXiv 2016 1602.07360
  6. [42]Basbrain A. Gan J.Q. One-Shot Only Real-Time Video Classification: A Case Study in Facial Emotion Recognition Proceedings of the Lecture Notes in Computer Science Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics Springer Berlin/Heidelberg, Germany 2020 Volume 12489
  7. [47]Jiang T. Wang J. Liu Z. Ling Y. Fusion-Extraction Network for Multimodal Sentiment Analysis Proceedings of the Lecture Notes in Computer Science Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics Springer Berlin/Heidelberg, Germany 2020 Volume 12085
  8. [48]Zadeh A. Chen M. Cambria E. Poria S. Morency L.P. Tensor Fusion Network for Multimodal Sentiment Analysis Proceedings of the EMNLP 2017-Conference on Empirical Methods in Natural Language Processing Association for Computational Linguistics Copenhagen, Denmark 2017
  9. [66]Simonyan K. Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition arXiv 2014 1409.1556
  10. [79]Qazi E.-H. Hussain M. AboAlsamh H. Ullah I. Automatic Emotion Recognition (AER) System Based on Two-Level Ensemble of Lightweight Deep CNN Models arXiv 2019 1904.13234
  11. [89]Castellano G. Kessous L. Caridakis G. Emotion Recognition through Multiple Modalities: Face, Body Gesture, Speech Proceedings of the Lecture Notes in Computer Science Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics Springer Berlin/Heidelberg, Germany 2008 Volume 4868

Cited by

Sorted by citations

Related

Authors
Muhammad, F., Hussain, M., & Aboalsamh, H.
Affiliations
King Saud University (SA)
Corresponding author
Muhammad, Farah
ORCID
Venue
Diagnostics
Publisher
MDPI AG
Published
2023-03-04
Volume
13
Issue
5
Pages
977
ISSN
2075-4418 (ISSN-L)
DOI
10.3390/diagnostics13050977
PMC
PMC10000366
OpenAlex
W4323312548
Cited by
36
References
77
Citations by year
2026: 10 · 2025: 13 · 2024: 10 · 2023: 3
FWCI
9.31
Type
journal article
Language
en
Open access
Licence
Locations
  • Diagnostics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Deputyship for Research and Innovation, Ministry of Education in Saudi Arabia (IFK-SURG-2-155)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
Keywords
multimodal emotion recognition, EEG, facial video, deep canonical correlation analysis, feature fusion, human-computer interaction, basic emotions, DEAP dataset, ResNet50, 1D convolutional neural network

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review