CoChat

Multimodal Emotion Recognition for Hearing-impaired Subjects by Fusing EEG Signals and Facial Expressions

Chen, Yirui, Bai, Zhongli, Cheng, Meiyi, Liu, Yang, Zhao, Xuewen, Song, Yu · 2023 · 2023 42nd Chinese Control Conference (CCC)

The DOI matches its Crossref record; 15 of the 17 references it cites were found in a registry or an index.

Abstract

With the development and improvement of artificial intelligence technology and multi-source information integration theory, emotional recognition has gradually transitioned from single-modality emotional recognition to multimodal. Due to the shortage of an emotional cognitive channel, the hearing-impaired subjects may have an emotional cognitive deviation in daily communication. Therefore, we classified the four kinds of emotions (happiness, fear, calmness, sadness) of the hearing-impaired subjects based on the feature-level fusion network which combined electroencephalography (EEG) signals and facial expressions. In this network, we adopted different feature extraction methods to obtain shallow features related to emotional changes in the two modalities, and then used multi -head cross-attention mechanism to the feature-level fusion layer between the two modalities. The results show that the average classification accuracy of four emotions recognition after multimodal fusion achieves 92.09%, which is higher than EEG signals (15.40%) and facial expressions (11.49%).

Overview

Integrity signals

Registry

Crossref · 10.23919/ccc58697.2023.10239741

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Research works printed but not linked: 4

  1. hoch, 0, Bimodal fusion of emot ional data in an automotive environment[C]//Proceedings.(IC ASSP'05), IEEE International Conference on Acoustics Spe-ech and Signal Processing, 2, ii/l085
  2. zhao, 2018, Personalized Emotion Recognition, Physiological Signals //IJCAI, 1660
  3. valentina, 0, Emotion from EEG and Facial Expressions: a Multimodal Approach[C], 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society(EMBC) IEEE
  4. 10.1016/j.measurement.2022.111724 · DOI

Cited by

See all 4

Metadata

Authors
Chen, Y., Bai, Z., Cheng, M., Liu, Y., Zhao, X., & Song, Y.
Affiliations
Tianjin University of Technology (CN)
Venue
2023 42nd Chinese Control Conference (CCC)
Publisher
IEEE
Published
2023-07-24
Pages
1-6
OpenAlex
W4386821089
Cited by
3
References
17
Citations by year
2025: 2 · 2024: 1
FWCI
0.34
Type
conference paper
Language
en
Open access
closed
Locations
Chinese Control Conference (CCC) (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
Keywords
multimodal emotion recognition, hearing-impaired individuals, EEG signals, facial expressions, feature-level fusion, cross-attention, emotion classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.