Multimodal Emotion Recognition for Hearing-impaired Subjects by Fusing EEG Signals and Facial Expressions
The DOI matches its Crossref record; 15 of the 17 references it cites were found in a registry or an index.
Abstract
With the development and improvement of artificial intelligence technology and multi-source information integration theory, emotional recognition has gradually transitioned from single-modality emotional recognition to multimodal. Due to the shortage of an emotional cognitive channel, the hearing-impaired subjects may have an emotional cognitive deviation in daily communication. Therefore, we classified the four kinds of emotions (happiness, fear, calmness, sadness) of the hearing-impaired subjects based on the feature-level fusion network which combined electroencephalography (EEG) signals and facial expressions. In this network, we adopted different feature extraction methods to obtain shallow features related to emotional changes in the two modalities, and then used multi -head cross-attention mechanism to the feature-level fusion layer between the two modalities. The results show that the average classification accuracy of four emotions recognition after multimodal fusion achieves 92.09%, which is higher than EEG signals (15.40%) and facial expressions (11.49%).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Recent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576
Differential entropy feature for EEG-based emotion classification
Duan, R.-N., Zhu, J.-Y., & Lu, B.-L. (2013) · 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER) · doi:10.1109/ner.2013.6695876 · Cited by 1,100
EEG-Based Emotion Recognition in Music Listening
Lin, Y.-P., Wang, C.-H., Jung, T.-P., Wu, T.-L., Jeng, S.-K., Duann, J.-R., & Chen, J.-H. (2010) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2010.2048568 · Cited by 1,021
Identifying Stable Patterns over Time for Emotion Recognition from EEG
Zheng, W.-L., Zhu, J.-Y., & Lu, B.-L. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2712143 · Cited by 990
Verma, G. K., & Tiwary, U. S. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.11.007 · Cited by 354
Emotion Recognition through Multiple Modalities: Face, Body Gesture, Speech
Castellano, G., Kessous, L., & Caridakis, G. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-85099-1_8 · Cited by 259
Fusion of facial expressions and EEG for implicit affective tagging
Koelstra, S., & Patras, I. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.10.002 · Cited by 228
Ta, V.-D., Liu, C.-M., & Tadesse, D. A. (2020) · Applied Sciences · doi:10.3390/app10020437 · Cited by 189
Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems
Ayata, D., Yaslan, Y., & Kamasak, M. E. (2020) · Journal of Medical and Biological Engineering · doi:10.1007/s40846-019-00505-7 · Cited by 182
Fusion of Facial Expressions and EEG for Multimodal Emotion Recognition
Huang, Y., Yang, J., Liao, P., & Pan, J. (2017) · Computational Intelligence and Neuroscience · doi:10.1155/2017/2107451 · Cited by 115 · cited 1×
“[12] proposed a method to combine facial expressions and EEG signals for emotion recognition to make up for its defect as a single information source under four basic emotional states.”
MethodDeep Feature Pyramid Network for Eeg Emotion Recognition
Hou, F., Gao, Q., Song, Y., Wang, Z., Bai, Z., Yang, Y., & Tian, Z. (2022) · Measurement · doi:10.1016/j.measurement.2022.111724 · Cited by 37
Self-citationLightSleepNet: Design of a Personalized Portable Sleep Staging System Based on Single-Channel EEG
Liao, Y., Zhang, C., Zhang, M., Wang, Z., & Xie, X. (2021) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2021.3086981 · Cited by 34
EEG-Based Emotion Recognition With Haptic Vibration by a Feature Fusion Method
Li, D., Yang, Z., Hou, F., Kang, Q., Liu, S., Song, Y., Gao, Q., & Dong, E. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3147882 · Cited by 31
Self-citationNovel high-dimensional phase space features for EEG emotion recognition
Goshvarpour, A., & Goshvarpour, A. (2022) · Signal Image and Video Processing · doi:10.1007/s11760-022-02248-6 · Cited by 23
Zhang, C., Liao, Y., Han, S., Zhang, M., Wang, Z., & Xie, X. (2022) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2022.3187205 · Cited by 21
Research works printed but not linked: 4
- hoch, 0, Bimodal fusion of emot ional data in an automotive environment[C]//Proceedings.(IC ASSP'05), IEEE International Conference on Acoustics Spe-ech and Signal Processing, 2, ii/l085
- zhao, 2018, Personalized Emotion Recognition, Physiological Signals //IJCAI, 1660
- valentina, 0, Emotion from EEG and Facial Expressions: a Multimodal Approach[C], 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society(EMBC) IEEE
- 10.1016/j.measurement.2022.111724 · DOI
Cited by
Ji, M., Gan, Y., Yang, M., & Hou, R. (2024) · IEEE Smart World Congress (SWC) · doi:10.1109/swc62898.2024.00190 · Cited by 1
“Currently, multimodal fusion employs speech and text [16], face and voice[17], electroencephalogram (EEG) and voice[18], or EEG and face[19] for recognition.”
Mahaseni, B., & Khan, N. (2025) · Empathic Computing · doi:10.70401/ec.2025.0010 · Cited by 4
Denoise and Attend: An AE-Transformer Approach to EEG Emotion Recognition
Luo, M. (2025) · International Conference on Electrical Engineering and Computer Technology (IC2ECT) · doi:10.1109/ic2ect66838.2025.11291002
Ji, M. (2024) · Poster Volume Ⅰ The 2024 Twentieth International Conference on Intelligent Computing August 5-8, 2024 Tianjin, China · doi:10.65286/icic.v20i2.94055
Metadata
- Authors
- Chen, Y., Bai, Z., Cheng, M., Liu, Y., Zhao, X., & Song, Y.
- Affiliations
- Tianjin University of Technology (CN)
- Venue
- 2023 42nd Chinese Control Conference (CCC)
- Publisher
- IEEE
- Published
- 2023-07-24
- Pages
- 1-6
- OpenAlex
- W4386821089
- Cited by
- 3
- References
- 17
- Citations by year
- 2025: 2 · 2024: 1
- FWCI
- 0.34
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Chinese Control Conference (CCC) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- multimodal emotion recognition, hearing-impaired individuals, EEG signals, facial expressions, feature-level fusion, cross-attention, emotion classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.