Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Emotion recognition plays an essential role in human–computer interaction. Previous studies have investigated the use of facial expression and electroencephalogram (EEG) signals from single modal for emotion recognition separately, but few have paid attention to a fusion between them. In this paper, we adopted a multimodal emotion recognition framework by combining facial expression and EEG, based on a valence-arousal emotional model. For facial expression detection, we followed a transfer learning approach for multi-task convolutional neural network (CNN) architectures to detect the state of valence and arousal. For EEG detection, two learning targets (valence and arousal) were detected by different support vector machine (SVM) classifiers, separately. Finally, two decision-level fusion methods based on the enumerate weight rule or an adaptive boosting technique were used to combine facial expression and EEG. In the experiment, the subjects were instructed to watch clips designed to elicit an emotional response and then reported their emotional state. We used two emotion datasets—a Database for Emotion Analysis using Physiological Signals (DEAP) and MAHNOB-human computer interface (MAHNOB-HCI)—to evaluate our method. In addition, we also performed an online experiment to make our method more robust. We experimentally demonstrated that our method produces state-of-the-art results in terms of binary valence/arousal classification, based on DEAP and MAHNOB-HCI data sets. Besides this, for the online experiment, we achieved 69.75% accuracy for the valence space and 70.00% accuracy for the arousal space after fusion, each of which has surpassed the highest performing single modality (69.28% for the valence space and 64.00% for the arousal space). The results suggest that the combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources. The novelty of this work is as follows. To begin with, we combined facial expression and EEG to improve the performance of emotion recognition. Furthermore, we used transfer learning techniques to tackle the problem of lacking data and achieve higher accuracy for facial expression. Finally, in addition to implementing the widely used fusion method based on enumerating different weights between two models, we also explored a novel fusion method, applying boosting technique.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Through the above optimization methods, the model proposed in this paper outperforms other similar studies mentioned above in the classification tasks of DEAP and MAHNOB-HCI datasets.”
From Multimodal Emotion Recognition From EEG Signals and Facial Expressions (2023) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 2×
“In all convolutional and fully connected layers, the rectified linear unit (ReLU) non-linearity approach was applied [18].”
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
Measuring emotion: The self-assessment manikin and the semantic differential
Bradley, M. M., & Lang, P. J. (1994) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(94)90063-9 · Cited by 9,538 · cited 4×
“The CNN network for multitask classification; “DO” indicates that the layers drop out [15].”
InfluentialHow transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093 · cited 3×
“, DEAP and MAHNOB-HCI) to test our models [13,14].”
What are emotions? And how can they be measured?
Scherer, K. R. (2005) · Social Science Information · doi:10.1177/0539018405058216 · Cited by 4,264 · cited 1×
“Psychologists have proposed various emotion representation models, including discrete and dimensional models [2].”
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335
Research works printed but not linked: 1
- Patel, 2017, Emotion recognition from speech with gaussian mixture models & via boosted gmm, Int. J. Res. Sci. Eng., 3, 47
Cited by
Items 101–124 of 124
Cao, Y., Yang, H., Xue, Y., Wang, F., Li, T., Zhao, L., & Fu, Y. (2026) · Cognitive Neurodynamics · doi:10.1007/s11571-025-10380-5 · Cited by 2
“In the study by Huang et al. (2019 ), facial expressions and EEG were combined, employing a multi-task Convo - lutional Neural Network (CNN) architecture and a SVM classifier for multimodal emotion recognition, with optimi - zation performed through two decision-level fusion meth - ods.”
High Order Frequency Features as Emotion Discriminators
Massaranduba, A. B. R., Coelho, B. F. O., Sampaio, L. R., & Ramos, R. P. (2021) · E-Health and Bioengineering Conference (EHB) · doi:10.1109/ehb52898.2021.9657606 · Cited by 1
A Robust Ensemble Approach to Face Expression Recognition and Image Sentiment Analysis
Moung, E. G., Wooi, C. C., Sufian, M. M., Dargham, J. A., & Khoo, J. J. D. (2024) · Internet of things · doi:10.1007/978-981-97-1432-2_12 · Cited by 1
Multimodal Emotion Recognition Method Based on MAT-MER
Qu, H., Yang, X., Shangguan, F., & Yang, H. (2024) · International Conference on Intelligent Computing and Signal Processing (ICSP) · doi:10.1109/icsp62122.2024.10743748 · Cited by 1
“Koelst et al. [11] 83.23 83.71 78.56 79.31 Huang et al. [12] 85.28 85.14 81.31 81.19 WU X et al. [13] 88.31 87.95 84.62 85.34 LI et al. [14] 90.11 90.43 88.84 89.07 Fang et al. [15] 91.94 91.49 89.36 90.17”
InfluentialKumar, J. L. M., Rashid, M., Musa, R. M., Razman, M. A. M., Sulaiman, N., Jailani, R., & Majeed, A. P. P. A. (2021) · Lecture notes in electrical engineering · doi:10.1007/978-981-33-4597-3_6
Zhang, S., Zheng, X., Zhang, M., Guo, G., & Ji, C. (2021) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-92635-9_36 · Cited by 1
Facial Emotion Recognition using Deep Learning: Advancements, Challenges, and Future DirectionsPreprint
Kaur, N., & Kaur, K. (2023) · Research Square · doi:10.21203/rs.3.rs-3244446/v1 · Cited by 1
Visibility Convolutional Neural Networks for Label Distribution Learning of Facial Emotions
Almowallad, A., & Sánchez, V. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3693351
“The basic emotions considered in these studies are: happiness, sadness, anger, fear, surprise, and disgust .”
Zhang, T., Dahlan, H. A., Xie, Z., Jin-feng, W., Chen, Y., Pan, Q., & Huang, Y. (2024) · Applied Mathematics and Nonlinear Sciences · doi:10.2478/amns-2024-2156 · Cited by 2
Self-citationEnhancing Sentiment Analysis Accuracy Through Multimodal Data Fusion: A Deep Learning Approach
Tiwari, S., Jain, A., & Mishra, A. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-6726-7_45
Multimodal Anti-Spoofing Emotion Recognition Using GRU and BERT
Ge, C., Peng, G., Guo, A., Zhu, W., Fan, Z., & Zhu, X. (2024) · doi:10.1109/ictech63197.2024.00096
A Hybrid Approach to Facial Emotion Recognition: Integrating AKAZE Algorithm and Deep Learning
Kaur, N., & Kaur, K. (2025) · Journal of The Institution of Engineers (India) Series B · doi:10.1007/s40031-025-01232-1 · Cited by 1
“Based on DL, there are various approaches available for recognizing the behavior of individuals such as electroencephalogram (EEG) signals, gesture, text, FER [2–17].”
Johar, K. P., & Maruthupermal, S. (2025) · International Conference on Inventive Research in Computing Applications (ICIRCA) · doi:10.1109/icirca65293.2025.11089897
“Thus, emotions are a crucial part of human experience and significantly influence how people understand their surroundings [7].”
Emotion recognition through integrated dual-channel EEG and peripheral physiological data
Kale, P., Srivastava, A., Roy, S., Band, S. M., Bhavsar, P., Ronickom, J. F. A., & Kumar, D. (2025) · International Journal of Advances in Engineering Sciences and Applied Mathematics · doi:10.1007/s12572-025-00411-z
“By leveraging diverse signal types to mutually support and fuse complementary information, significant improvements in recognition outcomes can be achieved [49].”
Fatima, T., Venkateswarlu, C. H., Allakonda, M., & Shiva, B. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-7477-0_50
Facial emotion recognition based on ResNet18 with multi-dimensional attention mechanisms
Yang, X., Wu, C., Meng, T., & Li, C. (2025) · Memetic Computing · doi:10.1007/s12293-025-00476-0
EEG- and Micro-Expression-Based Emotion Recognition and Consciousness Detection
Chen, Z., Liang, Y., Li, J., Bai, C., Xie, Q., & Pan, J. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3650482 · Cited by 3
“We benchmarked the STST model on MAHNOB-HCI against 6 representative models, including NBM [81], CNN+SVM [82], CNN+LSTM [83], AM-LSTM [84], EEG-PSD+LSTM [85], and HSAttentionNet [17].”
Self-citationYao, J., Tian, J., & Zhang, H. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356601
“Early attempts at joint video–EEG modeling include decision-level fusion [23, 24], dense graph convolution with joint cross-attention [25, 26], and the addition of micro-expression together with EEG and other physiological indicators like GSR and PPG [27, 28].”
Alumnus, A. S., Alumnus, N. K., Alumnus, S. J. R., & Jayashree, M. J. (2025) · International Conference on Intelligent Computing, Information and Control Systems (ICOIICS) · doi:10.1109/icoiics67115.2025.11390230
“[1] adopted the multi-model Emotion Recognition (ER) framework by merging EEG and facial expression data.”
Zha, L., Feng, Y., & Liang, Z. (2026) · Journal of Visualized Experiments · doi:10.3791/70600
Multimodal Sentiment Recognition via Residual‐Contraction Convolution and Cross‐Attention
Liu, F., Jiang, S., & Su, N. (2026) · IET Image Processing · doi:10.1049/ipr2.70370
Identifying Anger Emotion Using BVP Sensor: Heart Rate Variability (HRV)
Dehghanizadeh, Z., Dolatshahi, B., Nosratabadi, M., & Moradi, H. (2026) · Iranian Journal of Psychiatry and Behavioral Sciences · doi:10.5812/ijpbs-162423
Wen, S., Liu, Q., Wang, X., Chen, K., Ju, J., Ding, J., Min, B., & Zhao, X.-M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6876239
Song, Q., Zhang, G., Lyu, C., Xie, X., Yao, N., Chen, H., & Tian, F. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831177
Metadata
- Authors
- Huang, Y., Yang, J., Liu, S., & Pan, J.
- Affiliations
- South China Normal University (CN)
- Corresponding author
- Pan, Jiahui
- Venue
- Future Internet
- Publisher
- MDPI AG
- Published
- 2019-05-02
- Volume
- 11
- Issue
- 5
- Pages
- 105
- ISSN
- 1999-5903 (ISSN-L)
- OpenAlex
- W2943183446
- Cited by
- 124
- References
- 37
- Citations by year
- 2026: 9 · 2025: 23 · 2024: 28 · 2023: 16 · 2022: 21 · 2021: 16 · 2020: 10 · 2019: 1
- FWCI
- 14.13
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Future Internet (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Natural Science Foundation of China (61503143)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- emotion recognition, facial expression recognition, EEG, multimodal fusion, valence-arousal model, transfer learning, convolutional neural networks, support vector machine, decision fusion, ensemble boosting, DEAP dataset, physiological signals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.