Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Emotion recognition plays an essential role in human–computer interaction. Previous studies have investigated the use of facial expression and electroencephalogram (EEG) signals from single modal for emotion recognition separately, but few have paid attention to a fusion between them. In this paper, we adopted a multimodal emotion recognition framework by combining facial expression and EEG, based on a valence-arousal emotional model. For facial expression detection, we followed a transfer learning approach for multi-task convolutional neural network (CNN) architectures to detect the state of valence and arousal. For EEG detection, two learning targets (valence and arousal) were detected by different support vector machine (SVM) classifiers, separately. Finally, two decision-level fusion methods based on the enumerate weight rule or an adaptive boosting technique were used to combine facial expression and EEG. In the experiment, the subjects were instructed to watch clips designed to elicit an emotional response and then reported their emotional state. We used two emotion datasets—a Database for Emotion Analysis using Physiological Signals (DEAP) and MAHNOB-human computer interface (MAHNOB-HCI)—to evaluate our method. In addition, we also performed an online experiment to make our method more robust. We experimentally demonstrated that our method produces state-of-the-art results in terms of binary valence/arousal classification, based on DEAP and MAHNOB-HCI data sets. Besides this, for the online experiment, we achieved 69.75% accuracy for the valence space and 70.00% accuracy for the arousal space after fusion, each of which has surpassed the highest performing single modality (69.28% for the valence space and 64.00% for the arousal space). The results suggest that the combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources. The novelty of this work is as follows. To begin with, we combined facial expression and EEG to improve the performance of emotion recognition. Furthermore, we used transfer learning techniques to tackle the problem of lacking data and achieve higher accuracy for facial expression. Finally, in addition to implementing the widely used fusion method based on enumerating different weights between two models, we also explored a novel fusion method, applying boosting technique.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Through the above optimization methods, the model proposed in this paper outperforms other similar studies mentioned above in the classification tasks of DEAP and MAHNOB-HCI datasets.”
From Multimodal Emotion Recognition From EEG Signals and Facial Expressions (2023) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 2×
“In all convolutional and fully connected layers, the rectified linear unit (ReLU) non-linearity approach was applied [18].”
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
Measuring emotion: The self-assessment manikin and the semantic differential
Bradley, M. M., & Lang, P. J. (1994) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(94)90063-9 · Cited by 9,538 · cited 4×
“The CNN network for multitask classification; “DO” indicates that the layers drop out [15].”
InfluentialHow transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093 · cited 3×
“, DEAP and MAHNOB-HCI) to test our models [13,14].”
What are emotions? And how can they be measured?
Scherer, K. R. (2005) · Social Science Information · doi:10.1177/0539018405058216 · Cited by 4,264 · cited 1×
“Psychologists have proposed various emotion representation models, including discrete and dimensional models [2].”
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335
Research works printed but not linked: 1
- Patel, 2017, Emotion recognition from speech with gaussian mixture models & via boosted gmm, Int. J. Res. Sci. Eng., 3, 47
Cited by
Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion
Cimtay, Y., Ekmekcioglu, E., & Caglar-Ozhan, S. (2020) · IEEE Access · doi:10.1109/access.2020.3023871 · Cited by 200
Advances in Multimodal Emotion Recognition Based on Brain–Computer Interfaces
He, Z., Li, Z., Yang, F., Wang, L., Li, J., Zhou, C., & Pan, J. (2020) · Brain Sciences · doi:10.3390/brainsci10100687 · Cited by 151
Self-citationMultimodal Emotion Recognition From EEG Signals and Facial Expressions
Wang, S., Qu, J., Zhang, Y., & Zhang, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3263670 · Cited by 124
“For example, Huang et al. [31] proposed a multimodal emotion recognition framework combining EEG and facial expressions.”
BackgroundMethodUsing transformers for multimodal emotion recognition: Taxonomies and state of the art review
Hazmoune, S., & Bougamouza, F. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108339 · Cited by 169
Salama, E. S., El-Khoribi, R. A., Shoman, M. E., & Wahby Shalaby, M. A. (2021) · Egyptian Informatics Journal · doi:10.1016/j.eij.2020.07.005 · Cited by 117
Tan, C. N. W., Šarlija, M., & Kasabov, N. K. (2021) · Neurocomputing · doi:10.1016/j.neucom.2020.12.098 · Cited by 127
Emotion recognition based on convolutional neural networks and heterogeneous bio-signal data sources
Ngai, W. K., Xie, H., Zou, D., & Chou, K. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.007 · Cited by 108
Biosignal-Based Human–Machine Interfaces for Assistance and Rehabilitation: A Survey
Esposito, D., Centracchio, J., Andreozzi, E., Gargiulo, G. D., Naik, G. R., & Bifulco, P. (2021) · Sensors · doi:10.3390/s21206863 · Cited by 103
Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network
Zhang, Y., Cheng, C., & Zhang, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3049516 · Cited by 84
“The expression of emotions includes not only some physiological responses, such as skin temperature and heart rate, but also some nonphysiological responses, such as facial expressions and body language [1].”
BackgroundMultimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
Udahemuka, G., Djouani, K., & Kurien, A. M. (2024) · Applied Sciences · doi:10.3390/app14178071 · Cited by 140
“AdaBoost was employed to develop a fused method in which an SVM was used for processing the PSD features from EEG data, and the face data were processed using a CNN [217].”
A Review of Generalizable Transfer Learning in Automatic Emotion Recognition
Feng, K., & Chaspari, T. (2020) · Frontiers in Computer Science · doi:10.3389/fcomp.2020.00009 · Cited by 90
“…knowledge transfer in multi-modal methods include fine-tune well trained models to a specific type of signal (Vielzeuf et al., 2017; Yan et al., 2018; Huang et al., 2019; Ortega et al., 2019), or fine-tune different well-trained models to both speech and video signals (Ouyang et al., 2017; Zhang et…”
MethodA survey of multidisciplinary domains contributing to affective computing
Arya, R., Singh, J., & Kumar, A. P. (2021) · Computer Science Review · doi:10.1016/j.cosrev.2021.100399 · Cited by 93
Multimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70
Saffaryazdi, N., Wasim, S. T., Dileep, K., Nia, A. F., Nanayakkara, S., Broadbent, E., & Billinghurst, M. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.864047 · Cited by 69
“Later, they extended their work by improving facial expression recognition using a CNN model (Huang et al., 2019).”
BackgroundSingle-Channel Selection for EEG-Based Emotion Recognition Using Brain Rhythm Sequencing
Li, J. W., Barma, S., Mak, P. U., Chen, F., Li, C., Li, M. T., Vai, M. I., & Pun, S. H. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3148109 · Cited by 69
The Fusion of Electroencephalography and Facial Expression for Continuous Emotion Recognition
Li, D., Wang, Z., Wang, C., Liu, S., Chi, W., Dong, E., Song, X., Gao, Q., & Song, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2949707 · Cited by 63
“[8] performed binary classification of the statues of valence and arousal by combining the EEG and facial expressions.”
BackgroundMethodSelf-citationA Survey on Automatic Multimodal Emotion Recognition in the Wild
Sharma, G., & Dhall, A. (2020) · Intelligent systems reference library · doi:10.1007/978-3-030-51870-7_3 · Cited by 76
Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression
Sun, Y., Ayaz, H., & Akansu, A. N. (2020) · Brain Sciences · doi:10.3390/brainsci10020085 · Cited by 57
Driver Emotions Recognition Based on Improved Faster R-CNN and Neural Architectural Search Network
Zaman, K., Sun, Z., Shah, S. M., Shoaib, M. A., Pei, L., & Hussain, A. (2022) · Symmetry · doi:10.3390/sym14040687 · Cited by 70
Zhang, Y., Cheng, C., Wang, S., & Xia, T. (2022) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2022.103877 · Cited by 63
DRER: Deep Learning–Based Driver’s Real Emotion Recognizer
Oh, G., Ryu, J., Jeong, E., Yang, J. H., Hwang, S., Lee, S., & Lim, S. (2021) · Sensors · doi:10.3390/s21062166 · Cited by 63
“Owing to the lack of fusion between EEG and facial expressions, they applied a transfer learning approach for multi-task CNN architectures in a later study [42].”
MethodSelf-citationTan, C., Ceballos, G., Kasabov, N., & Puthanmadam Subramaniyam, N. (2020) · Sensors · doi:10.3390/s20185328 · Cited by 51
“Huang and colleagues obtained 50.57% for valence classification using appearance descriptors based facial features (Local binary pattern from three orthogonal planes, LBP-TOP) and 66.28% using fusion it with global EEG features [119].”
MethodInfluentialBindi: Affective Internet of Things to Combat Gender-Based Violence
Miranda, J., Rituerto-González, E., Luis-Mingueza, C., Canabal, M. F., Barcenas, A. R., Lanza-Gutiérrez, J. M., Peláez-Moreno, C., & López-Ongil, C. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3177256 · Cited by 59
“That is The three fusion strategies are based on the literature (e.g., [61]) and are proposed as a tradeoff between low computational complexity and robustness considering the confidence of the system in the predictions.”
BackgroundMethodVempati, R., & Sharma, L. D. (2023) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2023.109879 · Cited by 43
Recent Advances in Fatigue Detection Algorithm Based on EEG
Wang, F., Wan, Y., Li, M., Huang, H., Li, L., Hou, X., Pan, J., Wen, Z., & Li, J. (2022) · Intelligent Automation & Soft Computing · doi:10.32604/iasc.2023.029698 · Cited by 43
Self-citationDense Graph Convolutional With Joint Cross-Attention Network for Multimodal Emotion Recognition
Cheng, C., Liu, W., Feng, L., & Jia, Z. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3412074 · Cited by 45
Automated Facial Expression Recognition Framework Using Deep Learning
Saeed, S., Shah, A. A., Ehsan, M. K., Amirzada, M. R., Mahmood, A., & Mezgebo, T. (2022) · Journal of Healthcare Engineering · doi:10.1155/2022/5707930 · Cited by 50
“Padding can be categorized as valid padding or same padding [33].”
BackgroundMuhammad, F., Hussain, M., & Aboalsamh, H. (2023) · Diagnostics · doi:10.3390/diagnostics13050977 · Cited by 36
“Huang et al. [75] proposed two decision-level fusion approaches based on the enumerate weight rule and an adaptive boosting technique to combine facial expression and EEG data.”
Pradhan, A., & Srivastava, S. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.104624 · Cited by 36
Body Language Analysis in Healthcare: An Overview
Abdulghafor, R. A. A., Turaev, S., & Ali, M. A. H. (2022) · Healthcare · doi:10.3390/healthcare10071251 · Cited by 55
“[71] Combining Facial Expressions and Electroencephalog-raphy to Enhance Emotion Recognition • To recognize emotions by compounding EEG and facial expressions.”
BackgroundMultimodal EEG Emotion Recognition Based on the Attention Recurrent Graph Convolutional Network
Chen, J., Liu, Y., Xue, W., Hu, K., & Lin, W. (2022) · Information · doi:10.3390/info13110550 · Cited by 34
“Huang et al. [17] proposed a decision-level-fusion method based on Huang et al. [17] proposed a decision-level-fusion method based on the enumerated weight rules for the classification of facial expression and EEG. the enumerated weight rules for the classification of facial expression and EEG.”
MethodTian, F., Zhang, L., Zhu, L., Zhao, M., Liu, J., Dong, Q., & Zhao, Q. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3420445 · Cited by 33
Ensemble-based face expression recognition approach for image sentiment analysis
Moung, E. G., Wooi, C. C., Sufian, M. M., On, C. K., & Dargham, J. A. (2022) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v12i3.pp2588-2600 · Cited by 37
Brain Computer Interface based EEG for Emotion Recognition System: A Systematic Review
Bhise, P. R., Kulkarni, S. B., & Aldhaheri, T. A. (2020) · International Conference on Innovative Mechanisms for Industry Applications · doi:10.1109/icimia48430.2020.9074921 · Cited by 34
DeepVANet: A Deep End-to-End Network for Multi-modal Emotion Recognition
Zhang, Y., Hossain, M. Z., & Rahman, S. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-85613-7_16 · Cited by 29
Multi-modal emotion identification fusing facial expression and EEG
Wu, Y., & Li, J. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13711-4 · Cited by 34
“Huang [37] proposed a new decision-level fusion strategy based on adaptive enhancement, which achieved a fusion identification accuracy of 78.”
BackgroundMethodLi, J., & Peng, J. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3430310 · Cited by 40
“Thus, multimodal fusion has been used to recognizeemotionsinsomestudies,suchasfusingphysiological signalswithfacialexpressions[15],[16],[17].”
MethodOnline multi-hypergraph fusion learning for cross-subject emotion recognition
Pan, T., Ye, Y., Zhang, Y., Xiao, K., & Cai, H. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102338 · Cited by 32
Emotion recognition using cross-modal attention from EEG and facial expression
Cui, R., Chen, W., & Li, M. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112587 · Cited by 32
Singh, J., & Sharma, D. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17504-1 · Cited by 29
“…expense of data gathering involving human subjects and the ongoing development of experimental guidelines for multiple scenarios [25, 62, 92, 124, 125] . g) Validation Several research categories remain at the evidence demonstrating stage due to limited sample sizes and the lack of adequate…”
Multimodal Physiological Signals Fusion for Online Emotion Recognition
Pan, T., Ye, Y., Cai, H., Huang, S., Yang, Y., & Wang, G. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612555 · Cited by 36
“According to signal types, these methods can be divided into physical signals (such as facial expression or speech) based methods [19, 26, 46], physiological signals (such as ECG and EEG) based methods [1, 9, 35], and hybrid modalities signals based methods [18, 21, 44].”
MethodEmotion Recognition Based on EEG Brain Rhythm Sequencing Technique
Li, J., Barma, S., Pun, S. H., Vai, M. I., & Mak, P. U. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2022.3149953 · Cited by 24
FCAN–XGBoost: A Novel Hybrid Model for EEG Emotion Recognition
Zong, J., Xiong, X., Zhou, J., Ji, Y., Zhou, D., & Zhang, Q. (2023) · Sensors · doi:10.3390/s23125680 · Cited by 27
“This recognition strategy results in a lack of reliability [10].”
BackgroundJafari, M., Tao, X., Barua, P. D., Tan, R., & Acharya, U. R. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102982 · Cited by 32
WEMAC: Women and Emotion Multi-modal Affective Computing dataset
Miranda, J., Gutiérrez-Martín, L., Rituerto-González, E., Romero, E., Lanza-Gutiérrez, J. M., Peláez-Moreno, C., & López-Ongil, C. (2024) · Scientific Data · doi:10.1038/s41597-024-04002-8 · Cited by 35
Ma, Z., Li, A., Tang, J., Zhang, J., & Yin, Z. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110004 · Cited by 25
Saffaryazdi, N., Gunasekaran, T. S., Loveys, K., Broadbent, E., & Billinghurst, M. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2540500 · Cited by 38
Subject independent emotion recognition using EEG and physiological signals – a comparative study
Ramaswamy, M. P. A., & Palaniswamy, S. (2022) · Applied Computing and Informatics · doi:10.1108/aci-03-2022-0080 · Cited by 28
Li, J., Lin, D., Che, Y., Lv, J. J., Chen, R., Wang, L. J., Zeng, X. X., Ren, J. C., Zhao, H. M., & Lü, X. C. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1221512 · Cited by 19
Emotion Fusion-Sense (Emo Fu-Sense) – A novel multimodal emotion classification technique
Umair, M., Rashid, N., Khan, U. S., Hamza, A., & Iqbal, J. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106224 · Cited by 23
Toward the Construction of Affective Brain-Computer Interface: A Systematic Review
Chen, H., Li, J., He, H., Zhu, J., Sun, S., Li, X., & Hu, B. (2025) · ACM Computing Surveys · doi:10.1145/3712259 · Cited by 26
Attention-based 3D convolutional recurrent neural network model for multimodal emotion recognition
Du, Y., Li, P., Cheng, L., Zhang, X., Li, M., & Li, F. (2024) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1330077 · Cited by 21
Hierarchical fusion of visual and physiological signals for emotion recognition
Fang, Y., Rong, R., & Huang, J. (2021) · Multidimensional Systems and Signal Processing · doi:10.1007/s11045-021-00774-z · Cited by 17
Hidden Emotion Detection using Multi-modal Signals
Song, B. C., & Kim, D. H. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3451721 · Cited by 22
“However, since emotions are infuenced by various environmental factors, there may be a stronger emotional change in the inner state than in the facial expression, or vice versa [11].”
BackgroundDemystifying Mental Health by Decoding Facial Action Unit Sequences
Sharma, D., Singh, J., Sehra, S. S., & Sehra, S. K. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8070078 · Cited by 20
Fast facial expression recognition using Boosted Histogram of Oriented Gradient (BHOG) features
Saurav, S., Saini, R., & Singh, S. K. (2022) · Pattern Analysis and Applications · doi:10.1007/s10044-022-01112-0 · Cited by 16
“Techniques also exist for emotion recognition that uses combination of various physiological signals such as RSP and HRV [6], audio and visual [7], and EEG and visual signals [8, 9].”
BackgroundKhan, M. R., & Ahmad, M. (2024) · International Conference on Advances in Computing, Communication, Electrical, and Smart Systems (iCACCESS) · doi:10.1109/icaccess61735.2024.10499496 · Cited by 17
Sedehi, J. F., Dabanloo, N. J., Maghooli, K., & Sheikhani, A. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e36411 · Cited by 17
Hosseini, I., Hossain, M. Z., Zhang, Y., & Rahman, S. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104121 · Cited by 16
Gagliardi, G., Alfeo, A. L., Catrambone, V., Cimino, M. G. C. A., Vos, M. D., & Valenza, G. (2023) · International IEEE/EMBS Conference on Neural Engineering · doi:10.1109/ner52421.2023.10123758 · Cited by 12
“From a methodological perspective, several AI methodologies have been successfully applied to classify emotion perception from multimodal electrophysiological signals: e.g., convolutional neural networks (CNN), both with standard [8] or with transfer-learning methodologies [9].”
BackgroundDeterminants of Emotion Recognition System Adoption: Empirical Evidence from Malaysia
Yamin, M. N. M., Aziz, K. A., Siang, T. G., & Aziz, N. A. A. (2023) · Applied Sciences · doi:10.3390/app132111854 · Cited by 14
Research on driver’s anger recognition method based on multimodal data fusion
Sun, W., Liu, Y., Li, S., Tian, J., Wang, F., & Liu, D. (2024) · Traffic Injury Prevention · doi:10.1080/15389588.2023.2297658 · Cited by 12
“And in the study of driver emotion recognition, Huang et al. (2019) used a The study by Hieida et al. (2023) was able to successfully identify 74% of emotions.”
MethodPersian emotion elicitation film set and signal database
Shalchizadeh, F., Shamekhi, S., Sadeh, R. N., & Darvish, A. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.103290 · Cited by 12
Student's Emotion Recognition using Multimodality and Deep Learning
Kalaiyarasi, M., Prasad, B. V. V. S., Ramesh, J. V. N., Kushwaha, R. K., Patel, R., & Balajee, J. M. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3654797 · Cited by 10
“Facial expressions and EEG data are taken into consideration by a multimodal emotion identification paradigm [13].”
MethodMulti-Modal Emotion Recognition Combining Face Image and EEG Signal
Hu, Y., & Wang, F. (2022) · Journal of Circuits Systems and Computers · doi:10.1142/s0218126623501256 · Cited by 10
Muñoz, M. S., Torres, C. E. S., Salazar-Cabrera, R., López, D. M., & Vargas-Cañas, R. (2022) · Sustainability · doi:10.3390/su141811420 · Cited by 9
Shi, H., Chen, J., Feng, Z., Liu, T., Sun, D., & Zhou, X. (2025) · Buildings · doi:10.3390/buildings15030310 · Cited by 8
Real-time Engagement Detection from Facial Features
Marqués-Villarroya, S., Gamboa‐Montero, J. J., Bernardino, A. J. M., Gómez, M. M., Castillo, J. C., & Salichs, M. A. (2022) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/icdl53763.2022.9962228 · Cited by 10
A Reasonable Effectiveness of Features in Modeling Visual Perception of User Interfaces
Bakaev, M., Heil, S., & Gaedke, M. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7010030 · Cited by 10
EEG-based fusion approaches in multimodal emotion recognition: An in-depth review
Yılmaz, B. H., Yılmaz, Ç. M., & Köse, C. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.132235 · Cited by 14
Bamonte, M. F., Risk, M. R., & Herrero, V. (2024) · Electronics · doi:10.3390/electronics13163333 · Cited by 10
Personalized Clustering for Emotion Recognition Improvement
Gutiérrez-Martín, L., López-Ongil, C., Lanza-Gutiérrez, J. M., & Miranda, J. (2024) · Sensors · doi:10.3390/s24248110 · Cited by 9
“They are based on the literature [41] and are proposed as a trade-off between low computational complexity and robustness, considering the confidence of the personalized and general system in the predictions.”
Lixun, X. I. E., Sun, W., Zhang, J., & Zhao, X. (2025) · Digital Signal Processing · doi:10.1016/j.dsp.2025.105261 · Cited by 9
Yu, J., Ru, Y., Lei, B., & Chen, H. (2025) · Sensors · doi:10.3390/s25206397 · Cited by 11
“Unlike the static fusion strategies adopted by Huang [15] and Li et al. [30] for multimodal data, the present study introduces a dynamic gated attention mechanism.”
InfluentialResearch on emotion recognition of bimodal bioelectrical features based on D-S evidence theory
Qin, X., Yang, D., Pan, S., & Yuan, Q. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105538 · Cited by 8
SVD-Guided Multimodal Feature Fusion for Emotion Recognition From Facial Videos
Bao, J., Qian, J., & Yang, J. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3528636 · Cited by 8
“Additionally, LWF [49], DLF [5], MLK [50] and SVM-RFE [51] predict emotion results by combining the facial expression and physiological signals.”
Self-citationThe research of emotion recognition based on multi-source physiological signals with data fusion
Han, Y., & Xu, Y. (2022) · ITM Web of Conferences · doi:10.1051/itmconf/20224501038 · Cited by 5
“Huang et al.(2019) selected EEG data and expression data in the study, applied SVM and convolutional neural network(CNN) to classify and identify emotions respectively, and finally compared the application of enumeration weight method and adaptive enhancement method for decision-making level data…”
MethodA Multimodal Approach to Psycho-Emotional State Detection of a Vehicle Driver
Lashkov, I., & Kashevnik, A. M. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-82196-8_42 · Cited by 5
Affection of facial artifacts caused by micro-expressions on electroencephalography signals
Zeng, X., Zhao, X., Wang, S., Qin, J., Xie, J., Zhong, X., Chen, J., & Liu, G. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.1048199 · Cited by 5
“Therefore, researches on emotion recognition based on macro-expressions and EEG signals have attracted significant attention (Matlovic et al., 2016; Huang et al., 2019).”
BackgroundOQCNN: optimal quantum convolutional neural network for classification of facial expression
Sathya, T., & Sudha, S. (2023) · Neural Computing and Applications · doi:10.1007/s00521-022-08161-w · Cited by 9
“The state-of-the-art techniques often suffer from different complexities such as the use of a huge number of hyper-parameters, complex procedures to acquire the knowledge of pre-trained architectures, and complex loss functions [10].”
BackgroundElectroencephalography Emotion Recognition Based on Rhythm Information Entropy Extraction
Liu, Z., Xu, X. J., She, J., Yang, Z., & Chen, D. (2024) · Journal of Advanced Computational Intelligence and Intelligent Informatics · doi:10.20965/jaciii.2024.p1095 · Cited by 4
Massaranduba, A. B. R., Coelho, B. F. O., Souza, C. A. D. S., Viana, G. G., Brys, I., & Ramos, R. P. (2025) · Current Psychology · doi:10.1007/s12144-025-07337-6 · Cited by 5
“In addition, EEG signals contain relevant informa - tion in various frequency bands, such as alpha (8 to 13 Hz), beta (14 to 30 Hz), and gamma (above 30 Hz) (Huang et al., 2019).”
Affect Display Recognition Through Tactile and Visual Stimuli in a Social Robot
Marqués-Villarroya, S., Gamboa‐Montero, J. J., Jumela-Yedra, C., Castillo, J. C., & Salichs, M. A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-24667-8_12 · Cited by 5
de Oliveira, W. C., Gottardo, E., & Pimentel, A. R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-49663-0_8 · Cited by 4
A shared-private sentiment analysis approach based on cross-modal information interaction
Hou, Y., Zhong, X., Cao, H., Zhu, Z., Zhou, Y., & Zhang, J. (2024) · Pattern Recognition Letters · doi:10.1016/j.patrec.2024.05.009 · Cited by 4
Affective computing–driven virtual rehabilitation: a systematic survey
Zhao, L., Liu, D., Zhang, H., Yang, G., Zheng, D., Yan, J., & He, L. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00442-3 · Cited by 5
“Similarly, Huang et al. (Huang et al. 2019) applied linear SVM to text and audio data, demonstrating efficiency in simple tasks.”
Vietoris, V., Berčík, J., Veverka, M., Chhetri, G., Dvořák, M., & Korčok, M. (2025) · Applied Food Research · doi:10.1016/j.afres.2025.101217 · Cited by 4
A PROPOSED SIGN LANGUAGE MODEL FOR SPEECHLESS PERSONS USING EEG SIGNALS
Al-Anbary, A. H., & Al-Qaraawi, S. M. (2021) · Iraqi Journal of Information & Communications Technology · doi:10.31987/ijict.4.3.148 · Cited by 3
“The advances in machine learning and the availability of huge EEG data sets led to deep learning deployment in analyzing EEG signals and in the field of understanding brain functionality by defining collected information inside it [17], [18], [22]-[25].”
BackgroundUsing Multimodal Transformers in Affective Computing
Vazquez-Rodriguez, J. (2021) · International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) · doi:10.1109/aciiw52867.2021.9666396 · Cited by 4
Label Feature Co-Learning for Facial and EEG Emotion Recognition
Cai, M., Chen, C. L. P., Li, S., & Zhang, T. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3600267 · Cited by 3
“Baselines include traditional methods (SVM [55] and MLP [56]), CNN-based methods (Multi-task CNN [7], TCMA [57], DST-EIF [17]), GCN-based methods (UAGCFNet [43]), Transformer-based methods (UBVMT [58], Husformer [42]), and hybrid-architecture methods (PreVGG-LSTM [8], MindLink-Eumpy [9], MER-ESFE…”
Step-Wise Prompting Meets Uncertainty-Aware Dynamic Fusion for Robust EEG-Visual Emotion Recognition
Hao, D., Meng, M., Gao, Y., Lou, X., & Kong, W. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3632304 · Cited by 3
“Earlier methods, such as those by Huang et al. [51] and Jung et al. [52], achieve moderate performance, whereas more recent approaches [6], [53], [54] have achieved accuracy levels close to 98%.”
Transformer-Based Bimodal Emotion Recognition: Signal Measurement and Transformer Fusion Approach
Soleimani, S., & Osman, H. A. (2025) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2025.3615268 · Cited by 5
“The bimodal model presented in [82] shows a more reliable results under the enumerate fusion method; however, the proposed model achieves relatively higher train and test accuracy for both valence and arousal dimensions.”
InfluentialHuman States and Nonverbal Cues in Multi-party Facilitation: A Statistical Perspective
Chew, J. Y., & Jayarathne, I. (2024) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3610978.3640643 · Cited by 2
“Some studies employed fusion techniques, combining NOC and EEG data, to improve the identifcation of internal states in individuals [14, 15].”
MethodFacial Emotion Detection: A Comprehensive Survey
Prashanth, C. M., Lakshmi, D. S., Gangadhar, M. S., Swathi, K., & Rao, V. S. (2024) · International Conference on Cognitive Robotics and Intelligent Systems (ICC-ROBINS) · doi:10.1109/icc-robins60238.2024.10533999 · Cited by 2
Model-driven validation of visual explanations for multimodal emotion recognition
Gagliardi, G., Alfeo, A. L., Catrambone, V., Cimino, M. G. C. A., Vos, M. D., & Valenza, G. (2025) · Machine Learning · doi:10.1007/s10994-025-06921-y · Cited by 4
“3 IG explanation has been selected as optimal by the QT assessment while GC is the least optimal similar to the one reported in the literature (Huang et al., 2019 ), the BHI features consistently outperformed them across all classification tasks.”
Marandi, A. K., Jethava, G., RAJESH, A., Gupta, S. K., Sagar, S., & Sharma, S. (2022) · International Journal of Communication Networks and Information Security (IJCNIS) · doi:10.17762/ijcnis.v14i2.5462 · Cited by 2
Yin, X., & Chen, L. (2023) · International Journal of Data Warehousing and Mining · doi:10.4018/ijdwm.333854 · Cited by 2
“In this situation, multimodal sentiment analysis has emerged, which will extract and fuse features from multiple modalities of information published by users, thereby more accurately analyzing and predicting their emotions (Huang, Yang et al., 2019; Xiao et al., 2021).”
BackgroundKim, M. K. (2020) · Asian Journal of Beauty and Cosmetology · doi:10.20402/ajbc.2020.0040 · Cited by 2
Multimodal Emotion Recognition Model Using AI With the Aim of Addressing Gender-Based Violence
García-Hernández, R. A., Celaya-Padilla, J. M., & Luna-García, H. (2023) · Latin American conference on Human-Computer Interaction · doi:10.1145/3630970.3631065 · Cited by 1
“That is why current trends in emotion analysis have focused their attention in recent years on predicting emotions using physiological signals or mergers of physical and physiological signals, which promises to improve the performance and reliability of affective computing models [7],[11], [22].”
BackgroundDPMMN: A dual performer-multi-modal network for emotion recognition
Gornale, S. S., Shivakumara, P., Unki, A., & Vadera, S. (2025) · Signal Processing Image Communication · doi:10.1016/j.image.2025.117464 · Cited by 1
Metadata
- Authors
- Huang, Y., Yang, J., Liu, S., & Pan, J.
- Affiliations
- South China Normal University (CN)
- Corresponding author
- Pan, Jiahui
- Venue
- Future Internet
- Publisher
- MDPI AG
- Published
- 2019-05-02
- Volume
- 11
- Issue
- 5
- Pages
- 105
- ISSN
- 1999-5903 (ISSN-L)
- OpenAlex
- W2943183446
- Cited by
- 124
- References
- 37
- Citations by year
- 2026: 9 · 2025: 23 · 2024: 28 · 2023: 16 · 2022: 21 · 2021: 16 · 2020: 10 · 2019: 1
- FWCI
- 14.13
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Future Internet (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Natural Science Foundation of China (61503143)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- emotion recognition, facial expression recognition, EEG, multimodal fusion, valence-arousal model, transfer learning, convolutional neural networks, support vector machine, decision fusion, ensemble boosting, DEAP dataset, physiological signals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.