Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Emotion recognition plays an essential role in human–computer interaction. Previous studies have investigated the use of facial expression and electroencephalogram (EEG) signals from single modal for emotion recognition separately, but few have paid attention to a fusion between them. In this paper, we adopted a multimodal emotion recognition framework by combining facial expression and EEG, based on a valence-arousal emotional model. For facial expression detection, we followed a transfer learning approach for multi-task convolutional neural network (CNN) architectures to detect the state of valence and arousal. For EEG detection, two learning targets (valence and arousal) were detected by different support vector machine (SVM) classifiers, separately. Finally, two decision-level fusion methods based on the enumerate weight rule or an adaptive boosting technique were used to combine facial expression and EEG. In the experiment, the subjects were instructed to watch clips designed to elicit an emotional response and then reported their emotional state. We used two emotion datasets—a Database for Emotion Analysis using Physiological Signals (DEAP) and MAHNOB-human computer interface (MAHNOB-HCI)—to evaluate our method. In addition, we also performed an online experiment to make our method more robust. We experimentally demonstrated that our method produces state-of-the-art results in terms of binary valence/arousal classification, based on DEAP and MAHNOB-HCI data sets. Besides this, for the online experiment, we achieved 69.75% accuracy for the valence space and 70.00% accuracy for the arousal space after fusion, each of which has surpassed the highest performing single modality (69.28% for the valence space and 64.00% for the arousal space). The results suggest that the combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources. The novelty of this work is as follows. To begin with, we combined facial expression and EEG to improve the performance of emotion recognition. Furthermore, we used transfer learning techniques to tackle the problem of lacking data and achieve higher accuracy for facial expression. Finally, in addition to implementing the widely used fusion method based on enumerating different weights between two models, we also explored a novel fusion method, applying boosting technique.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Through the above optimization methods, the model proposed in this paper outperforms other similar studies mentioned above in the classification tasks of DEAP and MAHNOB-HCI datasets.”
From Multimodal Emotion Recognition From EEG Signals and Facial Expressions (2023) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 2×
“In all convolutional and fully connected layers, the rectified linear unit (ReLU) non-linearity approach was applied [18].”
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
Measuring emotion: The self-assessment manikin and the semantic differential
Bradley, M. M., & Lang, P. J. (1994) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(94)90063-9 · Cited by 9,538 · cited 4×
“The CNN network for multitask classification; “DO” indicates that the layers drop out [15].”
InfluentialHow transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093 · cited 3×
“, DEAP and MAHNOB-HCI) to test our models [13,14].”
What are emotions? And how can they be measured?
Scherer, K. R. (2005) · Social Science Information · doi:10.1177/0539018405058216 · Cited by 4,264 · cited 1×
“Psychologists have proposed various emotion representation models, including discrete and dimensional models [2].”
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335
Evidence for a three-factor theory of emotions
Russell, J. A., & Mehrabian, A. (1977) · Journal of Research in Personality · doi:10.1016/0092-6566(77)90037-x · Cited by 1,940 · cited 1×
“Valence ranges from unpleasant to pleasant, and arousal ranges from calm to excited, and can describe the intensity of emotion [4].”
Challenges in Representation Learning: A Report on Three Machine Learning Contests
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-42051-1_16 · Cited by 1,719
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664 · cited 3×
“The MAHNOBHCI dataset contained EEG, video, audio, gaze, and peripheral physiological recordings of 30 participants [13].”
Wellendorff, J., Lundgaard, K. T., Møgelhøj, A., Petzold, V., Landis, D. D., Nørskov, J. K., Bligaard, T., & Jacobsen, K. W. (2012) · Physical Review B · doi:10.1103/physrevb.85.235149 · Cited by 1,461
Understanding Neural Networks Through Deep Visualization
Yosinski, J., Clune, J., Nguyen, A. M., Fuchs, T. J., & Lipson, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06579 · Cited by 1,358 · cited 1×
“Our results are partially consistent with the findings of previous work [33,34].”
Ranjan, R., Patel, V. M., & Chellappa, R. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2781233 · Cited by 1,333
Identifying Stable Patterns over Time for Emotion Recognition from EEG
Zheng, W.-L., Zhu, J.-Y., & Lu, B.-L. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2712143 · Cited by 990 · cited 1×
“They systematically evaluated the performance of various popular feature extraction, feature selection, feature smoothing, and pattern classification methods with a Database for Emotion Analysis using Physiological Signals (DEAP) and a newly developed dataset called SJTU Emotion EEG Dataset (SEED) for this study [7].”
Challenges in representation learning: A report on three machine learning contests
Goodfellow, I. J., Erhan, D., Luc Carrier, P., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.005 · Cited by 697
Multiple SVM-RFE for Gene Selection in Cancer Classification With Expression Data
Duan, K.-B., Rajapakse, J. C., Wang, H., & Azuaje, F. (2005) · IEEE Transactions on Nanobioscience · doi:10.1109/tnb.2005.853657 · Cited by 468
Verma, G. K., & Tiwary, U. S. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.11.007 · Cited by 354 · cited 1×
“such that Cφ = ∫ ∞ −∞ (∣∣∣φ(u)∣∣∣2/|u|du), where φ(u) is the Fourier transform of φ(t) [20].”
Emotion representation, analysis and synthesis in continuous space: A survey
Gunes, H., Schuller, B., Pantic, M., & Cowie, R. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771357 · Cited by 322 · cited 1×
“The two-dimensional model of arousal and valence is the most widely used dimensional emotion model [3].”
Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?
Khorrami, P., Paine, T. L., & Huang, T. S. (2015) · 2015 IEEE International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2015.12 · Cited by 285 · cited 2×
“Our results are partially consistent with the findings of previous work [33,34].”
Fusion of facial expressions and EEG for implicit affective tagging
Koelstra, S., & Patras, I. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.10.002 · Cited by 228 · cited 5×
“Thus, we achieved high performance in detecting emotion compared to [10], especially for the face-based method, in which [10] achieved 64.”
InfluentialFrom individual to group-level emotion recognition: EmotiW 5.0
Dhall, A., Goecke, R., Ghosh, S., Joshi, J., Hoey, J., & Gedeon, T. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3143004 · Cited by 218
Improving multiview face detection with multi-task deep convolutional neural networks
Zhang, C., & Zhang, Z. (2014) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2014.6835990 · Cited by 208 · cited 1×
“Such a multi-task learning scheme can further improve the accuracy of the classifier [31,32].”
Evaluating a Computational Model of Emotion
Gratch, J., & Marsella, S. (2005) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-005-1081-1 · Cited by 161 · cited 1×
“These applications are manifested across a variety of levels and modalities [1].”
Combining Classifiers: From the Creation of Ensembles to the Decision Fusion
Ponti, M. A. (2011) · SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI) · doi:10.1109/sibgrapi-t.2011.9 · Cited by 148 · cited 1×
“For the second fusion method, we applied adaptive boosting (adaboost) techniques to decision-level fusion [22,23].”
Robust EEG emotion classification using segment level decision fusion
Rozgic, V., Vitaladevuni, S. N., & Prasad, R. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6637858 · Cited by 116 · cited 1×
“In the literature, based on the DEAP data set, the authors of [25] addressed the single-trial binary classification of emotion dimensions (arousal, valence, dominance, and liking) using EEG signals and achieved 76.”
Fusion of Facial Expressions and EEG for Multimodal Emotion Recognition
Huang, Y., Yang, J., Liao, P., & Pan, J. (2017) · Computational Intelligence and Neuroscience · doi:10.1155/2017/2107451 · Cited by 115
Self-citationChallenges in Representation Learning: A report on three machine learning contests
Goodfellow, I., Erhan, D., Carrier, P. L., Courville, A. C., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D., et al. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1307.0414 · Cited by 98 · cited 1×
“We discriminatively pre-trained the CNN using a large auxiliary data set (namely, fer2013) with image-level annotations [17].”
Continuous emotion detection using EEG signals and facial expressions
Soleymani, M., Asghari-Esfeden, S., Pantic, M., & Fu, Y. (2014) · 2014 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2014.6890301 · Cited by 95 · cited 4×
“In addition to implementing the widely used fusion method based on enumerating different weights between two models [10,11],”
InfluentialEEG-based classification of emotions using empirical mode decomposition and autoregressive model
Zhang, Y., Zhang, S., & Ji, X. (2018) · Multimedia Tools and Applications · doi:10.1007/s11042-018-5885-9 · Cited by 91 · cited 1×
“, [8]), high volatility is associated with facial expressions during emotion experiments because subjects are able to trick the machine if they know how to mimic certain facial expressions.”
Pan, J., Xie, Q., Huang, H., He, Y., Sun, Y., Yu, R., & Li, Y. (2018) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2018.00198 · Cited by 66 · cited 2×
“Our results are partially consistent with the findings of previous work [35].”
Self-citationIdentifying Stable Patterns over Time for Emotion Recognition from EEGPreprint
Zheng, W., Zhu, J., & Lu, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.02197 · Cited by 55 · cited 1×
“They systematically evaluated the performance of various popular feature extraction, feature selection, feature smoothing, and pattern classification methods with a Database for Emotion Analysis using Physiological Signals (DEAP) and a newly developed dataset called SJTU Emotion EEG Dataset (SEED) for this study [7].”
A New Fractional Random Wavelet Transform for Fingerprint Security
Bhatnagar, G., Wu, Q. M. J., & Raman, B. (2012) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/tsmca.2011.2147307 · Cited by 51 · cited 1×
“The one-dimensional sustained wavelet transform denoted by w f (s, τ) of a one-dimensional function f (t) is defined in (5) [19]:”
Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?Preprint
Khorrami, P., Paine, T. L., & Huang, T. S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.02969 · Cited by 29 · cited 2×
“Our results are partially consistent with the findings of previous work [33,34].”
Emotion Recognition from EEG Signals using Hierarchical Bayesian Network with Privileged Information
Gao, Z., & Wang, S. (2015) · International Conference on Multimedia Retrieval · doi:10.1145/2671188.2749364 · Cited by 23 · cited 2×
“The authors of [24] also tested their model on the DEAP data set and achieved 58.”
WT Feature Based Emotion Recognition from Multi-channel Physiological Signals with Decision Fusion
Xie, J., Xu, X., & Shu, L. (2018) · Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia) · doi:10.1109/aciiasia.2018.8470381 · Cited by 16
(1996)
Research works printed but not linked: 1
- Patel, 2017, Emotion recognition from speech with gaussian mixture models & via boosted gmm, Int. J. Res. Sci. Eng., 3, 47
Cited by
Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion
Cimtay, Y., Ekmekcioglu, E., & Caglar-Ozhan, S. (2020) · IEEE Access · doi:10.1109/access.2020.3023871 · Cited by 200
Advances in Multimodal Emotion Recognition Based on Brain–Computer Interfaces
He, Z., Li, Z., Yang, F., Wang, L., Li, J., Zhou, C., & Pan, J. (2020) · Brain Sciences · doi:10.3390/brainsci10100687 · Cited by 151
Self-citationMultimodal Emotion Recognition From EEG Signals and Facial Expressions
Wang, S., Qu, J., Zhang, Y., & Zhang, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3263670 · Cited by 124
“For example, Huang et al. [31] proposed a multimodal emotion recognition framework combining EEG and facial expressions.”
BackgroundMethodUsing transformers for multimodal emotion recognition: Taxonomies and state of the art review
Hazmoune, S., & Bougamouza, F. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108339 · Cited by 167
Salama, E. S., El-Khoribi, R. A., Shoman, M. E., & Wahby Shalaby, M. A. (2021) · Egyptian Informatics Journal · doi:10.1016/j.eij.2020.07.005 · Cited by 117
Metadata
- Authors
- Huang, Y., Yang, J., Liu, S., & Pan, J.
- Affiliations
- South China Normal University (CN)
- Corresponding author
- Pan, Jiahui
- Venue
- Future Internet
- Publisher
- MDPI AG
- Published
- 2019-05-02
- Volume
- 11
- Issue
- 5
- Pages
- 105
- ISSN
- 1999-5903 (ISSN-L)
- OpenAlex
- W2943183446
- Cited by
- 124
- References
- 37
- Citations by year
- 2026: 9 · 2025: 23 · 2024: 28 · 2023: 16 · 2022: 21 · 2021: 16 · 2020: 10 · 2019: 1
- FWCI
- 14.13
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Future Internet (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Natural Science Foundation of China (61503143)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- emotion recognition, facial expression recognition, EEG, multimodal fusion, valence-arousal model, transfer learning, convolutional neural networks, support vector machine, decision fusion, ensemble boosting, DEAP dataset, physiological signals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.