The Fusion of Electroencephalography and Facial Expression for Continuous Emotion Recognition
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Recently, the study of emotion recognition has received increasing attentions by the rapid development of noninvasive sensor technologies, machine learning algorithms and compute capability of computers. Compared with single modal emotion recognition, the multimodal paradigm introduces complementary information for emotion recognition. Hence, in this work, we presented a decision level fusion framework for detecting emotions continuously by fusing the Electroencephalography (EEG) and facial expressions. Three types of movie clips (positive, negative, and neutral) were utilized to elicit specific emotions of subjects, the EEG and facial expression signals were recorded simultaneously. The power spectrum density (PSD) features of EEG were extracted by time-frequency analysis, and then EEG features were selected for regression. For the facial expression, the facial geometric features were calculated by facial landmark localization. Long short-term memory networks (LSTM) were utilized to accomplish the decision level fusion and captured temporal dynamics of emotions. The results have shown that the proposed method achieved outstanding performance for continuous emotion recognition, and it yields 0.625± 0.029 of concordance correlation coefficient (CCC). From the results, the fusion of two modalities outperformed EEG and facial expression separately. Furthermore, different numbers of time-steps of LSTM was applied to analyze the temporal dynamic capturing.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093
What are emotions? And how can they be measured?
Scherer, K. R. (2005) · Social Science Information · doi:10.1177/0539018405058216 · Cited by 4,264 · cited 1×
“In order to represent emotions, the discrete models and dimensional models were proposed by psychologists [6].”
MethodIEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664
Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism
Li, Y., Zeng, J., Shan, S., & Chen, X. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2886767 · Cited by 916 · cited 1×
“CNNs and improved CNNs have shown the strong ability in facial expression recognition [49], [50].”
BackgroundThe International Affective Picture System (IAPS) in the Study of Emotion and Attention
Bradley, M. M., & Lang, P. J. (2007) · Handbook of Emotion Elicitation and Assessment · doi:10.1093/oso/9780195169157.003.0003 · Cited by 915
Emotional state classification from EEG data using machine learning approach
Wang, X.-W., Nie, D., & Lu, B.-L. (2014) · Neurocomputing · doi:10.1016/j.neucom.2013.06.046 · Cited by 755 · cited 1×
“audio stimuli in context makes movies seem to be the one of the most effective ways to elicit emotions [15].”
BackgroundAffective neuroscience and psychophysiology: Toward a synthesis
Davidson, R. J. (2003) · Psychophysiology · doi:10.1111/1469-8986.00067 · Cited by 535 · cited 1×
“PSD features of EEG signals in different bands are correlated with emotions, and theta bands (4-7 Hz), alpha bands (8-12 Hz), beta bands (13-30 Hz) and gamma bands (30-47 Hz) are the general selections [33].”
BackgroundNicolaou, M. A., Gunes, H., & Pantic, M. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.9 · Cited by 450 · cited 1×
“Moreover, bidirectional LSTM (BLSTM) was also used to explore the complex relationship between features and labels [27], [28].”
MethodGunes, H., & Schuller, B. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.016 · Cited by 360 · cited 1×
“The core of continuous emotion recognition is to analyze the time-varying affect phenomenon that are elicited by stimulus [10].”
BackgroundLSTM-Modeling of continuous emotions in an audiovisual affect recognition framework
Wöllmer, M., Kaiser, M., Eyben, F., Schuller, B., & Rigoll, G. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.03.001 · Cited by 288
Multimodal emotion recognition using EEG and eye tracking data
Zheng, W.-L., Dong, B.-N., & Lu, B.-L. (2014) · 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/embc.2014.6944757 · Cited by 230
(2015) · doi:10.1145/2808196.2811642 · Cited by 178
Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition
Huang, Y., Yang, J., Liu, S., & Pan, J. (2019) · Future Internet · doi:10.3390/fi11050105 · Cited by 124 · cited 2×
“[8] performed binary classification of the statues of valence and arousal by combining the EEG and facial expressions.”
BackgroundMethodSelf-citationFusion of Facial Expressions and EEG for Multimodal Emotion Recognition
Huang, Y., Yang, J., Liao, P., & Pan, J. (2017) · Computational Intelligence and Neuroscience · doi:10.1155/2017/2107451 · Cited by 115
Emotion Recognition from EEG and Facial Expressions: a Multimodal Approach
Chaparro, V., Gomez, A., Salgado, A., Quintero, O. L., Lopez, N., & Villa, L. F. (2018) · 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2018.8512407 · Cited by 27
Cited by
Zamkah, A., Hui, T., Andrews, S., Dey, N., Shi, F., & Sherratt, R. S. (2020) · Biosensors · doi:10.3390/bios10040040 · Cited by 110
“Happy and Routray [8] used image processing to detect emotional states in facial expression.”
BackgroundMultimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70
Wu, D., Zhang, J., & Zhao, Q. (2020) · IEEE Access · doi:10.1109/access.2020.3010311 · Cited by 51
MindLink-Eumpy: An Open-Source Python Toolbox for Multimodal Emotion Recognition
Li, R., Liang, Y., Liu, X., Wang, B., Huang, W., Cai, Z., Ye, Y., Qiu, L., & Pan, J. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.621493 · Cited by 36
Hierarchical fusion of visual and physiological signals for emotion recognition
Fang, Y., Rong, R., & Huang, J. (2021) · Multidimensional Systems and Signal Processing · doi:10.1007/s11045-021-00774-z · Cited by 17
“Some researchers treat emotion recognition as continuous problem (Soleymani et al. 2015; Sen and Sert 2018; Li et al. 2019).”
Background
Metadata
- Authors
- Li, D., Wang, Z., Wang, C., Liu, S., Chi, W., Dong, E., Song, X., Gao, Q., & Song, Y.
- Venue
- IEEE Access
- Published
- 2019
- Volume
- 7
- Pages
- 155724-155736
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2981754449
- Cited by
- 63
- References
- 17
- FWCI
- 8.11
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.