Multimodal Fused Emotion Recognition About Expression-EEG Interaction and Collaboration Using Deep Learning
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
The proposed emotion recognition model is based on the hierarchical long-short term memory neural network (LSTM) for video-electroencephalogram (Video-EEG) signal interaction. The inputs are facial-video and EEG signals from the subjects when they are watching the emotion-stimulated video. The outputs are the corresponding emotion recognition results. Facial-video features and corresponding EEG features are extracted based on a fully connected neural network (FC) at each time point. These features are fused through hierarchical LSTM to predict the key emotional signal frames at the next time point until the emotion recognition result is calculated at the last time point. Specially, a self-attention mechanism is applied to show the correlation of the stacked LSTM at different hierarchies. In this process, the “selective focus” is used to analyze the human-emotional temporal sequences in each model, which improves the utilization of the key spatial EEG signals. Moreover, the process includes the temporal attention mechanism to predict the key signal frame at next time point, which utilizes the key emotion data in temporal domain. The experimental results prove that the classification rate (CR) and F1-score of the proposed emotion recognition model are significantly increased by at least 2% and 0.015, respectively, compared to other methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Feature extraction and fusion are the key steps in multimodal emotion recognition [3].”
From Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network (2021) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Salovey, P., & Mayer, J. D. (1990) · Imagination, Cognition and Personality · doi:10.2190/dugg-p24e-52wk-6cdg · Cited by 10,484
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 1×
“Thebasic theory proves that when emotions arise, a variety reaction of human physiological and external behavior are activated [15].”
BackgroundDEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093
Toward machine emotional intelligence: analysis of affective physiological state
Picard, R. W., Vyzas, E., & Healey, J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.954607 · Cited by 2,334 · cited 1×
“Picard et al. [11] used four signals, including electromyogram, blood pressure, gal-vanic skin response and heart rate, to identify eight emotions (grief, pleasure, anger, annoyance, neutrality, and reverence), resulting in 81% classification accuracy.”
BackgroundAffect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 2×
“RELATED RESEARCH External features, such as the facial expression, body gesture and voice, are widely used in traditional emotion recognition methods [9].”
MethodEmotionMeter: A Multimodal Framework for Recognizing Human Emotions
Zheng, W.-L., Liu, W., Lu, Y., Lu, B.-L., & Cichocki, A. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2797176 · Cited by 1,276
Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks
Bashivan, P., Rish, I., Yeasin, M., & Codella, N. (2015) · arXiv · doi:10.48550/arxiv.1511.06448 · Cited by 836
EEG-based emotion recognition during watching movies
Nie, D., Wang, X.-W., Shi, L.-C., & Lu, B.-L. (2011) · 2011 5th International IEEE/EMBS Conference on Neural Engineering · doi:10.1109/ner.2011.5910636 · Cited by 360
Verma, G. K., & Tiwary, U. S. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.11.007 · Cited by 354
Multimodal emotion recognition using deep learning architectures
Ranganathan, H., Chakraborty, S., & Panchanathan, S. (2016) · 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2016.7477679 · Cited by 249
Exploring Fusion Methods for Multimodal Emotion Recognition with Missing Data
Wagner, J., Andre, E., Lingenfelser, F., & Kim, J. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.12 · Cited by 156 · cited 1×
“Forexample, Wagner et al. [16] extracted voice, face and gesture features and performed multimodal fusion in both feature and decision level.”
Multimodal and Temporal Perception of Audio-visual Cues for Emotion Recognition
Ghaleb, E., Popa, M., & Asteriadis, S. (2019) · 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII) · doi:10.1109/acii.2019.8925444 · Cited by 93
Zheng, W.-L., Guo, H.-T., & Lu, B.-L. (2015) · 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER) · doi:10.1109/ner.2015.7146583 · Cited by 63
The Fusion of Electroencephalography and Facial Expression for Continuous Emotion Recognition
Li, D., Wang, Z., Wang, C., Liu, S., Chi, W., Dong, E., Song, X., Gao, Q., & Song, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2949707 · Cited by 63
Duan, R.-N., Wang, X., & Lu, B. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-34478-7_57 · Cited by 53
Multimodal emotion recognition in response to videos (Extended abstract)
Soleymani, M., Pantić, M., & Pun, T. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344615 · Cited by 30
Multimodal mixed emotion detection
Patwardhan, A. (2017) · International Conference on Communication and Electronics Systems (ICCES) · doi:10.1109/cesys.2017.8321250 · Cited by 29
A multimodal emotion recognition system from video
Thushara, S., & Veni, S. (2016) · International Conference on Circuit, Power and Computing Technologies · doi:10.1109/iccpct.2016.7530161 · Cited by 27
Speech Emotion Recognition Adapted to Multimodal Semantic Repositories
Vryzas, N., Vrysis, L., Kotsakis, R., & Dimoulas, C. A. (2018) · International Workshop on Semantic and Social Media Adaptation and Personalization (SMAP) · doi:10.1109/smap.2018.8501881 · Cited by 21
Liebold, B., & Ohler, P. (2013) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2013.73 · Cited by 18
Transfer of multimodal emotion features in deep belief networks
Ranganathan, H., Chakraborty, S., & Panchanathan, S. (2016) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/acssc.2016.7869079 · Cited by 12
Mind Reader : A Facial Expression and EEG Based Emotion Recognizer
Islam, M. A., Hamza, A., Rahaman, M. H., Bhattacharjee, J., & Rahman, M. M. (2018) · International Conference on Computing Methodologies and Communication (ICCMC) · doi:10.1109/iccmc.2018.8487898 · Cited by 4
Research works printed but not linked: 2
- huang, 2018, Fusion of facial expressions and EEG for emotion recognition, Comput Syst Appl, 27, 9
- chang, 2017, Using reinforce learning to train multi attention model, ACTA Automatica Sinica, 43, 1563
Cited by
A systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 615
Emotion recognition in EEG signals using deep learning methods: A review
Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D. L., Gorriz, J. M., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107450 · Cited by 240
Multimodal Emotion Recognition with Deep Learning: Advancements, challenges, and future directions
A.V., G., T., M., D., P., & E., U. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102218 · Cited by 225
A review on EEG-based multimodal learning for emotion recognition
Pillalamarri, R., & Shanmugam, U. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11126-9 · Cited by 183
Deep Learning for Intelligent Human–Computer Interaction
Lv, Z., Poiesi, F., Dong, Q., Lloret, J., & Song, H. (2022) · Applied Sciences · doi:10.3390/app122211457 · Cited by 174
Metadata
- Authors
- Wu, D., Zhang, J., & Zhao, Q.
- Affiliations
- Qiqihar University (CN)
- Harbin Engineering University (CN)
- Venue
- IEEE Access
- Published
- 2020
- Volume
- 8
- Pages
- 133180-133189
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3043900038
- Cited by
- 51
- References
- 23
- Citations by year
- 2026: 7 · 2025: 16 · 2024: 6 · 2023: 12 · 2022: 8 · 2021: 2
- FWCI
- 5.31
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Natural Science Foundation of China (61672179, 61370083, 61402126)
- Specialized Research Fund for the Doctoral Program of Higher Education of China (20122304110012)
- Youth Science Foundation of Heilongjiang Province of China (QC2016083)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- multimodal emotion recognition, facial expressions, EEG, hierarchical LSTM, self-attention, temporal attention, EEG features
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.