Multimodal emotion recognition using EEG and eye tracking data
Abstract
This paper presents a new emotion recognition method which combines electroencephalograph (EEG) signals and pupillary response collected from eye tracker. We select 15 emotional film clips of 3 categories (positive, neutral and negative). The EEG signals and eye tracking data of five participants are recorded, simultaneously, while watching these videos. We extract emotion-relevant features from EEG signals and eye tracing data of 12 experiments and build a fusion model to improve the performance of emotion recognition. The best average accuracies based on EEG signals and eye tracking data are 71.77% and 58.90%, respectively. We also achieve average accuracies of 73.59% and 72.98% for feature level fusion strategy and decision level fusion strategy, respectively. These results show that both feature level fusion and decision level fusion combining EEG signals and eye tracking data can improve the performance of emotion recognition model.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Using eye movement signals for recognizing emotions has a certain degree of reliability.
“These results indicate that using eye movement signals for recognizing emotions has a certain degree of reliability.”
From A Lightweight Dual-Channel Multimodal Emotion Recognition Network Using Facial Expressions and Eye Movements (2024) · cites this paper as [12]
Multimodal approaches combining EEG with other data types have been shown to improve emotion recognition performance compared with unimodal methods.
“Using fusion techniques at the feature and decision levels, Zheng et al. [12] showed, for example, that combining EEG and eye movement data could improve emotion recognition performance.”
From Multimodal Emotion Recognition: Emotion Classification Through the Integration of EEG and Facial Expressions (2025) · cites this paper as [12]
Liu et al. and Zheng et al. incorporated eye movement data, obtaining accuracies of 91.01% and 73.59%, respectively.
“Additionally, Liu et al. [13] and Zheng et al. [12] incorporated eye movement data, obtaining accuracies of 91.01% and 73.59%, respectively.”
From Multimodal Emotion Recognition: Emotion Classification Through the Integration of EEG and Facial Expressions (2025) · cites this paper as [12]
Cites
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 1× · see citing works
“The early works are more based on non-physiological signals, such as text, facial expression, speech and gesture [1].”
EEG-based emotion classification using deep belief networks
Zheng, W.-L., Zhu, J.-Y., Peng, Y., & Lu, B.-L. (2014) · 2014 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2014.6890166 · Cited by 367 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
A multimodal approach to estimating vigilance using EEG and forehead EOG
Zheng, W.-L., & Lu, B.-L. (2017) · Journal of Neural Engineering · doi:10.1088/1741-2552/aa5a98 · Cited by 424 · see citing works
Self-citationEmotion Recognition from Multiband EEG Signals Using CapsNet
Chao, H., Dong, L., Liu, Y., & Lu, B. (2019) · Sensors · doi:10.3390/s19092212 · Cited by 287 · see citing works
Self-citationAutomatic Emotion Recognition Using Temporal Multimodal Deep Learning
Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V. (2020) · IEEE Access · doi:10.1109/access.2020.3027026 · Cited by 101 · see citing works
A hybrid latent space data fusion method for multimodal emotion recognition
Nemati, S., Rohani, R., Basiri, M. E., Abdar, M., Yen, N. Y., & Makarenkov, V. (2019) · IEEE Access · doi:10.1109/access.2019.2955637 · Cited by 98 · see citing works
Multimodal Emotion Recognition from Eye Image, Eye Movement and EEG Using Deep Neural Networks
Guo, J.-J., Zhou, R., Zhao, L.-M., & Lu, B.-L. (2019) · 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2019.8856563 · Cited by 73 · see citing works
Self-citationErdem Güler, S., & Patlar Akbulut, F. (2025) · IEEE Access · doi:10.1109/access.2025.3538642 · Cited by 21 · see citing works
Hierarchical fusion of visual and physiological signals for emotion recognition
Fang, Y., Rong, R., & Huang, J. (2021) · Multidimensional Systems and Signal Processing · doi:10.1007/s11045-021-00774-z · Cited by 17 · see citing works
Ji, M. (2024) · Poster Volume Ⅰ The 2024 Twentieth International Conference on Intelligent Computing August 5-8, 2024 Tianjin, China · doi:10.65286/icic.v20i2.94055
- Authors
- Zheng, W.-L., Dong, B.-N., & Lu, B.-L.
- Affiliations
- Shanghai Jiao Tong University (CN)
- Venue
- 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society
- Publisher
- IEEE
- Published
- 2014-08
- Volume
- 2014
- Pages
- 5040-5043
- ISSN
- 2375-7477 (ISSN-L)
- OpenAlex
- W2085497785
- Cited by
- 230
- References
- 2
- Citations by year
- 2026: 15 · 2025: 16 · 2024: 33 · 2023: 21 · 2022: 22 · 2021: 23 · 2020: 30 · 2019: 22 · 2018: 21 · 2017: 13 · 2016: 10 · 2015: 4
- FWCI
- 14.74
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Keywords
- multimodal emotion recognition, EEG, eye tracking, pupillary response, feature-level fusion, decision fusion, emotional film clips, affective computing
- MeSH terms
- Algorithms, Brain, Electroencephalography, Emotions, Eye Movements, Female, Humans, Iris, Male, Models, Psychological, Pattern Recognition, Automated, Young Adult, Multimodal Imaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
