Multimodal Attention Network for Continuous-Time Emotion Recognition Using Video and EEG Signals
Open accessThe DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Abstract
Emotion recognition is a very important technique for ultimate interactions between human beings and artificial intelligence systems. For effective emotion recognition in a continuous-time domain, this article presents a multimodal fusion network which integrates video modality and electroencephalogram (EEG) modality networks. To calculate the attention weights of facial video features and the corresponding EEG features in fusion, a multimodal attention network, that is utilizing bilinear pooling based on low-rank decomposition, is proposed. Finally, continuous domain valence values are computed by using two modality network outputs and attention weights. Experimental results show that the proposed fusion network provides an improved performance of about 6.9% over the video modality network for the MAHNOB human computer interface (MAHNOB-HCI) dataset. Also, we achieved the performance improvement even for our proprietary dataset.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“Next, CNN extracts the features from each facial image, and the CNN features pass through LSTM [25] to generate a video feature f vid whose length is 2048.”
MethodImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“Therefore, the deep convolutional encoder is pre-trained with a well-known ImageNet [27], which is a huge–scale image classification dataset with 1000 classes, and the pre-trained network is fine-tuned with the FER2013 dataset [28], which consists of 32,298 images with 7 emotional classes.”
BackgroundMethodDensely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093 · cited 1×
“TheDatabase for Emotion Analysis using Physiological signals (DEAP) dataset [17] and the MAHNOB-HCI dataset [18], which sensed EEG signals with facial videos when humans feel emotions, were released in 2012.”
MethodLucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303 · cited 2×
“Also, in interval [15], [35] of Fig.”
Method(2011) · Cited by 2,752
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642
Challenges in Representation Learning: A Report on Three Machine Learning Contests
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-42051-1_16 · Cited by 1,719
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664 · cited 3×
“Since two modality networks are trained independently and fusion is done through the multimodal attention network at the output stage, the proposed method is easier to learn than the conventional method [14], which conducts end-to-end learning with pre-fused multimodal input.”
BackgroundMethodInfluentialEEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks
Song, T., Zheng, W., Song, P., & Cui, Z. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2817622 · Cited by 1,597 · cited 5×
“Since then, many EEG-based emotion recognition algorithms have been developed [6]–[8], [52], [53].”
BackgroundMethodInfluential
Research works printed but not linked: 4
- DONG YOON CHOI (Graduate Student Member, IEEE) received the B.S. and M.S. degrees in elec- tronic engineering from Inha University, Incheon, South Korea, in 2014 and 2016, respectively, where he is currently pursuing the Ph.D. degree in electronic engineering. His research interests include image processing, computer vision, and multimodal deep-learning.
- BYUNG CHEOL SONG (Senior Member, IEEE) received the B.S., M.S., and Ph.D. degrees in electrical engineering from the Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea, in 1994, 1996, and 2001, respectively. From 2001 to 2008, he was a Senior Engineer with the Digital Media Research and Development Center, Samsung Electronics Com- pany Ltd., Suwon, South Korea. In March 2008, he joined the Department of Electronic Engineer- ing, Inha University, Incheon, South Korea, where he is currently a Professor. His research interests include image processing and computer vision.
- M. Valstar and M. Pantic, ''Induced disgust, happiness and surprise: An addition to the MMI facial expression database,'' in Proc. 3rd Intern. Workshop EMOTION (Satellite LREC), Corpora Res. Emotion Affect, 2010, p. 65.
- M. Cukic, D. Pokrajac, M. Stokic, S. Simic, V. Radivojevic, and M. Ljubisavljevic, ''EEG machine learning with Higuchi fractal dimension and sample entropy as features for successful detection of depression,'' 2018, arXiv:1803.05985. [Online]. Available: http://arxiv.org/abs/1803.05985 · arXiv
Cited by
A review on EEG-based multimodal learning for emotion recognition
Pillalamarri, R., & Shanmugam, U. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11126-9 · Cited by 183
Multimodal Emotion Recognition From EEG Signals and Facial Expressions
Wang, S., Qu, J., Zhang, Y., & Zhang, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3263670 · Cited by 124
“For example, Choi et al. [21] proposed a multimodal fusion network integrating video and EEG modalities.”
BackgroundMultimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network
Zhang, Y., Cheng, C., & Zhang, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3049516 · Cited by 84
“Each video clip was about 34 to 117 seconds [30].”
BackgroundMultimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70
Ding, Z., Ji, Y., Gan, Y., Wang, Y., & Xia, Y. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-18096-6 · Cited by 64
“In line with the multimodal nature of emotion recognition models, existing frameworks like CNN and Gated Recurrent Units (GRU) are also being used to extract emotional features from videos, actions, and even brain activity, aiming to enhance multi-dimensional recognition accuracy [87, 138].”
Speech Expression Multimodal Emotion Recognition Based on Deep Belief Network
Liu, D., Chen, L., Wang, Z., & Diao, G. (2021) · Journal of Grid Computing · doi:10.1007/s10723-021-09564-0 · Cited by 38
“Therefore, it will affect the application of multi-modal data in the field of deep learning [7, 8].”
BackgroundEEG-based affective brain–computer interfaces: recent advancements and future challenges
Chen, Y., Peng, Y. K., Tang, J., Camilleri, T. A., Camilleri, K. P., Kong, W., & Cichocki, A. S. (2025) · Journal of Neural Engineering · doi:10.1088/1741-2552/ade290 · Cited by 49
Tian, F., Zhang, L., Zhu, L., Zhao, M., Liu, J., Dong, Q., & Zhao, Q. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3420445 · Cited by 33
Al-Farouni, M. H. (2022) · Journal of Smart Internet of Things · doi:10.2478/jsiot-2022-0003 · Cited by 19
“Choi et al. [13] introduced a multi-modal fusion network that incorporates EEG and video modality networks.”
BackgroundIntegrating biological and machine intelligence: Attention mechanisms in brain–computer interfaces
Wang, J., Ye, W., He, J., Zhang, L., Huang, G., Yu, Z., & Liang, Z. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103417 · Cited by 34
Emotion Recognition with Pre-Trained Transformers Using Multimodal Signals
Vazquez-Rodriguez, J., Lefebvre, G., Cumin, J., & Crowley, J. L. (2022) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii55700.2022.9953852 · Cited by 28
“In a few cases, physiological signals have been used to improve recognition from image, audio and text [22]–[24].”
BackgroundResidual multimodal Transformer for expression‐EEG fusion continuous emotion recognition
Jin, X., Xiao, J., Jin, L., & Zhang, X. (2024) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.12346 · Cited by 23
Hidden Emotion Detection using Multi-modal Signals
Song, B. C., & Kim, D. H. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3451721 · Cited by 22
“2) We extract feature maps from image and EEG signals using CNNs [4], and calculate the emotional level based on the extracted feature maps.”
BackgroundMethodSelf-citationEmotion recognition with hybrid attentional multimodal fusion framework using cognitive augmentation
Kulkarni, S. V., Khot, S. S., & Angal, Y. (2025) · International Journal of Information Technology · doi:10.1007/s41870-025-02402-6 · Cited by 10
A Multimodal Approach to Psycho-Emotional State Detection of a Vehicle Driver
Lashkov, I., & Kashevnik, A. M. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-82196-8_42 · Cited by 5
Lan, Z., Li, Z., Yan, C., Xiang, X., Tang, D., Zhou, H., & Lai, J. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521793 · Cited by 7
“Many researchers have designed different fusion structures, such as middle fusion [21], [22] and late fusion [23].”
Dissanayake, T., Müller, K., & von Lühmann, A. (2025) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2025.3617858 · Cited by 9
“For instance, for ECG-EEG fusion, researchers have used mid/late fusion techniques [134], [136], and for EEG-Video fusion, researchers have used late fusion [137].”
Alotaibi, Y. A., & Vuyyuru, V. A. (2023) · AIMS Mathematics · doi:10.3934/math.20231169 · Cited by 7
Boosting Emotion Recognition in Context using Non-target Subject Information
Thuseethan, S., Rajasegarar, S., & Yearwood, J. L. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533637 · Cited by 4
“These works are classified as unimodal [17] or multimodal [18] based on the number of cues utilized.”
BackgroundAdvances In EEG-Based Multimodal Emotion Recognition: A Comprehensive Review
Pillalamarri, R., & Shanmugam, U. (2023) · Innovations in Power and Advanced Computing Technologies (i-PACT) · doi:10.1109/i-pact58649.2023.10434480 · Cited by 3
“By employing attention weights [53] the attention network identifies the more reliable modality by comparing the heterogeneous features.”
MethodStep-Wise Prompting Meets Uncertainty-Aware Dynamic Fusion for Robust EEG-Visual Emotion Recognition
Hao, D., Meng, M., Gao, Y., Lou, X., & Kong, W. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3632304 · Cited by 3
“To address this, Choi et al. [31] introduced a bilinear multimodal attention network with low-rank decomposition to capture cross-modal dependencies.”
PWO-C2A-AM: Attentional Deep Learning Model for Emotion Recognition Using EEG Signals
Khubani, J., & Kulkarni, S. S. (2025) · Biomedical Materials & Devices · doi:10.1007/s44174-025-00374-6 · Cited by 2
“This limits their practical accuracy and reliability in real-world applications [36].”
Model-driven validation of visual explanations for multimodal emotion recognition
Gagliardi, G., Alfeo, A. L., Catrambone, V., Cimino, M. G. C. A., Vos, M. D., & Valenza, G. (2025) · Machine Learning · doi:10.1007/s10994-025-06921-y · Cited by 4
“Typically, these methodologies are employed to integrate EEG signals with facial video data (Choi et al., 2020), leveraging attention mechanisms that enable the model to dynamically assess the significance of features across each modality, thus facilitating the selective integration of the most…”
Wei, S. (2025) · IEEE Access · doi:10.1109/access.2025.3561179 · Cited by 4
Home Robot Interaction Based on EEG Motor Imagery and Visual Perception Fusion
Zhou, T. H., Li, D., Jian, Z., Ding, W., & Wang, L. (2025) · Sensors · doi:10.3390/s25175568 · Cited by 1
“EEG and video signals were further fused using deep neural networks by Choi et al., facilitating real-time user interaction and the prediction of intentions [23].”
DPMMN: A dual performer-multi-modal network for emotion recognition
Gornale, S. S., Shivakumara, P., Unki, A., & Vadera, S. (2025) · Signal Processing Image Communication · doi:10.1016/j.image.2025.117464 · Cited by 1
Deng, Y., Li, M., & Xing, W. (2025) · IEEE International Conference on Cloud Computing Technology and Science · doi:10.1109/cloudcom67567.2025.11331494 · Cited by 2
“…short-term memory (LSTM) networks, convolutional neural networks (CNNs), and their hybrids have demonstrated strong capabilities in modeling both the temporal dependencies and spatial distributions of EEG activity, thereby capturing the complex patterns associated with emotional states [7]–[9].”
Tree-Based Mix-Order Polynomial Fusion Network for Multimodal Sentiment Analysis
Tang, J., Hou, M., Jin, X., Zhang, J., Zhao, Q., & Kong, W. (2023) · Systems · doi:10.3390/systems11010044 · Cited by 2
M2KD: Multi-Teacher Multi-Modal Knowledge Distillation for Aerial View Object Classification
Lan, Z., Li, Z., Yan, C., Xiang, X., Tang, D., & Zhou, H. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650915 · Cited by 1
“For instance, Choi et al. [8] proposed a multi-modal attention network based on video and EEG signals to recognize emotional changes.”
BackgroundLaksana, I. K. P. B., Yonatan, M. A., Parimartha, P. A., Hongastu, V. A., Parmonangan, I. H., & Diana, D. (2023) · IEEE International Conference on Computing, Engineering and Design (ICCED) · doi:10.1109/icced60214.2023.10425227 · Cited by 1
“Furthermore, Choi, et al. [10] proposed a multimodal fusion network that is integrated with video and electroencephalogram (EEG).”
BackgroundKomala, K. V., Jayadevappa, D., & Manjunatha Manjunatha, A. P. (2024) · Research Square · doi:10.21203/rs.3.rs-4129815/v1 · Cited by 1
The Influence of Emotion Recognition in Learning Processes
Llurba, C., Palau, R., & Recalde, J. M. (2023) · Advances in educational marketing, administration, and leadership book series · doi:10.4018/978-1-6684-8156-1.ch013
Towards Emotions Classification: Modeling Peripheral Signals with Deep Learning
Huang, X. (2025) · IEEE International Conference on Cyborg and Bionic Systems (CBS) · doi:10.1109/cbs65871.2025.11267613
“Choi et al. [11] further showed that integrating video with EEG via a multimodal attention mechanism employing bilinear pooling and low-rank factorization yields substantial gains, reporting a 6.9% improvement on MAHNOB-HCI over visual-only systems.”
InfluentialDenoise and Attend: An AE-Transformer Approach to EEG Emotion Recognition
Luo, M. (2025) · International Conference on Electrical Engineering and Computer Technology (IC2ECT) · doi:10.1109/ic2ect66838.2025.11291002
Reddy, C. H. N., Kumar, K. S. A., Manjunathachari, K., & Abdul, A. M. (2026) · Iranian Journal of Science and Technology Transactions of Electrical Engineering · doi:10.1007/s40998-025-00979-z
“As a result, EEG-based emotion recognition classifies the psychological reaction to a film as happiness, sorrow, contempt, fear, or calm (Choi et al. 2020).”
Deng, Y., Li, M., Liu, Q., Zhang, G., & Xing, W. (2026) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2026.3666740
Elavaar Kuzhali, S., Suttur, C. S., & R, D. V. T. (2026) · Research on Biomedical Engineering · doi:10.1007/s42600-026-00460-8
“(Choi et al. 2020).”
DTHFormer: Dual Token Hierarchical Transformer Network for Fine-Grained Emotion Recognition
Chen, B., Qing, C., Jin, J., & Xu, X. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3691981
“Choi et al. [27] fused video modality and EEG modality and used low-rank decomposition and bilinear pooling to calculate continuous emotion.”
Amini, A., Maqsood, S., Abbas, I., Sarwar, M. A., Maskeliūnas, R., & Damaševičius, R. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831113
“Recent work has used residualized multimodal transformers to stabilize token-level integration across modalities [18], whereas earlier EEG– video pipelines employed attention to modulate each modality’s contribution during fusion [4].”
Influential
Metadata
- Authors
- Choi, D. Y., Kim, D.-H., & Song, B. C.
- Affiliations
- Inha University (KR)
- Venue
- IEEE Access
- Published
- 2020
- Volume
- 8
- Pages
- 203814-203826
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3101150053
- Cited by
- 39
- References
- 58
- Citations by year
- 2026: 5 · 2025: 14 · 2024: 7 · 2023: 6 · 2022: 2 · 2021: 5
- FWCI
- 3.14
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Korea Evaluation Institute of Industrial Technology (10073154)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Keywords
- multimodal fusion, bilinear pooling, low-rank decomposition, EEG features, human-computer interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.