Multimodal Attention Network for Continuous-Time Emotion Recognition Using Video and EEG Signals
Open accessThe DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Abstract
Emotion recognition is a very important technique for ultimate interactions between human beings and artificial intelligence systems. For effective emotion recognition in a continuous-time domain, this article presents a multimodal fusion network which integrates video modality and electroencephalogram (EEG) modality networks. To calculate the attention weights of facial video features and the corresponding EEG features in fusion, a multimodal attention network, that is utilizing bilinear pooling based on low-rank decomposition, is proposed. Finally, continuous domain valence values are computed by using two modality network outputs and attention weights. Experimental results show that the proposed fusion network provides an improved performance of about 6.9% over the video modality network for the MAHNOB human computer interface (MAHNOB-HCI) dataset. Also, we achieved the performance improvement even for our proprietary dataset.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“Next, CNN extracts the features from each facial image, and the CNN features pass through LSTM [25] to generate a video feature f vid whose length is 2048.”
MethodImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“Therefore, the deep convolutional encoder is pre-trained with a well-known ImageNet [27], which is a huge–scale image classification dataset with 1000 classes, and the pre-trained network is fine-tuned with the FER2013 dataset [28], which consists of 32,298 images with 7 emotional classes.”
BackgroundMethodDensely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093 · cited 1×
“TheDatabase for Emotion Analysis using Physiological signals (DEAP) dataset [17] and the MAHNOB-HCI dataset [18], which sensed EEG signals with facial videos when humans feel emotions, were released in 2012.”
MethodLucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303 · cited 2×
“Also, in interval [15], [35] of Fig.”
Method(2011) · Cited by 2,752
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642
Challenges in Representation Learning: A Report on Three Machine Learning Contests
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-42051-1_16 · Cited by 1,719
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664 · cited 3×
“Since two modality networks are trained independently and fusion is done through the multimodal attention network at the output stage, the proposed method is easier to learn than the conventional method [14], which conducts end-to-end learning with pre-fused multimodal input.”
BackgroundMethodInfluentialEEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks
Song, T., Zheng, W., Song, P., & Cui, Z. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2817622 · Cited by 1,597 · cited 5×
“Since then, many EEG-based emotion recognition algorithms have been developed [6]–[8], [52], [53].”
BackgroundMethodInfluential
Research works printed but not linked: 3
- zhang, 2017, S3FD: Single shot scale-invariant face detector, Proc IEEE Int Conf Comput Vis, 192
- valstar, 2010, Induced disgust, happiness and surprise: An addition to the MMI facial expression database, Proc Third Int'l Workshop EMOTION (satellite of LREC) Corpora for Research on Emotion and Affect, 65
- cukic, 2018, EEG machine learning with Higuchi fractal dimension and sample entropy as features for successful detection of depression, arXiv 1803 05985
Cited by
A review on EEG-based multimodal learning for emotion recognition
Pillalamarri, R., & Shanmugam, U. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11126-9 · Cited by 183
Multimodal Emotion Recognition From EEG Signals and Facial Expressions
Wang, S., Qu, J., Zhang, Y., & Zhang, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3263670 · Cited by 124
“For example, Choi et al. [21] proposed a multimodal fusion network integrating video and EEG modalities.”
BackgroundMultimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network
Zhang, Y., Cheng, C., & Zhang, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3049516 · Cited by 84
“Each video clip was about 34 to 117 seconds [30].”
BackgroundMultimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70
Ding, Z., Ji, Y., Gan, Y., Wang, Y., & Xia, Y. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-18096-6 · Cited by 64
“In line with the multimodal nature of emotion recognition models, existing frameworks like CNN and Gated Recurrent Units (GRU) are also being used to extract emotional features from videos, actions, and even brain activity, aiming to enhance multi-dimensional recognition accuracy [87, 138].”
Metadata
- Authors
- Choi, D. Y., Kim, D.-H., & Song, B. C.
- Affiliations
- Inha University (KR)
- Venue
- IEEE Access
- Published
- 2020
- Volume
- 8
- Pages
- 203814-203826
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3101150053
- Cited by
- 39
- References
- 58
- Citations by year
- 2026: 5 · 2025: 14 · 2024: 7 · 2023: 6 · 2022: 2 · 2021: 5
- FWCI
- 3.14
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Korea Evaluation Institute of Industrial Technology (10073154)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Keywords
- multimodal fusion, bilinear pooling, low-rank decomposition, EEG features, human-computer interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal Attention Network for Continuous-Time Emotion Recognition Using Video and EEG Signals. https://app.cochat.ai/papers/10.1109/access.2020.3036877
- The paper
- Choi, D. Y., Kim, D.-H., & Song, B. C. (2020). Multimodal Attention Network for Continuous-Time Emotion Recognition Using Video and EEG Signals. IEEE Access. https://doi.org/10.1109/access.2020.3036877
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.