Automatic Emotion Recognition Using Temporal Multimodal Deep Learning
Open accessThe DOI matches its Crossref record; 41 of the 43 references it cites were found in a registry or an index.
Abstract
Emotion recognition using miniaturised wearable physiological sensors has emerged as a revolutionary technology in various applications. However, detecting emotions using the fusion of multiple physiological signals remains a complex and challenging task. When fusing physiological signals, it is essential to consider the ability of different fusion approaches to capture the emotional information contained within and across modalities. Moreover, since physiological signals consist of time-series data, it becomes imperative to consider their temporal structures in the fusion process. In this study, we propose a temporal multimodal fusion approach with a deep learning model to capture the non-linear emotional correlation within and across electroencephalography (EEG) and blood volume pulse (BVP) signals and to improve the performance of emotion classification. The performance of the proposed model is evaluated using two different fusion approaches - early fusion and late fusion. Specifically, we use a convolutional neural network (ConvNet) long short-term memory (LSTM) model to fuse the EEG and BVP signals to jointly learn and explore the highly correlated representation of emotions across modalities, after learning each modality with a single deep network. The performance of the temporal multimodal deep learning model is validated on our dataset collected from smart wearable sensors and is also compared with results of recent studies. The experimental results show that the temporal multimodal deep learning models, based on early and late fusion approaches, successfully classified human emotions into one of four quadrants of dimensional emotions with an accuracy of 71.61% and 70.17%, respectively.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Still, more and more researchers use multimodal models that integrate multiple data and features to carry out emotion recognition, which can extract more abundant features and improve the effect of emotion recognition.”
From Multimodal Emotion Recognition From EEG Signals and Facial Expressions (2023) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“Still, more and more researchers use multimodal models that integrate multiple data and features to carry out emotion recognition, which can extract more abundant features and improve the effect of emotion recognition.”
From Multimodal Emotion Recognition From EEG Signals and Facial Expressions (2023) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Akselrod, S., Gordon, D., Ubel, F. A., Shannon, D. C., Berger, A. C., & Cohen, R. J. (1981) · Science · doi:10.1126/science.6166045 · Cited by 5,079 · cited 1×
“There are some studies that focus on extracting features from the time and frequency domains from the BVP signal [22]–[24].”
Background(2011) · Cited by 2,752
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664 · cited 1×
“Each participant watched nine video clips used in the MAHNOB dataset [40] to induce different emotions.”
MethodEmotion recognition system using short-term monitoring of physiological signals
Kim, K. H., Bang, S. W., & Kim, S. R. (2004) · Medical & Biological Engineering & Computing · doi:10.1007/bf02344719 · Cited by 944 · cited 1×
“Inner emotional states can affect the body’s physiological signals such as EEG and BVP signals which originate from these two components [5], [6].”
BackgroundConvolutional Neural Networks for P300 Detection with Application to Brain-Computer Interfaces
Cecotti, H., & Graser, A. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.125 · Cited by 778 · cited 1×
“DL methods have strong capabilities in constructing reliable features in different domains like speech recognition [25] and time-series data analysis [26], [27].”
MethodMultimodal Emotion Recognition in Response to Videos
Soleymani, M., Pantic, M., & Pun, T. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.37 · Cited by 722 · cited 1×
“In recent research, multimodal data are utilised to improve the performance of emotion classification [7]–[9].”
MethodTime Series Classification Using Multi-Channels Deep Convolutional Neural Networks
Zheng, Y., Liu, Q., Chen, E., Ge, Y., & Zhao, J. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-08010-9_33 · Cited by 652 · cited 1×
“DL methods have strong capabilities in constructing reliable features in different domains like speech recognition [25] and time-series data analysis [26], [27].”
MethodEmotion Recognition based on EEG using LSTM Recurrent Neural Network
Alhagry, S., Aly, A., & A., R. (2017) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2017.081046 · Cited by 645
Mandryk, R. L., & Atkins, M. S. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2006.11.011 · Cited by 538 · cited 1×
“Automated emotion recognition using lightweight body sensors and advanced machine learning technologies has been used in different application domains such as computer games [1], e-health [2], [3] and road safety [4].”
Background
Research works printed but not linked: 2
- 2020, Frontiers|SAE+LSTM A New Framework for Emotion Recognition From Multi-Channel EEG | Frontiers in Neurorobotics
- nugraha, 2016, Classification of driver fatigue state based on EEG using emotiv EPOC, J Theor Appl Inf Technol, 86, 151
Cited by
Multimodal Emotion Recognition using Deep Learning
Abdullah, S. M. S. A., Ameen, S. Y. A., M. Sadeeq, M. A., & Zeebaree, S. (2021) · Journal of Applied Science and Technology Trends · doi:10.38094/jastt20291 · Cited by 347
“] [70] suggest a temporal fusion approach with deep learning model to capture the non-linear emotional association and enhance emotion classification efficiency.”
BackgroundMultimodal Emotion Recognition with Deep Learning: Advancements, challenges, and future directions
A.V., G., T., M., D., P., & E., U. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102218 · Cited by 225
Recognition of human emotions using EEG signals: A review
Rahman, M. M., Sarkar, A. K., Hossain, M. A., Hossain, M. S., Islam, M. R., Hossain, M. B., Quinn, J. M. W., & Moni, M. A. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104696 · Cited by 227
Emotion recognition in EEG signals using deep learning methods: A review
Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D. L., Gorriz, J. M., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107450 · Cited by 240
Driver stress detection via multimodal fusion using attention-based CNN-LSTM
Mou, L., Zhou, C., Zhao, P., Nakisa, B., Rastgoo, M. N., Jain, R., & Gao, W. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.114693 · Cited by 198
Self-citation
Metadata
- Authors
- Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V.
- Affiliations
- Deakin University (AU)
- Queensland University of Technology (AU)
- Venue
- IEEE Access
- Published
- 2020
- Volume
- 8
- Pages
- 225463-225474
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3089423349
- Cited by
- 101
- References
- 43
- Citations by year
- 2026: 15 · 2025: 22 · 2024: 19 · 2023: 22 · 2022: 15 · 2021: 8
- FWCI
- 10.88
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- QUT ePrints (Queensland University of Technology) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Griffith Research Online (submittedVersion, cc-by)
- Funders
- Queensland University of Technology (QUT) through the Centre for Robotics
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Keywords
- emotion recognition, multimodal fusion, temporal fusion, EEG, blood volume pulse, wearable sensors, physiological signals, feature-level fusion, decision fusion, LSTM, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Automatic Emotion Recognition Using Temporal Multimodal Deep Learning. https://app.cochat.ai/papers/10.1109/access.2020.3027026
- The paper
- Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V. (2020). Automatic Emotion Recognition Using Temporal Multimodal Deep Learning. IEEE Access. https://doi.org/10.1109/access.2020.3027026
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.