Automatic Emotion Recognition Using Temporal Multimodal Deep Learning
Open accessThe DOI matches its Crossref record; 41 of the 43 references it cites were found in a registry or an index.
Abstract
Emotion recognition using miniaturised wearable physiological sensors has emerged as a revolutionary technology in various applications. However, detecting emotions using the fusion of multiple physiological signals remains a complex and challenging task. When fusing physiological signals, it is essential to consider the ability of different fusion approaches to capture the emotional information contained within and across modalities. Moreover, since physiological signals consist of time-series data, it becomes imperative to consider their temporal structures in the fusion process. In this study, we propose a temporal multimodal fusion approach with a deep learning model to capture the non-linear emotional correlation within and across electroencephalography (EEG) and blood volume pulse (BVP) signals and to improve the performance of emotion classification. The performance of the proposed model is evaluated using two different fusion approaches - early fusion and late fusion. Specifically, we use a convolutional neural network (ConvNet) long short-term memory (LSTM) model to fuse the EEG and BVP signals to jointly learn and explore the highly correlated representation of emotions across modalities, after learning each modality with a single deep network. The performance of the temporal multimodal deep learning model is validated on our dataset collected from smart wearable sensors and is also compared with results of recent studies. The experimental results show that the temporal multimodal deep learning models, based on early and late fusion approaches, successfully classified human emotions into one of four quadrants of dimensional emotions with an accuracy of 71.61% and 70.17%, respectively.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Still, more and more researchers use multimodal models that integrate multiple data and features to carry out emotion recognition, which can extract more abundant features and improve the effect of emotion recognition.”
From Multimodal Emotion Recognition From EEG Signals and Facial Expressions (2023) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“Still, more and more researchers use multimodal models that integrate multiple data and features to carry out emotion recognition, which can extract more abundant features and improve the effect of emotion recognition.”
From Multimodal Emotion Recognition From EEG Signals and Facial Expressions (2023) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Akselrod, S., Gordon, D., Ubel, F. A., Shannon, D. C., Berger, A. C., & Cohen, R. J. (1981) · Science · doi:10.1126/science.6166045 · Cited by 5,079 · cited 1×
“There are some studies that focus on extracting features from the time and frequency domains from the BVP signal [22]–[24].”
Background(2011) · Cited by 2,752
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664 · cited 1×
“Each participant watched nine video clips used in the MAHNOB dataset [40] to induce different emotions.”
MethodEmotion recognition system using short-term monitoring of physiological signals
Kim, K. H., Bang, S. W., & Kim, S. R. (2004) · Medical & Biological Engineering & Computing · doi:10.1007/bf02344719 · Cited by 944 · cited 1×
“Inner emotional states can affect the body’s physiological signals such as EEG and BVP signals which originate from these two components [5], [6].”
BackgroundConvolutional Neural Networks for P300 Detection with Application to Brain-Computer Interfaces
Cecotti, H., & Graser, A. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.125 · Cited by 778 · cited 1×
“DL methods have strong capabilities in constructing reliable features in different domains like speech recognition [25] and time-series data analysis [26], [27].”
MethodMultimodal Emotion Recognition in Response to Videos
Soleymani, M., Pantic, M., & Pun, T. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.37 · Cited by 722 · cited 1×
“In recent research, multimodal data are utilised to improve the performance of emotion classification [7]–[9].”
MethodTime Series Classification Using Multi-Channels Deep Convolutional Neural Networks
Zheng, Y., Liu, Q., Chen, E., Ge, Y., & Zhao, J. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-08010-9_33 · Cited by 652 · cited 1×
“DL methods have strong capabilities in constructing reliable features in different domains like speech recognition [25] and time-series data analysis [26], [27].”
MethodEmotion Recognition based on EEG using LSTM Recurrent Neural Network
Alhagry, S., Aly, A., & A., R. (2017) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2017.081046 · Cited by 645
Mandryk, R. L., & Atkins, M. S. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2006.11.011 · Cited by 538 · cited 1×
“Automated emotion recognition using lightweight body sensors and advanced machine learning technologies has been used in different application domains such as computer games [1], e-health [2], [3] and road safety [4].”
BackgroundJirayucharoensak, S., Pan-Ngum, S., & Israsena, P. (2014) · The Scientific World Journal · doi:10.1155/2014/627892 · Cited by 486 · cited 1×
“Another study applied a stacked autoencoder to extract better EEG features [33].”
MethodEmotion Recognition Using Bio-sensors: First Steps towards an Automatic System
Haag, A., Goronzy, S., Schaich, P., & Williams, J. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24842-2_4 · Cited by 451 · cited 3×
“A strong correlation has been observed between such physiological signals and basic emotions like sadness, anger, surprise etc. [5], [6].”
BackgroundEmoNets: Multimodal deep learning approaches for emotion recognition in video
Kahou, S. E., Bouthillier, X., Lamblin, P., Gulcehre, C., Michalski, V., Konda, K., Jean, S., Froumenty, P., Dauphin, Y., Boulanger-Lewandowski, N., et al. (2016) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-015-0195-2 · Cited by 412 · cited 1×
“It has been shown that DL techniques are more reliable for effective modelling compared to the popular feature extraction-based methods [11], [28], [29].”
MethodDeep learning for robust feature generation in audiovisual emotion recognition
Kim, Y., Lee, H., & Provost, E. M. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638346 · Cited by 401
Emotion Assessment: Arousal Evaluation Using EEG’s and Peripheral Physiological Signals
Chanel, G., Kronegg, J., Grandjean, D., & Pun, T. (2006) · Lecture Notes in Computer Science · doi:10.1007/11848035_70 · Cited by 382
An empirical study of machine learning techniques for affect recognition in human–robot interaction
Rani, P., Liu, C., Sarkar, N., & Vanman, E. (2006) · Pattern Analysis and Applications · doi:10.1007/s10044-006-0025-y · Cited by 313
Learning deep physiological models of affect
Martinez, H. P., Bengio, Y., & Yannakakis, G. N. (2013) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2013.2247823 · Cited by 287 · cited 2×
“It has been shown that DL techniques are more reliable for effective modelling compared to the popular feature extraction-based methods [11], [28], [29].”
BackgroundMethodEmotion recognition from multi-channel EEG data through Convolutional Recurrent Neural Network
Li, X., Song, D., Zhang, P., Yu, G., Hou, Y., & Hu, B. (2016) · 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2016.7822545 · Cited by 283
Nakisa, B., Rastgoo, M. N., Tjondronegoro, D., & Chandran, V. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.09.062 · Cited by 233 · cited 3×
“In our previous study [21], we analysed the performance of conventional handcrafted feature extraction models using EEG and BVP signals.”
MethodInfluentialSelf-citationUtilizing Deep Learning Towards Multi-Modal Bio-Sensing and Vision-Based Affective Computing
Siddharth, Jung, T.-P., & Sejnowski, T. J. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2916015 · Cited by 232
Using Deep and Convolutional Neural Networks for Accurate Emotion Classification on DEAP Data
Tripathi, S., Acharya, S., Sharma, R., Mittal, S., & Bhattacharya, S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i2.19105 · Cited by 231 · cited 1×
“Deep learning has become the most effective method used in fusing different modalities in different domains, particularly in audio-visual speech recognition [10], [37] and affective computing [11], [37]–[39].”
MethodMultimodal emotion recognition using EEG and eye tracking data
Zheng, W.-L., Dong, B.-N., & Lu, B.-L. (2014) · 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/embc.2014.6944757 · Cited by 230 · cited 2×
“Some studies have shown early fusion can improve the performance of emotion recognition based on different modalities [7], [34], [35].”
BackgroundMethodMultimodal emotion recognition from expressive faces, body gestures and speech
Caridakis, G., Castellano, G., Kessous, L., Raouzaiou, A., Malatesta, L., Asteriadis, S., & Karpouzis, K. (2007) · IFIP The International Federation for Information Processing · doi:10.1007/978-0-387-74161-1_41 · Cited by 225 · cited 1×
“Some studies have shown early fusion can improve the performance of emotion recognition based on different modalities [7], [34], [35].”
BackgroundOnline Affect Detection and Robot Behavior Adaptation for Intervention of Children With Autism
Liu, C., Conn, K., Sarkar, N., & Stone, W. (2008) · IEEE Transactions on Robotics · doi:10.1109/tro.2008.2001362 · Cited by 209 · cited 1×
“Automated emotion recognition using lightweight body sensors and advanced machine learning technologies has been used in different application domains such as computer games [1], e-health [2], [3] and road safety [4].”
BackgroundMultimodal Emotion Recognition
Haq, S., & Jackson, P. J. B. (2010) · IGI Global eBooks · doi:10.4018/978-1-61520-919-4.ch017 · Cited by 203
Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data
Ringeval, F., Eyben, F., Kroupi, E., Yuce, A., Thiran, J.-P., Ebrahimi, T., Lalanne, D., & Schuller, B. (2015) · Pattern Recognition Letters · doi:10.1016/j.patrec.2014.11.007 · Cited by 202 · cited 1×
“Some studies have applied a ConvNet with a different number of layers to physiological signals to classify different emotions [28], [30]–[32].”
BackgroundAffect Recognition from Face and Body: Early Fusion vs. Late Fusion
Gunes, H., & Piccardi, M. (2006) · 2005 IEEE International Conference on Systems, Man and Cybernetics · doi:10.1109/icsmc.2005.1571679 · Cited by 191
Multimodal integration-a statistical view
Wu, L., Oviatt, S. L., & Cohen, P. R. (1999) · IEEE Transactions on Multimedia · doi:10.1109/6046.807953 · Cited by 190 · cited 1×
“Late fusion is most commonly used with a the combination of gestures and speech [36].”
BackgroundImproved Multimodal Deep Learning with Variation of Information
Sohn, K., Shang, W., & Lee, H. (2014) · neural information processing systems · Cited by 188
Koelstra, S., Yazdani, A., Soleymani, M., Mühl, C., Lee, J.-S., Nijholt, A., Pun, T., Ebrahimi, T., & Patras, I. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15314-3_9 · Cited by 186 · cited 2×
“It has been shown that emotion classification has often improved when EEG signals are combined with different modalities [15]–[17].”
BackgroundAudio-visual deep learning for noise robust speech recognition
Huang, J., & Kingsbury, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639140 · Cited by 179
Zhang, F., Haddad, S., Nakisa, B., Rastgoo, M. N., Cândido, C., Tjondronegoro, D., & de Dear, R. (2017) · Building and Environment · doi:10.1016/j.buildenv.2017.06.048 · Cited by 169 · cited 1×
“Although the accuracy of BVP is lower than that of electrocardiograms (ECGs), due to its simplicity BVP is widely used in biosensors developed for applications like office workers’ mental workload prediction [20].”
BackgroundSelf-citationMoon, S.-E., Jang, S., & Lee, J. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461315 · Cited by 160
Multi-modal Dimensional Emotion Recognition using Recurrent Neural Networks
Chen, S., & Jin, Q. (2015) · Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2808196.2811638 · Cited by 108
Remarks on SVM-based emotion recognition from multi-modal bio-potential signals
Takahashi, K. (2005) · IEEE RO-MAN · doi:10.1109/roman.2004.1374736 · Cited by 88
Remarks on emotion recognition from multi-modal bio-potential signals
Takahashi, K. (2005) · 2004 IEEE International Conference on Industrial Technology, 2004. IEEE ICIT '04. · doi:10.1109/icit.2004.1490720 · Cited by 86
Coverage of Emotion Recognition for Common Wearable Biosensors
Hui, T., & Sherratt, R. (2018) · Biosensors · doi:10.3390/bios8020030 · Cited by 80
Affective computing and medical informatics: state of the art in emotion-aware medical applications.
Luneski, A., Bamidis, P. D., & Hitoglou-Antoniadou, M. (2008) · PubMed · Cited by 34
Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective ComputingPreprint
Siddharth, S., Jung, T., & B, T. J. S. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07039 · Cited by 30 · cited 1×
“Some studies have applied a ConvNet with a different number of layers to physiological signals to classify different emotions [28], [30]–[32].”
BackgroundAffective Computing and Medical Informatics: State Of The Art in Emotion-Aware Medical Applications
Luneski, A., Bamidis, P., & Hitoglou-Antoniadou, M. (2008) · Medical Informatics Europe · doi:10.3233/978-1-58603-864-9-517 · Cited by 26 · cited 1×
“Automated emotion recognition using lightweight body sensors and advanced machine learning technologies has been used in different application domains such as computer games [1], e-health [2], [3] and road safety [4].”
BackgroundKhan, A. M., & Lawo, M. (2016) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-3-319-32703-7_248 · Cited by 20 · cited 1×
“One of the best indicators of different emotions is the BVP signal [5], [18], [19].”
Background
Research works printed but not linked: 2
- 2020, Frontiers|SAE+LSTM A New Framework for Emotion Recognition From Multi-Channel EEG | Frontiers in Neurorobotics
- nugraha, 2016, Classification of driver fatigue state based on EEG using emotiv EPOC, J Theor Appl Inf Technol, 86, 151
Cited by
Multimodal Emotion Recognition using Deep Learning
Abdullah, S. M. S. A., Ameen, S. Y. A., M. Sadeeq, M. A., & Zeebaree, S. (2021) · Journal of Applied Science and Technology Trends · doi:10.38094/jastt20291 · Cited by 347
“] [70] suggest a temporal fusion approach with deep learning model to capture the non-linear emotional association and enhance emotion classification efficiency.”
BackgroundMultimodal Emotion Recognition with Deep Learning: Advancements, challenges, and future directions
A.V., G., T., M., D., P., & E., U. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102218 · Cited by 225
Recognition of human emotions using EEG signals: A review
Rahman, M. M., Sarkar, A. K., Hossain, M. A., Hossain, M. S., Islam, M. R., Hossain, M. B., Quinn, J. M. W., & Moni, M. A. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104696 · Cited by 227
Emotion recognition in EEG signals using deep learning methods: A review
Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D. L., Gorriz, J. M., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107450 · Cited by 240
Driver stress detection via multimodal fusion using attention-based CNN-LSTM
Mou, L., Zhou, C., Zhao, P., Nakisa, B., Rastgoo, M. N., Jain, R., & Gao, W. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.114693 · Cited by 198
Self-citation
Metadata
- Authors
- Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V.
- Affiliations
- Deakin University (AU)
- Queensland University of Technology (AU)
- Venue
- IEEE Access
- Published
- 2020
- Volume
- 8
- Pages
- 225463-225474
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3089423349
- Cited by
- 101
- References
- 43
- Citations by year
- 2026: 15 · 2025: 22 · 2024: 19 · 2023: 22 · 2022: 15 · 2021: 8
- FWCI
- 10.88
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- QUT ePrints (Queensland University of Technology) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Griffith Research Online (submittedVersion, cc-by)
- Funders
- Queensland University of Technology (QUT) through the Centre for Robotics
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Keywords
- emotion recognition, multimodal fusion, temporal fusion, EEG, blood volume pulse, wearable sensors, physiological signals, feature-level fusion, decision fusion, LSTM, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.