Automatic Emotion Recognition Using Temporal Multimodal Deep Learning
Open accessThe DOI matches its Crossref record; 41 of the 43 references it cites were found in a registry or an index.
Abstract
Emotion recognition using miniaturised wearable physiological sensors has emerged as a revolutionary technology in various applications. However, detecting emotions using the fusion of multiple physiological signals remains a complex and challenging task. When fusing physiological signals, it is essential to consider the ability of different fusion approaches to capture the emotional information contained within and across modalities. Moreover, since physiological signals consist of time-series data, it becomes imperative to consider their temporal structures in the fusion process. In this study, we propose a temporal multimodal fusion approach with a deep learning model to capture the non-linear emotional correlation within and across electroencephalography (EEG) and blood volume pulse (BVP) signals and to improve the performance of emotion classification. The performance of the proposed model is evaluated using two different fusion approaches - early fusion and late fusion. Specifically, we use a convolutional neural network (ConvNet) long short-term memory (LSTM) model to fuse the EEG and BVP signals to jointly learn and explore the highly correlated representation of emotions across modalities, after learning each modality with a single deep network. The performance of the temporal multimodal deep learning model is validated on our dataset collected from smart wearable sensors and is also compared with results of recent studies. The experimental results show that the temporal multimodal deep learning models, based on early and late fusion approaches, successfully classified human emotions into one of four quadrants of dimensional emotions with an accuracy of 71.61% and 70.17%, respectively.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Still, more and more researchers use multimodal models that integrate multiple data and features to carry out emotion recognition, which can extract more abundant features and improve the effect of emotion recognition.”
From Multimodal Emotion Recognition From EEG Signals and Facial Expressions (2023) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“Still, more and more researchers use multimodal models that integrate multiple data and features to carry out emotion recognition, which can extract more abundant features and improve the effect of emotion recognition.”
From Multimodal Emotion Recognition From EEG Signals and Facial Expressions (2023) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Akselrod, S., Gordon, D., Ubel, F. A., Shannon, D. C., Berger, A. C., & Cohen, R. J. (1981) · Science · doi:10.1126/science.6166045 · Cited by 5,079 · cited 1×
“There are some studies that focus on extracting features from the time and frequency domains from the BVP signal [22]–[24].”
Background(2011) · Cited by 2,752
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664 · cited 1×
“Each participant watched nine video clips used in the MAHNOB dataset [40] to induce different emotions.”
MethodEmotion recognition system using short-term monitoring of physiological signals
Kim, K. H., Bang, S. W., & Kim, S. R. (2004) · Medical & Biological Engineering & Computing · doi:10.1007/bf02344719 · Cited by 944 · cited 1×
“Inner emotional states can affect the body’s physiological signals such as EEG and BVP signals which originate from these two components [5], [6].”
BackgroundConvolutional Neural Networks for P300 Detection with Application to Brain-Computer Interfaces
Cecotti, H., & Graser, A. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.125 · Cited by 778 · cited 1×
“DL methods have strong capabilities in constructing reliable features in different domains like speech recognition [25] and time-series data analysis [26], [27].”
MethodMultimodal Emotion Recognition in Response to Videos
Soleymani, M., Pantic, M., & Pun, T. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.37 · Cited by 722 · cited 1×
“In recent research, multimodal data are utilised to improve the performance of emotion classification [7]–[9].”
MethodTime Series Classification Using Multi-Channels Deep Convolutional Neural Networks
Zheng, Y., Liu, Q., Chen, E., Ge, Y., & Zhao, J. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-08010-9_33 · Cited by 652 · cited 1×
“DL methods have strong capabilities in constructing reliable features in different domains like speech recognition [25] and time-series data analysis [26], [27].”
MethodEmotion Recognition based on EEG using LSTM Recurrent Neural Network
Alhagry, S., Aly, A., & A., R. (2017) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2017.081046 · Cited by 645
Mandryk, R. L., & Atkins, M. S. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2006.11.011 · Cited by 538 · cited 1×
“Automated emotion recognition using lightweight body sensors and advanced machine learning technologies has been used in different application domains such as computer games [1], e-health [2], [3] and road safety [4].”
Background
Research works printed but not linked: 2
- 2020, Frontiers|SAE+LSTM A New Framework for Emotion Recognition From Multi-Channel EEG | Frontiers in Neurorobotics
- nugraha, 2016, Classification of driver fatigue state based on EEG using emotiv EPOC, J Theor Appl Inf Technol, 86, 151
Cited by
Multimodal Emotion Recognition using Deep Learning
Abdullah, S. M. S. A., Ameen, S. Y. A., M. Sadeeq, M. A., & Zeebaree, S. (2021) · Journal of Applied Science and Technology Trends · doi:10.38094/jastt20291 · Cited by 347
“] [70] suggest a temporal fusion approach with deep learning model to capture the non-linear emotional association and enhance emotion classification efficiency.”
BackgroundMultimodal Emotion Recognition with Deep Learning: Advancements, challenges, and future directions
A.V., G., T., M., D., P., & E., U. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102218 · Cited by 225
Recognition of human emotions using EEG signals: A review
Rahman, M. M., Sarkar, A. K., Hossain, M. A., Hossain, M. S., Islam, M. R., Hossain, M. B., Quinn, J. M. W., & Moni, M. A. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104696 · Cited by 227
Emotion recognition in EEG signals using deep learning methods: A review
Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D. L., Gorriz, J. M., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107450 · Cited by 240
Driver stress detection via multimodal fusion using attention-based CNN-LSTM
Mou, L., Zhou, C., Zhao, P., Nakisa, B., Rastgoo, M. N., Jain, R., & Gao, W. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.114693 · Cited by 198
Self-citationAudio based depression detection using Convolutional Autoencoder
Sardari, S., Nakisa, B., Rastgoo, M. N., & Eklund, P. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.116076 · Cited by 188
Self-citationDeep learning-based EEG emotion recognition: Current trends and future perspectives
Wang, X., Ren, Y., Luo, Z., He, W., Hong, J. S., & Huang, Y. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1126994 · Cited by 219
A review on EEG-based multimodal learning for emotion recognition
Pillalamarri, R., & Shanmugam, U. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11126-9 · Cited by 183
Multimodal Emotion Recognition From EEG Signals and Facial Expressions
Wang, S., Qu, J., Zhang, Y., & Zhang, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3263670 · Cited by 124
“For example, Nakisa et al. [11] proposed a time-series multimodal fusion method with a deep learning model to capture the nonlinear emotional correlation between and within EEG and blood volume pulse (BVP) signals and improve emotion classification performance.”
MethodDriver Emotion Recognition With a Hybrid Attentional Multimodal Fusion Framework
Mou, L., Zhao, Y., Zhou, C., Nakisa, B., Rastgoo, M. N., Ма, Л., Huang, T., Yin, B., Jain, R., & Gao, W. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3250460 · Cited by 90
“Nakisa showed that emotion recognition performance by multimodal data fusion is higher than that of single modality [29], [30].”
BackgroundSelf-citationEEG-based emotion recognition systems; comprehensive study
Hamzah, H. A., & Abdalla, K. K. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e31485 · Cited by 102
Driver Emotions Recognition Based on Improved Faster R-CNN and Neural Architectural Search Network
Zaman, K., Sun, Z., Shah, S. M., Shoaib, M. A., Pei, L., & Hussain, A. (2022) · Symmetry · doi:10.3390/sym14040687 · Cited by 70
“The authors of [3] developed a model specifically for drivers to determine whether or not they are in a stressful state.”
BackgroundCNN-LSTM for automatic emotion recognition using contactless photoplythesmographic signals
Mellouk, W., & Handouzi, W. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.104907 · Cited by 61
WeDea: A New EEG-Based Framework for Emotion Recognition
Kim, S.-H., Yang, H.-J., Nguyen, N. A. T., Prabhakar, S. K., & Lee, S.-W. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3091187 · Cited by 55
EEG-Based Emotion Recognition Using Spatial-Temporal-Connective Features via Multi-Scale CNN
Li, T., Fu, B., Wu, Z., & Liu, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3270317 · Cited by 54
“Nakisa use a convolutional neural network (ConvNet) long short-term memory (LSTM) model to fuse the EEG and BVP signals [11].”
MethodA Survey on EEG-Based Solutions for Emotion Recognition With a Low Number of Channels
Apicella, A., Arpaïa, P., Isgrò, F., Mastrati, G., & Moccaldi, N. (2022) · IEEE Access · doi:10.1109/access.2022.3219844 · Cited by 50
“The OpenBCI, the Mindlink, the Emotiv Insight, the Nexus10, the Nexus4, and the abmedica Helmate are employed in the minority of cases [82], [121], [122], [123], [124], [125], [126], [127].”
Facial expression recognition based on improved residual network
Zhang, W., Zhang, X., & Tang, Y. (2023) · IET Image Processing · doi:10.1049/ipr2.12743 · Cited by 44
Hybrid deep convolutional model-based emotion recognition using multiple physiological signals
Akbulut, F. P. (2022) · Computer Methods in Biomechanics & Biomedical Engineering · doi:10.1080/10255842.2022.2032682 · Cited by 47
“Several approaches have been presented to propose a robust emotion detection model with the multimodal data, one of them is utilized by Nakisa et al. (Nakisa et al. 2020).”
MethodMultimodal machine learning approach for emotion recognition using physiological signals
Ramadan, M. A., Salem, N. M., Mahmoud, L. N., & Sadek, I. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106553 · Cited by 45
Singh, P., Tripathi, M. K., Patil, M. B., Shivendra, & Neelakantappa, M. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19171-2 · Cited by 38
EEG-based affective brain–computer interfaces: recent advancements and future challenges
Chen, Y., Peng, Y. K., Tang, J., Camilleri, T. A., Camilleri, K. P., Kong, W., & Cichocki, A. S. (2025) · Journal of Neural Engineering · doi:10.1088/1741-2552/ade290 · Cited by 49
Multimodal EEG Emotion Recognition Based on the Attention Recurrent Graph Convolutional Network
Chen, J., Liu, Y., Xue, W., Hu, K., & Lin, W. (2022) · Information · doi:10.3390/info13110550 · Cited by 34
“Nakisa et al. [15] proposed a convolutional network and LSTM network cascaded Nakisa et al. [15] proposed a convolutional network and LSTM network cascaded model to capture the emotional correlation of EEG and BVP (blood volume pulse) modalities, which was experimentally performed on the MAHNOB…”
MethodTawsif, K., Aziz, N. A. A., Raja, J. E., Hossen, M. J., & Jesmeen, M. Z. H. (2022) · Emerging Science Journal · doi:10.28991/esj-2022-06-05-017 · Cited by 37
“[145] EEG , BVP MAHNOB/ 17 Arousal, Valence Video Sliding window strategy CNN, LSTM 71.”
MethodZou, Z., & Ergan, S. (2023) · Advanced Engineering Informatics · doi:10.1016/j.aei.2022.101868 · Cited by 35
Houssein, E. H., Hammad, A. S. A., Samee, N. M. A., Alohali, M. A., & Ali, A. A. (2024) · Cluster Computing · doi:10.1007/s10586-024-04590-5 · Cited by 36
“This will make it possible to employ it in the future for emotional real-time applications [70].”
Solar irradiance forecasting using a novel hybrid deep ensemble reinforcement learning algorithm
Jalali, S. M. J., Ahmadian, S., Nakisa, B., Khodayar, M., Khosravi, A., Nahavandi, S., Islam, S. M. S., Shafie‐khah, M., & Catalào, J. P. S. (2022) · Sustainable Energy Grids and Networks · doi:10.1016/j.segan.2022.100903 · Cited by 33
Self-citationHypercomplex Multimodal Emotion Recognition from EEG and Peripheral Physiological Signals
Lopez, E., Chiarantano, E., Grassucci, E., & Comminiello, D. (2023) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw59220.2023.10193329 · Cited by 44
“Thus in order to exploit the information contained in different modalities, recent studies adopt a multimodal approach for emotion recognition, some still relying on extracted features [2, 21, 22] and very few that directly employ raw data [23, 24, 25], where the latter focuses on perceived mental workload classificat…”
BackgroundSpeech-Based Depression Assessment: A Comprehensive Survey
Leal, S. S., Ntalampiras, S., & Sassi, R. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3521327 · Cited by 41
“In fact, the performance of the model depends significantly on the quality of the features [80].”
Al‐Shaery, A. M., Jassmi, H. A., Ahmed, S. G., Farooqi, N. S., Alhawsawi, A., Moussa, M., Tridane, A., & Alam, M. D. (2022) · IEEE Access · doi:10.1109/access.2022.3221771 · Cited by 25
eRAD-Fe: Emotion Recognition-Assisted Deep Learning Framework
Kim, S., Nguyen, N. A. T., Yang, H.-J., & Lee, S. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2021.3115195 · Cited by 28
Jafari, M., Tao, X., Barua, P. D., Tan, R., & Acharya, U. R. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102982 · Cited by 32
Amplified radio-over-fiber system linearization using recurrent neural networks
Pereira, L. A. M., Mendes, L. L., Bastos-Filho, C. J. A., & Arismar Cerqueira, S. (2023) · Journal of Optical Communications and Networking · doi:10.1364/jocn.474290 · Cited by 22
Driver stress levels detection system using hyperparameter optimization
Rastgoo, M. N., Nakisa, B., Rakotonirainy, A., Maire, F., & Chandran, V. (2022) · Journal of Intelligent Transportation Systems · doi:10.1080/15472450.2022.2140046 · Cited by 17
“…signals has been shown to be an efficient and effective approach to build a reliable system (J. Healey & Picard, 2000; Katsis et al., 2008; Nakisa et al., 2020; Singh et al., 2013), this approach presents a serious challenge to acceptability of the system by drivers in real-world…”
Self-citationAl‐Shaery, A. M., Ahmed, S. G., Jassmi, H. A., Alhawsawi, A., Maksoud, N., Tridane, A., & Farooqi, N. S. (2024) · IEEE Access · doi:10.1109/access.2024.3402230 · Cited by 19
Fiorini, L., Bossi, F., & Gruttola, F. D. (2024) · Scientific Reports · doi:10.1038/s41598-024-75263-x · Cited by 20
A novel signal channel attention network for multi-modal emotion recognition
Du, Z., Xia, Y., & Zhao, P. (2024) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2024.1442080 · Cited by 14
RETRACTED ARTICLE: Automatic emotion recognition using deep neural network
Sujatha, R., Chatterjee, J. M., Pathy, B., & Hu, Y. (2025) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20590-4 · Cited by 11
RetractedState-of-the-art Multimodal Emotion Recognition: A comprehensive survey and taxonomy
Yazici, A., Kucukyilmaz, T., Dokeroglu, T., Sharipbay, A., Lee, M.-H., & Tyler, B. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200642 · Cited by 11
Systematic Evaluation of Personalized Deep Learning Models for Affect Recognition
Han, Y., Zhang, P., park, M., & Lee, U. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3699724 · Cited by 23
“When considering a scenario of real-time affect state detection using wearable sensors, researchers typically use a 10-second window [12, 18, 54, 55].”
PHemoNet: A Multimodal Network for Physiological Signals
Lopez, E., Uncini, A., & Comminiello, D. (2024) · IEEE International Forum on Research and Technology for Society and Industry (RTSI) · doi:10.1109/rtsi61910.2024.10761462 · Cited by 18
“It begins to address challenges in the literature, such as extensive preprocessing requiring domain-specific knowledge [7], which prevents neural networks from learning important features from raw data, and the use of single modalities when emotional responses are intrinsically multimodal [8].”
Yılmaz, B. H., Köse, C., & Yılmaz, Ç. M. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10925-5 · Cited by 10
“Nakisa et al. proposed a fusion approach employing a deep learning model, assessing early and late fusion techniques using a CNN-LSTM model to fuse EEG and blood volume pulse (BVP) input data [7].”
Survey on Current Trend in Emotion Recognition Techniques Using Deep Learning
Adebiyi, M. O., Olaniyan, D., Adebiyi, A. A., & Okunola, A. A. (2023) · International Conference on Science, Engineering and Business for Sustainable Development Goals (SEB-SDG) · doi:10.1109/seb-sdg57117.2023.10124548 · Cited by 11
“12 [30] ConvNet and LSTM anger, happiness, fear, sadness, and surprise multimodal 71.”
BackgroundSelf-supervised representation learning using multimodal Transformer for emotion recognition
Goetz, T. I., Arora, P., Erick, F. X., Holzer, N., & Sawant, S. S. (2023) · International Workshop on Sensor-based Activity Recognition and Interaction (iWOAR) · doi:10.1145/3615834.3615837 · Cited by 15
“Lately, deep learning based approaches have been widely studied for effective emotion recognition and achieved promising results [9-11].”
BackgroundDeep Networks and Sensor Fusion for Personal Care Robot Tasks—A Review
Ramsai, N., & Sridharan, K. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3530007 · Cited by 8
“Another study [141] fuses EEG signals and blood volume pressure signals obtained from a headset and wristband, respectively, and gives this to a CNN-LSTM network for predicting human emotion.”
Physiology-based personalization of persuasive technology: a user modeling perspective
Spelt, H., Westerink, J. H. D. M., Frank, L. E., Ham, J. R. C., & IJsselsteijn, W. A. (2022) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-021-09313-8 · Cited by 11
“neural nets, fuzzy logic (Mandryk and Atkins 2007), temporal multimodal (Nakisa et al. 2020) or preference deep learning (Martinez et al.”
BackgroundMethodSingh, R., Gehlot, A., & Buddhi, D. (2022) · doi:10.13052/rp-9788770227667 · Cited by 9
Bio-Signal Based Multimodal Fusion with Bilinear Model for Emotion Recognition
Singh, A., Wittenberg, T. C., Salman, M.-M., Holzer, N., Göb, S., Pahl, J., Götz, T., & Sawant, S. S. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385273 · Cited by 10
“Since the information from different bio-signals can be complementary to each other, combining them can be beneficial than using individual bio-signal based emotion recognition [15].”
BackgroundMethodTakır, Ș., Toprak, E., Uluer, P., Barkana, D. E., & Köse, H. (2023) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3610661.3616240 · Cited by 9
“emotions into one of four quadrants of dimensional emotions [17].”
BackgroundSpace psychology: a comprehensive approach to the future of astronaut wellbeing
Nezami, A. (2025) · Frontiers in Virtual Reality · doi:10.3389/frvir.2024.1446796 · Cited by 6
Real-Time Emotion Detection in Live Video Streams using Multimodal Deep Learning Architectures
Rani, K. S., Muralidhar, A., Rao, G. S., Shareef, S. K., Sudheer, P., & Bhavsingh, M. (2025) · International Conference on Pervasive Computing and Social Networking (ICPCSN) · doi:10.1109/icpcsn65854.2025.11034807 · Cited by 5
“Based on this, Nakisa et al. [2] proposed a temporal deep learning model that efficiently models the dynamicity of emotional expression using multimodal inputs.”
Gu, X., & Hu, X. (2024) · Technology and Health Care · doi:10.1177/09287329241291376 · Cited by 5
“Nakisa et al. 16 designed a deep learning framework fusing time-series multimodal data, which efficiently captures the complex non-linear relationships between electroencephalogram (EEG) and blood volume pulse (BVP) signals, as well as within each of them, to optimize the recognition and…”
Zhao, Y., & Zhang, Y. (2024) · Alexandria Engineering Journal · doi:10.1016/j.aej.2024.12.045 · Cited by 7
Epileptic Seizure Detection Based on Attitude Angle Signal of Wearable Device
Wang, J., Hu, D., Lai, X., Jiang, T., Jiang, T., Gao, F., Vidal, P., & Cao, J. (2025) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2025.3529058 · Cited by 6
“The combined feature vector is subsequently used for training the relevant classifier [23].”
Pan, Y. (2025) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2025.0160613 · Cited by 6
“Nakisa et al. [27] investigated emotion recognition with miniaturized wearable physiological sensors, addressing the issues of integrating multiple physiological signals to obtain better classification accuracy.”
Enhancing Driver Emotion Recognition Through Deep Ensemble Classification
Zaman, F., Xu, Z., Hussain, A., Ullah, A., & Zaman, K. (2025) · Journal of Intelligent and Connected Vehicles · doi:10.26599/jicv.2025.9210055 · Cited by 4
A cheating detection system in online exams through real-time facial emotion recognition of students
Khabbachi, I., Mdaghri-Alaoui, G., Zouhair, A., & Mahboub, A. (2024) · International Conference on Computing, Internet of Things and Microwave Systems (ICCIMS) · doi:10.1109/iccims61672.2024.10690812 · Cited by 3
“They separated the internal state (driver's facial expressions) from the vehicle state into two distinct groups to collect experimental data for evaluation, with a focus on dashboard expressions [18].”
BackgroundMethodInfluentialEnd-to-End Emotion Recognition using Peripheral Physiological Signals
Pidgeon, M., Kanwal, N., Murray, N., & Asghar, M. N. (2022) · Electronic workshops in computing · doi:10.14236/ewic/hci2022.19 · Cited by 5
“Table 1: Feature Engineering for Emotion Recognition Reference Signal Feature Model Affective Rating This study GSR, Resp, BVP, Temp Raw CNN Arousal,Valence Nakisa et al. (2020) EEG Raw LSTM Quadrants Kanjo et al. (2019) HR,GSR,Temp Raw CNN-LSTM Valence Maier et al. (2019) BVP, EDA Raw CNN Discrete Martinez et al. (20…”
MethodClassifying Perceived Emotions based on Polarity of Arousal and Valence from Sound Events
Krishan, P., & Abri, F. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020353 · Cited by 5
“It is found that Blood Volume Pulse (BVP) and Electro Encephalo Gram (EEG) infer emotional states better [9].”
BackgroundMazzola, C., Romeo, M., Rea, F., Sciutti, A., & Cangelosi, A. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191452 · Cited by 4
“[27], which proved fusing streams between the convolutional and LSTM layers being beneficial rather than a late fusion after temporal training.”
BackgroundSawant, S. S., Erick, F. X., Arora, P., Pahl, J., Foltyn, A., Holzer, N., & Götz, T. (2023) · International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) · doi:10.1109/aciiw59127.2023.10388149 · Cited by 3
“Lately, deep learning based approaches have been gaining researcher’s attention due to automatic emotion-related feature extraction [9] [10] [11].”
BackgroundAdvances In EEG-Based Multimodal Emotion Recognition: A Comprehensive Review
Pillalamarri, R., & Shanmugam, U. (2023) · Innovations in Power and Advanced Computing Technologies (i-PACT) · doi:10.1109/i-pact58649.2023.10434480 · Cited by 3
“A hybrid temporal fusion model was used to combine the EEG and BVP signals [51].”
BackgroundMethodMifu-ER: Modality Quality Index-Based Incremental Fusion for Emotion Recognition
Kim, S.-H. (2025) · IEEE Access · doi:10.1109/access.2025.3584642 · Cited by 3
Abuali, N. A., Jadoon, I., Ullah, F., Hayajneh, M., & Alkobaisi, S. (2025) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2025.3580570 · Cited by 3
“The most sophisticated systems employ multimodal emotion sensing, integrating several methods for effective emotion sensing [14], [15].”
Liu, Y., & Rau, P. P. (2025) · International Journal of Social Robotics · doi:10.1007/s12369-025-01318-2 · Cited by 3
“While perfect emotion detection remains chal - lenging, recent advances in deep learning and sensor fusion have demonstrated sufficient reliability for basic emotional state classification (happy, sad, stressed, calm) in automo - tive contexts [ 10, 11].”
Comprehensive analysis of the state of art on emotion recognition using EEG
Mishra, A., & Ranjan, P. (2026) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2026.110958 · Cited by 5
A Systematic Literature Review of Multimodal Emotion Recognition
Rahayu, Y. D., Muharom, L. A., Windiarti, I. S., & Sugianto, A. H. (2023) · Scientific Journal of Informatics · doi:10.15294/sji.v10i2.43792 · Cited by 2
Automated Electroencephalogram-Based Emotion Detection: A Review
Gudikandula, N., Janapati, R., Sengupta, R., & Chintala, S. (2024) · International Conference on Intelligent Cyber Physical Systems and Internet of Things (ICoICI) · doi:10.1109/icoici62503.2024.10696472 · Cited by 3
“This study employs a ConvNet-LSTM model to fuse EEG and BVP signals, achieving 71.61% and 70.17% accuracy using early and late fusion methods, respectively, for classifying dimensional emotions [40].”
MethodInfluentialAn RFID-Based BCI System for Emotion Detection Using EEG patterns
Mishra, A., Singh, A., & Ujlayan, A. (2021) · IEEE International Conference on RFID Technology and Applications (RFID-TA) · doi:10.1109/rfid-ta53372.2021.9617423 · Cited by 3
Application of long short term memory algorithm in classification electroencephalogram
Huynh, V. Q., Nguyen, T., Tran, M. D., Le, A. N., Nguyen, P. T., & Huynh, T. V. (2021) · Science and Technology Development Journal - Natural Sciences · doi:10.32508/stdjns.v5i2.1006 · Cited by 1
Alves, C. L., Sallum, L. F., Rodrigues, F. A., Toutain, T. G. L. D. O., Aguiar, P. D. C., & Moeckel, M. (2025) · Journal of Physics Complexity · doi:10.1088/2632-072x/adf58a · Cited by 3
“This technique involves partitioning EEG signals into smaller time windows [56], typically lasting 10 s [43, 66, 70], which effectively increases the number of instances available for analysis.”
Pradhan, A., & Srivastava, S. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108541 · Cited by 1
Public Datasets for Multimodal Machine Learning-Driven Depression Detection: A Comprehensive Survey
Dou, Y., Li, S., Zhu, H., & Peng, S. (2025) · IEEE International Conference on Medical Artificial Intelligence (MedAI) · doi:10.1109/medai67139.2025.00016 · Cited by 2
“Filtering methods such as low-pass, high-pass, band-pass, and notch filters along with downsampling help reduce noise and improve SNR [46], [47].”
Pillalamarri, R., & Shanmugam, U. (2026) · Journal on Information Security · doi:10.1186/s13635-026-00230-0 · Cited by 1
Gharpure, N., Narayankar, M., & Tambat, I. (2022) · International Conference on Advances in Science and Technology (ICAST) · doi:10.1109/icast55766.2022.10039512 · Cited by 1
“In [18,19] a trimodal system was built for accurate predictions.”
BackgroundPhysiological Signals for Emotion Recognition
Taley, S. G., & Pund, M. A. (2023) · Algorithms for intelligent systems · doi:10.1007/978-981-99-4626-6_18 · Cited by 1
Hegh, A. N., Adeyelu, A. A., Iorliam, A., & Otor, S. U. (2025) · FUDMA Journal of Sciences · doi:10.33003/fjs-2025-0905-3412 · Cited by 1
Alves, C. L., Sallum, L. F., Rodrigues, F. A., Toutain, T. G. L. D. O., Aguiar, P. D. C., & Moeckel, M. (2024) · Research Square · doi:10.21203/rs.3.rs-4095364/v1 · Cited by 1
Abimala, T., Narmadha, T. V., & Raamesh, L. (2022) · Algorithms for intelligent systems · doi:10.1007/978-981-19-2126-1_21 · Cited by 1
Narayankar, M., Tambat, I., Gharpure, N., & Jayachandran, S. (2022) · IEEE India Conference · doi:10.1109/indicon56171.2022.10039746
“In [10], CNN and LSTM model is used to fuse the ECG and BVP signals to learn and explore the representation of emotions across modalities with an accuracy of 71.61%.”
MethodInfluentialPsychophysiology in the digital age
van der Mee, D. J. (2023) · doi:10.5463/thesis.258
A Wearable Multi-Sensor Fusion Approach for Gender Recognition based on Deep Learning
Roy, S., Nakisa, B., Pathirana, P. N., & Dazeley, R. (2023) · International Conference on Bioinformatics Research and Applications (ICBRA) · doi:10.1145/3632047.3632065
Self-citationJiang, L., Chen, D., Fan, L., & Mao, Z. (2026) · Complex & Intelligent Systems · doi:10.1007/s40747-026-02452-8
Channel and model selection for multi-channel EEG input to neural networks
Harachi, K., Yamamoto, Y., Muramatsu, A., Nagahara, H., Takemura, N., Shimojo, S., Furihata, D., & Mizuno‐Matsumoto, Y. (2024) · SICE Journal of Control Measurement and System Integration · doi:10.1080/18824889.2024.2385579 · Cited by 2
“Compared to early fusion, late fusion offers easier integration of multiple data modalities and leverages optimal classifiers for improved model performance [10].”
InfluentialEnhancing Sentiment Analysis Accuracy Through Multimodal Data Fusion: A Deep Learning Approach
Tiwari, S., Jain, A., & Mishra, A. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-6726-7_45
Machine Learning Algorithms for Emotion Detection using EEG Signals
Hossain, M. A., Biswas, S. K., Thounaojam, D. M., Khan, A., & Siddiquee, K. N. E. A. (2024) · IEEE International Conference on Computing, Applications and Systems (COMPAS) · doi:10.1109/compas60761.2024.10796614 · Cited by 2
“In order to improve the performance of emotion categorization, Nakisa et al. [15] introduced a temporal multimodal fusion strategy that uses a deep learning model to capture the non-linear emotional correlations inside and across electroencephalography (EEG) and blood volume pulse (BVP) signals.”
Machine Learning Approach of Audio-Visual Based Emotion Recognition: A Comparative Analysis
Selvi, R. S., Jeyalakshmi, C., Thangaraju, D., & Alagarsamy, M. (2025) · Journal of Circuits Systems and Computers · doi:10.1142/s0218126625502858
Bharaneedharan, M., & Kavya, G. (2025) · Journal of Circuits Systems and Computers · doi:10.1142/s0218126625503335
Cross-Model Implementation of Emotion Recognition Systems on Indian Datasets
Sharma, V., Aggarwal, N., & Kaur, N. (2025) · Cureus Journal of Computer Science. · doi:10.7759/s44389-025-08677-x
“Unimodal systems focus on a single modality, such as facial expressions, speech, to separately recognise emotions [5-7].”
Belabbaci, N. A., & Alam, M. A. U. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-10554-7_2
Human emotion recognition with deep learning model using EEG signals
Ramya, H. (2026) · AIP conference proceedings · doi:10.1063/5.0298198
Multimodal Signal Edge-Optimised Deep Learning Model for Real-Time Auditory Attention Classification
Prakash, A. J., Belkacem, A. N., Jelinek, H. F., Elfadel, I. M., & Atef, M. (2025) · IEEE Biomedical Circuits and Systems Conference (BioCAS) · doi:10.1109/biocas67066.2025.00013
Affect Recognition in Deaf Children Using Physiological Signal Measurements
Choudhary, K., & Prajapati, G. L. (2026) · SN Computer Science · doi:10.1007/s42979-025-04711-w
AI-Powered Emotion and Stress Detection: A WBAN-Based Approach for Real-Time Health Monitoring
Dhatrak, M., Jadhav, S. D., Vibhute, P. M., & Gupta, S. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-16099-7_8
The M-ArEye Database: A Multimodal Resource for Affect Recognition in Visually Impaired Children
Choudhary, K., & Prajapati, G. L. (2026) · IEEE International Conference on Interdisciplinary Approaches in Technology and Management for Social Innovation (IATMSI) · doi:10.1109/iatmsi68868.2026.11465269
Graph-based multimodal affect recognition in children using prototypical networks
Choudhary, K., & Prajapati, G. L. (2026) · Frontiers in Computer Science · doi:10.3389/fcomp.2026.1774796 · Cited by 1
Neuro-AI: Hybrid Model for Cognitive Enhancements
Sunanda, N., Jafrash, E., Goutham, M., Pranay, G., & Sathwik, K. (2026) · International Conference on Machine Learning and Autonomous Systems (ICMLAS) · doi:10.1109/icmlas67792.2026.11483969
“[5] Neuro-AI targets closed-loop cognitive support: (i) estimate cognitive state from EEG features using efficient ML; (ii) translate predictions into small, timely actions that fit study/work routines; [6] (iii) learn what works per user over time.”
Multilingual Speech Emotion Recognition Using End-to-End Multilingual Pre-trained Model
Das, A., & Haque, M. A. (2025) · IEEE International Conference on Signal Processing, Information, Communication and Systems (SPICSCON) · doi:10.1109/spicscon69221.2025.11504192
“A variety of modalities, including interactive data such as facial expressions [1], speech signals [2], [3], motion [4], text [4], [5] and physiological data such as EEG [6], ECG [7] etc. can be utilized for a comprehensive understanding of emotions.”
Deep Learning Approaches for Real-Time Affect Recognition
Selvi, R., & Vijayakumaran, C. (2026) · Automatic Control and Computer Sciences · doi:10.3103/s0146411626700070
“Zhang et al. [21] used DEAP and DECAF datasets to extract EEG and MOG features, including DCT, PSD, and HRV. Nakisa et al. [22] introduced a CNN–LSTM temporal multimodal model using late and early fusion, while Du et al. [23] combined HR and facial features in a BiLSTM–SOM network for emotion…”
Jiang, W., Mao, J., Quan, Z., Zhao, J., Xu, D., Long, Y., & Song, X. (2026) · Applied Spatial Analysis and Policy · doi:10.1007/s12061-026-09999-6
“The average accuracy of the fused model was above 0.68, which is similar to the indoor experiments (Liu et al., 2023; Nakisa et al., 2020; Zou et al., 2022).”
InfluentialCarvalho, C., Torres, J. M., & Moreira, R. S. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831116
Metadata
- Authors
- Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V.
- Affiliations
- Deakin University (AU)
- Queensland University of Technology (AU)
- Venue
- IEEE Access
- Published
- 2020
- Volume
- 8
- Pages
- 225463-225474
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3089423349
- Cited by
- 101
- References
- 43
- Citations by year
- 2026: 15 · 2025: 22 · 2024: 19 · 2023: 22 · 2022: 15 · 2021: 8
- FWCI
- 10.88
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- QUT ePrints (Queensland University of Technology) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Griffith Research Online (submittedVersion, cc-by)
- Funders
- Queensland University of Technology (QUT) through the Centre for Robotics
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Keywords
- emotion recognition, multimodal fusion, temporal fusion, EEG, blood volume pulse, wearable sensors, physiological signals, feature-level fusion, decision fusion, LSTM, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.