Multimodal Emotion Recognition using Deep Learning
Open accessThe DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.
Abstract
New research into human-computer interaction seeks to consider the consumer's emotional status to provide a seamless human-computer interface. This would make it possible for people to survive and be used in widespread fields, including education and medicine. Multiple techniques can be defined through human feelings, including expressions, facial images, physiological signs, and neuroimaging strategies. This paper presents a review of emotional recognition of multimodal signals using deep learning and comparing their applications based on current studies. Multimodal affective computing systems are studied alongside unimodal solutions as they offer higher accuracy of classification. Accuracy varies according to the number of emotions observed, features extracted, classification system and database consistency. Numerous theories on the methodology of emotional detection and recent emotional science address the following topics. This would encourage studies to understand better physiological signals of the current state of the science and its emotional awareness problems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Various experimental studies have attested to the superior performance of MER models over unimodal ones, placing MER at the vanguard of research within affective computing [77,78,79,80].”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [80]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
Zebari, D. A., Haron, H., Zeebaree, S. R. M., & Zeebaree, D. Q. (2019) · International Conference on Advanced Science and Engineering (ICOASE) · doi:10.1109/icoase.2019.8723779 · Cited by 46 · cited 1×
“To ensure meaningful communication, accurate interpretation of these emotions is important [5, 6].”
BackgroundSelf-citationRetractedEkman, P., & Friesen, W. V. (1975) · Cited by 4,758
Valence focus and arousal focus: Individual differences in the structure of affective experience.
Feldman, L. A. (1995) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.69.1.153 · Cited by 465
Automatic, Dimensional and Continuous Emotion Recognition
Gunes, H., & Pantic, M. (2010) · International Journal of Synthetic Emotions · doi:10.4018/jse.2010101605 · Cited by 426 · cited 1×
“However, emotional variations can be observed in physiological signals for a very short period of around 315seconds [73].”
BackgroundHigh-level feature representation using recurrent neural network for speech emotion recognition
Lee, J., & Tashev, I. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-336 · Cited by 388 · cited 1×
“[54] shows that using RNN bidirectional- (Bi-LSTM) is better for extracting essential speech characteristics for better speech”
MethodFacial emotion recognition using deep learning: review and insights
Mellouk, W., & Handouzi, W. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.07.101 · Cited by 339 · cited 1×
“Mehrabian observed that 7% of knowledge moves between people through writing, 38% through voice, and 55% through facial expression [36].”
BackgroundMulti-Sequence LSTM-RNN Deep Learning and Metaheuristics for Electric Load Forecasting
Bouktif, S., Fiaz, A., Ouni, A., & Serhani, M. A. (2020) · Energies · doi:10.3390/en13020391 · Cited by 256 · cited 2×
“In [60], to boost the efficiency of the multimodal fusion, a new algorithm and new architecture using the kernel matrix and a deep neural network are proposed.”
BackgroundMethodZhang, X., Liu, J., Shen, J., Li, S., Hou, K., Hu, B., Gao, J., Zhang, T., & Hu, B. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2987575 · Cited by 206 · cited 1×
“] [69] proposed a revolutionary multimodal fusion system with regularization focused on a new kernel matrix perspective and a deep network architecture.”
BackgroundMohan, K., Seal, A., Krejcar, O., & Yazidi, A. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2020.3031835 · Cited by 193 · cited 9×
“To efficiently map the subject's emotions with the extracted features from both ECG and PPG signals, a convolution neural network (CNN) was developed.”
BackgroundInfluentialLEARNet: Dynamic Imaging Network for Micro Expression Recognition
Verma, M., Vipparthi, S. K., Singh, G., & Murala, S. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2912358 · Cited by 174 · cited 9×
“Results show that the SVM algorithm has modified a lot to speech emotion recognition with the accuracy of 72.52 % using Berlin Emotional DB.”
BackgroundMethodInfluential
Research works printed but not linked: 5
- [4]4. O. F. Mohammad, M. S. M. Rahim, S. R. M. Zeebaree, and F. Y. Ahmed, "A survey and analysis of the image encryption methods," International Journal of Applied Engineering Research, vol. 12, pp. 13265-13280, 2017.
- [14]14. H. Dino, M. B. Abdulrazzaq, S. R. Zeebaree, A. B. Sallow, R. R. Zebari, H. M. Shukur, et al., "Facial Expression Recognition based on Hybrid Feature Extraction Techniques with Different Classifiers," TEST Engineering & Management, vol. 83, pp. 22319-22329, 2020.
- [15]15. S. S. R. Zeebaree, S. Ameen, and M. Sadeeq, "Social Media Networks Security Threats, Risks and Recommendation: A Case Study in the Kurdistan Region," International Journal of Innovation, Creativity and Change, vol. 13, pp. 349-365, 2020.
- [28]28. S. Y. Ameen, F. M. Almusailkh, and M. H. Al-Jammas, "FPGA Implementation of Neural Networks Based Symmetric Cryptosystem," in 6th International Conference: Sciences of Electronic, Technologies of Information and Telecommunications May, 2011, pp. 12-15.
- [59]59. F. Al-Naima, S. Y. Ameen, and A. F. Al-Saad, "Destroying steganography content in image files," in IEEE Proceedings of Fifth International Symposium on Communication Systems, Networks and Digital Signal Processing, University of Patras, Patras, Greece, 2006.
Cited by
Items 301–348 of 348
Al-Hagery, M. A., Abunadi, I., Alamri, M. Z., & Yaseen, I. (2025) · International Journal of Humanoid Robotics · doi:10.1142/s0219843625400201
Li, R., Han, H., & Ai, H. (2025) · IEEE International Conference on Sensors, Electronics and Computer Engineering (ICSECE) · doi:10.1109/icsece65727.2025.11257100
“Abdullah et al. [1] (2021) explored the performance of multimodal deep learning models in emotion recognition tasks and demonstrated their effectiveness in fusing hierarchical features.”
Recognizing pilot emotional State: An approach based on multimodal data
Wang, H., Zhao, Y., Si, H., Pan, Y., Li, G., Li, Y., & Yang, C. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130772 · Cited by 2
Improving Mandarin Emotional Vocabulary in Application Writing Using Deep Learning Models
Feng, L. (2025) · Applied Mathematics and Nonlinear Sciences · doi:10.2478/amns-2025-1062
IoT-Enabled Real-Time Vehicle Monitoring and Wait Time Prediction System for Smart Fuel Stations
Singh, P., Kansal, M., Jain, J., Mishra, P., & Asthana, A. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-4272-6.ch014
Vision Transformers in Emotion and Demographic Analysis: A Review of Techniques and Applications
Jain, P., Nair, P. S., & Sharma, R. K. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-8343-7_46
Selvamani, B. R., S, S. S., & Balakrishnan, M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6017139
A multimodal reconstruction network via dynamic gradient modulation for emotion recognition
Li, A., Wang, Z., & Liu, Y. (2026) · Alexandria Engineering Journal · doi:10.1016/j.aej.2025.12.016
Interaction Modes for Virtual Digital Humans in Museums Under Convolutional Neural Networks
Zhu, E., & Wang, L. (2026) · International Journal of Information Technologies and Systems Approach · doi:10.4018/ijitsa.398929
Classification of Stress States Using Cardiac Impedance and Ground Reaction Force in Virtual Reality
Malik, M. S., Shafique, L., Khalid, S., Cheema, M. O., Din, Z. M. U., & Gul, J. Z. (2025) · International Multi-Topic Conference · doi:10.1109/inmic65900.2025.11348443
“The combination of cardiac impedance with biomechanical parameters in machine learning models will enable single-modality models to classify stress states [19][22].”
Face Expression Images and EEG Signals Based Emotion Recognition
Peng, Y., Wang, H., Sun, L., Ji, S., Zheng, J., Ma, Y., & Cao, R. (2025) · International Conference on Advanced Cloud and Big Data · doi:10.1109/cbd69312.2025.00073 · Cited by 1
ALTSA: Adaptive Latent-space Teacher - Student Architecture for Emotion Recognition in Conversation
Bhattarai, A., Cai, C., Li, M., Zhang, K., Li, M., & Pan, Z. (2025) · ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry (VRCAI) · doi:10.1145/3779232.3779470
“Deep learning has helped mitigate environmental noise and capture subtle emotional cues [Abdullah et al. 2021], yet many approaches still overlook individual differences in emotional expression.”
A Deep Learning Approach to Speech Emotion Recognition in Bengali Code-Mixed Speech
Islam, M., Tabassuma, Singha, A., Ghosh, K. P., Paul, S. K., & Roy, A. (2025) · International Conference on Sustainable Technologies for Industry 5.0 (STI) · doi:10.1109/sti69347.2025.11367575
“CNN, LSTM, and MMAN on English and German achieved 73.98% with complex training and no Bangla data [18]; SERNAC [20] classified Urdu speech with high accuracy but was limited to Urdu. k-NN on Bangla speech achieved 87.50% but faced generalization and dataset size issues [17].”
InfluentialMulti-Dimensional Visual Analysis of Digital Media Emotion Based on Machine Learning
Lu, J., Peng, Y., & Hoon, K. J. (2026) · International Journal of Mobile Human Computer Interaction · doi:10.4018/ijmhci.400917
Zhou, R., & Huang, Y. (2025) · International Conference on Science Technology Engineering and Mathematics (ICONSTEM) · doi:10.1109/iconstem65670.2025.11374781
Liu, C., Bao, J., Chen, T., Huang, B., Gan, Y., Huang, F., & Tao, X. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6294407
Ray - An Offline Multimodal Desktop Assistant with Integrated Voice and Gesture Recognition
Indresh, T. B., Shanbhag, A., R, S. K., Yadav, D., & Manjunath, E. C. (2026) · International Conference on Intelligent and Innovative Technologies in Computing, Electrical and Electronics (IITCEE) · doi:10.1109/iitcee67948.2026.11393991
“However, most desktop environments still lack adaptive interfaces capable of seamlessly integrating both vision and speech [5], [6].”
Divya, A., Sangeetha, T., Ponmaniraj, S., Anbazhagan, K., Bharathikannan, B., & Sj, K. (2025) · IEEE International Conference for Women in Computing (InCoWoCo) · doi:10.1109/incowoco68239.2025.11406937
“Therefore, examining button presses is a simple and unobtrusive method for looking at psychology [4].”
Real-Time Emotion and Gender Recognition Using Mini-Xception CNN
Indhuja, E., Mugila., Y., Izza, H., Reddy, Y. S. K., & Suchith, P. (2026) · International Conference on Mobile Computing and Sustainable Informatics (ICMCSI) · doi:10.1109/icmcsi67283.2026.11412818
An efficient multimodal deep learning model for emotion recognition in social media videos
Sangeetha, J., & Anu, V. M. (2026) · Discover Computing · doi:10.1007/s10791-026-09914-0
Leveraging Multi-modal Emotion Detection for Yoga Therapy Recommendations
Pandiyarajan, P., Maheswaran, B., Manasa, K., Pavani, K. J. S., Seenaiah, D., & Kumar, M. P. (2025) · International Conference on Emerging Computation and Information Technologies (ICECIT) · doi:10.1109/icecit67774.2025.11451001
“Abdullah et al. [7] showed that a variety of deep-learning techniques are applied for emotion detection from facial expressions and body movements, and also found that the results are better when there is fusion of two or more data types rather than using one data type only.”
AI-Powered Emotion and Stress Detection: A WBAN-Based Approach for Real-Time Health Monitoring
Dhatrak, M., Jadhav, S. D., Vibhute, P. M., & Gupta, S. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-16099-7_8
Deep Learning-Based Multimodal System for Behavioral Analysis in Criminal Investigations
Baidya, S., Layek, A. S., Misra, B., Paul, S., Desai, L. R., & Dey, N. (2026) · International Conference on Emerging Trends and Innovations in ICT (ICEI) · doi:10.1109/icei65890.2026.11447701
Utami, P., Anggraini, Y., Sudira, P., Fajaryati, N., & Rinawan, R. B. (2025) · International Conference on Artificial Intelligence, Blockchain, Cloud Computing, and Data Analytics (ICoABCD) · doi:10.1109/icoabcd67551.2025.11470740 · Cited by 1
“Multimodal affective analytics that combine textual, vocal, and visual evidence can therefore provide a more complete view of teacher and student emotion in TVET settings [50], [51].”
A Transfer Learning Framework for Multimodal Emotion Detection using Wav2Vec, EfficientNet, and LSTM
M.Jagadeesh, Duraimurugan, N., Karthikeyan., U., & Siddhan, S. (2026) · International Conference on Visual Analytics and Data Visualization (ICVADV) · doi:10.1109/icvadv67766.2026.11470451
“The recognition of emotions is a major component of the interactions between people and machines [11].”
Du, J., Zhi, Y., Li, J., Wang, H., & Guo, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460629
Alshareef, A. M., Alyoubi, K. H., & Alsobhi, A. Y. (2026) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2026.2653077
Hallur, S., & Gavade, A. B. (2026) · Signal Image and Video Processing · doi:10.1007/s11760-026-05344-z
Rana, A., Singh, K. J., & Kapoor, D. S. (2026) · IEEE International Conference on AI Engineering and Innovations (AIEI) · doi:10.1109/aiei69164.2026.11497146
“The majority of the earlier research has employed eye movements in order to examine the interest of users, visual search process, and information processing [12].”
Emotion Classification from EEG Signals Using Capsule Networks
Krishna, S., Natarajan, R., Premkumar, A., Alfurhood, B. S., Mahadev, N., & Thangarasu, N. (2026) · International Conference on Next-Gen Quantum and Advanced Computing: Algorithms, Security, and Beyond (NQComp) · doi:10.1109/nqcomp68334.2026.11497749
Chen, H., Jia, Z., & Jiang, T. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3691107
“Therefore,someemotionrecognitionmethodsbasedon multi-modalphysiologicalsignalshaveachievedstate-of-the-art performance[6],[7],[8],[9].”
Pushpakumar, R., & Seethalakshmi, K. (2025) · International Conference on Sustainable Computing and Smart Systems (ICSCSS) · doi:10.1109/icscss64956.2025.11500784
Bhargavi, M. S., Nikhil, B., Chandralekha, M., & Raj, M. R. (2026) · International Conference on Data Science, Agents & Artificial Intelligence (ICDSAAI) · doi:10.1109/icdsaai69492.2026.11505161
“This has been proven effective with the works by Shehu et al. [11] and Abdullah et al. [12], where fusion techniques have enhanced the performance of their models using multi-layer perceptron networks [13], [14], [17], [19].”
Multimodal Sentiment Analysis and Emotion Recognition Using Deep Learning
Sawant, A., Jumb, V., Misquitta, Z., Patil, A., & Shaikh, M. A. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-3-032-20544-5_29
Kayes, M. I., Prova, S. A., Chakma, R., Tasnim, N., & Ahmed, M. (2025) · IEEE International Women in Engineering (WIE) Conference on Electrical and Computer Engineering (WIECON-ECE) · doi:10.1109/wiecon-ece69386.2025.11526575
Mind on the Wheel: EEG and Telematics Fusion for Emotion-Aware Driver State Classification
Madhuri, C. R., Manikonda, D., & Madhurya, C. S. (2026) · International Conference on Trends in Electronics and Informatics (ICOEI) · doi:10.1109/icoei68323.2026.11541311
“Only a limited number of studies have explored multimodal strategies that integrate external behavioral data with internal physiological signals [1], [20].”
Spatio-Temporal Facial Emotion Recognition Using a Hybrid CNN–LSTM Framework
Lokesh, G., Kaliappan, S., Sumalatha, V., & Sneha (2026) · ASU International Conference in Emerging Technologies for Sustainability and Intelligent Systems (ICETSIS) · doi:10.1109/icetsis68266.2026.11549392
“…al. [3] surveyed recent progress in multimodal sentiment analysis, noting a recent trend from unimodal analysis to multimodal affect systems, while Abdullah et al. [4] demonstrated that deep learning-based multimodal emotion recognition can increase robustness through complementary representations.”
Emotion recognition through multimodal deep learning sensing: Text, image, and speech
Rani, M., Ahammad, S. T., Rani, V. P., Reddy, T. R., Kumar, H. R., & Prathyusha, D. (2026) · AIP conference proceedings · doi:10.1063/5.0342799
Emotion-driven music recommendation system using facial recognition
Boovitha, D., Aishwarya, Karthikeyini, Sandhya, & Tamilarasan (2026) · AIP conference proceedings · doi:10.1063/5.0343573
Enhancing children’s emotional education with speech emotion recognition using deep learning
Cui, N. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3758
Kumar, A., Verma, K., Rana, A. K., & Salau, A. O. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116710
Said, Y., Saidani, T. S., Algarni, A. M., Alsheikhy, A. A., & Fawzy, M. S. (2026) · Journal of Disability Research · doi:10.57197/jdr-2026-0844
Pre-trained Deep Learning Models Comparative Evaluation for Facial Emotion Recognition Using RAF-DB
Abosaq, H. A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-25638-6_17
An Approach for Joint Recognition of Intent and Emotion Based on Multimodal Data
Liu, M., Liu, Z., Song, P., Yu, Z., Zhang, K., & Du, Z. (2026) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse72781.2026.00029
“Compared to unimodal approaches, multimodal fusion provides more comprehensive, accurate, and robust analysis [28], [29].”
Chien, S.-K., Chou, Y.-C., Chao, P.-C., Park, H. R., Lin, Y.-J., Chen, C.-Y., Hsiao, P.-Y., Wu, M.-X., Chen, S.-L., Li, K.-C., et al. (2026) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2026.3733583
“Most deep learning-based MER methods are built on neural network architectures and algorithmic designs [36], [37], and the structure of these models directly impacts classification accuracy.”
Amini, A., Maqsood, S., Abbas, I., Sarwar, M. A., Maskeliūnas, R., & Damaševičius, R. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831113
Metadata
- Authors
- Abdullah, S. M. S. A., Ameen, S. Y. A., M. Sadeeq, M. A., & Zeebaree, S.
- Affiliations
- Duhok Polytechnic University (IQ)
- Venue
- Journal of Applied Science and Technology Trends
- Published
- 2021-05-30
- Volume
- 2
- Issue
- 01
- Pages
- 73-79
- ISSN
- 2708-0757 (ISSN-L)
- OpenAlex
- W3156576211
- Cited by
- 347
- References
- 72
- Citations by year
- 2026: 40 · 2025: 91 · 2024: 70 · 2023: 66 · 2022: 46 · 2021: 34
- FWCI
- 52.20
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Journal of Applied Science and Technology Trends (publishedVersion, cc-by-nc-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- multimodal emotion recognition, affective computing, human-computer interaction, facial expression recognition, physiological signals, neuroimaging, deep learning, emotion classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.