Multimodal Emotion Recognition using Deep Learning
Open accessThe DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.
Abstract
New research into human-computer interaction seeks to consider the consumer's emotional status to provide a seamless human-computer interface. This would make it possible for people to survive and be used in widespread fields, including education and medicine. Multiple techniques can be defined through human feelings, including expressions, facial images, physiological signs, and neuroimaging strategies. This paper presents a review of emotional recognition of multimodal signals using deep learning and comparing their applications based on current studies. Multimodal affective computing systems are studied alongside unimodal solutions as they offer higher accuracy of classification. Accuracy varies according to the number of emotions observed, features extracted, classification system and database consistency. Numerous theories on the methodology of emotional detection and recent emotional science address the following topics. This would encourage studies to understand better physiological signals of the current state of the science and its emotional awareness problems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Various experimental studies have attested to the superior performance of MER models over unimodal ones, placing MER at the vanguard of research within affective computing [77,78,79,80].”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [80]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
Zebari, D. A., Haron, H., Zeebaree, S. R. M., & Zeebaree, D. Q. (2019) · International Conference on Advanced Science and Engineering (ICOASE) · doi:10.1109/icoase.2019.8723779 · Cited by 46 · cited 1×
“To ensure meaningful communication, accurate interpretation of these emotions is important [5, 6].”
BackgroundSelf-citationRetractedEkman, P., & Friesen, W. V. (1975) · Cited by 4,758
Valence focus and arousal focus: Individual differences in the structure of affective experience.
Feldman, L. A. (1995) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.69.1.153 · Cited by 465
Automatic, Dimensional and Continuous Emotion Recognition
Gunes, H., & Pantic, M. (2010) · International Journal of Synthetic Emotions · doi:10.4018/jse.2010101605 · Cited by 426 · cited 1×
“However, emotional variations can be observed in physiological signals for a very short period of around 315seconds [73].”
BackgroundHigh-level feature representation using recurrent neural network for speech emotion recognition
Lee, J., & Tashev, I. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-336 · Cited by 388 · cited 1×
“[54] shows that using RNN bidirectional- (Bi-LSTM) is better for extracting essential speech characteristics for better speech”
MethodFacial emotion recognition using deep learning: review and insights
Mellouk, W., & Handouzi, W. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.07.101 · Cited by 339 · cited 1×
“Mehrabian observed that 7% of knowledge moves between people through writing, 38% through voice, and 55% through facial expression [36].”
BackgroundMulti-Sequence LSTM-RNN Deep Learning and Metaheuristics for Electric Load Forecasting
Bouktif, S., Fiaz, A., Ouni, A., & Serhani, M. A. (2020) · Energies · doi:10.3390/en13020391 · Cited by 256 · cited 2×
“In [60], to boost the efficiency of the multimodal fusion, a new algorithm and new architecture using the kernel matrix and a deep neural network are proposed.”
BackgroundMethodZhang, X., Liu, J., Shen, J., Li, S., Hou, K., Hu, B., Gao, J., Zhang, T., & Hu, B. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2987575 · Cited by 206 · cited 1×
“] [69] proposed a revolutionary multimodal fusion system with regularization focused on a new kernel matrix perspective and a deep network architecture.”
BackgroundMohan, K., Seal, A., Krejcar, O., & Yazidi, A. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2020.3031835 · Cited by 193 · cited 9×
“To efficiently map the subject's emotions with the extracted features from both ECG and PPG signals, a convolution neural network (CNN) was developed.”
BackgroundInfluentialLEARNet: Dynamic Imaging Network for Micro Expression Recognition
Verma, M., Vipparthi, S. K., Singh, G., & Murala, S. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2912358 · Cited by 174 · cited 9×
“Results show that the SVM algorithm has modified a lot to speech emotion recognition with the accuracy of 72.52 % using Berlin Emotional DB.”
BackgroundMethodInfluential
Research works printed but not linked: 5
- [4]4. O. F. Mohammad, M. S. M. Rahim, S. R. M. Zeebaree, and F. Y. Ahmed, "A survey and analysis of the image encryption methods," International Journal of Applied Engineering Research, vol. 12, pp. 13265-13280, 2017.
- [14]14. H. Dino, M. B. Abdulrazzaq, S. R. Zeebaree, A. B. Sallow, R. R. Zebari, H. M. Shukur, et al., "Facial Expression Recognition based on Hybrid Feature Extraction Techniques with Different Classifiers," TEST Engineering & Management, vol. 83, pp. 22319-22329, 2020.
- [15]15. S. S. R. Zeebaree, S. Ameen, and M. Sadeeq, "Social Media Networks Security Threats, Risks and Recommendation: A Case Study in the Kurdistan Region," International Journal of Innovation, Creativity and Change, vol. 13, pp. 349-365, 2020.
- [28]28. S. Y. Ameen, F. M. Almusailkh, and M. H. Al-Jammas, "FPGA Implementation of Neural Networks Based Symmetric Cryptosystem," in 6th International Conference: Sciences of Electronic, Technologies of Information and Telecommunications May, 2011, pp. 12-15.
- [59]59. F. Al-Naima, S. Y. Ameen, and A. F. Al-Saad, "Destroying steganography content in image files," in IEEE Proceedings of Fifth International Symposium on Communication Systems, Networks and Digital Signal Processing, University of Patras, Patras, Greece, 2006.
Cited by
Zhang, S., Yang, Y., Chen, C., Zhang, X., Leng, Q., & Zhao, X. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121692 · Cited by 251
Emotion recognition in EEG signals using deep learning methods: A review
Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D. L., Gorriz, J. M., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107450 · Cited by 240
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
“Various experimental studies have attested to the superior performance of MER models over unimodal ones, placing MER at the vanguard of research within affective computing [77–80].”
BackgroundA systematic survey on multimodal emotion recognition using learning algorithms
Ahmed, N., Aghbari, Z. A., & Girija, S. (2023) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2022.200171 · Cited by 178
MSER: Multimodal speech emotion recognition using cross-attention with deep fusion
Khan, M., Gueaieb, W., El Saddik, A., & Kwon, S. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122946 · Cited by 161
Attack and Anomaly Detection in IoT Networks using Machine Learning Techniques: A Review
Haji, S. H., & Ameen, S. Y. (2021) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2021/v9i230218 · Cited by 125
Self-citationMultimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG
Pan, J., Fang, W., Zhang, Z., Chen, B., Zhang, Z., & Wang, S. (2024) · IEEE Open Journal of Engineering in Medicine and Biology · doi:10.1109/ojemb.2023.3240280 · Cited by 121
“The ways of expressing emotions can be broadly classified into two categories: external representations, such as facial expressions and speech, and internal representations, such as electroencephalography (EEG) and heart rate [1].”
BackgroundEEG-Based Multimodal Emotion Recognition: A Machine Learning Perspective
Liu, H., Lou, T., Zhang, Y., Wu, Y., Xiao, Y., Jensen, C. S., & Zhang, D. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3369130 · Cited by 177
“A considerable volume of studies exists in the field of emotion recognition using multimodal physiological signals, and several reviews [27], [28] have been published.”
BackgroundWu, Y., Mi, Q., & Gao, T. (2025) · Biomimetics · doi:10.3390/biomimetics10070418 · Cited by 183
“The integration of various data sources allows for a more comprehensive and accurate analysis of emotional states compared to unimodal recognition, which relies on only one type of data [1].”
IoT and ICT based Smart Water Management, Monitoring and Controlling System: A Review
Yasin, H. M., Zeebaree, S. R. M., Sadeeq, M. A. M., Ameen, S. Y., Ibrahim, I. M., Zebari, R. R., Ibrahim, R. K., & Sallow, A. B. (2021) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2021/v8i230198 · Cited by 112
Self-citationFrom Emotion AI to Cognitive AI
Zhao, G., Li, Y., & Xu, Q. (2022) · International Journal of Network Dynamics and Intelligence · doi:10.53941/ijndi0101006 · Cited by 100
Hassan, R. J., Zeebaree, S. R. M., Ameen, S. Y., Kak, S. F., Sadeeq, M. A. M., Ageed, Z. S., Al-Zebari, A., & Salih, A. A. (2021) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2021/v8i330202 · Cited by 90
“The platform layer provides a graphical user interface (GUI) [23].”
BackgroundSelf-citationA Survey of Data Mining Implementation in Smart City Applications
Ageed, Z. S., Zeebaree, S. R. M., Sadeeq, M. M., Kak, S. F., Rashid, Z. N., Salih, A. A., & Abduallah, W. M. (2021) · Qubahan Academic Journal · doi:10.48161/qaj.v1n2a52 · Cited by 89
“Big data mining is a service that collects the essential information and expertise from and provides the customer with a large, complex, competitive, high-volume, and lowdensity data set [19].”
BackgroundSelf-citationA Comprehensive Survey of 5G mm-Wave Technology Design Challenges
Jijo, B. T., Zeebaree, S. R. M., Zebari, R. R., Sadeeq, M. A. M., Sallow, A. B., Mohsin, S., & Ageed, Z. S. (2021) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2021/v8i130190 · Cited by 89
Self-citationComprehensive Survey for Cloud Computing Based Nature-Inspired Algorithms Optimization Scheduling
Yahia, H. S., Zeebaree, S. R. M., Sadeeq, M. A. M., Salim, N. O. M., Kak, S. F., Al-Zebari, A., Salih, A. A., & Hussein, H. A. (2021) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2021/v8i230195 · Cited by 74
Self-citationA State of Art Survey for Intelligent Energy Monitoring Systems
Ageed, Z. S., Zeebaree, S. R. M., Sadeeq, M. A. M., Abdulrazzaq, M. B., Salim, B. W., Salih, A. A., Yasin, H. M., & Ahmed, A. M. (2021) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2021/v8i130192 · Cited by 73
“The new IoT concept provides an open path to create applications that organize popular traditional computers[12].”
BackgroundSelf-citationArtificial intelligence and Machine Learning for Real-world problems (A survey)
Ghahremani-Nahr, J., Nozari, H., & Sadeghi, M. E. (2021) · International Journal of Innovation in Engineering · doi:10.59615/ijie.1.3.38 · Cited by 91
“Deep learning algorithms take less time to perform tests than machine learning algorithms that increase test time with data size (Abdullah et al., 2021).”
Mutawa, A. M., & Hassouneh, A. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105942 · Cited by 61
Liu, Y., Jia, Z., & Wang, H. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612277 · Cited by 88
Speech Databases, Speech Features, and Classifiers in Speech Emotion Recognition: A Review
Dar, G. H. M., & Delhibabu, R. (2024) · IEEE Access · doi:10.1109/access.2024.3476960 · Cited by 88
“A key component of human connection and communication is emotion detection, and researchers are increasingly concentrating on exploiting non-verbal clues like body language and facial expressions to enhance the precision of emotion recognition systems [139].”
BackgroundEarly Melanoma Detection Based on a Hybrid YOLOv5 and ResNet Technique
Elshahawy, M., Elnemr, A., Oproescu, M., Șchiopu, A.-G., Elgarayhi, A., Elmogy, M. M., & Sallah, M. (2023) · Diagnostics · doi:10.3390/diagnostics13172804 · Cited by 64
“In addition, it is associated with melanocytes, which cause malignant cells to change color, thus accounting for the bulk of pigmented skin cancer deaths [4].”
BackgroundEarly Melanoma Detection Based on a Hybrid YOLOv5 and ResNet Technique
Elshahawy, M., Elnemr, A., Oproescu, M., Schiopu, A.-G., Elgarayhi, A., Elmogy, M. M., & Sallah, M. (2023) · doi:10.20944/preprints202307.1970.v1 · Cited by 51
MFGCN: Multimodal fusion graph convolutional network for speech emotion recognition
Qi, X., Wen, Y., Zhang, P., & Huang, H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128646 · Cited by 61
CAMM: Cross-Attention Multimodal Classification of Disaster-Related Tweets
Khattar, A., & Quadri, S. M. K. (2022) · IEEE Access · doi:10.1109/access.2022.3202976 · Cited by 52
“It is being applied to several fields like visual question answering [18], [19], sentiment analysis [20], [21], neural machine translation [22], cross-modal retrieval [23], speech recognition [24] and image captioning.”
MethodEfficiency of Malware Detection in Android System: A Survey
Omer, M. A., Zeebaree, S. R. M., Sadeeq, M. A. M., Salim, B. W., Mohsin, S., Rashid, Z. N., & Haji, L. M. (2021) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2021/v7i430189 · Cited by 46
Self-citationAl-Andoli, M. N., Tan, S. C., Sim, K. S., Lim, C. P., & Goh, P. Y. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109756 · Cited by 49
Hybrid deep convolutional model-based emotion recognition using multiple physiological signals
Akbulut, F. P. (2022) · Computer Methods in Biomechanics & Biomedical Engineering · doi:10.1080/10255842.2022.2032682 · Cited by 47
“One of the studies (Abdullah et al. 2021) aims to analyze the physiological signals (ECG, EDA, and electromyography (EMG)) with deep methods, Deep Neural Networks (DNN) and LSTM were employed, and the results showed that the LSTM model recognizes emotions with higher accuracy than DNN.”
MethodTsalera, E., Papadakis, A. E., Σαμαράκου, Μ., & Voyiatzis, I. (2022) · Applied Sciences · doi:10.3390/app12178455 · Cited by 59
Gao, Q., Long, T., & Zhou, Z. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122111 · Cited by 46
Chandrasekaran, G., Dhanasekaran, S., Moorthy, C., & Oli, A. A. (2024) · Computer Methods in Biomechanics & Biomedical Engineering · doi:10.1080/10255842.2024.2313066 · Cited by 45
“Their ability to automatically learn complex patterns and representations from data, especially when applied to large-scale datasets, has made them particularly suitable for sentiment analysis tasks (Abdullah Abdullah et al. 2021; Sehar et al. 2021).”
A Novel Approach to Cross dataset studies in Facial Expression Recognition
Ramis, S., Rubio, J. M. B., Perales, F. J., & Manresa-Yee, C. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13117-2 · Cited by 51
“More recently, deep learning methods have contributed to improve many research areas [1, 2, 43] and FER is not an exception [6, 16, 17, 20, 27, 28, 41, 45, 52].”
BackgroundA Comprehensive Study of Kernel (Issues and Concepts) in Different Operating Systems
Malallah, H. S., Zeebaree, S. R. M., Zebari, R. R., Sadeeq, M. A. M., Ageed, Z. S., Ibrahim, I. M., Yasin, H. M., & Merceedi, K. J. (2021) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2021/v8i330201 · Cited by 38
Self-citationEfficiency of Semantic Web Implementation on Cloud Computing: A Review
Taher, K. I., Saeed, R. H., Ibrahim, R. K., Rashid, Z. N., Haji, L. M., Omar, N., & Dino, H. I. (2021) · Qubahan Academic Journal · doi:10.48161/qaj.v1n3a72 · Cited by 36
“The models are displayed in each solution for cloud models (S to denote SaaS, P to denote PaaS, I to denote IaaS, P to denote PaaS).”
MethodInfluentialSQL Injection Attacks Prevention System Technology: Review
Kareem, F. Q., Ameen, S. Y., Salih, A. A., Ahmed, D. M., Kak, S. F., Yasin, H. M., Ibrahim, I. M., Ahmed, A. M., Rashid, Z. N., & Omar, N. (2021) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2021/v10i330242 · Cited by 36
Self-citationAn Ensemble-Based Parallel Deep Learning Classifier With PSO-BP Optimization for Malware Detection
Al-Andoli, M. N., Sim, K. S., Tan, S. C., Goh, P. Y., & Lim, C. P. (2023) · IEEE Access · doi:10.1109/access.2023.3296789 · Cited by 44
“computer vision [8], emotion recognition [9].”
Emotional Intelligence in Robotics: A Scoping Review
Marcos-Pablos, S., & García‐Peñalvo, F. J. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-87687-6_7 · Cited by 45
Exploring Empathy with Digital Humans
Loveys, K., Sagar, M. A., Billinghurst, M., Saffaryazdi, N., & Broadbent, E. (2022) · IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) · doi:10.1109/vrw55335.2022.00055 · Cited by 46
Emotion Classification from Speech and Text in Videos Using a Multimodal Approach
Caschera, M. C., Grifoni, P., & Ferri, F. (2022) · Multimodal Technologies and Interaction · doi:10.3390/mti6040028 · Cited by 43
“An example of a situation involving a lying person is a police interrogation about a jewel theft, as discussed in Section 2.”
BackgroundUtilizing social media and machine learning for personality and emotion recognition using PERS
Talaat, F. M., El-Gendy, E. M., Saafan, M. M., & Gamel, S. A. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08962-7 · Cited by 42
Smart Classroom Monitoring Using Novel Real-Time Facial Expression Recognition System
Fakhar, S., Baber, J., Bazai, S. U., Marjan, S., Jasiński, M., Jasińska, E., Chaudhry, M. U., Leonowicz, Z. M., & Hussain, S. (2022) · Applied Sciences · doi:10.3390/app122312134 · Cited by 50
“The most implemented and successful technique is “deep learning” [20–24].”
MethodAutomatic Fingerprint Classification Using Deep Learning Technology (DeepFKTNet)
Saeed, F., Hussain, M., & Aboalsamh, H. (2022) · Mathematics · doi:10.3390/math10081285 · Cited by 35
“It has been used for a wide range of tasks, including image and video recognition [41], classification of images [42], medical image analysis [43], computer vision [44], and natural language processing [45].”
BackgroundReview on Natural Language Processing Based on Different Techniques
Maulud, D. H., Ameen, S. Y., Omar, N., Kak, S. F., Rashid, Z. N., Yasin, H. M., Ibrahim, I. M., Salih, A. A., Salim, N. O. M., & Ahmed, D. M. (2021) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2021/v10i130231 · Cited by 29
“NLP strategies offer a means to improve web data with semantics by inserting details about entities and relations and knowing the real-world things referenced to assign a URI to each object [9,10].”
BackgroundSelf-citationWeb Server Performance Improvement Using Dynamic Load Balancing Techniques: A Review
Ibrahim, I. M., Ameen, S. Y., Yasin, H. M., Omar, N., Kak, S. F., Rashid, Z. N., Salih, A. A., Salim, N. O. M., & Ahmed, D. M. (2021) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2021/v10i130234 · Cited by 29
“Thus increasing or decreasing the number of servers in the cluster will alter the distribution of workload among available servers without affecting the interaction of users with the system [19-21].”
BackgroundSelf-citationA Combined CNN Architecture for Speech Emotion Recognition
Begazo, R., Aguilera, A., Dongo, I., & Cardinale, Y. (2024) · Sensors · doi:10.3390/s24175797 · Cited by 39
“However, emotion recognition remains a dynamic process and a complex task, as each person expresses emotions differently [68].”
BackgroundMultimodal learning model based on video–audio–chat feature fusion for detecting e-sports highlights
Park, G., Hyun, H.-I., & Kwon, H.-Y. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109285 · Cited by 26
TC-Net: A Modest & Lightweight Emotion Recognition System Using Temporal Convolution Network
Ishaq, M., Khan, M., & Kwon, S. (2023) · Computer Systems Science and Engineering · doi:10.32604/csse.2023.037373 · Cited by 33
Battisti, D. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00857-w · Cited by 25
“…used for text, such as speech emotion recognition models (Madanian et al., 2023), facial expression recognition models (Ge et al., 2022), or multimodal emotion recognition (Abdullah et al., 2021) that combine data from various sources, including facial expressions, vocal tone, and body language.”
M2R2: Missing-Modality Robust Emotion Recognition Framework With Iterative Data Augmentation
Wang, N., Cao, H., Zhao, J., Chen, R., Yan, D., & Zhang, J. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3201809 · Cited by 35
“Multimodalemotion recognitionsystemsarestudiedalongsideunimodalsolutionsas theyofferhigheraccuracyofclassification[19].”
An Assessment of In-the-Wild Datasets for Multimodal Emotion Recognition
Aguilera, A., Mellado, D., & Rojas, F. (2023) · Sensors · doi:10.3390/s23115184 · Cited by 34
“These architectures have been trained for ER using multimodal signals such as facial and audio gestures, audio and written language, physiological signals and different variations in those modalities [27].”
MethodA Review Of TextMining Techniques: Trends, and Applications In Various Domains
Aleqabie, H. J., Safoq, M. S., Albeer, R. A., & Hadi, E. (2024) · Iraqi Journal for Computer Science and Mathematics · doi:10.52866/ijcsm.2024.05.01.009 · Cited by 40
Utilizing emotion recognition technology to enhance user experience in real-time
Xu, Y., Lin, Y., Zhou, X., & Shan, X. (2024) · Computing and artificial intelligence. · doi:10.59400/cai.v2i1.1388 · Cited by 24
A Hierarchical Cross-Modal Spatial Fusion Network for Multimodal Emotion Recognition
Xu, M. F., Shi, T., Zhang, H., Liu, Z., & He, X. Y. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3523250 · Cited by 28
“In recent years, deep learning has emerged as the predominant research approach for emotion recognition [12], [13], [14], [15].”
Mamodiya, U., Kishor, I., Syed, A. A., Sankalkar, P., & Naik, N. (2025) · IEEE Access · doi:10.1109/access.2025.3603738 · Cited by 22
“Successful HRI systems increasingly rely on real-time affective understanding and adaptive decision-making principles to provide more natural, intuitive, and empathetic interactions than were achieved by previous rule-based systems [15].”
Dimitri, G. M. (2022) · Computers · doi:10.3390/computers11110163 · Cited by 22
Web Phishing Detection Using Web Crawling, Cloud Infrastructure and Deep Learning Framework
Abdulrahman, L. M., Ahmed, S. H., Rashid, Z. N., Jghef, Y. S., Sami, T. M. G., & Jader, U. H. (2023) · Journal of Applied Science and Technology Trends · doi:10.38094/jastt401144 · Cited by 35
Hussain, M., Chen, C., Hussain, M., Anwar, M., Abaker, M., Abdelmaboud, A., & Yamin, I. (2025) · Scientific Reports · doi:10.1038/s41598-025-16001-9 · Cited by 53
“Recent studies have also explored multi-modal approaches that combine textual data with other modalities, such as images and videos, to enhance emotion classification performance further 32,33 .”
Evaluating the viewer experience of interactive virtual reality movies
Carpio, R., Baumann, O., & Birt, J. (2023) · Virtual Reality · doi:10.1007/s10055-023-00864-2 · Cited by 24
“Therefore, drawing from previous research (Abdullah et al. 2021; Marechal et al. 2019; Saganowski et al. 2020), we propose a multimodal approach to assess audience impact, which can also contribute to the cognitive movement in film theory.”
MethodPinitas, K., Renaudie, D., Thomsen, M., Barthet, M., Makantasis, K., Liapis, A., & Yannakakis, G. N. (2023) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3577190.3614203 · Cited by 32
“Other modalities such as physiology and speech (audio) have also been extensively used for modeling affect, either individually or in a multimodal setting [1, 2, 24, 47].”
BackgroundHybrid multi-modal emotion recognition framework based on InceptionV3DenseNet
Alamgir, F. M., & Alam, M. S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15066-w · Cited by 24
AI‐Based Emotion Detection System in Healthcare for Patient
Jain, A., & Jain, A. (2025) · Wiley eBooks · doi:10.1002/9781394280735.ch23 · Cited by 23
Feature-Enhanced Multimodal Interaction model for emotion recognition in conversation
Fu, Y., Yan, X., Chen, W., & Zhang, J. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112876 · Cited by 22
Inter-Modality and Intra-Sample Alignment for Multi-Modal Emotion Recognition
Wang, Y., Li, D., & Shen, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446571 · Cited by 20
Emotion Fusion-Sense (Emo Fu-Sense) – A novel multimodal emotion classification technique
Umair, M., Rashid, N., Khan, U. S., Hamza, A., & Iqbal, J. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106224 · Cited by 23
Supervised Contrastive Learning for Affect Modelling
Pinitas, K., Makantasis, K., Liapis, A., & Yannakakis, G. N. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536221.3556584 · Cited by 22
“modal affect modeling) has been an active research field [2, 42]; thus we will only focus on a few indicative studies.”
BackgroundBaradaran, F., Farzan, A., Danishvar, S., & Sheykhivand, S. (2023) · Electronics · doi:10.3390/electronics12102216 · Cited by 16
“Human–computer interaction (HCI) performance can be improved by reading and understanding human emotional states [2].”
BackgroundA State of Art for Survey of Combined Iris and Fingerprint Recognition Systems
Ahmed, D. M., Ameen, S. Y., Omar, N., Kak, S. F., Rashid, Z. N., Yasin, H. M., Ibrahim, I. M., Salih, A. A., Salim, N. O. M., & Ahmed, A. M. (2021) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2021/v10i130232 · Cited by 18
Self-citationZhou, N., Jika, L., Wang, W., Xie, Y., Du, Y., & Soh, Y. C. (2025) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2025.3590031 · Cited by 24
“Although recent advancements in multimodal emotion recognition [28], [29], [30], [31] have yielded promising results, many existing methods fail to fully exploit intermodal interactions and remain partially dependent on unimodal feature extraction.”
Xefteris, V.-R., Tsanousa, A., Georgakopoulou, N., Diplaris, S., Vrochidis, S., & Kompatsiaris, I. Y. (2022) · Sensors · doi:10.3390/s22218198 · Cited by 22
“Multimodal emotion recognition can be complemented by the unique physiological responses that each modality provides [10].”
BackgroundA Multi-Level Circulant Cross-Modal Transformer for Multimodal Speech Emotion Recognition
Gong, P., Liu, J., Wu, Z., Han, B., Wang, Y. K., & He, H. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.028291 · Cited by 17
The Multimodal Emotion Information Analysis of E-Commerce Online Pricing in Electronic Word of Mouth
Chen, J., Zhong, Z., Feng, Q., & Liu, L. (2022) · Journal of Global Information Management · doi:10.4018/jgim.315322 · Cited by 20
“Commonly used methods for multimodal emotion recognition technology are Naive Bayes, decision tree learning, maximum entropy, Support Vector Machines (SVM), and Random Forest (RF) (Abdullah et al., 2021).”
MethodSemi-Supervised Multimodal Emotion Recognition with Expression MAE
Cheng, Z., Lin, Y., Chen, Z., Li, X., Mao, S., Zhang, F., Ding, D., Zhang, B., & Peng, X. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612840 · Cited by 40
A Review on Automation Artificial Neural Networks based on Evolutionary Algorithms
Zebari, R. R., Zeebaree, S. R. M., Rashid, Z. N., Shukur, H. M., Alkhayyat, A. H., & Sadeeq, M. A. M. (2021) · International Conference on Developments in eSystems Engineering (DeSE) · doi:10.1109/dese54285.2021.9719492 · Cited by 15
Self-citationFrom the Lab to the Wild: Affect Modeling Via Privileged Information
Makantasis, K., Pinitas, K., Liapis, A., & Yannakakis, G. N. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3265072 · Cited by 24
“Seng and Ang [39] presented emotion modeling techniques based on mulitmodal unstructured big-data, while Abdullah et al. [40] presented a survey on the application of deep learning models to multimodal emotion recognition.”
MethodMultimodal Emotional Classification Based on Meaningful Learning
Filali, H., Riffi, J., Boulealam, C., Mahraz, M. A., & Tairi, H. (2022) · Big Data and Cognitive Computing · doi:10.3390/bdcc6030095 · Cited by 15
“…emotion challenge (AVEC) 2012 [26], which presented baseline models for concatenating the audio and visual features into a single feature vector and used support vector regression to predict the continuous affective values, was one of the major attempts to advance the state of the art in MER.”
BackgroundAnalyzing Emotional Impact of Speech Balloons on Comic Characters: A Study of Arousal Levels
Rishu, Kukreja, V., & Kumar, A. (2023) · International Conference on Computational Intelligence and Sustainable Engineering Solutions (CISES) · doi:10.1109/cises58720.2023.10183595 · Cited by 17
Retracted: Advanced Face Detection using Machine Learning And AI-based Algorithm
Phatak, S. S., Patil, H. S., Arshad, M. W., Jitkar, B., Patil, S. S., & Patil, J. (2022) · International Conference on Contemporary Computing and Informatics (IC3I) · doi:10.1109/ic3i56241.2022.10072527 · Cited by 13
“Other than these various other algorithms are being used for the face detection and recognition system such as the Principal Component Analysis (PCA), Independent Component Analysis (ICA), Linear Discriminant Analysis (LDA), Elastic Bunch Graph Matching (EBGM), and lastly the Fisherfaces [18].”
MethodRetractedVotintseva, A., Johnson, R., & Villa, I. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60405-8_26 · Cited by 18
WavFusion: Towards Wav2vec 2.0 Multimodal Speech Emotion Recognition
Li, F., Luo, J., & Xia, W. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-2071-5_24 · Cited by 24
Jemioło, P., Storman, D., Mamica, M., Szymkowski, M., Żabicka, W., Wojtaszek-Główka, M., & Ligęza, A. (2022) · Sensors · doi:10.3390/s22072538 · Cited by 19
“Typically, several signals are collected in order to improve the accuracy of the AI system used for AAER [24].”
MethodDixit, C., & Satapathy, S. M. (2023) · Multimedia Systems · doi:10.1007/s00530-023-01188-6 · Cited by 17
“These applications include both research perspectives and practical applications [60].”
Backgroundde Santana, M. A., Fonseca, F. S., Torcate, A. S., & dos Santos, W. P. (2023) · Research on Biomedical Engineering · doi:10.1007/s42600-023-00293-9 · Cited by 19
“These challenges can be overcome by associating facial expression data with data from other sources (Silva et al. 1997; Abdullah et al. 2021).”
BackgroundKumar, A., Singh, S. K., Bhardwaj, I., Singh, P. K., Khanna, A. K., & Brahma, B. (2024) · International Journal of Information Technology · doi:10.1007/s41870-024-02166-5 · Cited by 13
“Psychiatric practices of finding emotional stages [1] have been started earlier but found to be very slow and expensive.”
Improving EEG based brain computer interface emotion detection with EKO ALSTM model
Kanna, R. K., Shoran, P., Yadav, M., Ahmed, M. N., Burje, S., Shukla, G., Sinha, A., Hussain, M. R., & Khalid, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-07438-z · Cited by 12
“Studies such as Zhang et al. 3 and He et al. 9 used conventional EEG feature sets with accuracies ranging between 78 and 88% under fivefold cross-validation, while works like Abdullah et al. 4 and Rathour et al. 13 implemented multimodal pipelines achieving accuracy levels in the 85–90% range.”
Smart Android Graphical Password Strategy: A Review
Izadeen, G. Y., & Ameen, S. Y. (2021) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2021/v9i230220 · Cited by 12
Self-citationAccurate Solar Wind Speed Prediction with Multimodality Information
Sun, Y., Xie, Z., Chen, Y., & Hu, Q. (2022) · Space Science & Technology · doi:10.34133/2022/9805707 · Cited by 14
“Multimodality is one of the rapidly developing research fi elds in deep learning, which has extensively promoted the development of visual question answering (VQA) [18, 24], visual commonsense reasoning (VCR) [25, 26], speech emotion recognition (SER) [27], and so on [28, 29].”
BackgroundNon-Uniform Attention Network for Multi-modal Sentiment Analysis
Wang, B., Dong, G., Zhao, Y., Li, R., Cao, Q., & Chao, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-98358-1_48 · Cited by 16
Unbiased Missing-Modality Multimodal Learning
Dai, R., Li, C., Yan, Y., Mo, L., Qin, K., & He, T. (2025) · 2025 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02272 · Cited by 11
“Its effectiveness is demonstrated across diverse applications, including visual question answering (VQA) [25] and affective computing [1], etc.”
YOLOv7 ‐ XAI : Multi‐Class Skin Lesion Diagnosis Using Explainable AI With Fair Decision Making
Veeramani, N., & Premaladha, J. (2024) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.23214 · Cited by 11
Facial Expression Recognition with CNN-SVM for Emotional State Classification
Rishu, Kukreja, V., & Kumar, A. (2023) · International Conference on Computing Communication Control and automation · doi:10.1109/iccubea58933.2023.10392007 · Cited by 11
“It cannot be denied that the creation of the newest technological devices has been influenced by human emotions [1].”
BackgroundState-of-the-art Multimodal Emotion Recognition: A comprehensive survey and taxonomy
Yazici, A., Kucukyilmaz, T., Dokeroglu, T., Sharipbay, A., Lee, M.-H., & Tyler, B. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200642 · Cited by 11
Tripartite interaction representation learning for multi-modal sentiment analysis
Wang, B., Dong, G., Zhao, Y., Li, R., Yin, W., & Lu, L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126279 · Cited by 13
Goel, D. P., Mahajan, K., Nguyen, N. D., Natesan, S., & Lim, C. P. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07723-2 · Cited by 12
“In summary, the significance and contributions of this research are as follows: • There is a diversity of algorithms that adopt hybrid/multi-modal architectures or specialized network configurations to enhance SER classification accuracy [1, 27].”
BackgroundClassification of Mental Health and Emotion of Human from Text using Machine Learning Approaches
Midhan, T. M., Selvaraj, P., Raju., M. H. K., Reddy., M. B. P., & Bhaskar., T. (2023) · International Conference on Information Systems and Computer Networks (ISCON) · doi:10.1109/iscon57294.2023.10111973 · Cited by 11
“[19] DL Categorial Emotions Performance depends on the type emotion 72.”
BackgroundSemi-supervised Multimodal Emotion Recognition with Consensus Decision-making and Label Correction
Tian, J., Hu, D., Shi, X., He, J., Li, X., Gao, Y., Toda, T., Xu, X., & Hu, X. (2023) · International Workshop on Multimodal and Responsible Affective Computing (MRAC) · doi:10.1145/3607865.3613182 · Cited by 14
Li, W.-C., Yang, C.-J., Liu, B.-T., & Fang, W.-C. (2021) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc46164.2021.9630979 · Cited by 13
Review and Evaluation of End-to-End Video Compression with Deep-Learning
Yasin, H. M., & Ameen, S. Y. (2021) · International Conference of Modern Trends in Information and Communication Technology Industry (MTICTI) · doi:10.1109/mticti53925.2021.9664790 · Cited by 10
Self-citationSurvey on Current Trend in Emotion Recognition Techniques Using Deep Learning
Adebiyi, M. O., Olaniyan, D., Adebiyi, A. A., & Okunola, A. A. (2023) · International Conference on Science, Engineering and Business for Sustainable Development Goals (SEB-SDG) · doi:10.1109/seb-sdg57117.2023.10124548 · Cited by 11
“Apart from the above, [25] suggested that in order to create a seamless human-computer interface, new research into human-computer interaction aims to take the consumer's emotional state into account.”
BackgroundStudent's Emotion Recognition using Multimodality and Deep Learning
Kalaiyarasi, M., Prasad, B. V. V. S., Ramesh, J. V. N., Kushwaha, R. K., Patel, R., & Balajee, J. M. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3654797 · Cited by 10
“A person's emotional state affects the feelings associated with their acts, which implies that the emotions associated with each person's activities are unique [1, 22].”
BackgroundReal Time Bimodal Emotion Recognition using Hybridized Deep Learning Techniques
Bhat, A. A., Kavitha, S., Satapathy, S. M., & Kavipriya, J. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.04.168 · Cited by 10
Overview of Multimodal Machine Learning
Al‐Zoghby, A. M., Al-Awadly, E. M. K., Ebada, A. I., & Awad, W. A. (2025) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3701031 · Cited by 18
“By integrating multiple modalities—such as text, images, audio, video, and sensor data—multimodal learning enables the extraction of richer representations and facilitates a more comprehensive understanding of complex data [ 2 ].”
Metadata
- Authors
- Abdullah, S. M. S. A., Ameen, S. Y. A., M. Sadeeq, M. A., & Zeebaree, S.
- Affiliations
- Duhok Polytechnic University (IQ)
- Venue
- Journal of Applied Science and Technology Trends
- Published
- 2021-05-30
- Volume
- 2
- Issue
- 01
- Pages
- 73-79
- ISSN
- 2708-0757 (ISSN-L)
- OpenAlex
- W3156576211
- Cited by
- 347
- References
- 72
- Citations by year
- 2026: 40 · 2025: 91 · 2024: 70 · 2023: 66 · 2022: 46 · 2021: 34
- FWCI
- 52.20
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Journal of Applied Science and Technology Trends (publishedVersion, cc-by-nc-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- multimodal emotion recognition, affective computing, human-computer interaction, facial expression recognition, physiological signals, neuroimaging, deep learning, emotion classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.