Multimodal Emotion Recognition using Deep Learning
Open accessThe DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.
Abstract
New research into human-computer interaction seeks to consider the consumer's emotional status to provide a seamless human-computer interface. This would make it possible for people to survive and be used in widespread fields, including education and medicine. Multiple techniques can be defined through human feelings, including expressions, facial images, physiological signs, and neuroimaging strategies. This paper presents a review of emotional recognition of multimodal signals using deep learning and comparing their applications based on current studies. Multimodal affective computing systems are studied alongside unimodal solutions as they offer higher accuracy of classification. Accuracy varies according to the number of emotions observed, features extracted, classification system and database consistency. Numerous theories on the methodology of emotional detection and recent emotional science address the following topics. This would encourage studies to understand better physiological signals of the current state of the science and its emotional awareness problems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Various experimental studies have attested to the superior performance of MER models over unimodal ones, placing MER at the vanguard of research within affective computing [77,78,79,80].”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [80]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
Zebari, D. A., Haron, H., Zeebaree, S. R. M., & Zeebaree, D. Q. (2019) · International Conference on Advanced Science and Engineering (ICOASE) · doi:10.1109/icoase.2019.8723779 · Cited by 46 · cited 1×
“To ensure meaningful communication, accurate interpretation of these emotions is important [5, 6].”
BackgroundSelf-citationRetractedEkman, P., & Friesen, W. V. (1975) · Cited by 4,604
Valence focus and arousal focus: Individual differences in the structure of affective experience.
Feldman, L. A. (1995) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.69.1.153 · Cited by 465
Automatic, Dimensional and Continuous Emotion Recognition
Gunes, H., & Pantic, M. (2010) · International Journal of Synthetic Emotions · doi:10.4018/jse.2010101605 · Cited by 426 · cited 1×
“However, emotional variations can be observed in physiological signals for a very short period of around 315seconds [73].”
BackgroundHigh-level feature representation using recurrent neural network for speech emotion recognition
Lee, J., & Tashev, I. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-336 · Cited by 388 · cited 1×
“[54] shows that using RNN bidirectional- (Bi-LSTM) is better for extracting essential speech characteristics for better speech”
MethodFacial emotion recognition using deep learning: review and insights
Mellouk, W., & Handouzi, W. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.07.101 · Cited by 339 · cited 1×
“Mehrabian observed that 7% of knowledge moves between people through writing, 38% through voice, and 55% through facial expression [36].”
BackgroundMulti-Sequence LSTM-RNN Deep Learning and Metaheuristics for Electric Load Forecasting
Bouktif, S., Fiaz, A., Ouni, A., & Serhani, M. A. (2020) · Energies · doi:10.3390/en13020391 · Cited by 256 · cited 2×
“In [60], to boost the efficiency of the multimodal fusion, a new algorithm and new architecture using the kernel matrix and a deep neural network are proposed.”
BackgroundMethodZhang, X., Liu, J., Shen, J., Li, S., Hou, K., Hu, B., Gao, J., Zhang, T., & Hu, B. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2987575 · Cited by 206 · cited 1×
“] [69] proposed a revolutionary multimodal fusion system with regularization focused on a new kernel matrix perspective and a deep network architecture.”
BackgroundMohan, K., Seal, A., Krejcar, O., & Yazidi, A. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2020.3031835 · Cited by 193 · cited 9×
“To efficiently map the subject's emotions with the extracted features from both ECG and PPG signals, a convolution neural network (CNN) was developed.”
BackgroundInfluentialLEARNet: Dynamic Imaging Network for Micro Expression Recognition
Verma, M., Vipparthi, S. K., Singh, G., & Murala, S. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2912358 · Cited by 173 · cited 9×
“Results show that the SVM algorithm has modified a lot to speech emotion recognition with the accuracy of 72.52 % using Berlin Emotional DB.”
BackgroundMethodInfluentialFacial Expression Classification Based on SVM, KNN and MLP Classifiers
Dino, H. I., & Abdulrazzaq, M. B. (2019) · International Conference on Advanced Science and Engineering (ICOASE) · doi:10.1109/icoase.2019.8723728 · Cited by 158 · cited 1×
“Human beings, in general, communicate their feelings in different ways[3, 4].”
BackgroundYazdeen, A. A., Zeebaree, S. R. M., Sadeeq, M. M., Kak, S. F., Ahmed, O. M., & Zebari, R. R. (2021) · Qubahan Academic Journal · doi:10.48161/qaj.v1n2a38 · Cited by 147 · cited 2×
“Therefore, emotional signals from multiple modalities may be used to predict the emotional state of a subject [11-13].”
BackgroundMethodSelf-citationSurvey on AI-Based Multimodal Methods for Emotion Detection
Marechal, C., Mikołajewski, D., Tyburek, K., Prokopowicz, P., Bougueroua, L., Ancourt, C., & Węgrzyn-Wolska, K. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-16272-6_11 · Cited by 137 · cited 11×
“To efficiently map the subject's emotions with the extracted features from both ECG and PPG signals, a convolution neural network (CNN) was developed.”
BackgroundMethodInfluentialMultimodal Emotion Recognition Using Cross-Modal Attention and 1D Convolutional Neural Networks
Krishna, D. N. G., & Patil, A. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1190 · Cited by 134 · cited 5×
“In [50] and [56], the authors used LTSM for classification, and both of them record significant improvement in performance and accuracy reaches 73.”
BackgroundMethodInfluentialFacial Emotion Recognition of Students using Convolutional Neural Network
Lasri, I., Solh, A. R., & Elbelkacemi, M. (2019) · International Conference on Intelligent Computing in Data Sciences (ICDS) · doi:10.1109/icds47004.2019.8942386 · Cited by 133 · cited 1×
“FER also has a wide variety of social life uses, such as smart protection, lie detection, and smart medical practice[46],[47].”
BackgroundFacial Expression Recognition in Videos Using Dynamic Kernels
Perveen, N., Roy, D., & Mohan, C. K. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3011846 · Cited by 125 · cited 1×
“Recognition of emotions is a dynamic process that targets the emotional state of the person, which means that the emotions corresponding to each individual's actions are different [1, 2].”
BackgroundExpression-EEG Based Collaborative Multimodal Emotion Recognition Using Deep AutoEncoder
Zhang, H. (2020) · IEEE Access · doi:10.1109/access.2020.3021994 · Cited by 122 · cited 2×
“In [58],[62], they both used CNN, and two physiological signals are considered, but in [62], more than one classification algorithms are used, which had a great impact on the results with an accuracy rate of more than 97% on SEED database.”
BackgroundMethodA first look into a Convolutional Neural Network for speech emotion detection
Bertero, D., & Fung, P. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7953131 · Cited by 117 · cited 1×
“Convolution neural network has also been used for speech emotion recognition by [53].”
MethodAttention Driven Fusion for Multi-Modal Emotion Recognition
Priyasad, D., Fernando, T., Denman, S., Sridharan, S., & Fookes, C. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054441 · Cited by 117 · cited 2×
“] [64] presented a deep learning-based approach to protect codes that are characteristic emotion.”
BackgroundMethodSalama, E. S., El-Khoribi, R. A., Shoman, M. E., & Wahby Shalaby, M. A. (2021) · Egyptian Informatics Journal · doi:10.1016/j.eij.2020.07.005 · Cited by 117 · cited 1×
“This work would categorize the input into its variations of valence and arousal [42].”
BackgroundSiriwardhana, S., Reis, A., Weerasekera, R., & Nanayakkara, S. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1212 · Cited by 113
SAANet: Siamese action-units attention network for improving dynamic facial expression recognition
Liu, D., Ouyang, X., Xu, S., Zhou, P., He, K., & Wen, S. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.06.062 · Cited by 109 · cited 1×
“Facial expression recognition plays an important role in many applications, including humancomputer interaction and health care [34, 35].”
BackgroundLan, Y.-T., Liu, W., & Lu, B. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207625 · Cited by 106 · cited 2×
“In [50] and [56], the authors used LTSM for classification, and both of them record significant improvement in performance and accuracy reaches 73.”
BackgroundMethodSkin Lesion Classification Based on Deep Convolutional Neural Networks Architectures
Saeed, J. N., & Zeebaree, S. R. M. (2021) · Journal of Applied Science and Technology Trends · doi:10.38094/jastt20189 · Cited by 102 · cited 1×
“[27] reviewed facial expression recognition models based on deep learning techniques, including DBN, deep CNN, Long Short Term Memory (LSTM) [48, 49] and their combination.”
MethodSelf-citationAutomatic Emotion Recognition Using Temporal Multimodal Deep Learning
Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V. (2020) · IEEE Access · doi:10.1109/access.2020.3027026 · Cited by 101 · cited 1×
“] [70] suggest a temporal fusion approach with deep learning model to capture the non-linear emotional association and enhance emotion classification efficiency.”
BackgroundLearning deep multimodal affective features for spontaneous speech emotion recognition
Zhang, S., Tao, X., Chuang, Y., & Zhao, X. (2020) · Speech Communication · doi:10.1016/j.specom.2020.12.009 · Cited by 99
Multi-Modal Attention for Speech Emotion Recognition
Pan, Z., Luo, Z., Yang, J., & Li, H. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1653 · Cited by 95
Classifying subject ratings of emotional speech using acoustic features
Liscombe, J., Venditti, J., & Hirschberg, J. (2003) · 8th European Conference on Speech Communication and Technology (Eurospeech 2003) · doi:10.21437/eurospeech.2003-306 · Cited by 94 · cited 2×
“(2003) extracted a series of continuous speech features based on fundamental pitch, amplitude, and spectral tilt and evaluated its relationship with various emotions [51].”
BackgroundMultimodal and Temporal Perception of Audio-visual Cues for Emotion Recognition
Ghaleb, E., Popa, M., & Asteriadis, S. (2019) · 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII) · doi:10.1109/acii.2019.8925444 · Cited by 93 · cited 2×
“For the last few years, deep learning [23, 26], relying on the most modern system, has achieved great success in many areas, such as signal processing, artificial intelligence, and emotion detection.”
BackgroundClassifying Subject Ratings of Emotional Speech Using Acoustic Features
Liscombe, J., Venditti, J. J., & Hirschberg, J. (2003) · doi:10.7916/d8f76mwv · Cited by 87
Abdulrazzaq, M. B., Mahmood, M. R., Zeebaree, S. R. M., Abdulwahab, M. H., Zebari, R. R., & Sallow, A. B. (2021) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1804/1/012055 · Cited by 69 · cited 1×
“Moreover, there are strong variations in peoples' feelings when asked to express periodic emotions [39, 40].”
BackgroundSelf-citationNovel deep learning model for facial expression recognition based on maximum boosted CNN and LSTM
Rajan, S., Chenniappan, P., Devaraj, S., & Madian, N. (2020) · IET Image Processing · doi:10.1049/iet-ipr.2019.1188 · Cited by 68 · cited 1×
“[27] reviewed facial expression recognition models based on deep learning techniques, including DBN, deep CNN, Long Short Term Memory (LSTM) [48, 49] and their combination.”
MethodDeep Learning Models Based on Image Classification: A Review
Obaid, K. B., Zeebaree, S. R. M., & Ahmed, O. M. (2020) · International Journal of Science and Business · doi:10.5281/zenodo.4108432 · Cited by 64
Self-citationCombining Best Features Selection Using Three Classifiers in Intrusion Detection System
Salih, A. A., & Abdulrazzaq, M. B. (2019) · International Conference on Advanced Science and Engineering (ICOASE) · doi:10.1109/icoase.2019.8723671 · Cited by 63 · cited 1×
“Therefore, emotional signals from multiple modalities may be used to predict the emotional state of a subject [11-13].”
BackgroundSurvey on Semantic Similarity Based on Document Clustering
Ibrahim, R. K., Zeebaree, S. R. M., & Jacksi, K. (2019) · Advances in Science Technology and Engineering Systems Journal · doi:10.25046/aj040515 · Cited by 56 · cited 1×
“Various algorithms to recognize feelings in human speaking have been proposed [52] over the years.”
BackgroundSelf-citationDynamic Objectives Learning for Facial Expression Recognition
Wen, G., Chang, T., Li, H., & Jiang, L. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2966858 · Cited by 56 · cited 1×
“FER also has a wide variety of social life uses, such as smart protection, lie detection, and smart medical practice[46],[47].”
BackgroundClassification techniques’ performance evaluation for facial expression recognition
Mahmood, M. R., Abdulrazzaq, M. B., Zeebaree, S. R. M., Ibrahim, A. K., Zebari, R. R., & Dino, H. I. (2020) · Indonesian Journal of Electrical Engineering and Computer Science · doi:10.11591/ijeecs.v21.i2.pp1176-1184 · Cited by 56 · cited 1×
“Affective computation is implemented in the world of school [20, 21], smart cards, and the scene of motor vehicles [22], entertainment to health-care [23].”
BackgroundSelf-citationTwo-Stage Fuzzy Fusion Based-Convolution Neural Network for Dynamic Emotion Recognition
Wu, M., Su, W., Chen, L., Pedrycz, W., & Hirota, K. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2966440 · Cited by 55 · cited 1×
“However, emotional recognition in our everyday lives is important for social contact, and emotions play an important role in deciding human actions[7, 8].”
BackgroundA Comparison of Three Classification Algorithms for Handwritten Digit Recognition
Abdulrazzaq, M. B., & Saeed, J. N. (2019) · International Conference on Advanced Science and Engineering (ICOASE) · doi:10.1109/icoase.2019.8723702 · Cited by 54 · cited 1×
“This is one reason why emotional recognition should be treated as a multimodal problem [18, 19].”
BackgroundChen, J., Lv, Y., Xu, R., & Xu, C. (2019) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2019.04.017 · Cited by 50
Asghar, M. A., Khan, M. J., Rizwan, M., Mehmood, R. M., & Kim, S. (2020) · Sensors · doi:10.3390/s20133765 · Cited by 45 · cited 1×
“] [71] employed features from four previously trained DNN architectures to identify EEG data.”
MethodChen, A., Xing, H., & Wang, F. (2020) · IEEE Access · doi:10.1109/access.2020.2980060 · Cited by 42 · cited 2×
“In contrast, in [44] they combine image and text as signal to recognition.”
BackgroundMulti-modal feature fusion based on multi-layers LSTM for video emotion recognition
Nie, W., Yan, Y., Song, D., & Wang, K. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-08796-8 · Cited by 37 · cited 1×
“] [72] proposed a multi-layer LSTM architecture to retrieve the system emotion recognition's multimodal input function.”
BackgroundResearch on Multi-modal Music Emotion Classification Based on Audio and Lyirc
Liu, G., & Tan, Z. (2020) · IEEE Information Technology, Networking, Electronic and Automation Control Conference (ITNEC) · doi:10.1109/itnec48623.2020.9084846 · Cited by 30 · cited 1×
“] [66] A new multimodal music emotion grouping system was developed based on music audio quality and text for music lyrics.”
BackgroundA Convolution Neural Network Based Emotion Recognition System using Multimodal Physiological Signals
Yang, C.-J., Fahier, N., Li, W.-C., & Fang, W.-C. (2020) · IEEE International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan) · doi:10.1109/icce-taiwan49838.2020.9258341 · Cited by 30 · cited 1×
“] [68] proposed a method of emotion detection based on electrocardiogram (ECG) and photoplethysmogram (PPG) signals as target sources of data entry.”
MethodDhaouadi, S., & Khelifa, M. M. B. (2020) · International Conference on Advanced Technologies for Signal and Image Processing · doi:10.1109/atsip49331.2020.9231666 · Cited by 28 · cited 1×
“] [67] evaluate the introduction of (LSTM) and Deep Neural (DNN) networks in young gamers for real-time tension monitoring.”
BackgroundThe Impact of Test Case Generation Methods on the Software Performance: A Review
Hasan, D. A., Hussan, B. K., Zeebaree, S. R. M., Ahmed, D. M., Kareem, O. S., & Sadeeq, M. A. M. (2021) · RePEc: Research Papers in Economics · doi:10.5281/zenodo.4623939 · Cited by 23
Self-citationThe Impact of Test Case Generation Methods on the Software Performance: A Review
Hasan, D. A., Hussan, B. K., Zeebaree, S. R. M., Ahmed, D. M., Kareem, O. S., & Sadeeq, M. A. M. (2021) · International Journal of Science and Business · doi:10.5281/zenodo.4623940 · Cited by 22
Self-citationImage Steganalysis in Frequency Domain Using Co-Occurrence Matrix and Bpnn
Khalifa, I. A., Zeebaree, S. R. M., Ataş, M., & Khalifa, F. M. (2019) · Science Journal of University of Zakho · doi:10.25271/sjuoz.2019.7.1.574 · Cited by 21 · cited 2×
“such as Garbor wavelet, Weber Local Descriptor (WLD), Local Binary Pattern (LBP) [43], multi-feature fusion, etc.”
BackgroundSelf-citationResearch on Multi-modal Mandarin Speech Emotion Recognition Based on SVM
Chen, C. (2019) · IEEE International Conference on Power, Intelligent Computing and Systems (ICPICS) · doi:10.1109/icpics47731.2019.8942545 · Cited by 21 · cited 3×
“This applies in humancomputer interface architecture, intelligent robotics, safety control [24, 25] etc.”
MethodCSAERNet: An Efficient Deep Learning Architecture for Image Classification
Zeebaree, S. R. M., Ahmed, O., & Obid, K. (2020) · International Conference on Engineering Technology and its Applications (IICETA) · doi:10.1109/iiceta50496.2020.9318859 · Cited by 21 · cited 1×
“Moreover, there are strong variations in peoples' feelings when asked to express periodic emotions [39, 40].”
BackgroundSelf-citationA Sustainable Multi-Modal Multi-Layer Emotion-Aware Service at the Edge
Hu, L., Li, W., Yang, J., Fortino, G., & Chen, M. (2019) · IEEE Transactions on Sustainable Computing · doi:10.1109/tsusc.2019.2928316 · Cited by 19
Optimal Image Steganography Content Destruction Techniques
Ameen, S. Y., & Al-Badrany, M. R. (2021) · International Journal of Computers and Communications · doi:10.46300/91013.2021.15.14 · Cited by 19 · cited 1×
“The valence can be positive to negative, and the arousal can be calm to excited [41].”
BackgroundSelf-citationEmotional Stability Detection Using Convolutional Neural Networks
Hussein, E. S. A., Qidwai, U. A., & Al-Meer, M. H. (2020) · IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT) · doi:10.1109/iciot48696.2020.9089440 · Cited by 18 · cited 9×
“To efficiently map the subject's emotions with the extracted features from both ECG and PPG signals, a convolution neural network (CNN) was developed.”
BackgroundInfluentialComparison of Four Classification Algorithms for Facial Expression Recognition
Dino, H. I., & Abdulrazzaq, M. B. (2020) · Polytechnic Journal · doi:10.25156/ptj.v10n1y2020.pp74-80 · Cited by 18 · cited 5×
“[54] shows that using RNN bidirectional- (Bi-LSTM) is better for extracting essential speech characteristics for better speech recognition performance [55].”
BackgroundInfluentialSemi-supervised Multimodal Emotion Recognition with Improved Wasserstein GANs
Liang, J., Chen, S., & Jin, Q. (2019) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc47483.2019.9023144 · Cited by 17 · cited 1×
“Therefore, emotional signals from multiple modalities may be used to predict the emotional state of a subject [11-13].”
BackgroundChinese Multimodal Emotion Recognition in Deep and Traditional Machine Leaming Approaches
Miao, H., Zhang, Y., Li, W., Zhang, H., Wang, D., & Feng, S. (2018) · Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia) · doi:10.1109/aciiasia.2018.8470379 · Cited by 16 · cited 1×
“Deep belief networks (DBN) [27, 28], convolution neural network (CNN) [29-31], recurrent neural networks (RNN) [32], these are the most commonly used approaches for deep learning [33].”
BackgroundA Multi-modal Approach for Emotion Recognition of TV Drama Characters Using Image and Text
Lee, J., Kim, H.-J., & Cheong, Y.-G. (2020) · IEEE International Conference on Big Data and Smart Computing (BigComp) · doi:10.1109/bigcomp48618.2020.00-37 · Cited by 16 · cited 2×
“[65] CNN Natural Language Processin g (NLP) __ Asian Character from the TV drama series”
BackgroundDynamic facial expression recognition model based on BiLSTM-Attention
Chen, L., Ouyang, Y., Zeng, Y., & Li, Y. (2020) · International Conference on Computer Science & Education (ICCSE) · doi:10.1109/iccse49874.2020.9201892 · Cited by 15 · cited 1×
“However, emotional recognition in our everyday lives is important for social contact, and emotions play an important role in deciding human actions[7, 8].”
BackgroundComparison of algorithms for use in adaptive adjustment of digital data receivers
Bateman, S. C., & Ameen, S. Y. (1990) · IEE Proceedings I Communications Speech and Vision · doi:10.1049/ip-i-2.1990.0012 · Cited by 12
Self-citationReal-time multimodal emotion recognition system based on elderly accompanying robot
Dou, S., Feng, Z., Yang, X., & Tian, J. (2020) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1453/1/012093 · Cited by 12 · cited 2×
“The Deep Neural Network model application to facial expression analysis is the hottest subject these days in facial recognition [45].”
MethodTwo-Stage Fuzzy Fusion Based-Convolution Neural Network for Dynamic Emotion Recognition
Chen, L., Wu, M., Witold Pedrycz, C., & Hirota, K. (2020) · Studies in computational intelligence · doi:10.1007/978-3-030-61577-2_7 · Cited by 12
Coordinator and router investigation in IEEE802.15.14 ZigBee wireless sensor network
Ameen, S. Y., & Nourildean, S. W. (2013) · doi:10.1109/iceccpce.2013.6998748 · Cited by 11 · cited 1×
“With cannot decide someone is emotional simply by looking at a particular entity or occurrence in front of our eyes [16, 17].”
BackgroundSelf-citationDeep Learning for Multimodal Emotion Recognition-Attentive Residual Disconnected RNN
Chandra, E., & Hsu, J. Y.-J. (2019) · Conference on Technologies and Applications of Artificial Intelligence (TAAI) · doi:10.1109/taai48200.2019.8959913 · Cited by 10 · cited 1×
“This is one reason why emotional recognition should be treated as a multimodal problem [18, 19].”
BackgroundReal Time Implementation of Stegofirewall System
al., A.-S. M. R. E. (2019) · International Journal of Computing and Digital Systems · doi:10.12785/ijcds/080507 · Cited by 9 · cited 1×
“With cannot decide someone is emotional simply by looking at a particular entity or occurrence in front of our eyes [16, 17].”
BackgroundMultimodal emotion recognition using deep learning techniques
Nguyen, T. D. (2020) · Queensland University of Technology · doi:10.5204/thesis.eprints.180753 · Cited by 9
Puzzling Out Emotions: A Deep-Learning Approach to Multimodal Sentiment Analysis
Shrivastava, V., Richhariya, V., & Richhariya, V. (2018) · International Conference on Advanced Computation and Telecommunication (ICACAT) · doi:10.1109/icacat.2018.8933684 · Cited by 7 · cited 1×
“To ensure meaningful communication, accurate interpretation of these emotions is important [5, 6].”
BackgroundMulti-modal Attention for Speech Emotion RecognitionPreprint
Pan, Z., Luo, Z., Yang, J., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.04107 · Cited by 7 · cited 2×
“In [58],[62], they both used CNN, and two physiological signals are considered, but in [62], more than one classification algorithms are used, which had a great impact on the results with an accuracy rate of more than 97% on SEED database.”
BackgroundMethodAttention Driven Fusion For Multi-Modal Emotion RecognitionPreprint
Priyasad, D., Fernando, T., Denman, S., Sridharan, S., & Fookes, C. (2020) · arXiv (Cornell University) · doi:10.17023/hqsn-6n05 · Cited by 6
Ammen, S. Y., Alfarras, M., & Hadi, W. A. H. (2010) · Advances in Computer Science and Engineering · doi:10.2316/p.2010.689-085 · Cited by 4 · cited 1×
“The various ways people communicate their feelings, both verbally and nonverbally, including expressive speech, facial gestures, body languages, etc [9, 10].”
BackgroundHandwritten Numerals' Recognition in Kurdish Language Using Double Feature Selection
Abdulrazzaq, M. B., & Khalaf, K. I. (2019) · International Conference on Engineering Technology and its Applications (IICETA) · doi:10.1109/iiceta47481.2019.9013012 · Cited by 4 · cited 1×
“This applies in humancomputer interface architecture, intelligent robotics, safety control [24, 25] etc.”
A comparison of decision-feedback equalizers for a 9600 bit/s modem
Clark, A. P., Abdullah, S. N., & Ameen, S. Y. (1988) · Journal of the Institution of Electronic and Radio Engineers · doi:10.1049/jiere.1988.0012 · Cited by 3 · cited 2×
“Multimodal emotion processing continues to see the widespread application in science [9].”
BackgroundSelf-citationA Sustainable Multi-modal Multi-layer Emotion-aware Service at the EdgePreprint
Hu, L., Li, W., Yang, J., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01810 · Cited by 3 · cited 1×
“Affective computation is implemented in the world of school [20, 21], smart cards, and the scene of motor vehicles [22], entertainment to health-care [23].”
BackgroundThe Contextual Dynamics of Multimodal Emotion Recognition in VideosPreprint
Bhattacharya, P., Gupta, R. K., & Yang, Y. (2020) · arXiv (Cornell University) · Cited by 2
Research works printed but not linked: 5
- [4]4. O. F. Mohammad, M. S. M. Rahim, S. R. M. Zeebaree, and F. Y. Ahmed, "A survey and analysis of the image encryption methods," International Journal of Applied Engineering Research, vol. 12, pp. 13265-13280, 2017.
- [14]14. H. Dino, M. B. Abdulrazzaq, S. R. Zeebaree, A. B. Sallow, R. R. Zebari, H. M. Shukur, et al., "Facial Expression Recognition based on Hybrid Feature Extraction Techniques with Different Classifiers," TEST Engineering & Management, vol. 83, pp. 22319-22329, 2020.
- [15]15. S. S. R. Zeebaree, S. Ameen, and M. Sadeeq, "Social Media Networks Security Threats, Risks and Recommendation: A Case Study in the Kurdistan Region," International Journal of Innovation, Creativity and Change, vol. 13, pp. 349-365, 2020.
- [28]28. S. Y. Ameen, F. M. Almusailkh, and M. H. Al-Jammas, "FPGA Implementation of Neural Networks Based Symmetric Cryptosystem," in 6th International Conference: Sciences of Electronic, Technologies of Information and Telecommunications May, 2011, pp. 12-15.
- [59]59. F. Al-Naima, S. Y. Ameen, and A. F. Al-Saad, "Destroying steganography content in image files," in IEEE Proceedings of Fifth International Symposium on Communication Systems, Networks and Digital Signal Processing, University of Patras, Patras, Greece, 2006.
Cited by
Zhang, S., Yang, Y., Chen, C., Zhang, X., Leng, Q., & Zhao, X. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121692 · Cited by 251
Emotion recognition in EEG signals using deep learning methods: A review
Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D. L., Gorriz, J. M., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107450 · Cited by 240
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
“Various experimental studies have attested to the superior performance of MER models over unimodal ones, placing MER at the vanguard of research within affective computing [77–80].”
BackgroundA systematic survey on multimodal emotion recognition using learning algorithms
Ahmed, N., Aghbari, Z. A., & Girija, S. (2023) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2022.200171 · Cited by 178
MSER: Multimodal speech emotion recognition using cross-attention with deep fusion
Khan, M., Gueaieb, W., El Saddik, A., & Kwon, S. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122946 · Cited by 161
Metadata
- Authors
- Abdullah, S. M. S. A., Ameen, S. Y. A., M. Sadeeq, M. A., & Zeebaree, S.
- Affiliations
- Duhok Polytechnic University (IQ)
- Venue
- Journal of Applied Science and Technology Trends
- Published
- 2021-05-30
- Volume
- 2
- Issue
- 01
- Pages
- 73-79
- ISSN
- 2708-0757 (ISSN-L)
- OpenAlex
- W3156576211
- Cited by
- 347
- References
- 72
- Citations by year
- 2026: 40 · 2025: 91 · 2024: 70 · 2023: 66 · 2022: 46 · 2021: 34
- FWCI
- 52.20
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Journal of Applied Science and Technology Trends (publishedVersion, cc-by-nc-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- multimodal emotion recognition, affective computing, human-computer interaction, facial expression recognition, physiological signals, neuroimaging, deep learning, emotion classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.