Multimodal Emotion Recognition using Deep Learning
Open accessThe DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.
Abstract
New research into human-computer interaction seeks to consider the consumer's emotional status to provide a seamless human-computer interface. This would make it possible for people to survive and be used in widespread fields, including education and medicine. Multiple techniques can be defined through human feelings, including expressions, facial images, physiological signs, and neuroimaging strategies. This paper presents a review of emotional recognition of multimodal signals using deep learning and comparing their applications based on current studies. Multimodal affective computing systems are studied alongside unimodal solutions as they offer higher accuracy of classification. Accuracy varies according to the number of emotions observed, features extracted, classification system and database consistency. Numerous theories on the methodology of emotional detection and recent emotional science address the following topics. This would encourage studies to understand better physiological signals of the current state of the science and its emotional awareness problems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Various experimental studies have attested to the superior performance of MER models over unimodal ones, placing MER at the vanguard of research within affective computing [77,78,79,80].”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [80]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
Zebari, D. A., Haron, H., Zeebaree, S. R. M., & Zeebaree, D. Q. (2019) · International Conference on Advanced Science and Engineering (ICOASE) · doi:10.1109/icoase.2019.8723779 · Cited by 46 · cited 1×
“To ensure meaningful communication, accurate interpretation of these emotions is important [5, 6].”
BackgroundSelf-citationRetractedEkman, P., & Friesen, W. V. (1975) · Cited by 4,604
Valence focus and arousal focus: Individual differences in the structure of affective experience.
Feldman, L. A. (1995) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.69.1.153 · Cited by 465
Automatic, Dimensional and Continuous Emotion Recognition
Gunes, H., & Pantic, M. (2010) · International Journal of Synthetic Emotions · doi:10.4018/jse.2010101605 · Cited by 426 · cited 1×
“However, emotional variations can be observed in physiological signals for a very short period of around 315seconds [73].”
BackgroundHigh-level feature representation using recurrent neural network for speech emotion recognition
Lee, J., & Tashev, I. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-336 · Cited by 388 · cited 1×
“[54] shows that using RNN bidirectional- (Bi-LSTM) is better for extracting essential speech characteristics for better speech”
MethodFacial emotion recognition using deep learning: review and insights
Mellouk, W., & Handouzi, W. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.07.101 · Cited by 339 · cited 1×
“Mehrabian observed that 7% of knowledge moves between people through writing, 38% through voice, and 55% through facial expression [36].”
BackgroundMulti-Sequence LSTM-RNN Deep Learning and Metaheuristics for Electric Load Forecasting
Bouktif, S., Fiaz, A., Ouni, A., & Serhani, M. A. (2020) · Energies · doi:10.3390/en13020391 · Cited by 256 · cited 2×
“In [60], to boost the efficiency of the multimodal fusion, a new algorithm and new architecture using the kernel matrix and a deep neural network are proposed.”
BackgroundMethodZhang, X., Liu, J., Shen, J., Li, S., Hou, K., Hu, B., Gao, J., Zhang, T., & Hu, B. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2987575 · Cited by 206 · cited 1×
“] [69] proposed a revolutionary multimodal fusion system with regularization focused on a new kernel matrix perspective and a deep network architecture.”
BackgroundMohan, K., Seal, A., Krejcar, O., & Yazidi, A. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2020.3031835 · Cited by 193 · cited 9×
“To efficiently map the subject's emotions with the extracted features from both ECG and PPG signals, a convolution neural network (CNN) was developed.”
BackgroundInfluentialLEARNet: Dynamic Imaging Network for Micro Expression Recognition
Verma, M., Vipparthi, S. K., Singh, G., & Murala, S. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2912358 · Cited by 173 · cited 9×
“Results show that the SVM algorithm has modified a lot to speech emotion recognition with the accuracy of 72.52 % using Berlin Emotional DB.”
BackgroundMethodInfluential
Research works printed but not linked: 5
- [4]4. O. F. Mohammad, M. S. M. Rahim, S. R. M. Zeebaree, and F. Y. Ahmed, "A survey and analysis of the image encryption methods," International Journal of Applied Engineering Research, vol. 12, pp. 13265-13280, 2017.
- [14]14. H. Dino, M. B. Abdulrazzaq, S. R. Zeebaree, A. B. Sallow, R. R. Zebari, H. M. Shukur, et al., "Facial Expression Recognition based on Hybrid Feature Extraction Techniques with Different Classifiers," TEST Engineering & Management, vol. 83, pp. 22319-22329, 2020.
- [15]15. S. S. R. Zeebaree, S. Ameen, and M. Sadeeq, "Social Media Networks Security Threats, Risks and Recommendation: A Case Study in the Kurdistan Region," International Journal of Innovation, Creativity and Change, vol. 13, pp. 349-365, 2020.
- [28]28. S. Y. Ameen, F. M. Almusailkh, and M. H. Al-Jammas, "FPGA Implementation of Neural Networks Based Symmetric Cryptosystem," in 6th International Conference: Sciences of Electronic, Technologies of Information and Telecommunications May, 2011, pp. 12-15.
- [59]59. F. Al-Naima, S. Y. Ameen, and A. F. Al-Saad, "Destroying steganography content in image files," in IEEE Proceedings of Fifth International Symposium on Communication Systems, Networks and Digital Signal Processing, University of Patras, Patras, Greece, 2006.
Cited by
Zhang, S., Yang, Y., Chen, C., Zhang, X., Leng, Q., & Zhao, X. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121692 · Cited by 251
Emotion recognition in EEG signals using deep learning methods: A review
Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D. L., Gorriz, J. M., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107450 · Cited by 240
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
“Various experimental studies have attested to the superior performance of MER models over unimodal ones, placing MER at the vanguard of research within affective computing [77–80].”
BackgroundA systematic survey on multimodal emotion recognition using learning algorithms
Ahmed, N., Aghbari, Z. A., & Girija, S. (2023) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2022.200171 · Cited by 178
MSER: Multimodal speech emotion recognition using cross-attention with deep fusion
Khan, M., Gueaieb, W., El Saddik, A., & Kwon, S. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122946 · Cited by 161
Metadata
- Authors
- Abdullah, S. M. S. A., Ameen, S. Y. A., M. Sadeeq, M. A., & Zeebaree, S.
- Affiliations
- Duhok Polytechnic University (IQ)
- Venue
- Journal of Applied Science and Technology Trends
- Published
- 2021-05-30
- Volume
- 2
- Issue
- 01
- Pages
- 73-79
- ISSN
- 2708-0757 (ISSN-L)
- OpenAlex
- W3156576211
- Cited by
- 347
- References
- 72
- Citations by year
- 2026: 40 · 2025: 91 · 2024: 70 · 2023: 66 · 2022: 46 · 2021: 34
- FWCI
- 52.20
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Journal of Applied Science and Technology Trends (publishedVersion, cc-by-nc-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- multimodal emotion recognition, affective computing, human-computer interaction, facial expression recognition, physiological signals, neuroimaging, deep learning, emotion classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal Emotion Recognition using Deep Learning. https://app.cochat.ai/papers/10.38094/jastt20291
- The paper
- Abdullah, S. M. S. A., Ameen, S. Y. A., M. Sadeeq, M. A., & Zeebaree, S. (2021). Multimodal Emotion Recognition using Deep Learning. Journal of Applied Science and Technology Trends. https://doi.org/10.38094/jastt20291
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.