Multimodal emotion recognition: A comprehensive review, trends, and challenges
Abstract
Automatic emotion recognition is a burgeoning field of research and has its roots in psychology and cognitive science. This article comprehensively reviews multimodal emotion recognition, covering various aspects such as emotion theories, discrete and dimensional models, emotional response systems, datasets, and current trends. This article reviewed 179 multimodal emotion recognition literature papers from 2017 to 2023 to reflect on the current trends in multimodal affective computing. This article covers various modalities used in emotion recognition based on the emotional response system under four categories: subjective experience comprising text and self‐report; peripheral physiology comprising electrodermal, cardiovascular, facial muscle, and respiration activity; central physiology comprising EEG, neuroimaging, and EOG; behavior comprising facial, vocal, whole‐body behavior, and observer ratings. This review summarizes the measures and behavior of each modality under various emotional states. This article provides an extensive list of multimodal datasets and their unique characteristics. The recent advances in multimodal emotion recognition are grouped based on the research focus areas such as emotion elicitation strategy, data collection and handling, the impact of culture and modality on multimodal emotion recognition systems, feature extraction, feature selection, alignment of signals across the modalities, and fusion strategies. The recent multimodal fusion strategies are detailed in this article, as extracting shared representations of different modalities, removing redundant features from different modalities, and learning critical features from each modality are crucial for multimodal emotion recognition. This article summarizes the strengths and weaknesses of multimodal emotion recognition based on the review outcome, along with challenges and future work in multimodal emotion recognition. This article aims to serve as a lucid introduction, covering all aspects of multimodal emotion recognition for novices. This article is categorized under: Fundamental Concepts of Data and Knowledge > Human Centricity and User Interaction Technologies > Cognitive Computing Technologies > Artificial Intelligence
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01626-4 · Cited by 13,874 · see citing works
Evidence for a three-factor theory of emotions
Russell, J. A., & Mehrabian, A. (1977) · Journal of Research in Personality · doi:10.1016/0092-6566(77)90037-x · Cited by 1,940 · see citing works
Belief and feeling: Evidence for an accessibility model of emotional self-report.
Robinson, M. D., & Clore, G. L. (2002) · Psychological Bulletin · doi:10.1037/0033-2909.128.6.934 · Cited by 1,806 · see citing works
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644 · see citing works
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433 · see citing works
Wager, T. D., Phan, K. L., Liberzon, I., & Taylor, S. F. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00078-8 · Cited by 1,150 · see citing works
Inference of attitudes from nonverbal communication in two channels.
Mehrabian, A., & Ferris, S. R. (1967) · Journal of Consulting Psychology · doi:10.1037/h0024648 · Cited by 952 · see citing works
Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.94 · Cited by 852 · see citing works
Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review
Zhang, J., Yin, Z., Chen, P., & Nichele, S. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.01.011 · Cited by 806 · see citing works
Discrete Emotions or Dimensions? The Role of Valence Focus and Arousal Focus
Barrett, L. F. (1998) · Cognition & Emotion · doi:10.1080/026999398379574 · Cited by 645 · see citing works
Recognition of emotions using multimodal physiological signals and an ensemble deep learning model
Yin, Z., Zhao, M., Wang, Y., Yang, J., & Zhang, J. (2017) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2016.12.005 · Cited by 416 · see citing works
Considerations in the assessment of heart rate variability in biobehavioral research
Quintana, D. S., & Heathers, J. A. J. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00805 · Cited by 395 · see citing works
MSP-IMPROV: An Acted Corpus of Dyadic Interactions to Study Emotion Perception
Busso, C., Parthasarathy, S., Burmania, A., AbdelWahab, M., Sadoughi, N., & Provost, E. M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2515617 · Cited by 391 · see citing works
Psychophysiological stress and emg activity of the trapezius muscle
Lundberg, U., Kadefors, R., Melin, B., Palmerud, G., Hassmén, P., Engström, M., & Elfsberg Dohns, I. (1994) · International Journal of Behavioral Medicine · doi:10.1207/s15327558ijbm0104_5 · Cited by 381 · see citing works
Video-based emotion recognition in the wild using deep transfer learning and score fusion
Kaya, H., Gürpınar, F., & Salah, A. A. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.01.012 · Cited by 235 · see citing works
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224 · see citing works
Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems
Ayata, D., Yaslan, Y., & Kamasak, M. E. (2020) · Journal of Medical and Biological Engineering · doi:10.1007/s40846-019-00505-7 · Cited by 182 · see citing works
Automated emotion recognition: Current trends and future perspectives
Maithri, M., Raghavendra, U., Gudigar, A., Samanth, J., Barua, P. D., Murugappan, M., Chakole, Y., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106646 · Cited by 172 · see citing works
Pan, B., Hirota, K., Jia, Z., & Dai, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126866 · Cited by 142 · see citing works
Multimodal Emotion Recognition From EEG Signals and Facial Expressions
Wang, S., Qu, J., Zhang, Y., & Zhang, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3263670 · Cited by 124 · see citing works
Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition
Huang, Y., Yang, J., Liu, S., & Pan, J. (2019) · Future Internet · doi:10.3390/fi11050105 · Cited by 124 · see citing works
Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG
Pan, J., Fang, W., Zhang, Z., Chen, B., Zhang, Z., & Wang, S. (2024) · IEEE Open Journal of Engineering in Medicine and Biology · doi:10.1109/ojemb.2023.3240280 · Cited by 121 · see citing works
Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network
Zhang, Y., Cheng, C., & Zhang, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3049516 · Cited by 84 · see citing works
Multimodal emotion recognition based on manifold learning and convolution neural network
Zhang, Y., Cheng, C., & Zhang, Y. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13149-8 · Cited by 67 · see citing works
Deep Multimodal Emotion Recognition on Human Speech: A Review
Koromilas, P., & Giannakopoulos, T. (2021) · Applied Sciences · doi:10.3390/app11177962 · Cited by 58 · see citing works
Multimodal Attention Network for Continuous-Time Emotion Recognition Using Video and EEG Signals
Choi, D. Y., Kim, D.-H., & Song, B. C. (2020) · IEEE Access · doi:10.1109/access.2020.3036877 · Cited by 39 · see citing works
Muhammad, F., Hussain, M., & Aboalsamh, H. (2023) · Diagnostics · doi:10.3390/diagnostics13050977 · Cited by 36 · see citing works
Hierarchical fusion of visual and physiological signals for emotion recognition
Fang, Y., Rong, R., & Huang, J. (2021) · Multidimensional Systems and Signal Processing · doi:10.1007/s11045-021-00774-z · Cited by 17 · see citing works
Design of a hierarchy modular neural network and its application in multimodal emotion recognition
Li, W., Chu, M., & Qiao, J. (2019) · Soft Computing · doi:10.1007/s00500-018-03735-0 · Cited by 13 · see citing works
A Multi-Modal Fusion Method Based on Higher-Order Orthogonal Iteration Decomposition
Liu, F., Chen, J., Tan, W., & Cai, C. (2021) · Entropy · doi:10.3390/e23101349 · Cited by 13 · see citing works
Flagged first, then by citations
- Authors
- Ramaswamy, M. P. A., & Palaniswamy, S.
- Affiliations
- Amrita Vishwa Vidyapeetham (IN)
- Corresponding author
- Palaniswamy, Suja
- Venue
- WIREs Data Mining and Knowledge Discovery
- Publisher
- Wiley
- Published
- 2024-10-08
- Volume
- 14
- Issue
- 6
- ISSN
- 1942-4787 (ISSN-L)
- 1942-4795
- OpenAlex
- W4403262674
- Cited by
- 70
- References
- 30
- Citations by year
- 2026: 47 · 2025: 23
- FWCI
- 23.25
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- multimodal emotion recognition, affective computing, emotion theory, EEG, facial expressions, vocal emotion, multimodal dataset, emotion induction, cross-cultural emotion recognition, feature extraction, feature selection, multimodal fusion, signal alignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
