Deep Multimodal Emotion Recognition on Human Speech: A Review
Open accessThe DOI matches its Crossref record; 90 of the 90 references it cites were found in a registry or an index.
Abstract
This work reviews the state of the art in multimodal speech emotion recognition methodologies, focusing on audio, text and visual information. We provide a new, descriptive categorization of methods, based on the way they handle the inter-modality and intra-modality dynamics in the temporal dimension: (i) non-temporal architectures (NTA), which do not significantly model the temporal dimension in both unimodal and multimodal interaction; (ii) pseudo-temporal architectures (PTA), which also assume an oversimplification of the temporal dimension, although in one of the unimodal or multimodal interactions; and (iii) temporal architectures (TA), which try to capture both unimodal and cross-modal temporal dependencies. In addition, we review the basic feature representation methods for each modality, and we present aggregated evaluation results on the reported methodologies. Finally, we conclude this work with an in-depth analysis of the future challenges related to validation procedures, representation learning and method robustness.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“but failed to mention early fusion and late fusion techniques.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“With the advent of transformer networks [37], the architectures that produce word embeddings were further enriched with GPT (generative pre-trained) 1, 2 and 3 [38–40]; BERT (bidirectional encoder representations from transformers) [41]; ALBERT (a lite BERT) [42], etc.”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 2×
“GloVe (Global Vectors) [34] is one of the first models to produce robust word embeddings, followed by the FastText model [35], which also included sub-word information, and ELMo (Embeddings from Language Models) [36], where the notion of deep context-dependent word representations has been introduced.”
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“With the advent of transformer networks [37], the architectures that produce word embeddings were further enriched with GPT (generative pre-trained) 1, 2 and 3 [38–40]; BERT (bidirectional encoder representations from transformers) [41]; ALBERT (a lite BERT) [42], etc.”
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645 · cited 1×
“GloVe (Global Vectors) [34] is one of the first models to produce robust word embeddings, followed by the FastText model [35], which also included sub-word information, and ELMo (Embeddings from Language Models) [36], where the notion of deep context-dependent word representations has been…”
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · cited 5×
“ALBERT [42] embeddings for the text modality.”
InfluentialIEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · cited 1×
“One the contrary, IEMOCAP [74], the most widely used dataset, is collected in a detailed way in a lab environment.”
Ekman, P. (1993) · American Psychologist · doi:10.1037/0003-066x.48.4.384 · Cited by 3,441
librosa: Audio and Music Signal Analysis in Python
McFee, B., Raffel, C., Liang, D., Ellis, D., McVicar, M., Battenberg, E., & Nieto, O. (2015) · Proceedings of the Python in Science Conference · doi:10.25080/majora-7b98e3ed-003 · Cited by 3,033 · cited 1×
“There is a variety of open-source robust and efficient libraries, such as [22–25], that can be used in order to extract the aforementioned audio features.”
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3122291 · Cited by 2,983
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 2,912
(2011) · Cited by 2,752
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642 · cited 2×
“There is a variety of open-source robust and efficient libraries, such as [22–25], that can be used in order to extract the aforementioned audio features.”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369 · cited 2×
“…unsupervised methods for speech representation learning from audio are rapidly evolving and produce efficient architectures such as wav2vec [30], wav2vec 2.0 [31] and the newly produced HuBERT (Hidden-Unit BERT) [32], which is the currently state-of-the-art architecture for such representations.”
InfluentialMultimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259 · cited 9×
“MCTN [58] [70] UTA {A, T, V} Train-Val-Test 3-1-1 Sessions 84.”
BackgroundMethodInfluentialTensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 9×
“A different approach is that of tensor fusion, which was introduced by [6].”
BackgroundInfluentialMultimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433 · cited 4×
“For this reason, it is mostly used for sentiment analysis architectures (e.g., [63,66,67,70,83–85]).”
Influentialwav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors
Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1209 · Cited by 1,170 · cited 5×
“LMF [52] [52] NTA {A, T, V} Train-Val-Test 3-1-1 Sessions 87.”
BackgroundInfluentialContext-Dependent Sentiment Analysis in User-Generated Videos
Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1081 · Cited by 939 · cited 2×
“A work that tries to introduce the temporal dimension is proposed in [51].”
Speaker Recognition from Raw Waveform with SincNet
Ravanelli, M., & Bengio, Y. (2018) · 2018 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639585 · Cited by 896
Memory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855
Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions
Ringeval, F., Sonderegger, A., Sauer, J., & Lalanne, D. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553805 · Cited by 771 · cited 1×
“IEMOCAP [74] 11:28:12 10 {A, T, V} English CMU-MOSEI [63] 65:53:36 1000 {A, T, V} English RECOLA [75] 03:50:00 46 {A, V} French VAM [76] 12:00:00 20 {A, V} German”
COVAREP — A collaborative voice analysis repository for speech technologies
Degottex, G., Kane, J., Drugman, T., Raitio, T., & Scherer, S. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6853739 · Cited by 695
McKeown, G., Valstar, M., Cowie, R., Pantic, M., & Schroder, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.20 · Cited by 665
Collecting Large, Richly Annotated Facial-Expression Databases from Movies
Dhall, A., Goecke, R., Lucey, S., & Gedeon, T. (2012) · IEEE MultiMedia · doi:10.1109/mmul.2012.26 · Cited by 663 · cited 1×
“SEMAINE [79] 06:30:00 20 {A, V} English Greek Hebrew AFEW [80] 02:28:03 330 {A, V} English AFEW-VA [81] 00:40:00 240 {A, V} English Mimicry [82] 11:00:00 48 {A, V} English”
Emotion recognition from speech: a review
Koolagudi, S. G., & Rao, K. S. (2012) · International Journal of Speech Technology · doi:10.1007/s10772-011-9125-1 · Cited by 641 · cited 1×
“Such features are calculated in the frequency domain and are widely used since characteristics of the vocal tract are well represented in that space [12].”
Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis
Poria, S., Chaturvedi, I., Cambria, E., & Hussain, A. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0055 · Cited by 595 · cited 4×
“In [9], the authors, in order to learn representations from visual sequences, propose the convolutional recurrent multiple kernel learning (CRMKL) network which uses a convolutional RNN, i.”
InfluentialMao, Q., Dong, M., Huang, Z., & Zhan, Y. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2360798 · Cited by 568 · cited 1×
“For example, in [26–28], spectrograms were used as image inputs to CNNs in order to train classifiers for emotion recognition, and since then, this approach of combining spectrograms (or melgrams) with CNNs has been widely adopted for speech emotion recognition tasks.”
Evaluating deep learning architectures for Speech Emotion Recognition
Fayek, H. M., Lech, M., & Cavedon, L. (2017) · Neural Networks · doi:10.1016/j.neunet.2017.02.013 · Cited by 554 · cited 1×
“For example, in [26–28], spectrograms were used as image inputs to CNNs in order to train classifiers for emotion recognition, and since then, this approach of combining spectrograms (or melgrams) with CNNs has been widely adopted for speech emotion recognition tasks.”
Pham, H., Liang, P. P., Manzini, T., Morency, L.-P., & Póczos, B. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016892 · Cited by 512
Multi-attention Recurrent Network for Human Communication Comprehension
Zadeh, A., Liang, P. P., Poria, S., Vij, P., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12024 · Cited by 505
Neural Synchronization during Face-to-Face Communication
Jiang, J., Dai, B., Peng, D., Zhu, C., Liu, L., & Lu, C. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.2926-12.2012 · Cited by 489 · cited 2×
“Based on works by [3–5], the authors of [64] argue that the brain’s multimodal functionality is performed in a multi-stage manner, and, based on that, they introduce multi-stage fusion across modalities.”
Towards multimodal sentiment analysis: harvesting opinions from the web
Morency, L.-P., Mihalcea, R., & Doshi, P. (2011) · Proceedings of the 13th international conference on multimodal interfaces · doi:10.1145/2070481.2070509 · Cited by 470 · cited 2×
“In addition, as humans, we employ three modalities in a coordinated manner, in order to convey our intentions: language modality (words, phrases and sentences), vision modality (gestures and expressions) and acoustic modality (paralinguistics and changes in vocal tones) [1].”
BackgroundMethodWords Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
Wang, Y., Shen, Y., Liu, Z., Liang, P. P., Zadeh, A., & Morency, L.-P. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017216 · Cited by 466
pyAudioAnalysis: An Open-Source Python Library for Audio Signal Analysis
Giannakopoulos, T. (2015) · PLOS ONE · doi:10.1371/journal.pone.0144610 · Cited by 454
Speech Emotion Recognition from Spectrograms with Deep Convolutional Neural Network
Badshah, A. M., Ahmad, J., Rahim, N., & Baik, S. W. (2017) · 2017 International Conference on Platform Technology and Service (PlatCon) · doi:10.1109/platcon.2017.7883728 · Cited by 413
Sun, Z., Sarma, P., Sethares, W., & Liang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6431 · Cited by 410
Nonlinear feature based classification of speech under stress.
Zhou, G., Hansen, J. H. L., & Kaiser, J. F. (2001) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.905995 · Cited by 409 · cited 1×
“TEO-decomposed frequency modulation variation, normalized TEO autocorrelation and critical band-based TEO auto-correlation are three basic features produced by [21].”
The Vera am Mittag German audio-visual emotional speech database
Grimm, M., Kroschel, K., & Narayanan, S. (2008) · 2008 IEEE International Conference on Multimedia and Expo · doi:10.1109/icme.2008.4607572 · Cited by 407 · cited 1×
“IEMOCAP [74] 11:28:12 10 {A, T, V} English CMU-MOSEI [63] 65:53:36 1000 {A, T, V} English RECOLA [75] 03:50:00 46 {A, V} French VAM [76] 12:00:00 20 {A, V} German”
The Human Face as a Dynamic Tool for Social Communication
Jack, R. E., & Schyns, P. G. (2015) · Current Biology · doi:10.1016/j.cub.2015.05.052 · Cited by 405 · cited 1×
“According to [44], there are three main dimensions of facial variation—morphology, complexion and dynamics— with the third one being the most important for emotion recognition [45].”
Multimodal Speech Emotion Recognition Using Audio and Text
Yoon, S., Byun, S., & Jung, K. (2018) · 2018 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639583 · Cited by 351
MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · arXiv · doi:10.48550/arxiv.1606.06259 · Cited by 350 · cited 1×
“Late fusion, instead, trains unimodal classifiers independently and performs decision voting [10,11] or other classifier combination strategies.”
BackgroundSome useful properties of Teager's energy operators
Kaiser, J. F. (1993) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1993.319457 · Cited by 346 · cited 2×
“Teager Energy Operator-based features: TEO was introduced in [19,20] and was based on the observation that under stressful conditions, there is a change in fundamental frequency and critical bands of the hearing process due to the distribution of harmonics.”
MethodEvidence for Nonlinear Sound Production Mechanisms in the Vocal Tract
Teager, H. M., & Teager, S. M. (1990) · Speech Production and Speech Modelling · doi:10.1007/978-94-009-2037-8_10 · Cited by 326 · cited 2×
“Teager Energy Operator-based features: TEO was introduced in [19,20] and was based on the observation that under stressful conditions, there is a change in fundamental frequency and critical bands of the hearing process due to the distribution of harmonics.”
MethodLearning Factorized Multimodal Representations
Tsai, Y.-H. H., Liang, P. P., Zadeh, A., Morency, L.-P., & Salakhutdinov, R. (2018) · arXiv · doi:10.48550/arxiv.1806.06176 · Cited by 323 · cited 7×
“An example of such models is presented in [59], where the authors introduce the multimodal factorization model (MFM) that factorizes multimodal representations into multimodal discriminative factors and modality-specific generative factors.”
InfluentialDouglas-Cowie, E., Cowie, R., Sneddon, I., Cox, C., Lowry, O., McRorie, M., Martin, J.-C., Devillers, L., Abrilian, S., Batliner, A., et al. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74889-2_43 · Cited by 321 · cited 1×
“SEWA [77] 04:39:00 408 {A, V} Chinese English German Greek Hungarian Serbian HUMAINE [78] 04:11:00 4 {A, V} English”
BackgroundAutoencoder-based Unsupervised Domain Adaptation for Speech Emotion Recognition
Deng, J., Zhang, Z., Eyben, F., & Schuller, B. (2014) · IEEE Signal Processing Letters · doi:10.1109/lsp.2014.2324759 · Cited by 270 · cited 1×
“…for the problem of speech emotion recognition that are: (i) trained in a cross-dataset evaluation manner, while utilizing the power of unsupervised [90] or supervised [91] domain adaptation methodologies, in order to better evaluate their generalization power; (ii) capable of performing inference…”
Sewa db: A rich database for audio-visual emotion and sentiment research in the wild
Kossaifi, J., Walecki, R., Panagakis, Y., Shen, J., Schmitt, M., Ringeval, F., Han, J., Pandit, V., Toisoul, A., Schuller, B., et al. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2944808 · Cited by 243 · cited 1×
“SEWA [77] 04:39:00 408 {A, V} Chinese English German Greek Hungarian Serbian HUMAINE [78] 04:11:00 4 {A, V} English”
BackgroundDeep multimodal fusion for persuasiveness prediction
Nojavanasghari, B., Gopinath, D., Koushik, J., Baltrušaitis, T., & Morency, L.-P. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2993176 · Cited by 237 · cited 1×
“One of the basic fusion ideas is applied on the task of persuasiveness prediction by [50] but evaluated on multimodal SER in a variety of latter works.”
BackgroundMai, S., Hu, H., & Xing, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5347 · Cited by 237
(2013) · Cited by 236
AFEW-VA database for valence and arousal estimation in-the-wild
Kossaifi, J., Tzimiropoulos, G., Todorovic, S., & Pantic, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.02.001 · Cited by 236 · cited 1×
“SEMAINE [79] 06:30:00 20 {A, V} English Greek Hebrew AFEW [80] 02:28:03 330 {A, V} English AFEW-VA [81] 00:40:00 240 {A, V} English Mimicry [82] 11:00:00 48 {A, V} English”
Multimodal Language Analysis with Recurrent Multistage Fusion
Liang, P. P., Liu, Z., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1014 · Cited by 230 · cited 2×
“Based on works by [3–5], the authors of [64] argue that the brain’s multimodal functionality is performed in a multi-stage manner, and, based on that, they introduce multi-stage fusion across modalities.”
BackgroundSelect-additive learning: Improving generalization in multimodal sentiment analysis
Wang, H., Meghawat, A., Morency, L.-P., & Xing, E. P. (2017) · 2017 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2017.8019301 · Cited by 203 · cited 1×
“Late fusion, instead, trains unimodal classifiers independently and performs decision voting [10,11] or other classifier combination strategies.”
BackgroundSergent, J., & Signoret, J.-L. (1992) · Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rstb.1992.0007 · Cited by 186 · cited 1×
“…mechanism functions in order to perceive and understand communication: different regions independently process and understand different modalities [3,4], while, at the same time, these individual regions are interconnected using neural links to achieve multimodal information integration [5].”
Voice emotion recognition by cochlear-implanted children and their normally-hearing peers
Chatterjee, M., Zion, D. J., Deroche, M. L., Burianek, B. A., Limb, C. J., Goren, A. P., Kulkarni, A. M., & Christensen, J. A. (2015) · Hearing Research · doi:10.1016/j.heares.2014.10.003 · Cited by 182 · cited 1×
“More specifically, instead of the Mel-filter bank, it uses the Gammatone filter-bank on a range of sub-band energies [17].”
MethodMTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences
Yang, J., Wang, Y., Yi, R., Zhu, Y., Rehman, A., Zadeh, A., Poria, S., & Morency, L. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.79 · Cited by 172
Learning Alignment for Multimodal Emotion Recognition from Speech
Xu, H., Zhang, H., Han, K., Wang, Y., Peng, Y., & Li, X. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-3247 · Cited by 170 · cited 1×
“One of the first works in this category is proposed for the speech emotion recognition problem in [68] and utilizes a basic multimodal sequence combination using an attention layer.”
Shenoy, A., & Sardana, A. (2020) · doi:10.18653/v1/2020.challengehml-1.3 · Cited by 154
Extending Long Short-Term Memory for Multi-View Structured Learning
Rajagopalan, S. S., Morency, L., Baltrušaitis, T., & Goecke, R. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_21 · Cited by 139 · cited 1×
“For example, is has been proposed to adopt multimodal representation inside an LSTM’s architecture in [60], where the authors have constructed the multi-view LSTM, which partitions the memory cell and the gates into regions corresponding to multiple modalities or views.”
BackgroundSurvey on AI-Based Multimodal Methods for Emotion Detection
Marechal, C., Mikołajewski, D., Tyburek, K., Prokopowicz, P., Bougueroua, L., Ancourt, C., & Węgrzyn-Wolska, K. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-16272-6_11 · Cited by 137 · cited 1×
“According to [44], there are three main dimensions of facial variation—morphology, complexion and dynamics— with the third one being the most important for emotion recognition [45].”
BackgroundSpeaker-sensitive emotion recognition via ranking: Studies on acted and spontaneous speech
Cao, H., Verma, R., & Nenkova, A. (2014) · Computer Speech & Language · doi:10.1016/j.csl.2014.01.003 · Cited by 133 · cited 1×
“…speech: the participants are unaware of the recording while their speech and reactions are recorded with hidden mechanisms in a real environment [72]; (ii) acted speech: the emotional condition of the speakers is acted; (iii) elicited speech: where the speaker is placed in a situation which…”
Speech Emotion Recognition Using Multi-hop Attention Mechanism
Yoon, S., Byun, S., Dey, S., & Jung, K. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683483 · Cited by 128 · cited 4×
“However, the evaluation of this work states that the third attention step results in a worse performance compared to that achieved by MHA-2.”
InfluentialDe la Torre, F., Chu, W.-S., Xiong, X., Vicente, F., Ding, X., & Cohn, J. (2015) · 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2015.7163082 · Cited by 125 · cited 1×
“Existing tools for automatic facial recognition, such as IntraFace [48] and FACET [49], implement FACS and give automated facial feature tracking, head pose estimation, facial attribute recognition and facial expression analysis from video.”
A review on emotion recognition using speech
Basu, S., Chakraborty, J., Bag, A., & Aftabuddin, M. (2017) · International Conference Inventive Communication and Computational Technologies · doi:10.1109/icicct.2017.7975169 · Cited by 119 · cited 1×
“More specifically, these methods include one of the following: (i) spontaneous speech: the participants are unaware of the recording while their speech and reactions are recorded with hidden mechanisms in a real environment [72]; (ii) acted speech: the emotional condition of the speakers is acted; (iii) elicited speec…”
MethodAttention Driven Fusion for Multi-Modal Emotion Recognition
Priyasad, D., Fernando, T., Denman, S., Sridharan, S., & Fookes, C. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054441 · Cited by 117 · cited 2×
“ADF-III [56] [56] UTA {A, T,} 10-Fold Cross-Val 79.”
MethodGkoumas, D., Li, Q., Lioma, C., Yu, Y., & Song, D. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.09.005 · Cited by 117 · cited 1×
“For an empirical comparison of such architectures, which trains from scratch a wide range of the presented architectures, we refer to [88].”
BackgroundSiriwardhana, S., Reis, A., Weerasekera, R., & Nanayakkara, S. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1212 · Cited by 113
Metallinou, A., Lee, S., & Narayanan, S. (2010) · 2010 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2010.5494890 · Cited by 89 · cited 1×
“, angry, excited, fear, sad, surprised, frustrated, happy, disappointed and neutral), while beginning with [87], some works merge the happiness and excitement classes.”
BackgroundMai, S., Xing, S., & Hu, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3068598 · Cited by 89 · cited 1×
“For this reason, it is mostly used for sentiment analysis architectures (e.g., [63,66,67,70,83–85]).”
(2019) · neural information processing systems · Cited by 83
A Multi-Scale Fusion Framework for Bimodal Speech Emotion Recognition
Chen, M., & Zhao, X. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-3156 · Cited by 80 · cited 2×
“STSER [55] [55] UTA {A, T,} LO-SessionO 71.”
BackgroundSupervised domain adaptation for emotion recognition from speech
Abdelwahab, M., & Busso, C. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178934 · Cited by 77 · cited 1×
“Thus, in practice, more robust models need to be designed, and thus future works should produce multimodal temporal architectures for the problem of speech emotion recognition that are: (i) trained in a cross-dataset evaluation manner, while utilizing the power of unsupervised [90] or supervised [91] domain adaptation…”
BackgroundA comparative analysis of classifiers in emotion recognition through acoustic features
Kuchibhotla, S., Vankayalapati, H. D., Vaddi, R., & Anne, K. R. (2014) · International Journal of Speech Technology · doi:10.1007/s10772-014-9239-3 · Cited by 66 · cited 1×
“• Mel-Frequency Cepstral Coefficients (MFCC): These coefficients represent the short term power spectrum of the speech signal and consist of the most widely used spectral features for emotion recognition [14].”
BackgroundCombating Human Trafficking with Multimodal Deep Models
Tong, E., Zadeh, A., Jones, C., & Morency, L.-P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1142 · Cited by 63 · cited 1×
“…used for each emotion class, while for the latter, the accuracy over all emotions is monitored together with the weighted accuracy ( WA ), where, according to [86], where N is the total number of negatives, P is the total number of positives, TP is the true positives and TN is the true negatives.”
An analysis on LPC, RASTA and MFCC techniques in Automatic Speech recognition system
Gupta, K., & Gupta, D. (2016) · doi:10.1109/confluence.2016.7508170 · Cited by 62 · cited 1×
“One major drawback of LPC-produced features is that they are exposed to noise and thus need a processing technique to avoid additive noise error [16].”
BackgroundThe State of the Art of Feature Extraction Techniques in Speech Recognition
Gupta, D., Bansal, P., & Choudhary, K. (2017) · Advances in intelligent systems and computing · doi:10.1007/978-981-10-6626-9_22 · Cited by 61 · cited 1×
“Before calculating the cepstral coefficients, the signal is transformed using a Mel-filter bank on a number of sub-band energies [15].”
Analysis of speech features and personality traits
Guidi, A., Gentili, C., Scilingo, E. P., & Vanello, N. (2019) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2019.01.027 · Cited by 61 · cited 1×
“These are extremely useful since according to [18], the emotional content and voice quality of speech are correlated.”
BackgroundThe MAHNOB Mimicry Database: A database of naturalistic human interactions
Bilakhia, S., Petridis, S., Nijholt, A., & Pantic, M. (2015) · Pattern Recognition Letters · doi:10.1016/j.patrec.2015.03.005 · Cited by 57 · cited 1×
“SEMAINE [79] 06:30:00 20 {A, V} English Greek Hebrew AFEW [80] 02:28:03 330 {A, V} English AFEW-VA [81] 00:40:00 240 {A, V} English Mimicry [82] 11:00:00 48 {A, V} English”
Imaging first impressions: Distinct neural processing of verbal and nonverbal social information
Kuzmanovic, B., Bente, G., von Cramon, D. Y., Schilbach, L., Tittgemeyer, M., & Vogeley, K. T. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.12.046 · Cited by 54 · cited 1×
“Yet, why is it important for speech emotion recognition applications to adopt multimodality in the core of their architecture? To answer this question, let us see how the brain’s mechanism functions in order to perceive and understand communication: different regions independently process and understand different moda…”
BackgroundWords Can Shift: Dynamically Adjusting Word Representations Using Nonverbal BehaviorsPreprint
Wang, Y., Shen, Y., Liu, Z., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.09362 · Cited by 44 · cited 11×
“CTC + RAVEN [69] [70] {A, T, V} Train-Val-Test 3-1-1 Sessions 77.”
BackgroundInfluentialMultimodal Local-Global Ranking Fusion for Emotion Recognition
Liang, P. P., Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 20th ACM International Conference on Multimodal Interaction · doi:10.1145/3242969.3243019 · Cited by 43 · cited 2×
“The first one is found in [57], where the authors break down the emotion recognition problem to three sub-tasks in order to extract direct and relative predictions.”
BackgroundPham, H., Liang, P. P., Manzini, T., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.07809 · Cited by 38 · cited 6×
“MCTN [58] [70] UTA {A, T, V} Train-Val-Test 3-1-1 Sessions 84.”
BackgroundInfluentialSpeech emotion recognition using DWT
Lalitha, S., Mudupu, A., Nandyala, B. V., & Munagala, R. (2015) · IEEE International Conference on Computational Intelligence and Computing Research (ICCIC) · doi:10.1109/iccic.2015.7435630 · Cited by 35 · cited 1×
“Features such as the spectral centroid, spectral flux, spectral rolloff and spectral entropy have been widely used in traditional audio and music classification tasks and in some speech emotion recognition methods that use hand-crafted audio features [13].”
MethodLearning Alignment for Multimodal Emotion Recognition from Speech
(2019) · doi:10.48550/arxiv.1909.05645 · Cited by 6 · cited 1×
“One of the first works in this category is proposed for the speech emotion recognition problem in [68] and utilizes a basic multimodal sequence combination using an attention layer.”
Research works printed but not linked: 1
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
Cited by
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
“Koromilas et al. [22] exhaustively introduced an intermediate fusion method based on a deep multimodal learning architecture, but failed to mention early fusion and late fusion techniques.”
MethodKalateh, S., Estrada-Jimenez, L. A., Nikghadam-Hojjati, S., & Barata, J. (2024) · IEEE Access · doi:10.1109/access.2024.3430850 · Cited by 200
“Another review in MER (2021) [26] introduces a new categorization of methods based on temporal dynamics handling for speech, discusses feature representation meth-ods for each modality, and presents challenges related to validation procedures, representation learning, and method robustness.”
BackgroundAl-Saadawi, H. F. T., Daş, B., & Daş, R. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124852 · Cited by 103
Multimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70
Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN
Ma, F., Li, Y., Ni, S., Huang, S., & Zhang, L. (2022) · Applied Sciences · doi:10.3390/app12010527 · Cited by 81
Metadata
- Authors
- Koromilas, P., & Giannakopoulos, T.
- Affiliations
- National Centre of Scientific Research "Demokritos" (GR)
- Institute of Informatics and Telecommunications, National Center for Scientific Research—Demokritos, 15310 Athens, Greece
- Corresponding author
- Koromilas, Panagiotis
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2021-08-28
- Volume
- 11
- Issue
- 17
- Pages
- 7962
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W3198872525
- Cited by
- 58
- References
- 81
- Citations by year
- 2026: 10 · 2025: 14 · 2024: 13 · 2023: 12 · 2022: 8 · 2021: 1
- FWCI
- 8.21
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- speech emotion recognition, multimodal emotion recognition, temporal modeling, feature representation, representation learning, validation, robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.