Evaluating deep learning architectures for Speech Emotion Recognition
Abstract
Speech Emotion Recognition (SER) can be regarded as a static or dynamic classification problem, which makes SER an excellent test bed for investigating and comparing various deep learning architectures. We describe a frame-based formulation to SER that relies on minimal speech processing and end-to-end deep learning to model intra-utterance dynamics. We use the proposed SER system to empirically explore feed-forward and recurrent neural network architectures and their variants. Experiments conducted illuminate the advantages and limitations of these architectures in paralinguistic speech recognition and emotion recognition in particular. As a result of our exploration, we report state-of-the-art results on the IEMOCAP database for speaker-independent SER and present quantitative and qualitative assessments of the models' performances.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
(2014) · Cited by 33,419 · see citing works
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
(2016) · Cited by 11,679 · see citing works
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260 · see citing works
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · see citing works
(2011) · Cited by 6,993 · see citing works
Kaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 5,270 · see citing works
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · see citing works
Neural Information Processing Systems
(2018) · The MIT Press eBooks · doi:10.7551/mitpress/11474.003.0014 · Cited by 3,775 · see citing works
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120 · see citing works
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603 · see citing works
Convolutional Neural Networks for Speech Recognition
Abdel‐Hamid, O., Mohamed, A., Jiang, H., Deng, L., Penn, G., & Yu, D. (2014) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2014.2339736 · Cited by 2,268 · see citing works
Survey on speech emotion recognition: Features, classification schemes, and databases
Ayadi, M. M. H. E., Kamel, M. S., & Karray, F. (2010) · Pattern Recognition · doi:10.1016/j.patcog.2010.09.020 · Cited by 2,089 · see citing works
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818 · see citing works
Encyclopedia of Database Systems
LIU, L., & ÖZSU, M. T. (2009) · Springer eBooks · doi:10.1007/978-0-387-39940-9 · Cited by 1,748 · see citing works
Acoustic Modeling Using Deep Belief Networks
Mohamed, A., Dahl, G. E., & Hinton, G. E. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2109382 · Cited by 1,731 · see citing works
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2016) · doi:10.1109/cvpr33180.2016 · Cited by 1,569 · see citing works
Generalization of backpropagation with application to a recurrent gas market model
Werbos, P. J. (1988) · Neural Networks · doi:10.1016/0893-6080(88)90007-x · Cited by 947 · see citing works
The INTERSPEECH 2009 emotion challenge
Schuller, B., Steidl, S., & Batliner, A. (2009) · Interspeech 2009 · doi:10.21437/interspeech.2009-103 · Cited by 938 · see citing works
Emotional speech recognition: Resources, features, and methods
Ververidis, D., & Kotropoulos, C. L. (2006) · Speech Communication · doi:10.1016/j.specom.2006.04.003 · Cited by 930 · see citing works
Speech emotion recognition using deep neural network and extreme learning machine
Han, K., Yu, D., & Tashev, I. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-57 · Cited by 824 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky (2012) · Medical Entomology and Zoology · Cited by 799 · see citing works
2017 International Joint Conference on Neural Networks (IJCNN)
(2017) · doi:10.1109/ijcnn39090.2017 · Cited by 722 · see citing works
The INTERSPEECH 2010 paralinguistic challenge
Schuller, B. W., Steidl, S., Batliner, A. M., Burkhardt, F., Devillers, L. Y., Müller, C. A., & Narayanan, S. S. (2010) · Interspeech (USB) · doi:10.21437/interspeech.2010-739 · Cited by 577 · see citing works
Mao, Q., Dong, M., Huang, Z., & Zhan, Y. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2360798 · Cited by 568 · see citing works
Deep learning for robust feature generation in audiovisual emotion recognition
Kim, Y., Lee, H., & Provost, E. M. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638346 · Cited by 401 · see citing works
Emotion recognition using a hierarchical binary decision tree approach
Lee, C.-C., Mower, E., Busso, C., Lee, S., & Narayanan, S. S. (2011) · Speech Communication · doi:10.1016/j.specom.2011.06.004 · Cited by 359 · see citing works
Deep neural networks for acoustic emotion recognition: Raising the benchmarks
Stuhlsatz, A., Meyer, C., Eyben, F., Zielke, T., Meier, G., & Schuller, B. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947651 · Cited by 288 · see citing works
Acoustic emotion recognition: A benchmark comparison of performances
Schuller, B. W., Vlasenko, B., Eyben, F., Rigoll, G., & Wendemuth, A. (2009) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2009.5372886 · Cited by 275 · see citing works
Hybrid Deep Neural Network--Hidden Markov Model (DNN-HMM) Based Speech Emotion Recognition
Li, L., Zhao, Y., Jiang, D., Zhang, Y., Wang, F., Gonzalez, I., Valentin, E., & Sahli, H. (2013) · 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction · doi:10.1109/acii.2013.58 · Cited by 163 · see citing works
Towards real-time Speech Emotion Recognition using deep neural networks
Fayek, H. M., Lech, M., & Cavedon, L. (2015) · 2015 9th International Conference on Signal Processing and Communication Systems (ICSPCS) · doi:10.1109/icspcs.2015.7391796 · Cited by 103 · see citing works
Self-citationEmotion-Oriented Systems: The Humaine Handbook
Petta, P., Pélachaud, C., & Cowie, R. (2011) · Cognitive technologies · doi:10.1007/978-3-642-15184-2 · Cited by 101 · see citing works
Towards a Small Set of Robust Acoustic Features for Emotion Recognition: Challenges
Tahon, M., & Devillers, L. Y. (2015) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2015.2487051 · Cited by 98 · see citing works
Frame vs. Turn-Level: Emotion Recognition from Speech Considering Static and Dynamic Processing
Vlasenko, B., Schuller, B. W., Wendemuth, A., & Rigoll, G. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2_13 · Cited by 92 · see citing works
Exploring Cross-Modality Affective Reactions for Audiovisual Emotion Recognition
Mariooryad, S., & Busso, C. (2013) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2013.11 · Cited by 82 · see citing works
Modeling subjectiveness in emotion recognition with deep neural networks: Ensembles vs soft labels
Fayek, H. M., Lech, M., & Cavedon, L. (2016) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2016.7727250 · Cited by 73 · see citing works
Self-citationA computational model for the automatic recognition of affect in speech
Fernandez, R. C., & Picard, R. W. (2004) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 72 · see citing works
Emotion recognition in spontaneous and acted dialogues
Tian, L., Moore, J. D., & Lai, C. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344645 · Cited by 47 · see citing works
A multi-modal approach to emotion recognition using undirected topic models
Shah, M., Chakrabarti, C., & Spanias, A. (2014) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2014.6865245 · Cited by 27 · see citing works
Fayek, H. M., Lech, M., & Cavedon, L. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-868 · Cited by 26 · see citing works
Self-citationEnergy and F0 contour modeling with functional data analysis for emotional speech detection
Arias, J. P., Busso, C., & Yoma, N. B. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-253 · Cited by 23 · see citing works
Real-time robust recognition of speakers' emotions and characteristics on mobile platforms
Eyben, F., Huber, B., Marchi, E., Schuller, D., & Schuller, B. W. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344658 · Cited by 11 · see citing works
RECOGNIZING EMOTIONS IN DIALOGUES WITH DISFLUENCIES AND NON-VERBAL VOCALISATIONS
Tian, L., Lai, C., & Moore, J. D. (2015) · Cited by 11 · see citing works
Paiva, A., Prada, R., & Picard, R. W. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Deep learning modelling techniques: Current progress, applications, advantages, and challenges
Ahmed, S. F., Alam, M. S. B., Hassan, M., Rozbu, M. R., Ishtiak, T., Rafa, N., Mofijur, M., Shawkat Ali, A. B. M., & Gandomi, A. H. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10466-8 · Cited by 1,079 · see citing works
Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review
Zhang, J., Yin, Z., Chen, P., & Nichele, S. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.01.011 · Cited by 806 · see citing works
Emotion Recognition from Speech Using wav2vec 2.0 Embeddings
Pepino, L., Riera, P., & Ferrer, L. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-703 · Cited by 389 · see citing works
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
Automated emotion recognition: Current trends and future perspectives
Maithri, M., Raghavendra, U., Gudigar, A., Samanth, J., Barua, P. D., Murugappan, M., Chakole, Y., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106646 · Cited by 172 · see citing works
Machine learning to assist clinical decision-making during the COVID-19 pandemic
Debnath, S., Barnaby, D. P., Coppa, K., Makhnevich, A., Kim, E. J., Chatterjee, S., Tóth, V., Levy, T. J., Paradis, M. D., Cohen, S. L., et al. (2020) · Bioelectronic Medicine · doi:10.1186/s42234-020-00050-8 · Cited by 104 · see citing works
Deep Multimodal Emotion Recognition on Human Speech: A Review
Koromilas, P., & Giannakopoulos, T. (2021) · Applied Sciences · doi:10.3390/app11177962 · Cited by 58 · see citing works
A comprehensive survey and taxonomy on privacy-preserving deep learning
Tran, A.-T., Luong, T.-D., & Huynh, V.-N. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127345 · Cited by 21 · see citing works
Chou, H.-C., Goncalves, L., Leem, S.-G., Salman, A. N., Lee, C.-C., & Busso, C. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3411290 · Cited by 19 · see citing works
- Authors
- Fayek, H. M., Lech, M., & Cavedon, L.
- Affiliations
- RMIT University (AU)
- Corresponding author
- Fayek, Haytham M.
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2017-08
- Volume
- 92
- Pages
- 60-68
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W2602034649
- Cited by
- 554
- References
- 56
- Citations by year
- 2026: 31 · 2025: 42 · 2024: 44 · 2023: 78 · 2022: 72 · 2021: 92 · 2020: 92 · 2019: 67 · 2018: 28 · 2017: 8
- FWCI
- 69.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Networks (publishedVersion)
- PubMed (publishedVersion)
- RMIT Research Repository (RMIT University Library) (submittedVersion)
- Figshare (submittedVersion)
- Funders
- RMIT University
- NVIDIA Corporation
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- speech emotion recognition, deep learning architectures, recurrent neural networks, feedforward neural networks, paralinguistic features, IEMOCAP, speaker-independent emotion recognition, end-to-end learning
- MeSH terms
- Machine Learning, Emotions, Neural Networks, Computer, Speech Recognition Software
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review