Survey on speech emotion recognition: Features, classification schemes, and databases
The DOI matches its Crossref record; 132 of the 132 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Neural networks for pattern recognition
(1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,538
Object recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472
A Tutorial on Support Vector Machines for Pattern Recognition
Burges, C. J. C. (1998) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009715923555 · Cited by 16,429
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325 · cited 1×
“Bagging scheme is a method for generating multiple versions of the classifier and using them to get an aggregated classifier with higher classification accuracy [16].”
MethodAn Introduction to Support Vector Machines and Other Kernel-based Learning Methods
Cristianini, N., & Shawe-Taylor, J. (2000) · doi:10.1017/cbo9780511801389 · Cited by 13,855
Witten, I. H. (2008) · Cited by 9,143
Fundamentals of speech recognition
Rabiner, L. R., & Juang, B.-H. F. (1993) · Cited by 7,680
Research works printed but not linked: 21
- N. Amir, S. Ron, N. Laor, Analysis of an emotional speech corpus in Hebrew based on objective criteria, in: SpeechEmotion-2000, 2000, pp. 29–33.
- Le Bouquin, 1996, Enhancement of noisy speech signals: application to mobile radio communications, Speech Commun., 18, 3, 10.1016/0167-6393(95)00021-6 · doi:10.1016/0167-6393(95)00021-6
- Cahn, 1990, The generation of affect in synthesized speech, J. Am. Voice Input/Output Soc., 8, 1
- Abu El-Yazeed, 2004, On the determination of optimal model order for gmm-based text-independent speaker identification, EURASIP J. Appl. Signal Process., 8, 1078
- Gelfer, 1995, Comparisons of jitter, shimmer, and signal-to-noise ratio from directly digitized versus taped voice samples, J. Voice, 9, 378, 10.1016/S0892-1997(05)80199-7 · doi:10.1016/s0892-1997(05)80199-7
- K. Hirose, H. Fujisaki, M. Yamaguchi, Synthesis by rule of voice fundamental frequency contours of spoken Japanese from linguistic information, in: IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP ’84, vol. 9, 1984, pp. 597–600. · doi:10.1109/icassp.1984.1172814
- H. Hu, M. Xu, W. Wu, Dimensions of emotional meaning in speech, in: Proceedings of the ISCA ITRW on Speech and Emotion, 2000, pp. 25–28.
- J. Kaiser, On a simple algorithm to calculate the ‘energy’ of the signal, in: ICASSP-90, 1990, pp. 381–384.
- C. Lee, S. Narayanan, R. Pieraccini, Classifying emotions in human–machine spoken dialogs, in: Proceedings of the ICME’02, vol. 1, 2002, pp. 737–740.
- Leinonen, 1997, Expression of emotional-motivational connotations with a one-word utterance, J. Acoust. Soc. Am., 102, 1853, 10.1121/1.420109 · DOI · Paper page
- M. Lugger, B. Yang, The relevance of voice quality features in speaker independent emotion recognition, in: IEEE International Conference on Acoustics, Speech and Signal Processing, 2007, ICASSP 2007, vol. 4, April 2007, pp. IV-17–IV-20. · DOI
- J. Ma, H. Jin, L. Yang, J. Tsai, in: Ubiquitous Intelligence and Computing: Third International Conference, UIC 2006, Wuhan, China, September 3–6, 2006, Proceedings (Lecture Notes in Computer Science), Springer-Verlag, New York, Inc., Secaucus, NJ, USA, 2006.
- A. Oster, A. Risberg, The identification of the mood of a speaker by hearing impaired listeners, Speech Transmission Lab. Quarterly Progress Status Report 4, Stockholm, 1986, pp. 79–90.
- A. Razak, R. Komiya, M. Abidin, Comparison between fuzzy and nn method for speech emotion recognition, in: 3rd International Conference on Information Technology and Applications ICITA 2005, vol. 1, 2005, pp. 297–302. · doi:10.1109/icita.2005.101
- B. Schuller, Towards intuitive speech interaction by the integration of emotional aspects, in: 2002 IEEE International Conference on Systems, Man and Cybernetics, vol. 6, 2002, p. 6. · DOI
- B. Schuller, M. Lang, G. Rigoll, Robust acoustic speech emotion recognition by ensembles of classifiers, in: Proceedings of the DAGA’05, 31, Deutsche Jahrestagung für Akustik, DEGA, 2005, pp. 329–330.
- L.C. De Silva, T. Miyasato, R. Nakatsu, Facial emotion recognition using multi-modal information, in: Proceedings of 1997 International Conference on Information, Communications and Signal Processing, 1997, ICICS, vol. 1, September 1997, pp. 397–401. · DOI · Paper page
- Stevens, 1994, Classification of glottal vibration from acoustic measurements, Vocal Fold Physiol., 147
- D. Ververidis, C. Kotropoulos, Emotional speech classification using Gaussian mixture models and the sequential floating forward selection algorithm, in: IEEE International Conference on Multimedia and Expo, 2005. ICME 2005, July 2005, pp. 1500–1503. · DOI
- Wang, 2006, A dynamic conditional random field model for foreground and shadow segmentation, IEEE Trans. Pattern Anal. Mach. Intell., 28, 279, 10.1109/TPAMI.2006.25 · DOI
- C. Williams, K. Stevens, Vocal correlates of emotional states, Speech Evaluation in Psychiatry, Grune and Stratton, 1981, pp. 189–220.
Cited by
Recent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
A Survey on Deep Learning: Algorithms, Techniques, and Applications
Pouyanfar, S., Sadiq, S., Yan, Y., Tian, H., Tao, Y., Reyes, M. P., Shyu, M.-L., Chen, S.-C., & Iyengar, S. S. (2019) · ACM Computing Surveys · doi:10.1145/3234150 · Cited by 1,403
Speech emotion recognition using deep 1D & 2D CNN LSTM networks
Zhao, J., Mao, X., & Chen, L. (2019) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2018.08.035 · Cited by 1,044
Speech emotion recognition using deep neural network and extreme learning machine
Han, K., Yu, D., & Tashev, I. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-57 · Cited by 794
Akçay, M. B., & Oğuz, K. (2020) · Speech Communication · doi:10.1016/j.specom.2019.12.001 · Cited by 721
Speech Emotion Recognition Using Deep Learning Techniques: A Review
Khalil, R. A., Jones, E., Babar, M. I., Jan, T., Zafar, M. H., & Alhussain, T. (2019) · IEEE Access · doi:10.1109/access.2019.2936124 · Cited by 702
Emotion recognition from speech: a review
Koolagudi, S. G., & Rao, K. S. (2012) · International Journal of Speech Technology · doi:10.1007/s10772-011-9125-1 · Cited by 641
A systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 615
Mao, Q., Dong, M., Huang, Z., & Zhan, Y. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2360798 · Cited by 568
Emotion Recognition from Physiological Signal Analysis: A Review
Egger, M., Ley, M., & Hanke, S. (2019) · Electronic Notes in Theoretical Computer Science · doi:10.1016/j.entcs.2019.04.009 · Cited by 561
Evaluating deep learning architectures for Speech Emotion Recognition
Fayek, H. M., Lech, M., & Cavedon, L. (2017) · Neural Networks · doi:10.1016/j.neunet.2017.02.013 · Cited by 554
Speech emotion recognition: two decades in a nutshell, benchmarks, and ongoing trends
Schuller, B. W. (2018) · Communications of the ACM · doi:10.1145/3129340 · Cited by 541
Current State of Text Sentiment Analysis from Opinion to Emotion Mining
Yadollahi, A., Shahraki, A. G., & Zaiane, O. R. (2018) · ACM Computing Surveys · doi:10.1145/3057270 · Cited by 489
Emotion recognition using wireless signals
Zhao, M., Adib, F., & Katabi, D. (2016) · Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking · doi:10.1145/2973750.2973762 · Cited by 461
Zhang, S., Zhang, S., Huang, T., & Gao, W. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2766843 · Cited by 441
Speech Emotion Recognition from Spectrograms with Deep Convolutional Neural Network
Badshah, A. M., Ahmad, J., Rahim, N., & Baik, S. W. (2017) · 2017 International Conference on Platform Technology and Service (PlatCon) · doi:10.1109/platcon.2017.7883728 · Cited by 413
“One significant challenge in realizing this goal is the inability of machines to understand the emotional state hidden behind spoken words [1].”
BackgroundInfluentialSpeech Emotion Recognition Using Fourier Parameters
Wang, K., An, N., Li, B. N., Zhang, Y., & Li, L. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2392101 · Cited by 408
Speech emotion recognition using convolutional and Recurrent Neural Networks
Lim, W., Jang, D., & Lee, T. (2016) · 2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA) · doi:10.1109/apsipa.2016.7820699 · Cited by 405
Deep learning for robust feature generation in audiovisual emotion recognition
Kim, Y., Lee, H., & Provost, E. M. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638346 · Cited by 401
“This reduction is commonly accomplished using feature selection, a method that identifies a subset of the initial features that provide enhanced classification accuracy [8].”
MethodMohammad, S. M. (2016) · Emotion Measurement · doi:10.1016/b978-0-08-100508-8.00009-6 · Cited by 391
“See surveys by El Ayadi, Kamel, and Karray (2011) and Anagnostopoulos, Iliou, and Giannoukos (2015) for an overview of emotion detection in speech.”
BackgroundMSP-IMPROV: An Acted Corpus of Dyadic Interactions to Study Emotion Perception
Busso, C., Parthasarathy, S., Burmania, A., AbdelWahab, M., Sadoughi, N., & Provost, E. M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2515617 · Cited by 391
A Comprehensive Review of Speech Emotion Recognition Systems
Wani, T. M., Gunawan, T. S., Qadri, S. A. A., Kartiwi, M., & Ambikairajah, E. (2021) · IEEE Access · doi:10.1109/access.2021.3068045 · Cited by 376
Speech Emotion Recognition From 3D Log-Mel Spectrograms With Deep Learning Network
Meng, H., Yan, T., Yuan, F., & Wei, H. (2019) · IEEE Access · doi:10.1109/access.2019.2938007 · Cited by 356
Databases, features and classifiers for speech emotion recognition: a review
Swain, M., Routray, A., & Kabisatpathy, P. (2018) · International Journal of Speech Technology · doi:10.1007/s10772-018-9491-z · Cited by 356
Deep learning for affective computing: Text-based emotion recognition in decision support
Kratzwald, B., Ilić, S., Kraus, M., Feuerriegel, S., & Prendinger, H. (2018) · Decision Support Systems · doi:10.1016/j.dss.2018.09.002 · Cited by 352
Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition
Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2719043 · Cited by 346
“tures [34], respectively.”
Camacho, D., Panizo-LLedot, Á., Bello-Orgaz, G., Gonzalez-Pardo, A., & Cambria, E. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.05.009 · Cited by 329
Emotion Recognition for Human-Robot Interaction: Recent Advances and Future Perspectives
Spezialetti, M., Placidi, G., & Rossi, S. (2020) · Frontiers in Robotics and AI · doi:10.3389/frobt.2020.532279 · Cited by 293
Speech emotion recognition based on feature selection and extreme learning machine decision tree
Liu, Z.-T., Wu, M., Cao, W.-H., Mao, J.-W., Xu, J.-P., & Tan, G.-Z. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.07.050 · Cited by 267
Emotion detection from text and speech: a survey
Sailunaz, K., Dhaliwal, M., Rokne, J., & Alhajj, R. (2018) · Social Network Analysis and Mining · doi:10.1007/s13278-018-0505-2 · Cited by 260
Sewa db: A rich database for audio-visual emotion and sentiment research in the wild
Kossaifi, J., Walecki, R., Panagakis, Y., Shen, J., Schmitt, M., Ringeval, F., Han, J., Pandit, V., Toisoul, A., Schuller, B., et al. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2944808 · Cited by 243
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243
(2013) · Cited by 236
Meng, H., Huang, D., Wang, H., Yang, H., AI-Shuraifi, M., & Wang, Y. (2013) · Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge · doi:10.1145/2512530.2512532 · Cited by 225
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
An experimental study of speech emotion recognition based on deep convolutional neural networks
Zheng, W. Q., Yu, J. S., & Zou, Y. X. (2015) · 2015 International Conference on Affective Computing and Intelligent Interaction (ACII) · doi:10.1109/acii.2015.7344669 · Cited by 187
“However, it is uncertain whether these hand-tuned feature sets can sufficiently and efficiently reflect the emotional characteristics of speech [2].”
BackgroundHybrid Deep Neural Network--Hidden Markov Model (DNN-HMM) Based Speech Emotion Recognition
Li, L., Zhao, Y., Jiang, D., Zhang, Y., Wang, F., Gonzalez, I., Valentin, E., & Sahli, H. (2013) · 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction · doi:10.1109/acii.2013.58 · Cited by 163
Attention Based Fully Convolutional Network for Speech Emotion Recognition
Zhang, Y., Du, J., Wang, Z., Zhang, J., & Tu, Y. (2018) · 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipa.2018.8659587 · Cited by 161
Pan, B., Hirota, K., Jia, Z., & Dai, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126866 · Cited by 142
Evaluation of Speech-to-Gesture Generation Using Bi-Directional LSTM Network
Hasegawa, D., Kaneko, N., Shirakawa, S., Sakuta, H., & Sumi, K. (2018) · Proceedings of the 18th International Conference on Intelligent Virtual Agents · doi:10.1145/3267851.3267878 · Cited by 131
Chen, L., Wang, K., Li, M., Wu, M., Pedrycz, W., & Hirota, K. (2023) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2022.3150097 · Cited by 113
A hybrid latent space data fusion method for multimodal emotion recognition
Nemati, S., Rohani, R., Basiri, M. E., Abdar, M., Yen, N. Y., & Makarenkov, V. (2019) · IEEE Access · doi:10.1109/access.2019.2955637 · Cited by 98
Social support agents for older adults: longitudinal affective computing in the home
Ring, L., Shi, L., Totzke, K., & Bickmore, T. (2015) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-014-0157-0 · Cited by 97
Multimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70
Learning Discriminative Features from Spectrograms Using Center Loss for Speech Emotion Recognition
Dai, D., Wu, Z., Li, R., Wu, X., Jia, J., & Meng, H. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683765 · Cited by 58
DNN-based Emotion Recognition Based on Bottleneck Acoustic Features and Lexical Features
Kim, E., & Shin, J. W. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683077 · Cited by 54
“The biggest issue to successfully recognize human emotions from the speech and text modalities is the extraction of appropriate features that discriminate different emotions [1].”
BackgroundAutomatic analysis of speech F0 contour for the characterization of mood changes in bipolar patients
Guidi, A., Vanello, N., Bertschy, G., Gentili, C., Landini, L., & Scilingo, E. P. (2015) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2014.10.011 · Cited by 43
Hong, K., Liu, G., Chen, W., & Hong, S. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.12.013 · Cited by 34
Deep Implicit Distribution Alignment Networks for cross-Corpus Speech Emotion Recognition
Zhao, Y., Wang, J., Zong, Y., Zheng, W., Lian, H., & Zhao, L. (2023) · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49357.2023.10095388 · Cited by 19
Improving Facial Emotion Recognition Systems with Crucial Feature Extractors
Pandey, R. K., Karmakar, S., Ramakrishnan, A. G., & Saha, N. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-30642-7_24 · Cited by 17
(2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-72038-8 · Cited by 11
Improving Facial Emotion Recognition Systems Using Gradient and Laplacian Images
Pandey, R. K., Karmakar, S., Ramakrishnan, A. G., & Saha, N. (2019) · arXiv · doi:10.48550/arxiv.1902.05411 · Cited by 9
Chaspari, T., Soldatos, C., & Maragos, P. (2015) · The World Journal of Biological Psychiatry · doi:10.3109/15622975.2015.1012228 · Cited by 8
“A study surveying the features and classifi cation schemes for speech emotion recognition can be found in (El Ayadi et al. 2011).”
Background
Metadata
- Authors
- El Ayadi, M., Kamel, M. S., & Karray, F.
- Venue
- Pattern Recognition
- Publisher
- Elsevier BV
- Published
- 2011-03
- Volume
- 44
- Issue
- 3
- Pages
- 572-587
- ISSN
- 0031-3203 (ISSN-L)
- OpenAlex
- W2074788634
- Cited by
- 2,073
- References
- 132
- FWCI
- 53.94
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- doi.org/10.15223/policy-017 (stm-asf)
- doi.org/10.15223/policy-037 (stm-asf)
- doi.org/10.15223/policy-012 (stm-asf)
- doi.org/10.15223/policy-029 (stm-asf)
- doi.org/10.15223/policy-004 (stm-asf)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.