Class-level spectral features for emotion recognition
Open accessAbstract
The most common approaches to automatic emotion recognition rely on utterance level prosodic features. Recent studies have shown that utterance level statistics of segmental spectral features also contain rich information about expressivity and emotion. In our work we introduce a more fine-grained yet robust set of spectral features: statistics of Mel-Frequency Cepstral Coefficients computed over three phoneme type classes of interest-stressed vowels, unstressed vowels and consonants in the utterance. We investigate performance of our features in the task of speaker-independent emotion recognition using two publicly available datasets. Our experimental results clearly indicate that indeed both the richer set of spectral features and the differentiation between phoneme type classes are beneficial for the task. Classification accuracies are consistently higher for our features compared to prosodic or utterance-level spectral features. Combination of our phoneme class features with prosodic features leads to even further improvement. Given the large number of class-level spectral features, we expected feature selection will improve results even further, but none of several selection methods led to clear gains. Further analyses reveal that spectral features computed from consonant regions of the utterance contain more information about emotion than either stressed or unstressed vowel features. We also explore how emotion recognition accuracy depends on utterance length. We show that, while there is no significant dependence for utterance-level prosodic features, accuracy of emotion recognition using class-level spectral features increases with the utterance length.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Automatic emotion recognition based on utterance level prosodic features may play an important role within speaker-independent emotion recognition.
“Automatic emotion recognition based on utterance level prosodic features may play an important role within speaker-independent emotion recognition [30].”
From Survey on AI-Based Multimodal Methods for Emotion Detection (2019) · cites this paper as [30]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · see citing works
Libsvm : A library for support vector machines
Ferrari, V. (2008) · Medical Entomology and Zoology · Cited by 10,124 · see citing works
Praat, a system for doing phonetics by computer
Boersma, P. (2002) · UvA-DARE (University of Amsterdam) · Cited by 4,417 · see citing works
A database of German emotional speech
Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W. F., & Weiss, B. (2005) · Interspeech 2005 · doi:10.21437/interspeech.2005-446 · Cited by 2,365 · see citing works
Acoustic profiles in vocal emotion expression.
Banse, R., & Scherer, K. R. (1996) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.70.3.614 · Cited by 2,253 · see citing works
Young, S. J., Jansen, J., Odell, J. J., Ollason, D., & Woodland, P. (1995) · Cited by 1,427 · see citing works
Speech emotion recognition using hidden Markov models
Nwe, T. L., Foo, S. W., & De Silva, L. C. (2003) · Speech Communication · doi:10.1016/s0167-6393(03)00099-2 · Cited by 920 · see citing works
Dellaert, F., Polzin, T., & Waibel, A. (1996) · International Conference on Spoken Language Processing (ICSLP) · doi:10.1109/icslp.1996.608022 · Cited by 570 · see citing works
Practical feature subset selection for machine learning
Hall, M. A., & Smith, L. A. (1998) · Research Commons (University of Waikato) · Cited by 500 · see citing works
Mean and variance adaptation within the MLLR framework
Gales, M., & Woodland, P. C. (1996) · Computer Speech & Language · doi:10.1006/csla.1996.0013 · Cited by 393 · see citing works
Emotion recognition by speech signals
Kwon, O.-W., Chan, K., Hao, J., & Lee, T.-W. (2003) · 8th European Conference on Speech Communication and Technology (Eurospeech 2003) · doi:10.21437/eurospeech.2003-80 · Cited by 366 · see citing works
Emotion Recognition in Speech Using Neural Networks
Nicholson, J., Takahashi, K., & Nakatsu, R. (2000) · Neural Computing and Applications · doi:10.1007/s005210070006 · Cited by 345 · see citing works
EMOTION RECOGNITION BY SPEECH SIGNAL
Kwon, O., Chan, K.-H., & Hao, J. (2003) · Interspeech (USB) · Cited by 338 · see citing works
Feature subset selection: a correlation based filter approach
Hall, M. A., & Smith, L. A. (1997) · Research Commons (University of Waikato) · Cited by 255 · see citing works
Emotion recognition in spontaneous speech using GMMs
Neiberg, D., Elenius, K. O. E., & Laskowski, K. (2006) · Interspeech (USB) · doi:10.21437/interspeech.2006-277 · Cited by 247 · see citing works
Emotion Recognition based on Phoneme Classes
Lee, C. M., Yildirim, S., Bulut, M., Kazemzadeh, A., Busso, C., Deng, Z., Lee, S., & Narayanan, S. (2004) · Interspeech 2004 · doi:10.21437/interspeech.2004-322 · Cited by 228 · see citing works
GMM Supervector Based SVM with Spectral Features for Speech Emotion Recognition
Hu, H., Xu, M., & Wu, W. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366937 · Cited by 201 · see citing works
Modeling drivers’ speech under stress
Fernandez, R., & Picard, R. W. (2003) · Speech Communication · doi:10.1016/s0167-6393(02)00080-8 · Cited by 195 · see citing works
Recognition of emotions in interactive voice response systems
Yacoub, S., Simske, S., Lin, X., & Burns, J. (2003) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.2003-307 · Cited by 177 · see citing works
APPROACHING AUTOMATIC RECOGNITION OF EMOTION FROM VOICE: A ROUGH BENCHMARK
McGilloway, S., Cowie, R., Douglas‐Cowie, E., Gielen, S. C. A. M., Westerdijk, M., & Stroeve, S. (2000) · Cited by 167 · see citing works
Dellaert, F., Polzin, T., & Waibel, A. (1996) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.1996-462 · Cited by 158 · see citing works
Automatic emotion recognition using prosodic parameters
Luengo, I., Navas, E., Hernáez, I., & Sánchez, J. (2005) · Interspeech (USB) · doi:10.21437/interspeech.2005-324 · Cited by 144 · see citing works
Schuller, B. W., Müller, R., Lang, M. K., & Rigoll, G. (2005) · Interspeech (USB) · doi:10.21437/interspeech.2005-379 · Cited by 141 · see citing works
Shafran, I., Riley, M., & Mohri, M. (2004) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2003.1318399 · Cited by 112 · see citing works
Emotion Recognition using Mel-Frequency Cepstral Coefficients
Sato, N., & Obuchi, Y. (2007) · Journal of Natural Language Processing · doi:10.5715/jnlp.14.4_83 · Cited by 107 · see citing works
Frame vs. Turn-Level: Emotion Recognition from Speech Considering Static and Dynamic Processing
Vlasenko, B., Schuller, B. W., Wendemuth, A., & Rigoll, G. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2_13 · Cited by 92 · see citing works
Timing levels in segment-based speech emotion recognition
Schuller, B. W., & Rigoll, G. (2006) · Interspeech (USB) · doi:10.21437/interspeech.2006-502 · Cited by 86 · see citing works
Real-time Emotion Detection System using Speech: Multi-modal Fusion of Different Timescale Features
Kim, S., Georgiou, P., Lee, S., & Narayanan, S. S. (2007) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.2007.4412815 · Cited by 81 · see citing works
Segment-Based Approach to the Recognition of Emotions in Speech
Shami, M., & Kamel, M. S. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2005.1521436 · Cited by 63 · see citing works
Recognizing Human Emotion from Audiovisual Informaiton
Wang, Y., & Guan, L. (2006) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2005.1415607 · Cited by 55 · see citing works
Classifier Fusion for Emotion Recognition from Speech
Scherer, S. J., Schwenker, F., & Palm, G. (2009) · Springer eBooks · doi:10.1007/978-0-387-76485-6_5 · Cited by 45 · see citing works
Wolf, & Shashua (2003) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2003.1238369 · Cited by 45 · see citing works
Combining Categorical And Primitives-Based Emotion Recognition
Grimm, M., Kroschel, K., Mower, E., et al. (2006) · doi:10.5281/zenodo.39852 · Cited by 31 · see citing works
Toward A Speaker-Independent Real-Time Affect Detection System
Huang, R., & Ma, C. (2006) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2006.1127 · Cited by 28 · see citing works
Phonetic and speaker variations in automatic emotion classification
Sethu, V., Ambikairajah, E., & Epps, J. (2008) · Interspeech (USB) · doi:10.21437/interspeech.2008-197 · Cited by 27 · see citing works
Speech Emotion Classification on a Riemannian Manifold
Ye, C., Liu, J., Chen, C., et al. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-89796-5_7 · Cited by 26 · see citing works
Children's emotion recognition in an intelligent tutoring scenario
Hasegawa‐Johnson, M., Levinson, S. E., & Zhang, T. (2004) · Interspeech (USB) · doi:10.21437/interspeech.2004-552 · Cited by 24 · see citing works
Recognition of Emotions in German Speech Using Gaussian Mixture Models
Vondra, M., & Vích, R. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-00525-1_26 · Cited by 22 · see citing works
Detecting Emotions in Mandarin Speech
Pao, T.-L., Chen, Y.-T., Yeh, J.-H., et al. (2005) · Cited by 22 · see citing works
Classifier fusion for emotion recognition from speech
Scherer, S. J., Schwenker, F., & Palm, G. (2007) · doi:10.1049/cp:20070360 · Cited by 21 · see citing works
Detecting Emotions in Mandarin Speech
Pao, T.-L., Chen, Y.-T., Yeh, J.-H., & Lu, J.-J. (2004) · International Conference on Computational Linguistics · doi:10.30019/ijclclp.200509.0004 · Cited by 19 · see citing works
Combined Speech-Emotion Recognition for Spoken Human-Computer Interfaces
Meng, H., Pittermann, J., Pittermann, A., et al. (2007) · doi:10.1109/icspc.2007.4728535 · Cited by 19 · see citing works
On the Influence of Phonetic Content Variation for Acoustic Emotion Recognition
Vlasenko, B., Schuller, B. W., Wendemuth, A., et al. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-69369-7_24 · Cited by 16 · see citing works
Emotion Recognition Using Novel Speech Signal Features
Tabatabaei, T. S., Krishnan, S., & Guergachi, A. (2007) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2007.378460 · Cited by 14 · see citing works
Speech Emotion Recognition and Intensity Estimation
Song, M., Chen, C., Bu, J., et al. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24768-5_43 · Cited by 10 · see citing works
Improving emotion recognition using class-level spectral features
Bitouk, D., Nenkova, A., & Verma, R. (2009) · Interspeech (USB) · doi:10.21437/interspeech.2009-582 · Cited by 9 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Survey on AI-Based Multimodal Methods for Emotion Detection
Marechal, C., Mikołajewski, D., Tyburek, K., Prokopowicz, P., Bougueroua, L., Ancourt, C., & Węgrzyn-Wolska, K. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-16272-6_11 · Cited by 137 · see citing works
Bimodal fusion network with multi-head attention for multimodal sentiment analysis
Zhang, R., Xue, C., Qi, Q., Lin, L., Zhang, J., & Zhang, L. (2023) · Applied Sciences · doi:10.3390/app13031915 · Cited by 11 · see citing works
- Authors
- Bitouk, D., Verma, R., & Nenkova, A.
- Affiliations
- University of Pennsylvania (US)
- Corresponding author
- Bitouk, Dmitri
- Venue
- Speech Communication
- Publisher
- Elsevier BV
- Published
- 2010-07
- Volume
- 52
- Issue
- 7-8
- Pages
- 613-625
- ISSN
- 0167-6393 (ISSN-L)
- PMC
- PMC3686526
- OpenAlex
- W2058819080
- Cited by
- 219
- References
- 46
- Citations by year
- 2026: 4 · 2025: 7 · 2024: 7 · 2023: 14 · 2022: 13 · 2021: 17 · 2020: 19 · 2019: 14 · 2018: 16 · 2017: 10 · 2016: 11 · 2015: 24 · 2014: 19 · 2013: 16 · 2012: 21
- FWCI
- 18.42
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Speech Communication (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- ScholarlyCommons (University of Pennsylvania) (submittedVersion)
- Funders
- National Institutes of Health
- National Institute of Mental Health (R01MH073174)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- emotion recognition, Mel-frequency cepstral coefficients, stressed vowels, unstressed vowels, consonants, speaker-independent emotion recognition, prosodic features, utterance length, feature selection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
