A first look into a Convolutional Neural Network for speech emotion detection
Abstract
We propose a real-time Convolutional Neural Network model for speech emotion detection. Our model is trained from raw audio on a small dataset of TED talks speech data, manually annotated into three emotion classes: “Angry”, “Happy” and “Sad”. It achieves an average accuracy of 66.1%, 5% higher than a feature-based SVM baseline, with an evaluation time of few hundred milliseconds. We also provide an in-depth model visualization and analysis. We show how our neural network effectively activates during the speech sections of the waveform regardless of the emotion, ignoring the silence parts which do not contain information. On the frequency domain the CNN filters distribute throughout all the spectrum range, with higher concentration around the average pitch range related to that emotion. Each filter also activates at multiple frequency intervals, presumably due to the additional contribution of amplitude-related feature learning. Our work will allow faster and more accurate emotion detection modules for human-machine empathetic dialog systems and other related applications.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
What's basic about basic emotions?
Ortony, A., & Turner, T. J. (1990) · Psychological Review · doi:10.1037/0033-295x.97.3.315 · Cited by 2,015 · see citing works
Murray, I. R., & Arnott, J. L. (1993) · The Journal of the Acoustical Society of America · doi:10.1121/1.405558 · Cited by 1,098 · see citing works
The INTERSPEECH 2009 emotion challenge
Schuller, B., Steidl, S., & Batliner, A. (2009) · Interspeech 2009 · doi:10.21437/interspeech.2009-103 · Cited by 938 · see citing works
Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network
Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472669 · Cited by 904 · see citing works
Speech emotion recognition using deep neural network and extreme learning machine
Han, K., Yu, D., & Tashev, I. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-57 · Cited by 824 · see citing works
Visualizing and Understanding Neural Models in NLP
Li, J., Chen, X., Hovy, E. H., & Jurafsky, D. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1082 · Cited by 673 · see citing works
Mao, Q., Dong, M., Huang, Z., & Zhan, Y. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2360798 · Cited by 568 · see citing works
Perceptual Cues in Nonverbal Vocal Expressions of Emotion
Sauter, D. A., Eisner, F., Calder, A. J., & Scott, S. K. (2010) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470211003721642 · Cited by 358 · see citing works
Analysis of CNN-based speech recognition system using raw speech as input
Palaz, D., Magimai-Doss, M., & Collobert, R. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-3 · Cited by 288 · see citing works
Analysis of Emotionally Salient Aspects of Fundamental Frequency for Emotion Detection
Busso, C., Lee, S., & Narayanan, S. S. (2009) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2008.2009578 · Cited by 278 · see citing works
Learning a better representation of speech soundwaves using restricted boltzmann machines
Jaitly, N., & Hinton, G. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947700 · Cited by 231 · cited 1× · see citing works
“Deep learning from raw audio is now replacing traditional feature-based learning in all speech-related tasks, with Automatic Speech Recognition the most prominent field [6, 7, 8].”
BackgroundEnhancing the TED-LIUM Corpus with Selected Data for Language Modeling and More TED Talks
Rousseau, A., Deléglise, P., & Estève, Y. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/52t9tccogrod · Cited by 213 · see citing works
Real-Time Speech Emotion and Sentiment Recognition for Interactive Dialogue Systems
Bertero, D., Siddique, F. B., Wu, C.-S., Wan, Y., Chan, R. H. Y., & Fung, P. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1110 · Cited by 131 · see citing works
Self-citationConvolutional neural networks for acoustic modeling of raw time signal in LVCSR
Golik, P., Tüske, Z., Schlüter, R., & Ney, H. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-6 · Cited by 101 · see citing works
Towards Empathetic Human-Robot Interactions
Fung, P., Bertero, D., Wan, Y., Dey, A., Chan, R. H. Y., Siddique, F. B., Yang, Y., Wu, C.-S., & Lin, R. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-75487-1_14 · Cited by 63 · see citing works
Self-citationZara The Supergirl: An Empathetic Personality Recognition System
Fung, P., Dey, A., Siddique, F. B., Lin, R., Yang, Y., Wan, Y., & Chan, H. Y. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-3018 · Cited by 34 · see citing works
Self-citationTowards Empathetic Human-Robot InteractionsPreprint
Fung, P., Bertero, D., Wan, Y., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.04072 · Cited by 21 · see citing works
Self-citationFung, P. (2015) · Scientific American · doi:10.1038/scientificamerican1115-60 · Cited by 17 · see citing works
Self-citationTowards a corpus of speech emotion for interactive dialog systems
Bertero, D., Siddique, F. B., & Fung, P. (2016) · International Conference on Speech Database and Assessments · doi:10.1109/icsda.2016.7919019 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Multimodal Emotion Recognition using Deep Learning
Abdullah, S. M. S. A., Ameen, S. Y. A., M. Sadeeq, M. A., & Zeebaree, S. (2021) · Journal of Applied Science and Technology Trends · doi:10.38094/jastt20291 · Cited by 347 · see citing works
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
- Authors
- Bertero, D., & Fung, P.
- Affiliations
- Hong Kong University of Science and Technology (HK)
- Venue
- 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2017-03
- Pages
- 5115-5119
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2707551695
- Cited by
- 117
- References
- 20
- Citations by year
- 2025: 10 · 2024: 20 · 2023: 16 · 2022: 13 · 2021: 21 · 2020: 10 · 2019: 16 · 2018: 9 · 2017: 2
- FWCI
- 11.62
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Keywords
- convolutional neural networks, real-time emotion recognition, TED Talks, frequency domain analysis, pitch, amplitude features, empathy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
