Toward detecting emotions in spoken dialogs
Abstract
The importance of automatically recognizing emotions from human speech has grown with the increasing role of spoken language interfaces in human-computer interaction applications. This paper explores the detection of domain-specific emotions using language and discourse information in conjunction with acoustic correlates of emotion in speech signals. The specific focus is on a case study of detecting negative and non-negative emotions using spoken language data obtained from a call center application. Most previous studies in emotion recognition have used only the acoustic information contained in speech. In this paper, a combination of three sources of information-acoustic, lexical, and discourse-is used for emotion recognition. To capture emotion information at the language level, an information-theoretic notion of emotional salience is introduced. Optimization of the acoustic correlates of emotion with respect to classification error was accomplished by investigating different feature sets obtained from feature selection, followed by principal component analysis. Experimental results on our call center data show that the best results are obtained when acoustic and language information are combined. Results show that combining all the information, rather than using only acoustic information, improves emotion classification by 40.7% for males and 36.4% for females (linear discriminant classifier used for acoustic information).
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The information channels (keystroke dynamics, written/oral content, and facial video) synergistically support investigations into participant emotions triggered by experimental stressors.
“They combine keystroke dynamics, written/oral content, and facial video. These information channels synergistically support investigations into participant emotions9–12, triggered by the experimental stressors.”
From Stress and productivity patterns of interrupted, synergistic, and antagonistic office activities (2019) · cites this paper as [11]
Cited by
Sorted by citations
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · see citing works
Self-citationA review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644 · see citing works
Analysis of emotion recognition using facial expressions, speech and multimodal information
Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., Lee, S., Neumann, U., & Narayanan, S. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027968 · Cited by 840 · see citing works
Self-citationMao, Q., Dong, M., Huang, Z., & Zhan, Y. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2360798 · Cited by 568 · see citing works
MoodScope: building a mood sensor from smartphone usage patterns
LiKamWa, R., Liu, Y., Lane, N. D., & Zhong, L. (2013) · Proceeding of the 11th annual international conference on Mobile systems, applications, and services · doi:10.1145/2462456.2464449 · Cited by 444 · see citing works
Deep learning for robust feature generation in audiovisual emotion recognition
Kim, Y., Lee, H., & Provost, E. M. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638346 · Cited by 401 · see citing works
MSP-IMPROV: An Acted Corpus of Dyadic Interactions to Study Emotion Perception
Busso, C., Parthasarathy, S., Burmania, A., AbdelWahab, M., Sadoughi, N., & Provost, E. M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2515617 · Cited by 391 · see citing works
Ensemble methods for spoken emotion recognition in call-centres
Morrison, D., Wang, R., & De Silva, L. C. (2007) · Speech Communication · doi:10.1016/j.specom.2006.11.004 · Cited by 320 · see citing works
Learning deep physiological models of affect
Martinez, H. P., Bengio, Y., & Yannakakis, G. N. (2013) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2013.2247823 · Cited by 287 · see citing works
Emotion Recognition based on Phoneme Classes
Lee, C. M., Yildirim, S., Bulut, M., Kazemzadeh, A., Busso, C., Deng, Z., Lee, S., & Narayanan, S. (2004) · Interspeech 2004 · doi:10.21437/interspeech.2004-322 · Cited by 228 · see citing works
Self-citationYao, Z., Wang, Z., Liu, W., Liu, Y., & Pan, J. (2020) · Speech Communication · doi:10.1016/j.specom.2020.03.005 · Cited by 152 · see citing works
A survey of affect recognition methods: audio, visual and spontaneous expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2007) · Proceedings of the 9th international conference on Multimodal interfaces · doi:10.1145/1322192.1322216 · Cited by 144 · see citing works
Error weighted semi-coupled hidden Markov model for audio-visual emotion recognition
Lin, J.-C., Wu, C.-H., & Wei, W.-L. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2011.2171334 · Cited by 121 · see citing works
Chen, L., Wang, K., Li, M., Wu, M., Pedrycz, W., & Hirota, K. (2023) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2022.3150097 · Cited by 113 · see citing works
Review of Extracting Information From the Social Web for Health Personalization
Fernandez-Luque, L., Karlsen, R., & Bonander, J. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1432 · Cited by 61 · see citing works
Temporal Bayesian Fusion for Affect Sensing: Combining Video, Audio, and Lexical Modalities
Savran, A., Cao, H., Nenkova, A., & Verma, R. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2362101 · Cited by 53 · see citing works
Stress and productivity patterns of interrupted, synergistic, and antagonistic office activities
Zaman, S., Wesley, A., Silva, D. R. D. C., Buddharaju, P., Akbar, F., Gao, G., Mark, G., Gutierrez-Osuna, R., & Pavlidis, I. (2019) · Scientific Data · doi:10.1038/s41597-019-0249-5 · Cited by 17 · see citing works
Chaspari, T., Soldatos, C., & Maragos, P. (2015) · The World Journal of Biological Psychiatry · doi:10.3109/15622975.2015.1012228 · Cited by 8 · see citing works
- Authors
- Lee, C. M., & Narayanan, S. S.
- Venue
- IEEE Transactions on Speech and Audio Processing
- Published
- 2005-03
- Volume
- 13
- Issue
- 2
- Pages
- 293-303
- ISSN
- 1063-6676
- Cited by
- 655
- References
- 0
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review