The DOI matches its Crossref record; 38 of the 39 references it cites were found in a registry or an index.
Deep learning for robust feature generation in audiovisual emotion recognition
Abstract
Automatic emotion recognition systems predict high-level affective content from low-level human-centered signal cues. These systems have seen great improvements in classification accuracy, due in part to advances in feature selection methods. However, many of these feature selection methods capture only linear relationships between features or alternatively require the use of labeled data. In this paper we focus on deep learning techniques, which can overcome these limitations by explicitly capturing complex non-linear feature interactions in multimodal data. We propose and evaluate a suite of Deep Belief Network models, and demonstrate that these models show improvement in emotion classification performance over baselines that do not employ deep learning. This suggests that the learned high-order non-linear relationships are effective for emotion recognition.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Kim et al. [7] developed a suite of deep belief net- work models that showed improvements in emotion clas- sification performance over baselines that do not use deep learning.”
From Multimodal emotion recognition using deep learning architectures (2016) · cites this paper as [7]
In the citing paper
“Kim et al. [159] and others (e.g., [149], [148]) demonstrated how this can be achieved with DBNs.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [159]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 7,330
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 5,050
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 1×
“The parameters of the RBM (W, b, c) are learned using contrastive divergence as in [33].”
MethodIEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
(2011) · Cited by 2,752
Self-citationSurvey on speech emotion recognition: Features, classification schemes, and databases
Ayadi, M. M. H. E., Kamel, M. S., & Karray, F. (2010) · Pattern Recognition · doi:10.1016/j.patcog.2010.09.020 · Cited by 2,139
Acoustic Modeling Using Deep Belief Networks
Mohamed, A., Dahl, G. E., & Hinton, G. E. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2109382 · Cited by 1,756
Information processing in dynamical systems: foundations of harmony theory
Smolensky, P. (1986) · The MIT Press eBooks · Cited by 1,673
Toward detecting emotions in spoken dialogs
Lee, C. M., & Narayanan, S. S. (2005) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2004.838534 · Cited by 988
The INTERSPEECH 2009 emotion challenge
Schuller, B., Steidl, S., & Batliner, A. (2009) · Interspeech 2009 · doi:10.21437/interspeech.2009-103 · Cited by 938
(2007) · Cited by 849
Self-citationAnalysis of emotion recognition using facial expressions, speech and multimodal information
Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., Lee, S., Neumann, U., & Narayanan, S. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027968 · Cited by 840 · cited 1×
“Previous research has demonstrated the benefit of using multimodal data in emotion recognition tasks and has identified various techniques for generating robust multimodal features [2] [3] [4] [5] [6].”
BackgroundModeling Human Motion Using Binary Latent Variables
Taylor, G. W., Hinton, G. E., & Roweis, S. T. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/7503.003.0173 · Cited by 710
Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011
Anagnostopoulos, C.-N., Iliou, T., & Giannoukos, I. (2015) · Artificial Intelligence Review · doi:10.1007/s10462-012-9368-5 · Cited by 428
Unsupervised learning of hierarchical representations with convolutional deep belief networks
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2011) · Communications of the ACM · doi:10.1145/2001269.2001295 · Cited by 396
Self-citationFeature selection using principal feature analysis
Lu, Y., Cohen, I., Zhou, X. S., & Tian, Q. C. (2007) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1291233.1291297 · Cited by 350
Deep neural networks for acoustic emotion recognition: Raising the benchmarks
Stuhlsatz, A., Meyer, C., Eyben, F., Zielke, T., Meier, G., & Schuller, B. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947651 · Cited by 288
Comparing Feature Sets for Acted and Spontaneous Speech in View of Automatic Emotion Recognition
Vogt, T., & André, E. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2005.1521463 · Cited by 223
A Framework for Automatic Human Emotion Classification Using Emotion Profiles
Mower, E., Mataric, M. J., & Narayanan, S. (2011) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2010.2076804 · Cited by 215
The INTERSPEECH 2012 speaker trait challenge
Schuller, B., Steidl, S., Batliner, A., Nöth, E., Vinciarelli, A., Burkhardt, F., Son, R. V., Weninger, F., Eyben, F., Bocklet, T., et al. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-86 · Cited by 211
Wöllmer, M., Metallinou, A., Eyben, F., Schuller, B., & Narayanan, S. S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-646 · Cited by 201 · cited 1×
“PCA and its variants (e.g., Principal Feature Analysis, or PFA [21]) are broadly used in the emotion recognition literature [22] [23] [24].”
MethodVerveridis, D., & Kotropoulos, C. L. (2008) · Signal Processing · doi:10.1016/j.sigpro.2008.07.001 · Cited by 155
Interrelation Between Speech and Facial Gestures in Emotional Utterances: A Single Subject Study
Busso, C., & Narayanan, S. S. (2007) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2007.905145 · Cited by 149
Deep and Wide: Multiple Layers in Automatic Speech Recognition
Morgan, N. H. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2116010 · Cited by 128
Sohn, K., Jung, D. Y., Lee, H., & Hero, A. O. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126554 · Cited by 116
Self-citationSteidl, S., Levit, M., Batliner, A. M., Nöth, E., & Niemann, H. J. (2006) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2005.1415114 · Cited by 96
(2010) · Cited by 94
Metallinou, A., Lee, S., & Narayanan, S. (2010) · 2010 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2010.5494890 · Cited by 89
Using neutral speech models for emotional speech analysis
Busso, C., Lee, S., & Narayanan, S. S. (2007) · Interspeech (USB) · doi:10.21437/interspeech.2007-605 · Cited by 82
Sparse Multilayer Perceptron for Phoneme Recognition
Sivaram, G. S. V. S., & Hynek Hermansky, B. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2129510 · Cited by 65
LOW-LEVEL FUSION OF AUDIO AND VIDEO FEATURE FOR MULTI-MODAL EMOTION RECOGNITION
Wimmer, M., Schuller, B. W., Arsić, D., Rigoll, G., & Radig, B. (2008) · International Conference on Computer Vision Theory and Applications · doi:10.5220/0001082801450151 · Cited by 52
Visual emotion recognition using compact facial representations and viseme information
Metallinou, A., Busso, C., Lee, S., & Narayanan, S. S. (2010) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2010.5494893 · Cited by 52
Emotion classification in children's speech using fusion of acoustic and linguistic features
Polzehl, T., Sundaram, S., Ketabdar, H., Wagner, M., & Metze, F. (2009) · Interspeech (USB) · doi:10.21437/interspeech.2009-110 · Cited by 39
Likability classification - a not so deep neural network approach
Brueckner, R., & Schuller, B. W. (2012) · Interspeech (USB) · doi:10.21437/interspeech.2012-95 · Cited by 29
Multimodal Emotion Recognition from Low-Level Cues
Pantić, M., Caridakis, G., André, E., Kim, J., Karpouzis, K. C., & Kollias, S. (2010) · Cognitive technologies · doi:10.1007/978-3-642-15184-2_8 · Cited by 28
Feature Selection Based on Information Theory Filters
Duch, W., Biesiada, J., Winiarski, T., Grudziński, K., & Grąbczewski, K. (2003) · Physica-Verlag eBooks · doi:10.1007/978-3-7908-1902-1_23 · Cited by 15
Cited by
Sorted by citations
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
A comprehensive survey on support vector machine classification: Applications, challenges and trends
Cervantes, J., Garcia-Lamont, F., Rodríguez-Mazahua, L., & Lopez, A. (2020) · Neurocomputing · doi:10.1016/j.neucom.2019.10.118 · Cited by 2,381
Speech Recognition Using Deep Neural Networks: A Systematic Review
Nassif, A. B., Shahin, I. M. A., Attili, I. B., Azzeh, M., & Shaalan, K. F. (2019) · IEEE Access · doi:10.1109/access.2019.2896880 · Cited by 1,215
Speech emotion recognition using deep 1D & 2D CNN LSTM networks
Zhao, J., Mao, X., & Chen, L. (2019) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2018.08.035 · Cited by 1,044
Fadlullah, Z. M., Tang, F., Mao, B., Kato, N., Akashi, O., Inoue, T., & Mizutani, K. (2017) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2017.2707140 · Cited by 810
Metadata
- Authors
- Kim, Y., Lee, H., & Provost, E. M.
- Affiliations
- University of Michigan (US)
- Venue
- 2013 IEEE International Conference on Acoustics, Speech and Signal Processing
- Publisher
- IEEE
- Published
- 2013-05
- Pages
- 3687-3691
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2009059481
- Cited by
- 401
- References
- 39
- Citations by year
- 2026: 5 · 2025: 19 · 2024: 22 · 2023: 18 · 2022: 33 · 2021: 36 · 2020: 44 · 2019: 50 · 2018: 49 · 2017: 47 · 2016: 29 · 2015: 30 · 2014: 17 · 2013: 2
- FWCI
- 35.38
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- audio-visual emotion recognition, deep belief network, feature selection, multimodal data, emotion classification, higher-order interactions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
