Deep learning for robust feature generation in audiovisual emotion recognition
Abstract
Automatic emotion recognition systems predict high-level affective content from low-level human-centered signal cues. These systems have seen great improvements in classification accuracy, due in part to advances in feature selection methods. However, many of these feature selection methods capture only linear relationships between features or alternatively require the use of labeled data. In this paper we focus on deep learning techniques, which can overcome these limitations by explicitly capturing complex non-linear feature interactions in multimodal data. We propose and evaluate a suite of Deep Belief Network models, and demonstrate that these models show improvement in emotion classification performance over baselines that do not employ deep learning. This suggests that the learned high-order non-linear relationships are effective for emotion recognition.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Kim et al. [7] developed a suite of deep belief network models that showed improvements in emotion classification performance over baselines that do not use deep learning.
“Kim et al. [7] developed a suite of deep belief net- work models that showed improvements in emotion clas- sification performance over baselines that do not use deep learning.”
From Multimodal emotion recognition using deep learning architectures (2016) · cites this paper as [7]
Kim et al. and others demonstrated that joint feature representations can be learned with DBNs.
“Kim et al. [159] and others (e.g., [149], [148]) demonstrated how this can be achieved with DBNs.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [159]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · see citing works
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 7,048 · see citing works
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 5,037 · see citing works
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 1× · see citing works
“The parameters of the RBM (W, b, c) are learned using contrastive divergence as in [33].”
MethodIEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · see citing works
(2011) · Cited by 2,752 · see citing works
Self-citationSurvey on speech emotion recognition: Features, classification schemes, and databases
Ayadi, M. M. H. E., Kamel, M. S., & Karray, F. (2010) · Pattern Recognition · doi:10.1016/j.patcog.2010.09.020 · Cited by 2,089 · see citing works
Acoustic Modeling Using Deep Belief Networks
Mohamed, A., Dahl, G. E., & Hinton, G. E. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2109382 · Cited by 1,731 · see citing works
Information processing in dynamical systems: foundations of harmony theory
Smolensky, P. (1986) · The MIT Press eBooks · Cited by 1,673 · see citing works
The INTERSPEECH 2009 emotion challenge
Schuller, B., Steidl, S., & Batliner, A. (2009) · Interspeech 2009 · doi:10.21437/interspeech.2009-103 · Cited by 938 · see citing works
(2007) · Cited by 849 · see citing works
Self-citationAnalysis of emotion recognition using facial expressions, speech and multimodal information
Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., Lee, S., Neumann, U., & Narayanan, S. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027968 · Cited by 840 · cited 1× · see citing works
“Previous research has demonstrated the benefit of using multimodal data in emotion recognition tasks and has identified various techniques for generating robust multimodal features [2] [3] [4] [5] [6].”
BackgroundModeling Human Motion Using Binary Latent Variables
Taylor, G. W., Hinton, G. E., & Roweis, S. T. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/7503.003.0173 · Cited by 708 · see citing works
Toward detecting emotions in spoken dialogs
Lee, C. M., & Narayanan, S. S. (2005) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2004.838534 · Cited by 655 · see citing works
Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011
Anagnostopoulos, C.-N., Iliou, T., & Giannoukos, I. (2015) · Artificial Intelligence Review · doi:10.1007/s10462-012-9368-5 · Cited by 428 · see citing works
Unsupervised learning of hierarchical representations with convolutional deep belief networks
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2011) · Communications of the ACM · doi:10.1145/2001269.2001295 · Cited by 386 · see citing works
Self-citationFeature selection using principal feature analysis
Lu, Y., Cohen, I., Zhou, X. S., & Tian, Q. C. (2007) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1291233.1291297 · Cited by 343 · see citing works
Deep neural networks for acoustic emotion recognition: Raising the benchmarks
Stuhlsatz, A., Meyer, C., Eyben, F., Zielke, T., Meier, G., & Schuller, B. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947651 · Cited by 288 · see citing works
Comparing Feature Sets for Acted and Spontaneous Speech in View of Automatic Emotion Recognition
Vogt, T., & André, E. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2005.1521463 · Cited by 223 · see citing works
A Framework for Automatic Human Emotion Classification Using Emotion Profiles
Mower, E., Mataric, M. J., & Narayanan, S. (2011) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2010.2076804 · Cited by 215 · see citing works
The INTERSPEECH 2012 speaker trait challenge
Schuller, B., Steidl, S., Batliner, A., Nöth, E., Vinciarelli, A., Burkhardt, F., Son, R. V., Weninger, F., Eyben, F., Bocklet, T., et al. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-86 · Cited by 211 · see citing works
Wöllmer, M., Metallinou, A., Eyben, F., Schuller, B., & Narayanan, S. S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-646 · Cited by 201 · cited 1× · see citing works
“PCA and its variants (e.g., Principal Feature Analysis, or PFA [21]) are broadly used in the emotion recognition literature [22] [23] [24].”
MethodVerveridis, D., & Kotropoulos, C. L. (2008) · Signal Processing · doi:10.1016/j.sigpro.2008.07.001 · Cited by 155 · see citing works
Interrelation Between Speech and Facial Gestures in Emotional Utterances: A Single Subject Study
Busso, C., & Narayanan, S. S. (2007) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2007.905145 · Cited by 149 · see citing works
Deep and Wide: Multiple Layers in Automatic Speech Recognition
Morgan, N. H. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2116010 · Cited by 128 · see citing works
Sohn, K., Jung, D. Y., Lee, H., & Hero, A. O. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126554 · Cited by 118 · see citing works
Self-citationSteidl, S., Levit, M., Batliner, A. M., Nöth, E., & Niemann, H. J. (2006) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2005.1415114 · Cited by 96 · see citing works
(2010) · Cited by 94 · see citing works
Metallinou, A., Lee, S., & Narayanan, S. (2010) · 2010 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2010.5494890 · Cited by 89 · see citing works
Using neutral speech models for emotional speech analysis
Busso, C., Lee, S., & Narayanan, S. S. (2007) · Interspeech (USB) · doi:10.21437/interspeech.2007-605 · Cited by 82 · see citing works
Sparse Multilayer Perceptron for Phoneme Recognition
Sivaram, G. S. V. S., & Hynek Hermansky, B. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2129510 · Cited by 65 · see citing works
LOW-LEVEL FUSION OF AUDIO AND VIDEO FEATURE FOR MULTI-MODAL EMOTION RECOGNITION
Wimmer, M., Schuller, B. W., Arsić, D., Rigoll, G., & Radig, B. (2008) · International Conference on Computer Vision Theory and Applications · doi:10.5220/0001082801450151 · Cited by 52 · see citing works
Visual emotion recognition using compact facial representations and viseme information
Metallinou, A., Busso, C., Lee, S., & Narayanan, S. S. (2010) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2010.5494893 · Cited by 52 · see citing works
Emotion classification in children's speech using fusion of acoustic and linguistic features
Polzehl, T., Sundaram, S., Ketabdar, H., Wagner, M., & Metze, F. (2009) · Interspeech (USB) · doi:10.21437/interspeech.2009-110 · Cited by 39 · see citing works
Likability classification - a not so deep neural network approach
Brueckner, R., & Schuller, B. W. (2012) · Interspeech (USB) · doi:10.21437/interspeech.2012-95 · Cited by 29 · see citing works
Multimodal Emotion Recognition from Low-Level Cues
Pantić, M., Caridakis, G., André, E., Kim, J., Karpouzis, K. C., & Kollias, S. (2010) · Cognitive technologies · doi:10.1007/978-3-642-15184-2_8 · Cited by 28 · see citing works
Feature Selection Based on Information Theory Filters
Duch, W., Biesiada, J., Winiarski, T., Grudziński, K., & Grąbczewski, K. (2003) · Physica-Verlag eBooks · doi:10.1007/978-3-7908-1902-1_23 · Cited by 15 · see citing works
Flagged first, then by citations
1 of the 39 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · see citing works
A comprehensive survey on support vector machine classification: Applications, challenges and trends
Cervantes, J., Garcia-Lamont, F., Rodríguez-Mazahua, L., & Lopez, A. (2020) · Neurocomputing · doi:10.1016/j.neucom.2019.10.118 · Cited by 2,381 · see citing works
Speech Recognition Using Deep Neural Networks: A Systematic Review
Nassif, A. B., Shahin, I. M. A., Attili, I. B., Azzeh, M., & Shaalan, K. F. (2019) · IEEE Access · doi:10.1109/access.2019.2896880 · Cited by 1,159 · see citing works
Speech emotion recognition using deep 1D & 2D CNN LSTM networks
Zhao, J., Mao, X., & Chen, L. (2019) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2018.08.035 · Cited by 1,044 · see citing works
Fadlullah, Z. M., Tang, F., Mao, B., Kato, N., Akashi, O., Inoue, T., & Mizutani, K. (2017) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2017.2707140 · Cited by 810 · see citing works
Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review
Zhang, J., Yin, Z., Chen, P., & Nichele, S. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.01.011 · Cited by 806 · see citing works
End-to-End Multimodal Emotion Recognition using Deep Neural Networks
Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2017) · arXiv · doi:10.48550/arxiv.1704.08619 · Cited by 752 · see citing works
Speech emotion recognition with deep convolutional neural networks
Issa, D., Demirci, M. F., & Yazıcı, A. (2020) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2020.101894 · Cited by 621 · see citing works
A Review of Human Activity Recognition Methods
Vrigkas, M., Nikou, C., & Kakadiaris, I. A. (2015) · Frontiers in Robotics and AI · doi:10.3389/frobt.2015.00028 · Cited by 571 · see citing works
Evaluating deep learning architectures for Speech Emotion Recognition
Fayek, H. M., Lech, M., & Cavedon, L. (2017) · Neural Networks · doi:10.1016/j.neunet.2017.02.013 · Cited by 554 · see citing works
Schuller, B. W. (2018) · Communications of the ACM · doi:10.1145/3129340 · Cited by 547 · see citing works
Deep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535 · see citing works
Emotion recognition using deep learning approach from audio–visual emotional big data
Hossain, M. S., & Muhammad, G. (2018) · Information Fusion · doi:10.1016/j.inffus.2018.09.008 · Cited by 454 · see citing works
DeepX: A Software Accelerator for Low-Power Deep Learning Inference on Mobile Devices
Lane, N. D., Bhattacharya, S., Georgiev, P., Forlivesi, C., Jiao, L., Qendro, L., & Kawsar, F. (2016) · 2016 15th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) · doi:10.1109/ipsn.2016.7460664 · Cited by 450 · see citing works
Speech emotion recognition using convolutional and Recurrent Neural Networks
Lım, W., Jang, D., & Lee, T. (2016) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipa.2016.7820699 · Cited by 421 · see citing works
Speech Emotion Recognition Using CNN
Huang, Z.-W., Dong, M., Mao, Q., & Zhan, Y. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654984 · Cited by 416 · see citing works
High-level feature representation using recurrent neural network for speech emotion recognition
Lee, J., & Tashev, I. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-336 · Cited by 408 · see citing works
Pacheco, F., Expósito, E., Gineste, M., Baudoin, C., & Aguilar, J. (2018) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2018.2883147 · Cited by 371 · see citing works
Speech Emotion Recognition From 3D Log-Mel Spectrograms With Deep Learning Network
Meng, H., Yan, T., Yuan, F., & Wei, H. (2019) · IEEE Access · doi:10.1109/access.2019.2938007 · Cited by 356 · see citing works
Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition
Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2719043 · Cited by 346 · see citing works
Hierarchical Convolutional Neural Networks for EEG-Based Emotion Recognition
Li, J., Zhang, Z., & He, H. (2017) · Cognitive Computation · doi:10.1007/s12559-017-9533-x · Cited by 340 · see citing works
MMTM: Multimodal Transfer Module for CNN Fusion
Joze, H. R. V., Shaban, A., Iuzzolino, M. L., & Koishida, K. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01330 · Cited by 323 · see citing works
Multimodal Deep Learning for Activity and Context Recognition
Radu, V., Tong, C., Bhattacharya, S., Lane, N. D., Mascolo, C., Marina, M. K., & Kawsar, F. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3161174 · Cited by 281 · see citing works
Emotion-aware connected healthcare big data towards 5G
Hossain, M. S., & Muhammad, G. (2018) · IEEE Internet of Things Journal · doi:10.1109/jiot.2017.2772959 · Cited by 273 · see citing works
M3ER: Multiplicative Multimodal Emotion Recognition using Facial, Textual, and Speech Cues
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i02.5492 · Cited by 23 · see citing works
Survey on Deep Neural Networks in Speech and Vision Systems
Alam, M., Samad, M. D., Vidyaratne, L., Glandon, A., & Iftekharuddin, K. M. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.07.053 · Cited by 257 · see citing works
Multimodal Sentiment Analysis: Addressing Key Issues and Setting Up the Baselines
Poria, S., Majumder, N., Hazarika, D., Cambria, E., Gelbukh, A. F., & Hussain, A. (2018) · IEEE Intelligent Systems · doi:10.1109/mis.2018.2882362 · Cited by 254 · see citing works
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
Multimodal emotion recognition using deep learning architectures
Ranganathan, H., Chakraborty, S., & Panchanathan, S. (2016) · 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2016.7477679 · Cited by 249 · see citing works
Facial Expression Recognition in Video with Multiple Feature Fusion
Chen, J., Chen, Z., Chi, Z., & Fu, H. C. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2593719 · Cited by 225 · see citing works
Analyzing Personality through Social Media Profile Picture Choice
Leqi, L., Preoțiuc-Pietro, D., Samani, Z. R., Moghaddam, M. E., & Ungar, L. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v10i1.14738 · Cited by 219 · see citing works
Scene Recognition by Manifold Regularized Deep Learning Architecture
Yuan, Y., Mou, L., & Lu, X. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2359471 · Cited by 214 · see citing works
Kim, Y., Soyata, T., & Feyzi-Behnagh, R. (2018) · IEEE Access · doi:10.1109/access.2018.2791861 · Cited by 216 · see citing works
Self-citationDeep Learning on Multi Sensor Data for Counter UAV Applications—A Systematic Review
Samaras, S., Diamantidou, E., Ataloglou, D., Sakellariou, N., Vafeiadis, A., Magoulianitis, V., Lalas, A. X., Dimou, A., Zarpalas, D., Votis, K., et al. (2019) · Sensors · doi:10.3390/s19224837 · Cited by 219 · see citing works
Abdelhamid, A. A., El‐kenawy, E. M., Alotaibi, B., Amer, G. M., Abdelkader, M. Y., Ibrahim, A. F., & Eid, M. M. (2022) · IEEE Access · doi:10.1109/access.2022.3172954 · Cited by 194 · see citing works
Emotion recognition from unimodal to multimodal analysis: A review
Ezzameli, K., & Mahersia, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101847 · Cited by 195 · see citing works
Multimodal Transformer Fusion for Continuous Emotion Recognition
Huang, J., Tao, J., Liu, B., Lian, Z., & Niu, M. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053762 · Cited by 187 · see citing works
A Multi-Task Learning Framework for Emotion Recognition Using 2D Continuous Space
Xia, R., & Liu, Y. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2512598 · Cited by 182 · see citing works
Multimodal Measurement of Depression Using Deep Learning Models
Yang, L., Jiang, D., Xia, X., Pei, E., Oveneke, M. C., & Hichem Sahli, B. (2017) · doi:10.1145/3133944.3133948 · Cited by 175 · see citing works
Ensemble of Deep Neural Networks with Probability-Based Fusion for Facial Expression Recognition
Wen, G., Hou, Z., Li, H., Li, D., Jiang, L., & Xun, E. (2017) · Cognitive Computation · doi:10.1007/s12559-017-9472-6 · Cited by 164 · see citing works
A DBN-based resampling SVM ensemble learning paradigm for credit classification with imbalanced data
Yu, L., Zhou, R., Tang, L., & Chen, R. (2018) · Applied Soft Computing · doi:10.1016/j.asoc.2018.04.049 · Cited by 161 · see citing works
3D CNN-Based Speech Emotion Recognition Using K-Means Clustering and Spectrograms
Hajarolasvadi, N., & Demirel, H. (2019) · Entropy · doi:10.3390/e21050479 · Cited by 157 · see citing works
CNN+LSTM Architecture for Speech Emotion Recognition with Data Augmentation
Etienne, C., Fidanza, G., Petrovskii, A., Devillers, L. Y., & Schmauch, B. (2018) · Workshop on Speech, Music and Mind (SMM) · doi:10.21437/smm.2018-5 · Cited by 161 · see citing works
A systematic literature review of speech emotion recognition approaches
Singh, Y. B., & Goel, S. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.04.028 · Cited by 150 · see citing works
Improved Multimodal Deep Learning with Variation of Information
Sohn, K., Shang, W., & Lee, H. (2014) · neural information processing systems · Cited by 142 · see citing works
Self-citationLearning deep features to recognise speech emotion using merged deep CNN
Zhao, J., Mao, X., & Chen, L. (2018) · IET Signal Processing · doi:10.1049/iet-spr.2017.0320 · Cited by 142 · see citing works
Audiovisual Fusion: Challenges and New Approaches
Katsaggelos, A. K., Bahaadini, S., & Molina, R. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2459017 · Cited by 140 · see citing works
Medical image analysis using wavelet transform and deep belief networks
Khatami, A., Khosravi, A., Nguyen, T. T., Lim, C. P., & Nahavandi, S. (2017) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.05.073 · Cited by 136 · see citing works
Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V. (2018) · IEEE Access · doi:10.1109/access.2018.2868361 · Cited by 135 · see citing works
Cai, Y., Li, X., & Li, J. (2023) · Sensors · doi:10.3390/s23052455 · Cited by 143 · see citing works
Gideon, J., McInnis, M. G., & Provost, E. M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2916092 · Cited by 128 · see citing works
Self-citationLearning in high-dimensional multimedia data: the state of the art
Gao, L., Song, J., Liu, X., Shao, J., Liu, J., & Shao, J. (2015) · Multimedia Systems · doi:10.1007/s00530-015-0494-1 · Cited by 127 · see citing works
Representation Learning for Speech Emotion Recognition
Ghosh, S., Laksana, E., Morency, L., & Scherer, S. J. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-692 · Cited by 125 · see citing works
D-vlog: Multimodal Vlog Dataset for Depression Detection
Yoon, J., Kang, C., Kim, S., & Han, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21483 · Cited by 126 · see citing works
Cooperative Learning and its Application to Emotion Recognition from Speech
Zhang, Z., Coutinho, E., Deng, J., & Schuller, B. W. (2014) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2014.2375558 · Cited by 121 · see citing works
Alzheimer's disease and automatic speech analysis: A review
Pulido, M. L. B., Hernández, J. B. A., Ballester, M. Á. F., González, C. M. T., Mekyska, J., & Smékal, Z. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113213 · Cited by 120 · see citing works
The Human in Emotion Recognition on Social Media: Attitudes, Outcomes, Risks
Andalibi, N., & Buss, J. (2020) · International Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376680 · Cited by 125 · see citing works
Emotion recognition from spontaneous speech using Hidden Markov models with deep belief networks
Le, D., & Provost, E. M. (2013) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2013.6707732 · Cited by 120 · see citing works
Self-citationIntegrating Deep and Shallow Models for Multi-Modal Depression Analysis—Hybrid Architectures
Yang, L., Jiang, D., & Hichem Sahli, B. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2870398 · Cited by 121 · see citing works
Machot, F. A., Elmachot, A., Ali, M., Machot, E. A., & Kyamakya, K. (2019) · Sensors · doi:10.3390/s19071659 · Cited by 113 · see citing works
Supervised Committee of Convolutional Neural Networks in Automated Facial Expression Analysis
Pons, G., & Masip, D. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2753235 · Cited by 109 · see citing works
A survey of multimodal deep generative models
Suzuki, M., & Matsuo, Y. (2022) · Advanced Robotics · doi:10.1080/01691864.2022.2035253 · Cited by 109 · see citing works
“[82] propose a multimodal generative model based on deep belief networks [28, 29], which has since been used in various multimodal applications [34, 41].”
BackgroundFeature learning based on SAE–PCA network for human gesture recognition in RGBD images
Li, S., Yu, B., Wu, W., Su, S., & Ji, R. (2014) · Neurocomputing · doi:10.1016/j.neucom.2014.06.086 · Cited by 108 · see citing works
Multimodal Deep Convolutional Neural Network for Audio-Visual Emotion Recognition
Zhang, S., Zhang, S. S., Huang, T., & Gao, W. (2016) · International Conference on Multimedia Retrieval · doi:10.1145/2911996.2912051 · Cited by 107 · see citing works
Deep spatio-temporal feature fusion with compact bilinear pooling for multimodal emotion recognition
Nguyen, D., Nguyen, K., Sridharan, S., Dean, D., & Fookes, C. (2018) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2018.06.005 · Cited by 106 · see citing works
Emotion Recognition in the Wild with Feature Fusion and Multiple Kernel Learning
Chen, J., Chen, Z., Chi, Z., & Fu, H. C. (2014) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2663204.2666277 · Cited by 104 · see citing works
Emotion recognition using secure edge and cloud computing
Hossain, M. S., & Muhammad, G. (2019) · Information Sciences · doi:10.1016/j.ins.2019.07.040 · Cited by 104 · see citing works
Towards Robust Discriminative Projections Learning via Non-Greedy -Norm MinMax
Nie, F., Wang, Z., Wang, R., Wang, Z., & Li, X. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2961877 · Cited by 104 · see citing works
Yan, J., Zheng, W., Xu, Q., Lu, G., Li, H., & Wang, B. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2557721 · Cited by 102 · see citing works
Automatic Emotion Recognition Using Temporal Multimodal Deep Learning
Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V. (2020) · IEEE Access · doi:10.1109/access.2020.3027026 · Cited by 101 · see citing works
Deep Neural Networks for Emotion Recognition Combining Audio and Transcripts
Cho, J., Pappagari, R. R., Kulkarni, P., Villalba, J., Carmiel, Y., & Dehak, N. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2466 · Cited by 100 · see citing works
Yoo, Y., Tang, L. Y. W., Brosch, T., Li, D. K. B., Kolind, S., Vavasour, I. M., Rauscher, A., MacKay, A. L., Traboulsee, A. L., & Tam, R. C. (2017) · NeuroImage Clinical · doi:10.1016/j.nicl.2017.10.015 · Cited by 97 · see citing works
Recognition of emotion in music based on deep convolutional neural network
Sarkar, R., Choudhury, S., Dutta, S., Roy, A., & Saha, S. K. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-08192-x · Cited by 94 · see citing works
Multimodal Deep Learning Framework for Mental Disorder Recognition
Zhang, Z., Lin, W., Liu, M., & Mahmoud, M. (2020) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg47880.2020.00033 · Cited by 89 · see citing works
A Research of Speech Emotion Recognition Based on Deep Belief Network and SVM
Huang, C., Gong, W. Y., Fu, W., & Feng, D. (2014) · Mathematical Problems in Engineering · doi:10.1155/2014/749604 · Cited by 87 · see citing works
Subramaniam, A., Patel, V., Mishra, A. K., Balasubramanian, P., & Mittal, A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-49409-8_27 · Cited by 86 · see citing works
Deep Spatio-Temporal Features for Multimodal Emotion Recognition
Nguyen, D., Nguyen, K., Sridharan, S., Ghasemi, A., Dean, D., & Fookes, C. (2017) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2017.140 · Cited by 82 · see citing works
A LSTM based deep learning network for recognizing emotions using wireless brainwave driven system
Sakalle, A., Tomar, P., Bhardwaj, H., Acharya, D., & Bhardwaj, A. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114516 · Cited by 82 · see citing works
Zou, Z., Zhao, X., Zhao, P., Qi, F., & Wang, N. (2019) · Journal of Cultural Heritage · doi:10.1016/j.culher.2019.02.002 · Cited by 81 · see citing works
Campos, V., Salvador, A., Giró-i-Nieto, X., & Jou, B. (2015) · doi:10.1145/2813524.2813530 · Cited by 78 · see citing works
Multi-Modal Sequence Fusion via Recursive Attention for Emotion Recognition
Beard, R., Das, R., Ng, R. W. M., Gopalakrishnan, P. G. K., Eerens, L., Świętojański, P., & Mikšík, O. (2018) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k18-1025 · Cited by 77 · see citing works
Recognizing Induced Emotions of Movie Audiences from Multimodal Information
Muszyński, M., Tian, L., Lai, C., Moore, J. D., Κωστούλας, Θ., Lombardo, P., Pun, T., & Chanel, G. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2902091 · Cited by 78 · see citing works
Real-time speech emotion recognition using deep learning and data augmentation
Barhoumi, C., & BenAyed, Y. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-11065-x · Cited by 76 · see citing works
Using Synthetic Data to Improve Facial Expression Analysis with 3D Convolutional Networks
Abbasnejad, I., Sridharan, S., Nguyen, D., Denman, S., Fookes, C., & Lucey, S. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.189 · Cited by 73 · see citing works
Machine Intelligence in Healthcare and Medical Cyber Physical Systems: A Survey
Shishvan, O. R., Zois, D., & Soyata, T. (2018) · IEEE Access · doi:10.1109/access.2018.2866049 · Cited by 73 · see citing works
Touati, R., Mignotte, M., & Dahmane, M. (2020) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2020.2964409 · Cited by 74 · see citing works
Learning Human Activities through Wi-Fi Channel State Information with Multiple Access Points
Li, H., Ota, K., Dong, M., & Guo, M. (2018) · IEEE Communications Magazine · doi:10.1109/mcom.2018.1700083 · Cited by 73 · see citing works
Modeling subjectiveness in emotion recognition with deep neural networks: Ensembles vs soft labels
Fayek, H. M., Lech, M., & Cavedon, L. (2016) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2016.7727250 · Cited by 73 · see citing works
Han, J., Zhang, Z., Cummins, N., & Schuller, B. W. (2019) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2019.2901088 · Cited by 86 · see citing works
Improved SMOTE Algorithm to Deal with Imbalanced Activity Classes in Smart Homes
Guo, S., Liu, Y., Chen, R., Sun, X. N., & Wang, X. (2018) · Neural Processing Letters · doi:10.1007/s11063-018-9940-3 · Cited by 71 · see citing works
Emotion Recognition by Fusing Time Synchronous and Time Asynchronous Representations
Wu, W., Zhang, C., & Woodland, P. C. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414880 · Cited by 70 · see citing works
Audio-visual emotion recognition using multi-directional regression and Ridgelet transform
Hossain, M. S., & Muhammad, G. (2015) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-015-0207-2 · Cited by 68 · see citing works
Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN
Ma, F., Li, Y., Ni, S., Huang, S., & Zhang, L. (2022) · Applied Sciences · doi:10.3390/app12010527 · Cited by 69 · see citing works
Qadri, S. F., Ai, D., Hu, G., Ahmad, M., Huang, Y., Wang, Y., & Yang, J. (2018) · Applied Sciences · doi:10.3390/app9010069 · Cited by 67 · see citing works
Deep Canonical Time Warping for Simultaneous Alignment and Representation Learning of Sequences
Trigeorgis, G., Nicolaou, M. A., Schuller, B. W., & Zafeiriou, S. P. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2710047 · Cited by 62 · see citing works
A deep-learning approach to facial expression recognition with candid images
Li, W., Li, M., Su, Z., & Zhu, Z. (2015) · Machine Vision Applications · doi:10.1109/mva.2015.7153185 · Cited by 62 · see citing works
A virtual environment for learning computer coding using gamification and emotion recognition
Cabada, R. Z., Estrada, M. L. B., Ríos-Félix, J. M., & Alor‐Hernández, G. (2018) · Interactive Learning Environments · doi:10.1080/10494820.2018.1558256 · Cited by 61 · see citing works
Guo, L., Wang, L., Dang, J., Liu, Z., & Guan, H. (2019) · IEEE Access · doi:10.1109/access.2019.2921390 · Cited by 61 · see citing works
Siddiqui, M. F. H., & Javaid, A. Y. (2020) · Multimodal Technologies and Interaction · doi:10.3390/mti4030046 · Cited by 62 · see citing works
Deep Learning based Emotion Recognition System Using Speech Features and TranscriptionsPreprint
Tripathi, S., Kumar, A., Ramesh, A., Singh, C., & Yenigalla, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05681 · Cited by 76 · see citing works
- Authors
- Kim, Y., Lee, H., & Provost, E. M.
- Affiliations
- University of Michigan (US)
- Venue
- 2013 IEEE International Conference on Acoustics, Speech and Signal Processing
- Publisher
- IEEE
- Published
- 2013-05
- Pages
- 3687-3691
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2009059481
- Cited by
- 401
- References
- 39
- Citations by year
- 2026: 5 · 2025: 19 · 2024: 22 · 2023: 18 · 2022: 33 · 2021: 36 · 2020: 44 · 2019: 50 · 2018: 49 · 2017: 47 · 2016: 29 · 2015: 30 · 2014: 17 · 2013: 2
- FWCI
- 35.38
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- audio-visual emotion recognition, deep belief network, feature selection, multimodal data, emotion classification, higher-order interactions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
