LSTM-Modeling of continuous emotions in an audiovisual affect recognition framework
Open accessAbstract
Automatically recognizing human emotions from spontaneous and non-prototypical real-life data is currently one of the most challenging tasks in the field of affective computing. This article presents our recent advances in assessing dimensional representations of emotion, such as arousal, expectation, power, and valence, in an audiovisual human-computer interaction scenario. Building on previous studies which demonstrate that long-range context modeling tends to increase accuracies of emotion recognition, we propose a fully automatic audiovisual recognition approach based on Long Short-Term Memory (LSTM) modeling of word-level audio and video features. LSTM networks are able to incorporate knowledge about how emotions typically evolve over time so that the inferred emotion estimates are produced under consideration of an optimal amount of context. Extensive evaluations on the Audiovisual Sub-Challenge of the 2011 Audio/Visual Emotion Challenge show how acoustic, linguistic, and visual features contribute to the recognition of different affective dimensions as annotated in the SEMAINE database. We apply the same acoustic features as used in the challenge baseline system whereas visual features are computed via a novel facial movement feature extractor. Comparing our results with the recognition scores of all Audiovisual Sub-Challenge participants, we find that the proposed LSTM-based technique leads to the best average recognition performance that has been reported for this task so far.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Data Mining: Practical Machine Learning Tools and Techniques
Witten, I. H., Frank, E., & Hall, M. A. (2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,493 · see citing works
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · see citing works
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398 · see citing works
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097 · see citing works
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · see citing works
Witten, I. H. (2008) · Cited by 9,115 · see citing works
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 5,882 · see citing works
Learning to Forget: Continual Prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 5,869 · see citing works
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120 · see citing works
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642 · see citing works
Self-citationEmotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603 · see citing works
Facial expression recognition based on Local Binary Patterns: A comprehensive study
Shan, C., Gong, S., & McOwan, P. W. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.08.005 · Cited by 2,300 · see citing works
Li, S. Z., Jain, A. K., Deng, J., & service), S. (. (2011) · Springer eBooks · doi:10.1007/978-0-85729-932-1 · Cited by 1,849 · see citing works
Computer Vision Face Tracking For Use in a Perceptual User Interface
Gary, R. B. (1998) · Cited by 1,400 · see citing works
Gradient Flow in Recurrent Nets: The Difficulty of Learning LongTerm Dependencies
(2009) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544037.ch14 · Cited by 1,312 · see citing works
The world of emotions is not two-dimensional
Fontaine, J. R. J., Scherer, K. R., Roesch, E. B., & Ellsworth, P. C. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.02024.x · Cited by 1,179 · see citing works
Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies
Hochreiter, S., Bengio, Y., Informatik, F. F., Bengio, Y., Frasconi, P., & Schmidhuber, J. (2001) · Cited by 863 · see citing works
Kotsia, I., & Pitas, I. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.884954 · Cited by 687 · see citing works
Hidden Markov model-based speech emotion recognition
Schuller, B. W., Rigoll, G., & Lang, M. (2003) · doi:10.1109/icme.2003.1220939 · Cited by 579 · see citing works
Self-citationNicolaou, M. A., Gunes, H., & Pantic, M. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.9 · Cited by 450 · see citing works
Audio-visual integration of emotion expression
Collignon, O., Girard, S., Gosselin, F., Roy, S., Saint‐Amour, D., Lassonde, M. C., & Leporé, F. (2008) · Brain Research · doi:10.1016/j.brainres.2008.04.023 · Cited by 353 · see citing works
Probabilistic and Bottle-Neck Features for LVCSR of Meetings
Grézl, F., Karafiát, M., Kontar, S., & Černocký, J. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.367023 · Cited by 340 · see citing works
Wöllmer, M., Eyben, F., Reiter, S. A., Schuller, B. W., Cox, C., Douglas‐Cowie, E., & Cowie, R. (2008) · Interspeech (USB) · doi:10.21437/interspeech.2008-192 · Cited by 329 · see citing works
Self-citationEmotion representation, analysis and synthesis in continuous space: A survey
Güneş, H., Schuller, B. W., Pantić, M., & Cowie, R. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771357 · Cited by 328 · see citing works
Self-citationDouglas‐Cowie, E., Cowie, R., Sneddon, I., Cox, C., Lowry, O. M., McRorie, M., Martin, J., Devillers, L. Y., Abrilian, S., Batliner, A. M., et al. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2_43 · Cited by 323 · see citing works
openSMILE - The Munich Versatile and Fast Open-Source Audio Feature Extractor
Eyben, F. ;. (2013) · mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) · Cited by 322 · see citing works
Self-citationThe SEMAINE corpus of emotionally coloured character interactions
McKeown, G. J., Valstar, M., Cowie, R., & Pantić, M. (2010) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2010.5583006 · Cited by 282 · see citing works
Acoustic emotion recognition: A benchmark comparison of performances
Schuller, B. W., Vlasenko, B., Eyben, F., Rigoll, G., & Wendemuth, A. (2009) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2009.5372886 · Cited by 275 · see citing works
Self-citationRecent Development of Open-Source Speech Recognition Engine Julius
Lee, A., & Kawahara, T. (2009) · Hokkaido University Collection of Scholarly and Academic Papers (Hokkaido University) · Cited by 271 · see citing works
AVEC 2011–The First International Audio/Visual Emotion Challenge
Schuller, B. W., Valstar, M., Eyben, F., McKeown, G. J., Cowie, R., & Pantić, M. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_53 · Cited by 270 · see citing works
Self-citationAn Application of Recurrent Neural Networks to Discriminative Keyword Spotting
Fernández, S., Graves, A., & Schmidhuber, J. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74695-9_23 · Cited by 251 · see citing works
Context Is Routinely Encoded During Emotion Perception
Barrett, L. F., & Kensinger, E. A. (2010) · Psychological Science · doi:10.1177/0956797610363547 · Cited by 242 · see citing works
The INTERSPEECH 2011 speaker state challenge
Schuller, B. W., Steidl, S., Batliner, A. M., Schiel, F., & Krajewski, J. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-801 · Cited by 225 · see citing works
Self-citationAn analysis of facial expression recognition under partial facial image occlusion
Kotsia, I., Buciu, I., & Pitas, I. (2007) · Image and Vision Computing · doi:10.1016/j.imavis.2007.11.004 · Cited by 223 · see citing works
Wöllmer, M., Metallinou, A., Eyben, F., Schuller, B., & Narayanan, S. S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-646 · Cited by 201 · cited 4× · see citing works
“Apart from preliminary experiments using facial marker information as additional input modality [13] and a recent study on subject dependent recognition of arousal and valence [27], LSTM architectures have hardly been applied for audiovisual emotion recognition.”
BackgroundMethodResultInfluentialSelf-citationSchuller, B. W., Müller, R., Eyben, F., Gast, J., Hörnler, B., Wöllmer, M., Rigoll, G., Hoethker, A., & Konosu, H. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2009.02.013 · Cited by 194 · see citing works
Self-citationBuilding autonomous sensitive artificial listeners
Schroder, M., Bevacqua, E., Cowie, R., Eyben, F., Gunes, H., Heylen, D., ter Maat, M., McKeown, G., Pammi, S., Pantic, M., et al. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.34 · Cited by 185 · see citing works
Self-citationOnline Driver Distraction Detection Using Long Short-Term Memory
Wöllmer, M., Blaschke, C., Schindl, T., Schuller, B. W., Färber, B., Mayer, S., & Trefflich, B. (2011) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2011.2119483 · Cited by 177 · see citing works
Self-citationWöllmer, M., Schuller, B. W., Eyben, F., & Rigoll, G. (2010) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2010.2057200 · Cited by 164 · see citing works
Self-citationEyben, F., Wöllmer, M., Graves, A., Schuller, B. W., Douglas‐Cowie, E., & Cowie, R. (2009) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-009-0032-6 · Cited by 137 · see citing works
Self-citationAudio–Visual Affective Expression Recognition Through Multistream Fused HMM
Zeng, Z., Tu, J., Pianfetti, B., & Huang, T. S. (2008) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2008.921737 · Cited by 134 · see citing works
Multiple Classifier Systems for the Classification of Audio-Visual Emotional States
Glodek, M., Tschechne, S., Layher, G., Schels, M., Brosch, T., Scherer, S. J., Kächele, M., Schmidt, M., Heiko Neumann, B., Palm, G., et al. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_47 · Cited by 133 · see citing works
Miller, G. (2004) · Science · doi:10.1126/science.304.5667.34 · Cited by 130 · see citing works
Support Vector Regression for Automatic Recognition of Spontaneous Emotions in Speech
Grimm, M., Kroschel, K., & Narayanan, S. S. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.367262 · Cited by 114 · see citing works
On the Role of Emotion in Embodied Cognitive Architectures: From Organisms to Robots
Tom Ziemke, B., & Lowe, R. J. (2009) · Cognitive Computation · doi:10.1007/s12559-009-9012-0 · Cited by 111 · see citing works
Audio-Visual Emotion Recognition Using Gaussian Mixture Models for Face and Voice
Metallinou, A., Lee, S., & Narayanan, S. S. (2008) · IEEE International Symposium on Multimedia (ISM) · doi:10.1109/ism.2008.40 · Cited by 97 · see citing works
Boosting encoded dynamic features for facial expression recognition
Yang, P., Liu, Q., & Metaxas, D. (2008) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.03.014 · Cited by 93 · see citing works
Modeling Latent Discriminative Dynamic of Multi-dimensional Affective Signals
Ramírez, G. A., Baltrušaitis, T., & Morency, L. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_51 · Cited by 91 · see citing works
Pose-Invariant Facial Expression Recognition Using Variable-Intensity Templates
Kumano, S., Otsuka, K., Yamato, J., Maeda, E., & Sato, Y. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-008-0185-x · Cited by 87 · see citing works
Wöllmer, M., Al-Hames, M., Eyben, F., Schuller, B. W., & Rigoll, G. (2009) · Neurocomputing · doi:10.1016/j.neucom.2009.08.005 · Cited by 86 · see citing works
Self-citationWöllmer, M., Eyben, F., Graves, A., Schuller, B. W., & Rigoll, G. (2010) · Cognitive Computation · doi:10.1007/s12559-010-9041-8 · Cited by 80 · see citing works
Self-citationAn Asynchronous Hidden Markov Model for Audio-Visual Speech Recognition
Samy Bengio, A. (2002) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 71 · see citing works
Wöllmer, M., Eyben, F., Keshet, J., Graves, A., Schuller, B. W., & Rigoll, G. (2009) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2009.4960492 · Cited by 66 · see citing works
Self-citationBrute-forcing hierarchical functionals for paralinguistics: A waste of feature space?
Schuller, B. W., Wimmer, M., Mösenlechner, L., Kern, C., Arsić, D., & Rigoll, G. (2008) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2008.4518656 · Cited by 60 · see citing works
Self-citationMeng, H., & Bianchi‐Berthouze, N. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_49 · Cited by 57 · see citing works
String-based audiovisual fusion of behavioural events for the assessment of dimensional affect
Eyben, F., Wöllmer, M., Valstar, M., Güneş, H., Schuller, B. W., & Pantić, M. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771417 · Cited by 54 · see citing works
Self-citationReal-life emotion-related states detection in call centers: a cross-corpora study
Devillers, L. Y., Vaudable, C., & Chastagnol, C. (2010) · Interspeech (USB) · doi:10.21437/interspeech.2010-643 · Cited by 52 · see citing works
A multi-stream ASR framework for BLSTM modeling of conversational speech
Wöllmer, M., Eyben, F., Schuller, B. W., & Rigoll, G. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947444 · Cited by 42 · see citing works
Self-citationThe hinterland of emotions: Facing the open-microphone challenge
Steidl, S., Batliner, A. M., Schuller, B. W., & Seppi, D. (2009) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2009.5349499 · Cited by 37 · see citing works
Self-citation2007 IEEE Workshop on Applications of Computer Vision (WACV '07)
(2007) · doi:10.1109/wacv.2007 · Cited by 32 · see citing works
Stupakov, A., Hanusa, E., Vijaywargi, D., Fox, D., & Bilmes, J. (2011) · Computer Speech & Language · doi:10.1016/j.csl.2010.12.003 · Cited by 29 · see citing works
Wöllmer, M., Metallinou, A., Katsamanis, N., Schuller, B. W., & Narayanan, S. S. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288834 · Cited by 29 · see citing works
Self-citationA Psychologically-Inspired Match-Score Fusion Model for Video-Based Facial Expression Recognition
Cruz, A. C., Bhanu, B., & Yang, S. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_45 · Cited by 28 · see citing works
Continuous Emotion Recognition Using Gabor Energy Filters
Dahmane, M., & Meunier, J. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_46 · Cited by 27 · see citing works
Investigating the Use of Formant Based Features for Detection of Affective Dimensions in Speech
Kim, J. C., Rao, H., & Clements, M. A. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_48 · Cited by 26 · see citing works
Tandem decoding of children's speech for keyword detection in a child-robot interaction scenario
Wöllmer, M., Schuller, B., Batliner, A., Steidl, S., & Seppi, D. (2011) · ACM Transactions on Speech and Language Processing · doi:10.1145/1998384.1998386 · Cited by 18 · see citing works
Self-citationSayedelahl, A., Fewzee, P., Kamel, M. S., & Karray, F. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_52 · Cited by 17 · see citing works
The CASIA Audio Emotion Recognition Method for Audio/Visual Emotion Challenge 2011
Pan, S., Tao, J., & Li, Y. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_50 · Cited by 14 · see citing works
Wöllmer, M., Schuller, B. W., & Rigoll, G. (2011) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2011.6163902 · Cited by 14 · see citing works
Self-citationSpeech Emotion Recognition System Based on L1 Regularized Linear Regression and Decision Fusion
Cen, L., Yu, Z. L., & Dong, M. H. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_44 · Cited by 11 · see citing works
Video object detection speedup using staggered sampling
Greig, D. (2009) · doi:10.1109/wacv.2009.5403129 · Cited by 10 · see citing works
A hierarchical approach for visual suspicious behavior detection in aircrafts
Arsić, D., Hörnler, B., Schuller, B. W., & Rigoll, G. (2009) · IEEE International Conference on Digital Signal Processing (DSP) · doi:10.1109/icdsp.2009.5201258 · Cited by 6 · see citing works
Self-citation
Flagged first, then by citations
7 of the 81 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · see citing works
Beyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. D. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299101 · Cited by 2,121 · see citing works
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644 · see citing works
Gandhi, A., Adhvaryu, K., Poria, S., Cambria, E., & Hussain, A. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.09.025 · Cited by 783 · see citing works
A Review and Meta-Analysis of Multimodal Affect Detection Systems
D’Mello, S. K., & Kory-Westlund, J. M. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 654 · see citing works
A survey of multimodal sentiment analysis
Soleymani, M., García, D., Jou, B., Schuller, B. W., Chang, S., & Pantić, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.08.003 · Cited by 620 · see citing works
Self-citationA survey of sentiment analysis in social media
Yue, L., Chen, W., Li, X., Zuo, W., & Yin, M. (2018) · Knowledge and Information Systems · doi:10.1007/s10115-018-1236-4 · Cited by 594 · see citing works
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527 · see citing works
Zhang, S., Zhang, S. S., Huang, T., & Gao, W. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2766843 · Cited by 444 · see citing works
Youtube movie reviews: Sentiment analysis in an audio-visual context
Wollmer, M., Weninger, F., Knaup, T., Schuller, B., Sun, C., Sagae, K., & Morency, L.-P. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.34 · Cited by 403 · see citing works
Self-citationLearning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition
Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2719043 · Cited by 346 · see citing works
Deep Learning Techniques for Speech Emotion Recognition, from Databases to Models
Abbaschian, B., Sierra-Sosa, D., & Elmaghraby, A. S. (2021) · Sensors · doi:10.3390/s21041249 · Cited by 302 · see citing works
Non-ferrous metals price forecasting based on variational mode decomposition and LSTM network
Liu, Y., Yang, C., Huang, K., & Gui, W. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.105006 · Cited by 254 · see citing works
Beyond Short Snippets: Deep Networks for Video ClassificationPreprint
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.08909 · Cited by 276 · see citing works
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
Wearable-Based Affect Recognition—A Review
Schmidt, P., Reiss, A., Dürichen, R., & Laerhoven, K. V. (2019) · Sensors · doi:10.3390/s19194079 · Cited by 267 · see citing works
Facial Expression Recognition: A Survey
Huang, Y., Chen, F., Lv, S., & Wang, X. (2019) · Symmetry · doi:10.3390/sym11101189 · Cited by 195 · see citing works
A survey of emotion recognition methods with emphasis on E-Learning environments
Imani, M., & Montazer, G. A. (2019) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2019.102423 · Cited by 190 · see citing works
Deep learning the dynamic appearance and shape of facial action units
Jaiswal, S., & Valstar, M. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477625 · Cited by 165 · see citing works
Audio-visual emotion fusion (AVEF): A deep efficient weighted approach
Ma, Y., Hao, Y., Chen, M., Chen, J., Lu, P., & Košir, A. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.06.003 · Cited by 155 · see citing works
Shlizerman, E., Dery, L., Schoen, H., & Kemelmacher-Shlizerman, I. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00790 · Cited by 154 · see citing works
He, L., Jiang, D., Yang, L., Pei, E., Wu, P., & Hichem Sahli, B. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811641 · Cited by 149 · see citing works
Deep neural architectures for prediction in healthcare
Kollias, D., Tagaris, A., Stafylopatis, A., Kollias, S., & Tagaris, G. L. (2017) · Complex & Intelligent Systems · doi:10.1007/s40747-017-0064-6 · Cited by 147 · see citing works
A systematic literature review of speech emotion recognition approaches
Singh, Y. B., & Goel, S. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.04.028 · Cited by 150 · see citing works
Transformer Encoder With Multi-Modal Multi-Head Attention for Continuous Affect Recognition
Chen, H., Jiang, D., & Hichem Sahli, B. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3037496 · Cited by 142 · see citing works
Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V. (2018) · IEEE Access · doi:10.1109/access.2018.2868361 · Cited by 135 · see citing works
D-vlog: Multimodal Vlog Dataset for Depression Detection
Yoon, J., Kang, C., Kim, S., & Han, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21483 · Cited by 126 · see citing works
Long Short Term Memory Recurrent Neural Network based Multimodal Dimensional Emotion Recognition
Chao, L., Tao, J., Yang, M., Li, Y., & Wen, Z. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811634 · Cited by 124 · see citing works
Robust continuous prediction of human emotions using multiscale dynamic cues
Nicolle, J., Rapp, V., Bailly, K., Prévost, L., & Chétouani, M. (2012) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2388676.2388783 · Cited by 113 · see citing works
A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition
Praveen, R. G., de Melo, W. C., Ullah, N., Aslam, M. H., Zeeshan, O., Denorme, T., Pedersoli, M., Koerich, A. L., Bacon, S., Cardinal, P., et al. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00278 · Cited by 115 · see citing works
Multi-modal Dimensional Emotion Recognition using Recurrent Neural Networks
Chen, S., & Jin, Q. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811638 · Cited by 111 · see citing works
Deep spatio-temporal feature fusion with compact bilinear pooling for multimodal emotion recognition
Nguyen, D., Nguyen, K., Sridharan, S., Dean, D., & Fookes, C. (2018) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2018.06.005 · Cited by 106 · see citing works
ShEMO: a large-scale validated database for Persian speech emotion detection
Nezami, O. M., Lou, P. J., & Karami, M. (2018) · Language Resources and Evaluation · doi:10.1007/s10579-018-9427-x · Cited by 97 · see citing works
Multi-modal Conditional Attention Fusion for Dimensional Emotion Prediction
Chen, S., & Jin, Q. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2967286 · Cited by 87 · see citing works
Impact of training data size on the LSTM performances for rainfall–runoff modeling
Boulmaiz, T., Guermoui, M., & Boutaghane, H. (2020) · Modeling Earth Systems and Environment · doi:10.1007/s40808-020-00830-w · Cited by 87 · see citing works
Ji, Y., Hu, Y., Yang, Y., Shen, F., & Shen, H. T. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.12.037 · Cited by 86 · see citing works
Zhou, H., Du, J., Zhang, Y., Wang, Q., Liu, Q.-F., & Lee, C.-H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3096037 · Cited by 85 · see citing works
Adaptive confidence learning for the personalization of pain intensity estimation systems
Kächele, M., Amirian, M., Thiam, P., Werner, P., Walter, S., Palm, G., & Schwenker, F. (2016) · Evolving Systems · doi:10.1007/s12530-016-9158-4 · Cited by 82 · see citing works
Sun, L., Lian, Z., Tao, J., Liu, B., & Niu, M. (2020) · Proceedings of the 1st International on Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop · doi:10.1145/3423327.3423672 · Cited by 81 · see citing works
Multimedia Intelligence: When Multimedia Meets Artificial Intelligence
Zhu, W., Wang, X. Y., & Gao, W. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2969791 · Cited by 80 · see citing works
Fine-Grained Facial Expression Recognition in the Wild
Liang, L., Lang, C., Li, Y., Feng, S., & Zhao, J. C. (2020) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2020.3007327 · Cited by 81 · see citing works
Modeling Emotion in Complex Stories: The Stanford Emotional Narratives Dataset
Ong, D. C., Wu, Z., Zhi‐Xuan, T., Reddan, M. C., Kahhalé, I., Mattek, A. M., & Zaki, J. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2955949 · Cited by 77 · see citing works
Recognizing Induced Emotions of Movie Audiences from Multimodal Information
Muszyński, M., Tian, L., Lai, C., Moore, J. D., Κωστούλας, Θ., Lombardo, P., Pun, T., & Chanel, G. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2902091 · Cited by 78 · see citing works
Wearable Assistive Technologies for Autism: Opportunities and Challenges
Mansouri-Benssassi, E., Gómez, J. C., Boyd, L. E., Hayes, G. R., & Ye, J. (2018) · IEEE Pervasive Computing · doi:10.1109/mprv.2018.022511239 · Cited by 76 · see citing works
Unsupervised Adversarial Domain Adaptation for Cross-Lingual Speech Emotion Recognition
Latif, S., Qadir, J., & Bilal, M. (2019) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2019.8925513 · Cited by 64 · see citing works
The Fusion of Electroencephalography and Facial Expression for Continuous Emotion Recognition
Li, D., Wang, Z., Wang, C., Liu, S., Chi, W., Dong, E., Song, X., Gao, Q., & Song, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2949707 · Cited by 64 · see citing works
Audio–Visual Fusion for Emotion Recognition in the Valence–Arousal Space Using Joint Cross-Attention
Praveen, R. G., Cardinal, P., & Granger, É. (2023) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2022.3233083 · Cited by 62 · see citing works
Learning Facial Action Units from Web Images with Scalable Weakly Supervised Clustering
Zhao, K., Chu, W.-S., & Martı́nez, A. M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00223 · Cited by 62 · see citing works
Deep Auto-Encoders With Sequential Learning for Multimodal Dimensional Emotion Recognition
Nguyen, D., Nguyen, T. T., Zeng, R., Nguyen, T. T., Tran, S. N., Nguyen, T., Sridharan, S., & Fookes, C. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3063612 · Cited by 60 · see citing works
Ahuja, C., Ma, S., Morency, L., & Sheikh, Y. A. (2019) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3340555.3353725 · Cited by 59 · see citing works
Mmlatch: Bottom-Up Top-Down Fusion For Multimodal Sentiment Analysis
Paraskevopoulos, G., Georgiou, E., & Potamianos, A. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746418 · Cited by 59 · see citing works
Automatic Recognition of Facial Displays of Unfelt Emotions
Kulkarni, K., Corneanu, C. A., Ofodile, I., Escalera, S., Baró, X., Hyniewska, S., Allïk, J., & Anbarjafari, G. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2874996 · Cited by 61 · see citing works
Emotion recognition in never-seen languages using a novel ensemble method with emotion profiles
Albornoz, E. M., & Milone, D. H. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2503757 · Cited by 57 · see citing works
Li, X., & Akagi, M. (2019) · Speech Communication · doi:10.1016/j.specom.2019.04.004 · Cited by 56 · see citing works
Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition
Praveen, R. G., Granger, É., & Cardinal, P. (2021) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg52635.2021.9667055 · Cited by 55 · see citing works
Strength modelling for real-worldautomatic continuous affect recognition from audiovisual signals
Han, J., Zhang, Z., Cummins, N., Ringeval, F., & Schuller, B. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2016.11.020 · Cited by 53 · see citing works
Self-citationMa, X., Weisi, L., Dongyan, H., Dong, M., & Li, H. (2017) · IEEE International Conference on Signal and Image Processing (ICSIP) · doi:10.1109/siprocess.2017.8124509 · Cited by 54 · see citing works
Continuous sign language recognition from wearable IMUs using deep capsule networks and game theory
Suri, K., & Gupta, R. (2019) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2019.08.006 · Cited by 54 · see citing works
Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network
Huang, J., Li, Y., Tao, J., Lian, Z., Wen, Z., Yang, M., & Yi, J. (2017) · Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge · doi:10.1145/3133944.3133946 · Cited by 54 · see citing works
Temporal Bayesian Fusion for Affect Sensing: Combining Video, Audio, and Lexical Modalities
Savran, A., Cao, H., Nenkova, A., & Verma, R. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2362101 · Cited by 53 · see citing works
Goncalves, L., & Busso, C. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3216993 · Cited by 53 · see citing works
Multi task sequence learning for depression scale prediction from video
Chao, L., Tao, J., Yang, M., & Li, Y. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344620 · Cited by 53 · see citing works
Expression Recognition Method Based on a Lightweight Convolutional Neural Network
Zhao, G., Yang, H., & Yu, M. (2020) · IEEE Access · doi:10.1109/access.2020.2964752 · Cited by 53 · see citing works
Exploring the Contextual Factors Affecting Multimodal Emotion Recognition in Videos
Bhattacharya, P., Gupta, R., & Yang, Y. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3071503 · Cited by 49 · see citing works
(2019) · doi:10.1109/smc.2019.8914655 · Cited by 52 · see citing works
Boosting crop classification by hierarchically fusing satellite, rotational, and contextual data
Barrière, V., Claverie, M., Schneider, M., Lemoine, G., & d’Andrimont, R. (2024) · Remote Sensing of Environment · doi:10.1016/j.rse.2024.114110 · Cited by 51 · see citing works
A Lightweight Residual Networks Framework for DDoS Attack Classification Based on Federated Learning
Qin, T., Guang, C., Chen, W., & Si, W. (2021) · IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) · doi:10.1109/infocomwkshps51825.2021.9484622 · Cited by 50 · see citing works
Two-Stage Fuzzy Fusion Based-Convolution Neural Network for Dynamic Emotion Recognition
Wu, M., Su, W., Chen, L., Pedrycz, W., & Hirota, K. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2966440 · Cited by 48 · see citing works
Li, Z., Guo, Q., Pan, Y., Ding, W., Yu, J., Zhang, Y., Liu, W., Chen, H., Wang, H., & Xie, Y. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101891 · Cited by 49 · see citing works
Raza, M. A., Karim, A., Alqarni, M., Al‐Khasawneh, M. A., Jumani, T. A., Aman, M. M., & Masud, M. I. (2025) · Energies · doi:10.3390/en18133324 · Cited by 49 · see citing works
Survey on psychotherapy chatbots
Xu, B., & Zhuang, Z. (2020) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.6170 · Cited by 46 · see citing works
Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition
Praveen, R. G., & Alam, M. J. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00483 · Cited by 49 · see citing works
Multimodal Content Analysis for Effective Advertisements on YouTube
Vedula, N., Sun, W., Lee, H., Gupta, H., Ogihara, M., Johnson, J., Ren, G., & Parthasarathy, S. (2017) · Proceedings · doi:10.1109/icdm.2017.149 · Cited by 48 · see citing works
Dang, T., Stasak, B., Huang, Z., Jayawardena, S., Atcheson, M., Hayat, M., Le, P. N., Sethu, V., Goecke, R., & Epps, J. (2017) · doi:10.1145/3133944.3133952 · Cited by 46 · see citing works
Pre-trained Deep Convolution Neural Network Model With Attention for Speech Emotion Recognition
Zhang, H., Gou, R., Shang, J., Shen, F., Wu, Y., & Dai, G. (2021) · Frontiers in Physiology · doi:10.3389/fphys.2021.643202 · Cited by 45 · see citing works
Inferring Depression and Affect from Application Dependent Meta Knowledge
Kächele, M., Schels, M., & Schwenker, F. (2014) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2661806.2661813 · Cited by 43 · see citing works
Early prediction for physical human robot collaboration in the operating room
Zhou, T., & Wachs, J. P. (2017) · Autonomous Robots · doi:10.1007/s10514-017-9670-9 · Cited by 44 · see citing works
Peng, Z., Dang, J., Unoki, M., & Akagi, M. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.03.027 · Cited by 42 · see citing works
Multimodal Continuous Emotion Recognition with Data Augmentation Using Recurrent Neural Networks
Huang, J., Li, Y., Tao, J., Lian, Z., Niu, M., & Yang, M. (2018) · Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop · doi:10.1145/3266302.3266304 · Cited by 40 · see citing works
Reconstruction-error-based learning for continuous emotion recognition in speech
Han, J., Zhang, Z., Ringeval, F., & Schuller, B. W. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952580 · Cited by 41 · see citing works
Self-citationHigher-order Multivariable Polynomial Regression to Estimate Human Affective States
Wei, J., Chen, T., Liu, G., & Yang, J. (2016) · Scientific Reports · doi:10.1038/srep23384 · Cited by 41 · see citing works
Forecasting Renewable Energy Consumption under Zero Assumptions
Ma, J., Oppong, A., Acheampong, K. N., & Abruquah, L. A. (2018) · Sustainability · doi:10.3390/su10030576 · Cited by 40 · see citing works
Multi-scale Temporal Modeling for Dimensional Emotion Recognition in Video
Chao, L., Tao, J., Yang, M., Li, Y., & Wen, Z. (2014) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2661806.2661811 · Cited by 39 · see citing works
Transformer-Based Spiking Neural Networks for Multimodal Audiovisual Classification
Guo, L., Gao, Z., Qu, J., Zheng, S., Jiang, R., Lu, Y., & Qiao, H. (2023) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2023.3327081 · Cited by 38 · see citing works
Pham, P., & Wang, J. (2017) · International Conference on Intelligent User Interfaces · doi:10.1145/3025171.3025186 · Cited by 31 · see citing works
Joint Multi-Scale Multimodal Transformer for Emotion Using Consumer Devices
Khan, M., Ahmad, J., Gueaieb, W., Masi, G., Karray, F. O., & Saddik, A. E. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2025.3532322 · Cited by 33 · see citing works
Speech Emotion Recognition Using Deep Learning Techniques
Ganapathy, A. (2016) · ABC Journal of Advanced Research · doi:10.18034/abcjar.v5i2.550 · Cited by 34 · see citing works
End-to-End Continuous Emotion Recognition from Video Using 3D Convlstm Networks
Huang, J., Li, Y., Tao, J., Lian, Z., & Yi, J. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461963 · Cited by 32 · see citing works
K., T., Aziz, N. A. A., Raja, J., Hossen, J., & H., J. M. Z. (2022) · Emerging Science Journal · doi:10.28991/esj-2022-06-05-017 · Cited by 27 · see citing works
Measuring two aspects of emotion recognition ability: Accuracy vs. sensitivity
Lyusin, D., & Ovsyannikova, V. (2015) · Learning and Individual Differences · doi:10.1016/j.lindif.2015.04.010 · Cited by 32 · see citing works
Moorthy, S., & Moon, Y.-K. (2025) · Mathematics · doi:10.3390/math13071100 · Cited by 31 · see citing works
Exploring Multimodal Visual Features for Continuous Affect Recognition
Sun, B., Cao, S., Li, L., He, J. Y. D., & Yu, L. (2016) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2988257.2988270 · Cited by 31 · see citing works
Adaptation and contextualization of deep neural network models
Kollias, D., Yu, M., Tagaris, A., Leontidis, G., Stafylopatis, A., & Kollias, S. (2017) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci.2017.8280975 · Cited by 31 · see citing works
Multi-Factor RFG-LSTM Algorithm for Stock Sequence Predicting
Su, Z., Xie, H., & Han, L. (2020) · Computational Economics · doi:10.1007/s10614-020-10008-2 · Cited by 31 · see citing works
Wearable affect and stress recognition: A reviewPreprint
Schmidt, P., Reiss, A., Duerichen, R., & Laerhoven, K. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.08854 · Cited by 31 · see citing works
Predicting Latent Narrative Mood Using Audio and Physiologic Data
Alhanai, T., & Ghassemi, M. M. (2017) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10625 · Cited by 37 · see citing works
A Survey of Computational Approaches and Challenges in Multimodal Sentiment Analysis
Huddar, M. G., Sannakki, S. S., & Rajpurohit, V. S. (2019) · International Journal of Computer Sciences and Engineering · doi:10.26438/ijcse/v7i1.876883 · Cited by 31 · see citing works
Hybrid Mutimodal Fusion for Dimensional Emotion Recognition
Ma, Z., Ma, F., Sun, B., & Shutao Li, A. (2021) · doi:10.1145/3475957.3484457 · Cited by 29 · see citing works
Factored Latent-Dynamic Conditional Random Fields for single and multi-label sequence modeling
Neogi, S., & Dauwels, J. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108236 · Cited by 28 · see citing works
- Authors
- Wöllmer, M., Kaiser, M., Eyben, F., Schuller, B., & Rigoll, G.
- Affiliations
- Technical University of Munich (DE)
- Corresponding author
- Wöllmer, Martin
- Venue
- Image and Vision Computing
- Publisher
- Elsevier BV
- Published
- 2013-02
- Volume
- 31
- Issue
- 2
- Pages
- 153-163
- ISSN
- 0262-8856 (ISSN-L)
- OpenAlex
- W2026243162
- Semantic Scholar
- 752c6738a1ef1c18943636bc3c98a4dbd6455b33
- Cited by
- 288
- References
- 81
- Citations by year
- 2026: 4 · 2025: 15 · 2024: 16 · 2023: 13 · 2022: 17 · 2021: 39 · 2020: 22 · 2019: 38 · 2018: 32 · 2017: 31 · 2016: 21 · 2015: 19 · 2014: 11 · 2013: 8 · 2012: 2
- FWCI
- 17.14
- Type
- journal article
- Language
- en
- Open access
- green
- mediatum.ub.tum.de
- Licence
- Locations
- Image and Vision Computing (publishedVersion)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion, other-oa)
- OPUS (Augsburg University) (submittedVersion, cc-by-nc-nd)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- audio-visual emotion recognition, long short-term memory, arousal, valence, expectations, power, long-range context modeling, video features, human-computer interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review