The DOI matches its Crossref record; 74 of the 81 references it cites were found in a registry or an index.
LSTM-Modeling of continuous emotions in an audiovisual affect recognition framework
Open accessAbstract
Automatically recognizing human emotions from spontaneous and non-prototypical real-life data is currently one of the most challenging tasks in the field of affective computing. This article presents our recent advances in assessing dimensional representations of emotion, such as arousal, expectation, power, and valence, in an audiovisual human-computer interaction scenario. Building on previous studies which demonstrate that long-range context modeling tends to increase accuracies of emotion recognition, we propose a fully automatic audiovisual recognition approach based on Long Short-Term Memory (LSTM) modeling of word-level audio and video features. LSTM networks are able to incorporate knowledge about how emotions typically evolve over time so that the inferred emotion estimates are produced under consideration of an optimal amount of context. Extensive evaluations on the Audiovisual Sub-Challenge of the 2011 Audio/Visual Emotion Challenge show how acoustic, linguistic, and visual features contribute to the recognition of different affective dimensions as annotated in the SEMAINE database. We apply the same acoustic features as used in the challenge baseline system whereas visual features are computed via a novel facial movement feature extractor. Comparing our results with the recognition scores of all Audiovisual Sub-Challenge participants, we find that the proposed LSTM-based technique leads to the best average recognition performance that has been reported for this task so far.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Data Mining: Practical Machine Learning Tools and Techniques
Witten, I. H., Frank, E., & Hall, M. A. (2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,543
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Witten, I. H. (2008) · Cited by 9,143
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 6,152
Learning to Forget: Continual Prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 6,122
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642
Self-citationEmotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
Facial expression recognition based on Local Binary Patterns: A comprehensive study
Shan, C., Gong, S., & McOwan, P. W. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.08.005 · Cited by 2,300
Li, S. Z., Jain, A. K., Deng, J., & service), S. (. (2011) · Springer eBooks · doi:10.1007/978-0-85729-932-1 · Cited by 1,867
Computer Vision Face Tracking For Use in a Perceptual User Interface
Gary, R. B. (1998) · Cited by 1,400
Gradient Flow in Recurrent Nets: The Difficulty of Learning LongTerm Dependencies
(2009) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544037.ch14 · Cited by 1,312
The world of emotions is not two-dimensional
Fontaine, J. R. J., Scherer, K. R., Roesch, E. B., & Ellsworth, P. C. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.02024.x · Cited by 1,179
Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies
Hochreiter, S., Bengio, Y., Informatik, F. F., Bengio, Y., Frasconi, P., & Schmidhuber, J. (2001) · Cited by 863
Kotsia, I., & Pitas, I. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.884954 · Cited by 687
Hidden Markov model-based speech emotion recognition
Schuller, B. W., Rigoll, G., & Lang, M. (2003) · doi:10.1109/icme.2003.1220939 · Cited by 579
Self-citationNicolaou, M. A., Gunes, H., & Pantic, M. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.9 · Cited by 450
Audio-visual integration of emotion expression
Collignon, O., Girard, S., Gosselin, F., Roy, S., Saint‐Amour, D., Lassonde, M. C., & Leporé, F. (2008) · Brain Research · doi:10.1016/j.brainres.2008.04.023 · Cited by 376
Emotion representation, analysis and synthesis in continuous space: A survey
Güneş, H., Schuller, B. W., Pantić, M., & Cowie, R. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771357 · Cited by 344
Self-citationProbabilistic and Bottle-Neck Features for LVCSR of Meetings
Grézl, F., Karafiát, M., Kontar, S., & Černocký, J. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.367023 · Cited by 340
Wöllmer, M., Eyben, F., Reiter, S. A., Schuller, B. W., Cox, C., Douglas‐Cowie, E., & Cowie, R. (2008) · Interspeech (USB) · doi:10.21437/interspeech.2008-192 · Cited by 333
Self-citationDouglas‐Cowie, E., Cowie, R., Sneddon, I., Cox, C., Lowry, O. M., McRorie, M., Martin, J., Devillers, L. Y., Abrilian, S., Batliner, A. M., et al. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2_43 · Cited by 323
openSMILE - The Munich Versatile and Fast Open-Source Audio Feature Extractor
Eyben, F. ;. (2013) · mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) · Cited by 322
Self-citationThe SEMAINE corpus of emotionally coloured character interactions
McKeown, G. J., Valstar, M., Cowie, R., & Pantić, M. (2010) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2010.5583006 · Cited by 282
Acoustic emotion recognition: A benchmark comparison of performances
Schuller, B. W., Vlasenko, B., Eyben, F., Rigoll, G., & Wendemuth, A. (2009) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2009.5372886 · Cited by 275
Self-citationRecent Development of Open-Source Speech Recognition Engine Julius
Lee, A., & Kawahara, T. (2009) · Hokkaido University Collection of Scholarly and Academic Papers (Hokkaido University) · Cited by 271
AVEC 2011–The First International Audio/Visual Emotion Challenge
Schuller, B. W., Valstar, M., Eyben, F., McKeown, G. J., Cowie, R., & Pantić, M. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_53 · Cited by 270
Self-citationAn Application of Recurrent Neural Networks to Discriminative Keyword Spotting
Fernández, S., Graves, A., & Schmidhuber, J. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74695-9_23 · Cited by 263
Context Is Routinely Encoded During Emotion Perception
Barrett, L. F., & Kensinger, E. A. (2010) · Psychological Science · doi:10.1177/0956797610363547 · Cited by 242
The INTERSPEECH 2011 speaker state challenge
Schuller, B. W., Steidl, S., Batliner, A. M., Schiel, F., & Krajewski, J. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-801 · Cited by 233
Self-citationAn analysis of facial expression recognition under partial facial image occlusion
Kotsia, I., Buciu, I., & Pitas, I. (2007) · Image and Vision Computing · doi:10.1016/j.imavis.2007.11.004 · Cited by 223
Wöllmer, M., Metallinou, A., Eyben, F., Schuller, B., & Narayanan, S. S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-646 · Cited by 201 · cited 4×
“Apart from preliminary experiments using facial marker information as additional input modality [13] and a recent study on subject dependent recognition of arousal and valence [27], LSTM architectures have hardly been applied for audiovisual emotion recognition.”
BackgroundMethodResultInfluentialSelf-citationSchuller, B. W., Müller, R., Eyben, F., Gast, J., Hörnler, B., Wöllmer, M., Rigoll, G., Hoethker, A., & Konosu, H. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2009.02.013 · Cited by 194
Self-citationBuilding autonomous sensitive artificial listeners
Schroder, M., Bevacqua, E., Cowie, R., Eyben, F., Gunes, H., Heylen, D., ter Maat, M., McKeown, G., Pammi, S., Pantic, M., et al. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.34 · Cited by 185
Self-citationOnline Driver Distraction Detection Using Long Short-Term Memory
Wöllmer, M., Blaschke, C., Schindl, T., Schuller, B. W., Färber, B., Mayer, S., & Trefflich, B. (2011) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2011.2119483 · Cited by 181
Self-citationWöllmer, M., Schuller, B. W., Eyben, F., & Rigoll, G. (2010) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2010.2057200 · Cited by 165
Self-citationEyben, F., Wöllmer, M., Graves, A., Schuller, B. W., Douglas‐Cowie, E., & Cowie, R. (2009) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-009-0032-6 · Cited by 137
Self-citationMultiple Classifier Systems for the Classification of Audio-Visual Emotional States
Glodek, M., Tschechne, S., Layher, G., Schels, M., Brosch, T., Scherer, S. J., Kächele, M., Schmidt, M., Heiko Neumann, B., Palm, G., et al. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_47 · Cited by 135
Audio–Visual Affective Expression Recognition Through Multistream Fused HMM
Zeng, Z., Tu, J., Pianfetti, B., & Huang, T. S. (2008) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2008.921737 · Cited by 134
Miller, G. (2004) · Science · doi:10.1126/science.304.5667.34 · Cited by 130
Support Vector Regression for Automatic Recognition of Spontaneous Emotions in Speech
Grimm, M., Kroschel, K., & Narayanan, S. S. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.367262 · Cited by 114
On the Role of Emotion in Embodied Cognitive Architectures: From Organisms to Robots
Ziemke, T., & Lowe, R. J. (2009) · Cognitive Computation · doi:10.1007/s12559-009-9012-0 · Cited by 111
Audio-Visual Emotion Recognition Using Gaussian Mixture Models for Face and Voice
Metallinou, A., Lee, S., & Narayanan, S. S. (2008) · IEEE International Symposium on Multimedia (ISM) · doi:10.1109/ism.2008.40 · Cited by 97
Boosting encoded dynamic features for facial expression recognition
Yang, P., Liu, Q., & Metaxas, D. (2008) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.03.014 · Cited by 94
Modeling Latent Discriminative Dynamic of Multi-dimensional Affective Signals
Ramírez, G. A., Baltrušaitis, T., & Morency, L. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_51 · Cited by 91
Pose-Invariant Facial Expression Recognition Using Variable-Intensity Templates
Kumano, S., Otsuka, K., Yamato, J., Maeda, E., & Sato, Y. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-008-0185-x · Cited by 88
Wöllmer, M., Al-Hames, M., Eyben, F., Schuller, B. W., & Rigoll, G. (2009) · Neurocomputing · doi:10.1016/j.neucom.2009.08.005 · Cited by 87
Self-citationWöllmer, M., Eyben, F., Graves, A., Schuller, B. W., & Rigoll, G. (2010) · Cognitive Computation · doi:10.1007/s12559-010-9041-8 · Cited by 81
Self-citationAn Asynchronous Hidden Markov Model for Audio-Visual Speech Recognition
Samy Bengio, A. (2002) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 71
Wöllmer, M., Eyben, F., Keshet, J., Graves, A., Schuller, B. W., & Rigoll, G. (2009) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2009.4960492 · Cited by 69
Self-citationBrute-forcing hierarchical functionals for paralinguistics: A waste of feature space?
Schuller, B. W., Wimmer, M., Mösenlechner, L., Kern, C., Arsić, D., & Rigoll, G. (2008) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2008.4518656 · Cited by 60
Self-citationMeng, H., & Bianchi‐Berthouze, N. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_49 · Cited by 57
String-based audiovisual fusion of behavioural events for the assessment of dimensional affect
Eyben, F., Wöllmer, M., Valstar, M., Güneş, H., Schuller, B. W., & Pantić, M. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771417 · Cited by 54
Self-citationReal-life emotion-related states detection in call centers: a cross-corpora study
Devillers, L. Y., Vaudable, C., & Chastagnol, C. (2010) · Interspeech (USB) · doi:10.21437/interspeech.2010-643 · Cited by 52
A multi-stream ASR framework for BLSTM modeling of conversational speech
Wöllmer, M., Eyben, F., Schuller, B. W., & Rigoll, G. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947444 · Cited by 42
Self-citationThe hinterland of emotions: Facing the open-microphone challenge
Steidl, S., Batliner, A. M., Schuller, B. W., & Seppi, D. (2009) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2009.5349499 · Cited by 37
Self-citation2007 IEEE Workshop on Applications of Computer Vision (WACV '07)
(2007) · doi:10.1109/wacv.2007 · Cited by 32
Wöllmer, M., Metallinou, A., Katsamanis, N., Schuller, B. W., & Narayanan, S. S. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288834 · Cited by 30
Self-citationStupakov, A., Hanusa, E., Vijaywargi, D., Fox, D., & Bilmes, J. (2011) · Computer Speech & Language · doi:10.1016/j.csl.2010.12.003 · Cited by 29
A Psychologically-Inspired Match-Score Fusion Model for Video-Based Facial Expression Recognition
Cruz, A. C., Bhanu, B., & Yang, S. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_45 · Cited by 28
Continuous Emotion Recognition Using Gabor Energy Filters
Dahmane, M., & Meunier, J. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_46 · Cited by 27
Investigating the Use of Formant Based Features for Detection of Affective Dimensions in Speech
Kim, J. C., Rao, H., & Clements, M. A. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_48 · Cited by 26
Tandem decoding of children's speech for keyword detection in a child-robot interaction scenario
Wöllmer, M., Schuller, B., Batliner, A., Steidl, S., & Seppi, D. (2011) · ACM Transactions on Speech and Language Processing · doi:10.1145/1998384.1998386 · Cited by 18
Self-citationSayedelahl, A., Fewzee, P., Kamel, M. S., & Karray, F. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_52 · Cited by 17
The CASIA Audio Emotion Recognition Method for Audio/Visual Emotion Challenge 2011
Pan, S., Tao, J., & Li, Y. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_50 · Cited by 14
Wöllmer, M., Schuller, B. W., & Rigoll, G. (2011) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2011.6163902 · Cited by 14
Self-citationSpeech Emotion Recognition System Based on L1 Regularized Linear Regression and Decision Fusion
Cen, L., Yu, Z. L., & Dong, M. H. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_44 · Cited by 11
Video object detection speedup using staggered sampling
Greig, D. (2009) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2009.5403129 · Cited by 10
A hierarchical approach for visual suspicious behavior detection in aircrafts
Arsić, D., Hörnler, B., Schuller, B. W., & Rigoll, G. (2009) · IEEE International Conference on Digital Signal Processing (DSP) · doi:10.1109/icdsp.2009.5201258 · Cited by 6
Self-citation
Cited by
Sorted by citations
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Beyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. D. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299101 · Cited by 2,152
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Gandhi, A., Adhvaryu, K., Poria, S., Cambria, E., & Hussain, A. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.09.025 · Cited by 783
A Review and Meta-Analysis of Multimodal Affect Detection Systems
D’Mello, S. K., & Kory-Westlund, J. M. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 695
Metadata
- Authors
- Wöllmer, M., Kaiser, M., Eyben, F., Schuller, B., & Rigoll, G.
- Affiliations
- Technical University of Munich (DE)
- Corresponding author
- Wöllmer, Martin
- Venue
- Image and Vision Computing
- Publisher
- Elsevier BV
- Published
- 2013-02
- Volume
- 31
- Issue
- 2
- Pages
- 153-163
- ISSN
- 0262-8856 (ISSN-L)
- OpenAlex
- W2026243162
- Semantic Scholar
- 752c6738a1ef1c18943636bc3c98a4dbd6455b33
- Cited by
- 288
- References
- 81
- Citations by year
- 2026: 4 · 2025: 15 · 2024: 16 · 2023: 13 · 2022: 17 · 2021: 39 · 2020: 22 · 2019: 38 · 2018: 32 · 2017: 31 · 2016: 21 · 2015: 19 · 2014: 11 · 2013: 8
- FWCI
- 17.14
- Type
- journal article
- Language
- en
- Open access
- green
- mediatum.ub.tum.de
- Licence
- Locations
- Image and Vision Computing (publishedVersion)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion, other-oa)
- OPUS (Augsburg University) (submittedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- audio-visual emotion recognition, long short-term memory, arousal, valence, expectations, power, long-range context modeling, video features, human-computer interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
