Cited by 288Emotion and Mood RecognitionSpeech and Audio ProcessingSentiment Analysis and Opinion Mining

LSTM-Modeling of continuous emotions in an audiovisual affect recognition framework

Open access
Wöllmer, Martin, Kaiser, Moritz, Eyben, Florian, Schuller, Björn, Rigoll, Gerhard·2013·Image and Vision Computing

Abstract

Automatically recognizing human emotions from spontaneous and non-prototypical real-life data is currently one of the most challenging tasks in the field of affective computing. This article presents our recent advances in assessing dimensional representations of emotion, such as arousal, expectation, power, and valence, in an audiovisual human-computer interaction scenario. Building on previous studies which demonstrate that long-range context modeling tends to increase accuracies of emotion recognition, we propose a fully automatic audiovisual recognition approach based on Long Short-Term Memory (LSTM) modeling of word-level audio and video features. LSTM networks are able to incorporate knowledge about how emotions typically evolve over time so that the inferred emotion estimates are produced under consideration of an optimal amount of context. Extensive evaluations on the Audiovisual Sub-Challenge of the 2011 Audio/Visual Emotion Challenge show how acoustic, linguistic, and visual features contribute to the recognition of different affective dimensions as annotated in the SEMAINE database. We apply the same acoustic features as used in the challenge baseline system whereas visual features are computed via a novel facial movement feature extractor. Comparing our results with the recognition scores of all Audiovisual Sub-Challenge participants, we find that the proposed LSTM-based technique leads to the best average recognition performance that has been reported for this task so far.

Integrity signals

Registry

Crossref · 10.1016/j.imavis.2012.03.001

Cites

74 of 74 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Cites

Flagged first, then by citations

7 of the 81 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Wöllmer, M., Kaiser, M., Eyben, F., Schuller, B., & Rigoll, G.
Affiliations
Technical University of Munich (DE)
Corresponding author
Wöllmer, Martin
Venue
Image and Vision Computing
Publisher
Elsevier BV
Published
2013-02
Volume
31
Issue
2
Pages
153-163
ISSN
0262-8856 (ISSN-L)
OpenAlex
W2026243162
Cited by
288
References
81
Citations by year
2026: 4 · 2025: 15 · 2024: 16 · 2023: 13 · 2022: 17 · 2021: 39 · 2020: 22 · 2019: 38 · 2018: 32 · 2017: 31 · 2016: 21 · 2015: 19 · 2014: 11 · 2013: 8 · 2012: 2
FWCI
17.14
Type
journal article
Language
en
Open access
Locations
  • Image and Vision Computing (publishedVersion)
  • mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion, other-oa)
  • OPUS (Augsburg University) (submittedVersion, cc-by-nc-nd)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
audio-visual emotion recognition, long short-term memory, arousal, valence, expectations, power, long-range context modeling, video features, human-computer interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review