CoChat
Sign Up
Cited by 21Emotion and Mood RecognitionSentiment Analysis and Opinion MiningSpeech Recognition and Synthesis

Automatic recognition of self-reported and perceived emotion: does joint modeling help?

Zhang, Biqiao, Essl, Georg, Mower Provost, Emily·2016·Proceedings of the 18th ACM International Conference on Multimodal Interaction
DOI 10.1145/2993148.2993173
Open this paper in CoChat — free account

Abstract

Emotion labeling is a central component of automatic emotion recognition. Evaluators are asked to estimate the emotion label given a set of cues, produced either by themselves (self-report label) or others (perceived label). This process is complicated by the mismatch between the intentions of the producer and the interpretation of the perceiver. Traditionally, emotion recognition systems use only one of these types of labels when estimating the emotion content of data. In this paper, we explore the impact of jointly modeling both an individual's self-report and the perceived label of others. We use deep belief networks (DBN) to learn a representative feature space, and model the potentially complementary relationship between intention and perception using multi-task learning. We hypothesize that the use of DBN feature-learning and multi-task learning of self-report and perceived emotion labels will improve the performance of emotion recognition systems. We test this hypothesis on the IEMOCAP dataset, an audio-visual and motion-capture emotion corpus. We show that both DBN feature learning and multi-task learning offer complementary gains. The results demonstrate that the perceived emotion tasks see greatest performance gain for emotionally subtle utterances, while the self-report emotion tasks see greatest performance gain for emotionally clear utterances. Our results suggest that the combination of knowledge from the self-report and perceived emotion labels lead to more effective emotion recognition systems.

Integrity signals

Registry

Crossref · 10.1145/2993148.2993173

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Zhang et al. [148] and Ranganathan et al. [149] used fully-connected DBN models to learn from aggregated facial feature point trajectories, both improving recognition accuracies over shallow aggregation strategies.

    “For example, Zhang et al. [148] and Ranganathan et al. [149] used fully-connected DBN models to learn from aggregated facial feature point trajectories, both improving recognition accuracies over shallow aggregation strategies.”

    From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [148]

  • Kim et al. and others demonstrated that joint feature representations can be learned with DBNs.

    “Kim et al. [159] and others (e.g., [149], [148]) demonstrated how this can be achieved with DBNs.”

    From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [148]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Zhang, B., Essl, G., & Mower Provost, E.
Affiliations
  • University of Michigan (US)
  • University of Wisconsin–Milwaukee (US)
  • University of Michigan, USA
ORCID
Venue
Proceedings of the 18th ACM International Conference on Multimodal Interaction
Publisher
ACM
Published
2016-10-31
Pages
217-224
DOI
10.1145/2993148.2993173
OpenAlex
W2548171908
Cited by
21
References
39
Citations by year
2026: 2 · 2025: 2 · 2024: 2 · 2023: 1 · 2021: 1 · 2020: 1 · 2019: 2 · 2018: 4 · 2017: 5 · 2016: 1
FWCI
4.63
Type
conference paper
Language
en
Open access
closed
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
automatic emotion recognition, self-reported emotion, perceived emotion, joint modeling, multi-task learning, deep belief network, IEMOCAP, representation learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review