CoChat
Sign Up
Cited by 49Emotion and Mood RecognitionSentiment Analysis and Opinion MiningSpeech Recognition and Synthesis

The DOI matches its Crossref record; 31 of the 33 references it cites were found in a registry or an index.

Multimodal and Multi-view Models for Emotion Recognition

Open access
Aguilar, Gustavo, Rozgic, Viktor, Wang, Weiran, Wang, Chao·2019·Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
DOI 10.18653/v1/p19-1095
Open this paper in CoChat — free accountOpen PDF

Abstract

Studies on emotion recognition (ER) show that combining lexical and acoustic information results in more robust and accurate models.The majority of the studies focus on settings where both modalities are available in training and evaluation.However, in practice, this is not always the case; getting ASR output may represent a bottleneck in a deployment pipeline due to computational complexity or privacyrelated constraints.To address this challenge, we study the problem of efficiently combining acoustic and lexical modalities during training while still providing a deployable acoustic model that does not require lexical inputs.We first experiment with multimodal models and two attention mechanisms to assess the extent of the benefits that lexical information can provide.Then, we frame the task as a multi-view learning problem to induce semantic information from a multimodal model into our acoustic-only network using a contrastive loss function.Our multimodal model outperforms the previous state of the art on the USC-IEMOCAP dataset reported on lexical and acoustic information.Additionally, our multi-view-trained acoustic network significantly surpasses models that have been exclusively trained with acoustic features.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/p19-1095

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 The multimodal model. The shadowed box incloses the acoustic word mechanism, whose output is fed into the GMU unit along with the lexical word representation at each timestep. The model can have N layers of BLSTM at the frame and word levels.
Figure 1 The multimodal model. The shadowed box incloses the acoustic word mechanism, whose output is fed into the GMU unit along with the lexical word representation at each timestep. The model can have N layers of BLSTM at the frame and word levels.CC-BY · From the PDF
Figure 2 The multi-view models. The view on the left is the acoustic model, and the view on the right is the multimodal model. The shadowed box in the middle is the contrastive loss module.
Figure 2 The multi-view models. The view on the left is the acoustic model, and the view on the right is the multimodal model. The shadowed box in the middle is the contrastive loss module.CC-BY · From the PDF
Figure 3 Multimodal Attention. The figure shows the attention mechanisms at the modality and utterance levels. The bars over the words are the average of z in Eq. 1, and they show how much acoustic (left bar in blue) or lexical (right bar in red) information was used. The highlights in the background of the words are the attention probabilities, where the higher the probability the darker the word.
Figure 3 Multimodal Attention. The figure shows the attention mechanisms at the modality and utterance levels. The bars over the words are the average of z in Eq. 1, and they show how much acoustic (left bar in blue) or lexical (right bar in red) information was used. The highlights in the background of the words are the attention probabilities, where the higher the probability the darker the word.CC-BY · From the PDF
Figure 4 Histogram of frame sequences for every utterance.
Figure 4 Histogram of frame sequences for every utterance.CC-BY · From the PDF
Image 5
CC-BY · From the PDF
Figure 6 Histogram of word lengths in terms of frames.
Figure 6 Histogram of word lengths in terms of frames.CC-BY · From the PDF
Image 7
CC-BY · From the PDF
Image 8
CC-BY · From the PDF
Figure 9 Confusion matrix of the acoustic model BACO-1.
Figure 9 Confusion matrix of the acoustic model BACO-1.CC-BY · From the PDF
Figure 10 Confusion matrix of the acoustic model BACO-1 trained in a multi-view learning setting.
Figure 10 Confusion matrix of the acoustic model BACO-1 trained in a multi-view learning setting.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 33 references

Research works printed but not linked: 2

  1. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
  2. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Ten- sor fusion network for multimodal sentiment analy- sis. CoRR, abs/1707.07250. · arXiv

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Aguilar, G., Rozgic, V., Wang, W., & Wang, C.
Affiliations
University of Houston (US)
ORCID
Venue
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Publisher
Association for Computational Linguistics
Published
2019
Pages
991-1002
ISSN
0736-587X (ISSN-L)
DOI
10.18653/v1/p19-1095
OpenAlex
W2952307697
Cited by
45
References
33
Citations by year
2026: 5 · 2025: 3 · 2024: 6 · 2023: 4 · 2022: 10 · 2021: 7 · 2020: 9 · 2019: 1
FWCI
8.51
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
emotion recognition, multimodal fusion, lexical data, acoustic features, multi-view learning, contrastive loss, attention mechanism

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review