CoChat

Multi-modal Continuous Dimensional Emotion Recognition Using Recurrent Neural Network and Self-Attention Mechanism

Sun, Licai, Lian, Zheng, Tao, Jianhua, Liu, Bin, Niu, Mingyue · 2020 · Proceedings of the 1st International on Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop

The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.

Abstract

Automatic perception and understanding of human emotion or sentiment has a wide range of applications and has attracted increasing attention nowadays. The Multimodal Sentiment Analysis in Real-life Media (MuSe) 2020 provides a testing bed for recognizing human emotion or sentiment from multiple modalities (audio, video, and text) in the wild scenario. In this paper, we present our solutions to the MuSe-Wild sub-challenge of MuSe 2020. The goal of this sub-challenge is to perform continuous emotion (arousal and valence) predictions on a car review database, Muse-CaR. To this end, we first extract both handcrafted features and deep representations from multiple modalities. Then, we utilize the Long Short-Term Memory (LSTM) recurrent neural network as well as the self-attention mechanism to model the complex temporal dependencies in the sequence. The Concordance Correlation Coefficient (CCC) loss is employed to guide the model to learn local variations and the global trend of emotion simultaneously. Finally, two fusion strategies, early fusion and late fusion, are adopted to further boost the model's performance by exploiting complementary information from different modalities. Our proposed method achieves CCC of 0.4726 and 0.5996 for arousal and valence respectively on the test set, which outperforms the baseline system with corresponding CCC of 0.2834 and 0.2431.

Overview

Integrity signals

Registry

Crossref · 10.1145/3423327.3423672

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 45 references

Research works printed but not linked: 4

  1. Jimmy Lei Ba , Jamie Ryan Kiros, and Geoffrey E Hinton . 2016 . Layer normalization. arXiv preprint arXiv:1607.06450 (2016). Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016). · arXiv
  2. Diederik P Kingma and Jimmy Ba . 2014 . Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). · arXiv
  3. Yann LeCun , Yoshua Bengio , and Geoffrey Hinton . 2015. Deep learning. nature , Vol. 521 , 7553 ( 2015 ), 436--444. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature , Vol. 521, 7553 (2015), 436--444.
  4. Qi Liu , Matt J Kusner , and Phil Blunsom . 2020. A Survey on Contextual Embeddings. arXiv preprint arXiv:2003.07278 ( 2020 ). Qi Liu, Matt J Kusner, and Phil Blunsom. 2020. A Survey on Contextual Embeddings. arXiv preprint arXiv:2003.07278 (2020). · arXiv

Cited by

See all 81

Metadata

Authors
Sun, L., Lian, Z., Tao, J., Liu, B., & Niu, M.
Affiliations
Chinese Academy of Sciences (CN)
Venue
Proceedings of the 1st International on Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop
Publisher
ACM
Published
2020-10-15
Pages
27-34
OpenAlex
W3097075955
Cited by
81
References
44
Citations by year
2026: 8 · 2025: 9 · 2024: 16 · 2023: 20 · 2022: 10 · 2021: 18
FWCI
11.68
Type
conference paper
Language
en
Open access
closed
Funders
  • Key Program of the Natural Science Foundation of Tianjin (No. 18JCZDJC36300)
  • National Key Research & Development Plan of China (No.2017YFB1002804)
  • National Natural Science Foundation of China (No.61831022, No.61771472, No.61773379, No.61901473)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
continuous emotion recognition, multimodal sentiment analysis, LSTM, self-attention, feature-level fusion, decision fusion, temporal dependencies

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Multi-modal Continuous Dimensional Emotion Recognition Using Recurrent Neural Network and Self-Attention Mechanism. https://app.cochat.ai/papers/10.1145/3423327.3423672
The paper
Sun, L., Lian, Z., Tao, J., Liu, B., & Niu, M. (2020). Multi-modal Continuous Dimensional Emotion Recognition Using Recurrent Neural Network and Self-Attention Mechanism. Proceedings of the 1st International on Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop. https://doi.org/10.1145/3423327.3423672

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.