The DOI matches its Crossref record; the references it cites are not checked yet.
Semi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching
Open accessAbstract
Automatic emotion recognition is an active research topic with wide range of applications. Due to the high manual annotation cost and inevitable label ambiguity, the development of emotion recognition dataset is limited in both scale and quality. Therefore, one of the key challenges is how to build effective models with limited data resource. Previous works have explored different approaches to tackle this challenge including data enhancement, transfer learning, and semi-supervised learning etc. However, the weakness of these existing approaches includes such as training instability, large performance loss during transfer, or marginal improvement. In this work, we propose a novel semi-supervised multi-modal emotion recognition model based on cross-modality distribution matching, which leverages abundant unlabeled data to enhance the model training under the assumption that the inner emotional status is consistent at the utterance level across modalities. We conduct extensive experiments to evaluate the proposed model on two benchmark datasets, IEMOCAP and MELD. The experiment results prove that the proposed semi-supervised learning model can effectively utilize unlabeled data and combine multi-modalities to boost the emotion recognition performance, which outperforms other state-of-the-art approaches under the same condition. The proposed model also achieves competitive capacity compared with existing approaches which take advantage of additional auxiliary information such as speaker and interaction context.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Liang et al. [94] postulate that despite the potential misalignment of emotional expressions at the frame level across different modalities, the overall emotional states should exhibit similarity at a coarse-grained discourse level. Leveraging this assumption, they train the encoder via semi-supervised learning, using unlabeled video data to secure improved encoder representations.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [94]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In the literature, most previous MERMC studies primarily focus on the text and audio modalities (Poria et al., 2019; Liang et al., 2020; Mao et al., 2021; Chen et al., 2021)”
From A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party Conversations (2023) · cites this paper as Liang et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“OpenSMILE toolkit (Eyben et al., 2010) with the configuration of 'IS13 ComParE' is used to extract frame-level features, which have similar performance with the IS10 utterance-level acoustic features used in (Liang et al., 2020).”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Liang et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
The citation lists are queued
They are fetched later and appear here then.
References 51 · Cited by 73
Research works printed but not linked: 12
- 10.1145/3240508.3240578 · DOI · Paper page
- 10.5555/3298023.3298146 · doi:10.5555/3298023.3298146
- Eyben Florian, 2010, ACM International Conference on Multimedia. 1459--1462
- Ringeval Fabien, 2018, Proceedings of the 8th International Workshop on Audio/Visual Emotion Challenge, AVEC'18
- Goodfellow Ian J., 2014, International Conference on Neural Information Processing Systems. 2672--2680
- Herve Jegou Matthijs Douze and Cordelia Schmid. 2010b. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence Vol. 33 1 (2010) 117--128. Herve Jegou Matthijs Douze and Cordelia Schmid. 2010b. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence Vol. 33 1 (2010) 117--128. · DOI · Paper page
- 10.18653/v1/P17-1081 · DOI · Paper page
- Rozgić Viktor, 2012, INTERSPEECH
- Salimans Tim, 2016, Improved Techniques for Training GANs. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016
- Yutaka Sasaki et al. 2007. The truth of the F-measure. Teach Tutor mater Vol. 1 5 (2007) 1--5. Yutaka Sasaki et al. 2007. The truth of the F-measure. Teach Tutor mater Vol. 1 5 (2007) 1--5.
- Srivastava Nitish, 2015, International conference on machine learning. 843--852
- Zadeh Amir, 2018, Thirty-Second AAAI Conference on Artificial Intelligence.
Metadata
- Authors
- Liang, J., Li, R., & Jin, Q.
- Affiliations
- Renmin University of China (CN)
- ORCID
- Venue
- Proceedings of the 28th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2020-10-12
- Pages
- 2852-2861
- OpenAlex
- W3083650259
- Cited by
- 73
- References
- 51
- Citations by year
- 2026: 9 · 2025: 13 · 2024: 15 · 2023: 15 · 2022: 13 · 2021: 8
- FWCI
- 8.78
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Beijing Natural Science Foundation (4192028)
- National Natural Science Foundation of China (61772535)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- emotion recognition, semi-supervised learning, multimodal fusion, unlabeled data, limited annotations, IEMOCAP, MELD
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Semi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching. https://app.cochat.ai/papers/10.1145/3394171.3413579
- The paper
- Liang, J., Li, R., & Jin, Q. (2020). Semi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching. Proceedings of the 28th ACM International Conference on Multimedia. https://doi.org/10.1145/3394171.3413579
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.