CoChat
Sign Up
Cited by 35Emotion and Mood RecognitionSpeech Recognition and SynthesisDomain Adaptation and Few-Shot Learning

PEFT-SER: On the Use of Parameter Efficient Transfer Learning Approaches For Speech Emotion Recognition Using Pre-trained Speech Models

Open access
Feng, Tiantian, Narayanan, Shrikanth·2023·2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII)
DOI 10.1109/acii59096.2023.10388152
Open this paper in CoChat — free accountOpen PDF

Abstract

Many recent studies have focused on fine-tuning pretrained models for speech emotion recognition (SER), resulting in promising performance compared to traditional methods that rely largely on low-level, knowledge-inspired acoustic features. These pre-trained speech models learn general-purpose speech representations using self-supervised or weakly-supervised learning objectives from large-scale datasets. Despite the significant advances made in SER through the use of pre-trained architecture, fine-tuning these large pre-trained models for different datasets requires saving copies of entire weight parameters, rendering them impractical to deploy in real-world settings. As an alternative, this work explores parameter-efficient fine-tuning (PEFT) approaches for adapting pre-trained speech models for emotion recognition. Specifically, we evaluate the efficacy of adapter tuning, embedding prompt tuning, and LoRa (Low-rank approximation) on four popular SER testbeds. Our results reveal that LoRa achieves the best fine-tuning performance in emotion recognition while enhancing fairness and requiring only a minimal extra amount of weight parameters. Furthermore, our findings offer novel insights into future research directions in SER, distinct from existing approaches focusing on directly fine-tuning the model architecture. Our code is publicly available under: https://github.com/usc-sail/peft-ser.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/acii59096.2023.10388152

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The Whisper encoder encodes rich audio events and speech paralinguistic information, but it encodes information in the representations of different layers.

    “although the Whisper encoder encodes rich audio events and speech paralinguistic information [3,15,16], it encodes information in the representations of different layers.”

    From Joint Audio and Speech Understanding (2023) · cites this paper as [15]

    The Downstream Model Performance section reports that Whisper models perform well on speech emotion recognition, supporting the claim that their encoders capture speech paralinguistic information. The Background section lists voice activity detection among Whisper’s training objectives, but the paper does not establish that its encoder captures a rich range of audio events. The claim about information in different layers follows the citation and is not attributed to this source. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The Whisper encoder features not only encode linguistic information, but also encode rich general background sound information and paralinguistic and other information such as emotion and language development.

    “it was found that the Whisper encoder features not only encode linguistic information, but also encode rich general background sound information [3] and paralinguistic and other information (e.g., emotion [15] and language development [16]).”

    From Joint Audio and Speech Understanding (2023) · cites this paper as [15]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

2 of the 38 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Feng, T., & Narayanan, S.
Affiliations
University of Southern California (US)
ORCID
Narayanan, Shrikanth
Venue
2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII)
Publisher
IEEE
Published
2023-09-10
Pages
1-8
ISSN
2156-8103 (ISSN-L)
DOI
10.1109/acii59096.2023.10388152
OpenAlex
W4390874621
Cited by
34
References
38
Citations by year
2026: 6 · 2025: 15 · 2024: 10 · 2023: 3
FWCI
12.54
Type
conference paper
Language
en
Open access
Licence
Locations
  • International Conference on Affective Computing and Intelligent Interaction and workshops (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
speech emotion recognition, parameter-efficient fine-tuning, pre-trained speech models, LoRa, adapter tuning, prompt tuning, low-rank approximation, speech representation learning, model compression, fairness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review