Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 37 of the 41 references it cites were found in a registry or an index; 4 of the 7 sources its claims cite were checked for support.

Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation

Open access
Wu, Yusong, Chen, Ke, Zhang, Tianyu, Hui, Yuchen, Berg-Kirkpatrick, Taylor, Dubnov, Shlomo · 2023 · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Abstract

Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zero-shot setting and is able to obtain performance comparable to models’ results in the non-zero-shot setting. LAION-Audio-630K1and the proposed model2are both available to the public.

Overview

CoChat Summary

## TL;DR

The paper proposes a contrastive language-audio pretraining pipeline built on the newly released LAION-Audio-630K dataset (633,526 audio-text pairs, 4,325.39 hours), plus feature fusion for variable-length audio and keyword-to-caption augmentation. It reports improved text-to-audio retrieval and state-of-the-art zero-shot audio classification, with the strongest gains from large-scale data, feature fusion, and keyword-to-caption augmentation.

## Key findings

  • LAION-Audio-630K contains 633,526 audio-text pairs and 4,325.39 hours of audio; the paper says it is larger than prior public audio-text datasets and increases training captions to 2.5 million when combined with other data and caption augmentation.
  • In text-to-audio retrieval on AudioCaps and Clotho, the best model is HTSAT-RoBERTa with feature fusion and keyword-to-caption augmentation, reported as outperforming prior methods; the paper highlights R@1 = 36.7% on AudioCaps and R@1 = 18.2% on Clotho.
  • On zero-shot audio classification, the paper says its models achieve new state-of-the-art on ESC-50, VGGSound, and UrbanSound8K.
  • The paper reports that feature fusion helps especially on longer-audio datasets and that keyword-to-caption augmentation improves performance over simple template prompting on most metrics.

## Methods

  • The study trains a contrastive language-audio encoder with separate audio and text encoders, projecting both into a shared 512-dimensional space and optimizing a CLIP-style contrastive loss.
  • Audio encoders tested: PANN and HTSAT; text encoders tested: CLIP transformer, BERT, and RoBERTa.
  • Data/settings: training uses AudioCaps + Clotho (~55K pairs), LAION-Audio-630K (~630K pairs), and AudioSet (1.9 million audio samples); audio longer than 10 seconds is handled with feature fusion, and labels/tags are expanded into captions via template prompting or a T5-based keyword-to-caption model.
  • Evaluation covers text-to-audio retrieval, zero-shot audio classification on ESC-50, VGGSound, UrbanSound8K, and supervised classification on FSD50K and VGGSound.

## Limitations

  • The paper explicitly notes a trade-off when adding more data from different sources: performance can improve on one dataset while dropping on another; it says adding more data can boost generalization but move the distribution away from AudioCaps-like audio.
  • The paper states that keyword-to-caption augmentation is excluded for samples shorter than 2 seconds because the audio is “merely a single event” and matches poorly with generated captions.
  • Evident from the text: several reported results are only partially visible in the provided tables, so some performance details are not recoverable from this excerpt.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/icassp49357.2023.10095969

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

1 Valid · 4 Not valid · 2 Not checked · 45 Own results and interpretations

19 more claims

Create a free account to see all 52 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 13
  • It may state two separate claims as one. 18
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 41 references

Research works printed but not linked: 1

  1. wu, 2022, Text-to-audio retrieval via large-scale contrastive learning, Proc DCASE

Cited by

See all 459

Metadata

Authors
Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., & Dubnov, S.
Affiliations
  • Mila - Quebec Artificial Intelligence Institute (CA)
  • University of California San Diego (US)
Venue
ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Publisher
IEEE
Published
2023-06-04
Pages
1-5
ISSN
1520-6149 (ISSN-L)
OpenAlex
W4372260310
Cited by
459
References
41
Citations by year
2026: 130 · 2025: 202 · 2024: 116 · 2023: 11
FWCI
95.08
Type
conference paper
Language
en
Open access
Locations
  • IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, cc-by-nc)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
contrastive learning, multimodal representation learning, feature fusion, audio classification, audio encoder, text encoder

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation. https://app.cochat.ai/papers/10.1109/icassp49357.2023.10095969
The paper
Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., & Dubnov, S. (2023). Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation. ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). https://doi.org/10.1109/icassp49357.2023.10095969

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.