CoChat
Sign Up
Cited by 449Music and Audio ProcessingSpeech Recognition and SynthesisDomain Adaptation and Few-Shot Learning

CLAP Learning Audio Concepts from Natural Language Supervision

Open access
Elizalde, Benjamin, Deshmukh, Soham, Ismail, Mahmoud Al, Wang, Huaming·2023·ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
DOI 10.1109/icassp49357.2023.10095889
Open this paper in CoChat — free accountOpen PDF

Abstract

Mainstream machine listening models are trained to learn audio concepts under the paradigm of one class label to many recordings focusing on one task. Learning under such restricted supervision limits the flexibility of models because they require labeled audio for training and can only predict the predefined categories. Instead, we propose to learn audio concepts from natural language supervision. We call our approach Contrastive Language-Audio Pretraining (CLAP), which connects language and audio by using two encoders and a contrastive learning objective, bringing audio and text descriptions into a joint multimodal space. We trained CLAP with 128k audio and text pairs and evaluated it on 16 downstream tasks across 7 domains, such as classification of sound events, scenes, music, and speech. CLAP establishes state-of-the-art (SoTA) in Zero-Shot performance. Also, we evaluated CLAP’s audio encoder in a supervised learning setup and achieved SoTA in 5 tasks. The Zero-Shot capability removes the need of training with class labeled audio, enables flexible class prediction at inference time, and generalizes well in multiple downstream tasks. Code is available at: https://github.com/microsoft/CLAP.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/icassp49357.2023.10095889

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LTU-AS does not require any pre-defined label set.

    “Compared with CLIP-like models [48,47], LTU-AS does not require any pre-defined label set and directly outputs predictions in natural language”

    From Joint Audio and Speech Understanding (2023) · cites this paper as [48]

    Section 2.2 (Zero-Shot Linear Classification) says CLAP scores each audio against supplied class labels, supporting the comparison with a model that outputs predictions directly in natural language. But the Introduction explicitly says CLAP has no predefined categories: its labels are supplied for a particular prediction, not fixed in advance. The paper does not establish LTU-AS’s capabilities. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The accuracy of LTU-AS is nearly twice that of CLAP on the zero-shot GTZAN music genre classification task.

    “E.g., the accuracy of LTU-AS is nearly twice that of CLAP [48] on the zero-shot GTZAN music genre classification task”

    From Joint Audio and Speech Understanding (2023) · cites this paper as [48]

  • LTU-AS directly outputs predictions in natural language.

    “Compared with CLIP-like models [48,47], LTU-AS does not require any pre-defined label set and directly outputs predictions in natural language”

    From Joint Audio and Speech Understanding (2023) · cites this paper as [48]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 23

  1. Richard F Lyon, "Machine hearing: An emerging field [exploratory dsp]," IEEE signal processing magazine, vol. 27, pp. 131-139, 2010.
  2. Annamaria Mesaros, Aleksandr Diment, Benjamin Elizalde, Toni Heittola, Emmanuel Vincent, Bhiksha Raj, and Tuomas Virtanen, "Sound event detection in the dcase 2017 challenge," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 6, pp. 992-1006, 2019.
  3. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, "wav2vec 2.0: A framework for self-supervised learning of speech representations," Ad- vances in Neural Information Processing Systems, vol. 33, pp. 12449-12460, 2020.
  4. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al., "Learning transferable visual models from natural lan- guage supervision," in International Conference on Ma- chine Learning. PMLR, 2021, pp. 8748-8763.
  5. Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al., "Florence: A new foundation model for computer vision," arXiv preprint arXiv:2111.11432, 2021. · arXiv
  6. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig, "Scaling up visual and vision- language representation learning with noisy text super- vision," in International Conference on Machine Learn- ing. PMLR, 2021, pp. 4904-4916.
  7. Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello, "Wav2clip: Learning robust audio representations from clip," in ICASSP 2022 -2022 IEEE International Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), 2022, pp. 4563-4567.
  8. Andrey Guzhov, Federico Raue, Jörn Hees, and An- dreas Dengel, "Audioclip: Extending clip to image, text and audio," in ICASSP 2022 -2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 976-980.
  9. Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter, "Audio set: An ontology and human-labeled dataset for audio events," in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, pp. 776- 780.
  10. Joseph Turian, Jordie Shier, Humair Raj Khan, Bhik- sha Raj, Björn W Schuller, Christian J Steinmetz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk Mc- Nally, et al., "Hear 2021: Holistic evaluation of au- dio representations," arXiv preprint arXiv:2203.03022, 2022. · arXiv
  11. Qiuqiang Kong, Yong Xu, Wenwu Wang, and Mark D Plumbley, "Sound event detection of weakly labelled data with cnn-transformer and automatic threshold opti- mization," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2450-2460, 2020.
  12. Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, and Gerhard Widmer, "Efficient training of audio transform- ers with patchout," arXiv preprint arXiv:2110.05069, 2021. · arXiv
  13. Yoonchang Han, Jeongsoo Park, and Kyogu Lee, "Con- volutional neural networks with binaural representations and background subtraction for acoustic scene classi- fication," the Detection and Classification of Acoustic Scenes and Events (DCASE), pp. 1-5, 2017.
  14. Cristina Luna-Jiménez, Ricardo Kleinlein, David Griol, Zoraida Callejas, Juan M. Montero, and Fernando Fernández-Martínez, "A proposal for multimodal emo- tion recognition using aural transformers and action units on ravdess dataset," Applied Sciences, vol. 12, no. 1, 2022.
  15. Byeonggeun Kim, Simyung Chang, Jinkyu Lee, and Dooyong Sung, "Broadcasted residual learn- ing for efficient keyword spotting," arXiv preprint arXiv:2106.04140, 2021. · arXiv
  16. Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra, "Fsd50k: An open dataset of human-labeled sound events," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 829-852, 2022.
  17. Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen, "Clotho: an audio captioning dataset," in ICASSP 2020 -2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 736-740.
  18. Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim, "AudioCaps: Generating Captions for Audios in The Wild," in NAACL-HLT, 2019.
  19. Irene Martín-Morató and Annamaria Mesaros, "What is the ground truth? reliability of multi-annotator data for audio tagging," in 2021 29th European Signal Process- ing Conference (EUSIPCO). IEEE, 2021, pp. 76-80.
  20. Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley, "Panns: Large- scale pretrained audio neural networks for audio pat- tern recognition," IEEE/ACM Trans. Audio, Speech and Lang. Proc., vol. 28, pp. 2880-2894, jan 2020.
  21. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, "BERT: Pre-training of deep bidi- rectional transformers for language understanding," in Proceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, June 2019, pp. 4171-4186, Association for Computational Linguistics.
  22. Diederik P. Kingma and Jimmy Ba, "Adam: A method for stochastic optimization," in ICLR (Poster), 2015.
  23. Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lu- cas Beyer, "Lit: Zero-shot transfer with locked-image text tuning," arXiv preprint arXiv:2111.07991, 2021. · arXiv

Cited by

Sorted by citations

Authors
Elizalde, B., Deshmukh, S., Ismail, M. A., & Wang, H.
Affiliations
Microsoft (United States) (US)
ORCID
Venue
ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Publisher
IEEE
Published
2023-06-04
Pages
1-5
ISSN
1520-6149 (ISSN-L)
DOI
10.1109/icassp49357.2023.10095889
OpenAlex
W4372266552
Cited by
449
References
4
Citations by year
2026: 126 · 2025: 186 · 2024: 108 · 2023: 27 · 2022: 2
FWCI
92.83
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
natural language supervision, contrastive learning, sound event classification, scene classification, music classification, speech classification, audio encoder

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review