The DOI matches its Crossref record; 37 of the 41 references it cites were found in a registry or an index; 4 of the 7 sources its claims cite were checked for support.
Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
Open accessAbstract
Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zero-shot setting and is able to obtain performance comparable to models’ results in the non-zero-shot setting. LAION-Audio-630K1and the proposed model2are both available to the public.
Overview
CoChat Summary
## TL;DR
The paper proposes a contrastive language-audio pretraining pipeline built on the newly released LAION-Audio-630K dataset (633,526 audio-text pairs, 4,325.39 hours), plus feature fusion for variable-length audio and keyword-to-caption augmentation. It reports improved text-to-audio retrieval and state-of-the-art zero-shot audio classification, with the strongest gains from large-scale data, feature fusion, and keyword-to-caption augmentation.
## Key findings
- LAION-Audio-630K contains 633,526 audio-text pairs and 4,325.39 hours of audio; the paper says it is larger than prior public audio-text datasets and increases training captions to 2.5 million when combined with other data and caption augmentation.
- In text-to-audio retrieval on AudioCaps and Clotho, the best model is HTSAT-RoBERTa with feature fusion and keyword-to-caption augmentation, reported as outperforming prior methods; the paper highlights R@1 = 36.7% on AudioCaps and R@1 = 18.2% on Clotho.
- On zero-shot audio classification, the paper says its models achieve new state-of-the-art on ESC-50, VGGSound, and UrbanSound8K.
- The paper reports that feature fusion helps especially on longer-audio datasets and that keyword-to-caption augmentation improves performance over simple template prompting on most metrics.
## Methods
- The study trains a contrastive language-audio encoder with separate audio and text encoders, projecting both into a shared 512-dimensional space and optimizing a CLIP-style contrastive loss.
- Audio encoders tested: PANN and HTSAT; text encoders tested: CLIP transformer, BERT, and RoBERTa.
- Data/settings: training uses AudioCaps + Clotho (~55K pairs), LAION-Audio-630K (~630K pairs), and AudioSet (1.9 million audio samples); audio longer than 10 seconds is handled with feature fusion, and labels/tags are expanded into captions via template prompting or a T5-based keyword-to-caption model.
- Evaluation covers text-to-audio retrieval, zero-shot audio classification on ESC-50, VGGSound, UrbanSound8K, and supervised classification on FSD50K and VGGSound.
## Limitations
- The paper explicitly notes a trade-off when adding more data from different sources: performance can improve on one dataset while dropping on another; it says adding more data can boost generalization but move the distribution away from AudioCaps-like audio.
- The paper states that keyword-to-caption augmentation is excluded for samples shorter than 2 seconds because the audio is “merely a single event” and matches poorly with generated captions.
- Evident from the text: several reported results are only partially visible in the provided tables, so some performance details are not recoverable from this excerpt.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 4 Not valid · 2 Not checked · 45 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“with the total duration of 4,325.39 hours”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we first release LAION-Audio-630K , a large collection of 633,526 audio-text pairs from different data sources.”
- Also stated in: § 2.1. LAION-Audio-630K · plain
“We collect LAION-Audio-630K, a large-scale audio-text dataset consisting of 633,526 pairs”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Also stated in: § 2.1. LAION-Audio-630K · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and is able to obtain performance comparable to models' results in the non-zero-shot setting.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“And the keyword-to-caption augmentation does bring in better performance than the simple template text prompting method on most metrics.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Combining all the datasets, we increase the total number of audio samples with text caption to 2.5 million.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
19 more claims
Create a free account to see all 52 claims and every source verdict.
- Probably not a substantial claim. 13
- It may state two separate claims as one. 18
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Wu et al. [22] introduce keyword-to-caption augmentation (K2C Aug) to generate captions based on the ground truth tags of audio clips in AudioSet.They used a pre-trained T5 model without any instruction.”
From LP-MusicCaps: LLM-Based Pseudo Music Captioning (2023) · cites this paper as [22]
Experimental — accuracy not yet measuredIn the citing paper
“Re-AudioLDM [116] retrieves relevant caption-audio pairs with dense retriever CLAP [26] for generation.”
From Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024) · cites this paper as [26]
In the citing paper
“RECAP [323] and [43] leverages dense retrievers, CLAP [26] and VGGish [69] respectively, to retrieve related captions given audio data.”
From Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 2.1. LAION-Audio-630K
“We collect LAION-Audio-630K, a large-scale audio-text dataset consisting of 633,526 pairs with the total duration of 4,325.39 hours.”
§ D. DETAILS OF LAION-AUDIO-630K
“| BBC sound effects | 15973 | 463.48hrs | 1 caption per audio, audio | | Free To Use Sounds | 6370 | 175.73hrs | Filename as caption, audio | | Sonniss Game effects | 5049 | 84.6hrs | Filename as caption, audio | | We Sound Effects | 488 | 12.00hrs | Filename as caption, audio |”
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 2×
“During training, we use the Adam [23] optimizer with β 1 = 0 .”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“In audio event classification task, for instance, some text descriptions of an event can be mapped to the corresponding audio.”
BackgroundGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×
“PANN is a CNN-based [19] audio classification model with 7 downsampling CNN blocks and 7 upsampling blocks.”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 1×
“HTSAT is a transformer-based model with 4 groups of swin-transformer blocks [20], which achieves SOTAs on three audio classification datasets.”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · cited 2×
“3) Audioset [14] consists of 1.9 million audio samples with only labels available for each sample.”
BackgroundMethodInfluential
Research works printed but not linked: 1
- wu, 2022, Text-to-audio retrieval via large-scale contrastive learning, Proc DCASE
Cited by
Retrieval-Augmented Generation for AI-Generated Content: A Survey
Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., Jiang, J., & Cui, B. (2024) · arXiv · doi:10.48550/arxiv.2402.19473 · Cited by 240
“In the present era, efficient information retrieval systems can handle document collections on the order of billions [21], [22].”
AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining
Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q., Tian, Q., Wang, Y., Wang, W., Wang, Y., & Plumbley, M. D. (2024) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2024.3399607 · Cited by 180
Mei, X., Meng, C., Liu, H., Kong, Q., Ko, T., Zhao, C., Plumbley, M. D., Zou, Y., & Wang, W. (2024) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2024.3419446 · Cited by 150
“Unlike traditional large language models such as T5 [54] used in LAION-Audio-630K [53], ChatGPT has been shown to excel at generating human-like responses to natural language prompts, and has garnered widespread attention for its powerful understanding, reasoning, and dialogue abilities.”
BackgroundMethodResultInfluentialA survey on knowledge-oriented retrieval-augmented generation
Cheng, M., Luo, Y., Ouyang, J., Liu, Q., Liu, H., Li, L., Yu, S., Zhang, B., Cao, J., Ma, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.10677 · Cited by 140
“…4096 text-embedding[178] 768 Specific Domain Encoders ru-en-RoSBERTa[219] 1024 USTORY[263] 768 E5-Base-4k[289] 768 Image Vision Transformer (ViT)[55] 768 (Base) CLIP[196] 512 Audio Wav2Vec 2.0[16] 768 (Base) CLAP[247] 512 Video ViViT[9] 768 (Base) space for unified processing and retrieval.”
InfluentialFrom Large Language Models to Large Multimodal Models: A Literature Review
Huang, D., Yan, C., Li, Q., & Peng, X. (2024) · Applied Sciences · doi:10.3390/app14125068 · Cited by 152
Metadata
- Authors
- Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., & Dubnov, S.
- Affiliations
- Mila - Quebec Artificial Intelligence Institute (CA)
- University of California San Diego (US)
- Venue
- ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2023-06-04
- Pages
- 1-5
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W4372260310
- Cited by
- 459
- References
- 41
- Citations by year
- 2026: 130 · 2025: 202 · 2024: 116 · 2023: 11
- FWCI
- 95.08
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, cc-by-nc)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- contrastive learning, multimodal representation learning, feature fusion, audio classification, audio encoder, text encoder
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation. https://app.cochat.ai/papers/10.1109/icassp49357.2023.10095969
- The paper
- Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., & Dubnov, S. (2023). Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation. ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). https://doi.org/10.1109/icassp49357.2023.10095969
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.