CoChat

MusCaps: Generating Captions for Music Audio

Open access
Manco, Ilaria, Benetos, Emmanouil, Quinton, Elio, Fazekas, Gyorgy · 2021 · 2021 International Joint Conference on Neural Networks (IJCNN)

The DOI matches its Crossref record; 53 of the 57 references it cites were found in a registry or an index; 3 of the 11 sources its claims cite were checked for support.

Abstract

Content-based music information retrieval has seen rapid progress with the adoption of deep learning. Current approaches to high-level music description typically make use of classification models, such as in auto-tagging or genre and mood classification. In this work, we propose to address music description via audio captioning, defined as the task of generating a natural language description of music audio content in a human-like manner. To this end, we present the first music audio captioning model, MusCaps, consisting of an encoder-decoder with temporal attention. Our method combines convolutional and recurrent neural network architectures to jointly process audio-text inputs through a multimodal encoder and leverages pretraining on audio data to obtain representations that effectively capture and summarise musical features in the input. Evaluation of the generated captions through automatic metrics shows that our method outperforms a baseline designed for non-music audio captioning. Through an ablation study, we unveil that this performance boost can be mainly attributed to pre-training of the audio encoder, while other design choices - modality fusion, decoding strategy and the use of attention - contribute only marginally. Our model represents a shift away from classification-based music description and combines tasks requiring both auditory and linguistic understanding to bridge the semantic gap in music information retrieval11Code available at https://github.com/ilaria-manco/muscaps.

Overview

CoChat Summary

## TL;DR

MusCaps is presented as the first music audio captioning model, using a multimodal CNN-LSTM encoder-decoder with optional temporal attention and pre-trained music tagging features. On a private dataset of 6,035 audio-caption pairs, it outperforms a DCASE 2020 audio-captioning baseline, with the strongest gains coming from pre-training the audio feature extractor rather than from fusion or attention design choices.

## Key findings

  • The cleaned dataset contains 6,035 audio-caption pairs, split 60/20/20 into train/validation/test.
  • Compared with the DCASE baseline, MusCaps-Musicnn reaches BLEU-1 34.3, BLEU-4 5.4, METEOR 29.3, CIDEr 33.0, SPIDEr 28.6; MusCaps-HCNN reaches BLEU-1 37.3, BLEU-4 5.9, METEOR 29.6, CIDEr 36.9, SPIDEr 30.2.
  • In the ablation study, removing pre-training lowers performance; for example, MusCaps Early with pre-training has CIDEr 37.6 versus 29.2 without pre-training, and the paper says this also causes slower convergence.
  • Retrieval results are modest but improve with MusCaps variants: Recall@1 rises from 1.2 for the baseline to 5.7 for MusCaps Late; Median Rank improves from 41.8 to 37.8.

## Methods

  • The model is an encoder-decoder architecture for music captioning with a text embedding module, audio feature extractor, multimodal encoder, attention mechanism, and LSTM decoder.
  • Audio features come from two pre-trained music auto-tagging CNNs: Musicnn and Harmonic CNN, both trained on MagnaTagATune; the classification heads are removed and the feature extractors are frozen in the main comparison.
  • Experiments use a private production music dataset of audio-caption pairs, with captions tokenized and special tokens added; training uses teacher forcing, cross-entropy loss, Adam, up to 200 epochs, batch size 16, and beam search is evaluated as an alternative decoder.

## Limitations

  • The paper states that only one caption per example is available, which makes automatic caption metrics less reliable; it says additional data collection or augmentation may be needed to obtain several reference captions per example.
  • The retrieval evaluation may be ill-defined because multiple textual descriptions can be acceptable for one audio clip and vice versa, while the dataset and evaluation do not account for this.
  • Evident from the text: the dataset is private and there is no benchmark dataset, so comparability to prior music-captioning work is limited.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/ijcnn52387.2021.9533461

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 Valid · 1 Not valid · 8 Not checked · 46 Own results and interpretations

20 more claims

Create a free account to see all 57 claims and to verify the 3 references still open.

See all claims — free account
  • Probably not a substantial claim. 16
  • It may state two separate claims as one. 20

Create a free account to see these claims.

Citations

Cites

Show all 53 references

Research works printed but not linked: 1

  1. tian, 0, Music autotagging as captioning, Proceedings of the 1st Workshop on NLP for Music and Audio (NLP4MusA)

Cited by

See all 36

Metadata

Authors
Manco, I., Benetos, E., Quinton, E., & Fazekas, G.
Affiliations
Queen Mary University of London (GB)
Venue
2021 International Joint Conference on Neural Networks (IJCNN)
Publisher
IEEE
Published
2021-07-18
Pages
1-8
ISSN
2161-4407 (ISSN-L)
OpenAlex
W3159642309
Cited by
36
References
57
Citations by year
2026: 4 · 2025: 9 · 2024: 7 · 2023: 12 · 2022: 4
FWCI
3.51
Type
conference paper
Language
en
Open access
Locations
  • International Joint Conference on Neural Networks (IJCNN) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • UK Research and Innovation (EP/S022694/1)
  • Queen Mary University of London
Topics
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
  • Social Sciences › Arts and Humanities › Music › Music Technology and Sound Studies
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
music description, multimodal encoder, semantic gap, music information retrieval, convolutional recurrent neural network, modality fusion

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: MusCaps: Generating Captions for Music Audio. https://app.cochat.ai/papers/10.1109/ijcnn52387.2021.9533461
The paper
Manco, I., Benetos, E., Quinton, E., & Fazekas, G. (2021). MusCaps: Generating Captions for Music Audio. 2021 International Joint Conference on Neural Networks (IJCNN). https://doi.org/10.1109/ijcnn52387.2021.9533461

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.