MusCaps: Generating Captions for Music Audio
Open accessThe DOI matches its Crossref record; 53 of the 57 references it cites were found in a registry or an index; 3 of the 11 sources its claims cite were checked for support.
Abstract
Content-based music information retrieval has seen rapid progress with the adoption of deep learning. Current approaches to high-level music description typically make use of classification models, such as in auto-tagging or genre and mood classification. In this work, we propose to address music description via audio captioning, defined as the task of generating a natural language description of music audio content in a human-like manner. To this end, we present the first music audio captioning model, MusCaps, consisting of an encoder-decoder with temporal attention. Our method combines convolutional and recurrent neural network architectures to jointly process audio-text inputs through a multimodal encoder and leverages pretraining on audio data to obtain representations that effectively capture and summarise musical features in the input. Evaluation of the generated captions through automatic metrics shows that our method outperforms a baseline designed for non-music audio captioning. Through an ablation study, we unveil that this performance boost can be mainly attributed to pre-training of the audio encoder, while other design choices - modality fusion, decoding strategy and the use of attention - contribute only marginally. Our model represents a shift away from classification-based music description and combines tasks requiring both auditory and linguistic understanding to bridge the semantic gap in music information retrieval11Code available at https://github.com/ilaria-manco/muscaps.
Overview
CoChat Summary
## TL;DR
MusCaps is presented as the first music audio captioning model, using a multimodal CNN-LSTM encoder-decoder with optional temporal attention and pre-trained music tagging features. On a private dataset of 6,035 audio-caption pairs, it outperforms a DCASE 2020 audio-captioning baseline, with the strongest gains coming from pre-training the audio feature extractor rather than from fusion or attention design choices.
## Key findings
- The cleaned dataset contains 6,035 audio-caption pairs, split 60/20/20 into train/validation/test.
- Compared with the DCASE baseline, MusCaps-Musicnn reaches BLEU-1 34.3, BLEU-4 5.4, METEOR 29.3, CIDEr 33.0, SPIDEr 28.6; MusCaps-HCNN reaches BLEU-1 37.3, BLEU-4 5.9, METEOR 29.6, CIDEr 36.9, SPIDEr 30.2.
- In the ablation study, removing pre-training lowers performance; for example, MusCaps Early with pre-training has CIDEr 37.6 versus 29.2 without pre-training, and the paper says this also causes slower convergence.
- Retrieval results are modest but improve with MusCaps variants: Recall@1 rises from 1.2 for the baseline to 5.7 for MusCaps Late; Median Rank improves from 41.8 to 37.8.
## Methods
- The model is an encoder-decoder architecture for music captioning with a text embedding module, audio feature extractor, multimodal encoder, attention mechanism, and LSTM decoder.
- Audio features come from two pre-trained music auto-tagging CNNs: Musicnn and Harmonic CNN, both trained on MagnaTagATune; the classification heads are removed and the feature extractors are frozen in the main comparison.
- Experiments use a private production music dataset of audio-caption pairs, with captions tokenized and special tokens added; training uses teacher forcing, cross-entropy loss, Adam, up to 200 epochs, batch size 16, and beam search is evaluated as an alternative decoder.
## Limitations
- The paper states that only one caption per example is available, which makes automatic caption metrics less reliable; it says additional data collection or augmentation may be needed to obtain several reference captions per example.
- The retrieval evaluation may be ill-defined because multiple textual descriptions can be acceptable for one audio clip and vice versa, while the dataset and evaluation do not account for this.
- Evident from the text: the dataset is private and there is no benchmark dataset, so comparability to prior music-captioning work is limited.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 1 Not valid · 8 Not checked · 46 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“This is consistent with observations from previous studies [41], which attribute the performance degradation to overfitting or discrepancy between optimisation of the metrics and of the learning objective.”
What the sources say
- Supportedcites [41] [41] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, 'Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 39, no. 4, pp. 652-663, 4 2017. source
Section 5.2.5 (p. 9) reports that a small beam size of 3 performed best and interprets this as an indication of either overfitting or a mismatch between the training objective (likelihood) and human judgement. The section also says beam sizes were selected using CIDEr, supporting the claim’s description of a metric–objective discrepancy. (LLM, conf=0.93)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, the method proposed in [24] fails to generate grammatically correct sentences”
What the sources say
- Supportedcites [24] [24] K. Choi, G. Fazekas, M. Sandler, B. Mcfee, and K. Cho, 'Towards Music Captioning: Generating Music Playlist Descriptions,' arXiv preprint , 2016. - source
Section 4 'Experiments and Conclusions' (p. 2) states outright that 'The model currently overfits and fails to generate correct sentences', and illustrates this with a generated word sequence ('dramatic motivating the intense epic action adventure soaring soaring soaring gloriously Roger Deakins cinematography Maryse Alberti'), which is ungrammatical — so 'grammatically correct' is a fair paraphrase of 'correct'. The tested model is the pre-training sequence-to-sequence approach of Section 3.1 (p. 2) trained on a private 374-album dataset, and the paper attributes the failure to overfitting from limited data rather than to a flaw in the architecture; the citing sentence drops that causal qualifier but does not misstate the paper's point. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, we do not observe a significant difference across these three variants.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we also train MusCaps with pre-training using shorter, randomly selected chunks as the audio input, to ensure that this does not significantly affect performance, and find that only a small decrease in some of the scores is observed in this case.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Through an ablation study, we unveil that this performance boost can be mainly attributed to pre-training of the audio encoder, while other design choices - modality fusion, decoding strategy and the use of attention -contribute only marginally.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 57 claims and to verify the 3 references still open.
- Probably not a substantial claim. 16
- It may state two separate claims as one. 20
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“More recently, the success of BERT [11] has motivated research into its adaptation to V&L tasks, allowing to achieve state-of-the-art (SOTA) performance in all of them, both for still image and video [12], [13].”
BackgroundGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“This is initialised with 300-dimensional GloVe word embeddings [29], pre-”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 1×
“com/audio-captioning/dcase-2020-baseline/ [34], METEOR [35] and ROUGEL [36], is a set of metrics first proposed to evaluate text generated through machine translation; the second, including CIDEr [37], SPICE [38] and SPIDEr [39], was instead introduced for the evaluation of image captioning models.”
Background(2004) · Cited by 9,718
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“The attention model used is a soft top-down attention mechanism, similar to approaches first proposed for the image captioning task in [5], [32].”
BackgroundMethodShow and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 4×
“Our work is inspired by CNN-RNN architectures developed for image and audio captioning, but focusses on how such approaches can be extended to the music domain for the first time.”
BackgroundMethodInfluentialBottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00636 · Cited by 4,948 · cited 2×
“The attention model used is a soft top-down attention mechanism, similar to approaches first proposed for the image captioning task in [5], [32].”
MethodCIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 7×
“com/audio-captioning/dcase-2020-baseline/ [34], METEOR [35] and ROUGEL [36], is a set of metrics first proposed to evaluate text generated through machine translation; the second, including CIDEr [37], SPICE [38] and SPIDEr [39], was instead introduced for the evaluation of image captioning models.”
BackgroundResultInfluentialA Learning Algorithm for Continually Running Fully Recurrent Neural Networks
Williams, R. J., & Zipser, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.270 · Cited by 4,369 · cited 1×
“Our caption generation model is trained using teacher forcing [33].”
Method
Research works printed but not linked: 1
- tian, 0, Music autotagging as captioning, Proceedings of the 1st Workshop on NLP for Music and Audio (NLP4MusA)
Cited by
Bandi, A., Adapa, P. V. S. R., & Kuchi, Y. E. V. P. K. (2023) · Future Internet · doi:10.3390/fi15080260 · Cited by 638
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.04226 · Cited by 463
The Internet of Sounds: Convergent Trends, Insights, and Future Directions
Turchet, L., Lagrange, M., Rottondi, C., Fazekas, G., Peters, N., Østergaard, J., Font, F., Bäckström, T., & Fischione, C. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3253602 · Cited by 111
“This involves the use of an acoustic model and a large language model [108].”
BackgroundMethodSelf-citationRetrieving Multimodal Information for Augmented Generation: A Survey
Zhao, R., Chen, H., Wang, W., Jiao, F., Long, D., Qin, C., Ding, B., Guo, X., Li, M., Li, X., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.314 · Cited by 63
Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning
Liu, S., Hussain, A. S., Sun, C., & Shan, Y. (2024) · ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp48485.2024.10447027 · Cited by 38
“Several models have been proposed for generating cap-tions for music, including MusCaps [5], audio captioning transformer [6], audio captioning with audio-text retrieval pretraining [7], Whisper tiny audio captioning 2 , and LP-MusicCaps [8].”
Metadata
- Authors
- Manco, I., Benetos, E., Quinton, E., & Fazekas, G.
- Affiliations
- Queen Mary University of London (GB)
- Venue
- 2021 International Joint Conference on Neural Networks (IJCNN)
- Publisher
- IEEE
- Published
- 2021-07-18
- Pages
- 1-8
- ISSN
- 2161-4407 (ISSN-L)
- OpenAlex
- W3159642309
- Semantic Scholar
- f13bbe25e59fef5a209091b6192841367b2a87a4
- Cited by
- 36
- References
- 57
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 7 · 2023: 12 · 2022: 4
- FWCI
- 3.51
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- International Joint Conference on Neural Networks (IJCNN) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
- arXiv (Cornell University)
- Funders
- UK Research and Innovation (EP/S022694/1)
- Queen Mary University of London
- Topics
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Social Sciences › Arts and Humanities › Music › Music Technology and Sound Studies
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- music description, multimodal encoder, semantic gap, music information retrieval, convolutional recurrent neural network, modality fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: MusCaps: Generating Captions for Music Audio. https://app.cochat.ai/papers/10.1109/ijcnn52387.2021.9533461
- The paper
- Manco, I., Benetos, E., Quinton, E., & Fazekas, G. (2021). MusCaps: Generating Captions for Music Audio. 2021 International Joint Conference on Neural Networks (IJCNN). https://doi.org/10.1109/ijcnn52387.2021.9533461
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.