CoChat
Sign Up
PreprintCited by 151Music and Audio ProcessingMusic Technology and Sound StudiesTopic Modeling

The DOI matches its DataCite record; 30 of the 30 references it cites were found in a registry or an index.

MuLan: A Joint Embedding of Music Audio and Natural Language

Open access
Huang, Qingqing, Jansen, Aren, Lee, Joonseok, Ganti, Ravi, Li, Judith Yue, Ellis, Daniel P. W.·2022·arXiv
DOI 10.48550/arxiv.2208.12415arXiv 2208.12415
Open this paper in CoChat — free accountOpen PDF

Abstract

Music tagging and content-based retrieval systems have traditionally been constructed using pre-defined ontologies covering a rigid set of music attributes or text queries. This paper presents MuLan: a first attempt at a new generation of acoustic models that link music audio directly to unconstrained natural language music descriptions. MuLan takes the form of a two-tower, joint audio-text embedding model trained using 44 million music recordings (370K hours) and weakly-associated, free-form text annotations. Through its compatibility with a wide range of music genres and text styles (including conventional music tags), the resulting audio-text representation subsumes existing ontologies while graduating to true zero-shot functionalities. We demonstrate the versatility of the MuLan embeddings with a range of experiments including transfer learning, zero-shot music tagging, language understanding in the music domain, and cross-modal retrieval applications.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2208.12415

Cites

0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 16

  1. [1][1] C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig, 'Scaling up visual and vision-language representation learning with noisy text supervision,' in Proc. of the International Conference on Machine Learning (ICML) , 2021. -
  2. [2][2] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , 'Learning transferable visual models from natural language supervision,' in Proc. of the International Conference on Machine Learning (ICML) , 2021. -
  3. [9][9] --, 'Learning contextual tag embeddings for crossmodal alignment of audio and tags,' in Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2021. -
  4. [12][12] P. Hamel, M. E. Davies, K. Yoshii, and M. Goto, 'Transfer learning in MIR: Sharing learned latent representations for music audio classification and similarity,' in Proc. of International Conference on Music Information Retrieval (ISMIR) , 2013. -
  5. [13][13] A. Van Den Oord, S. Dieleman, and B. Schrauwen, 'Transfer learning by supervised pre-training for 14. audio-based music classification,' in Proc. of International Conference on Music Information Retrieval (ISMIR) , 2014. -
  6. [14][14] K. Choi, G. Fazekas, M. Sandler, and K. Cho, 'Transfer learning for music classification and regression tasks,' in Proc. of International Conference on Music Information Retrieval (ISMIR) , 2017. -
  7. [18][18] T. Bertin-Mahieux, D. P. Ellis, B. Whitman, and P. Lamere, 'The million song dataset,' in Proc. of International Conference on Music Information Retrieval (ISMIR) , 2011. -
  8. [21][21] N. Turpault, R. Serizel, and E. Vincent, 'Semisupervised triplet loss based learning of ambient audio embeddings,' in Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing , 2019. -
  9. [23][23] R. Castellon, C. Donahue, and P. Liang, 'Codified audio language modeling learns useful representations for music information retrieval,' in Proc. of International Conference on Music Information Retrieval (ISMIR) , 2021. -
  10. [24][24] S. Oramas, O. Nieto, F. Barbieri, and X. Serra, 'Multilabel music genre classification from audio, text, and images using deep features,' in Proc. of International Conference on Music Information Retrieval (ISMIR) , 2017. -
  11. [25][25] Q. Huang, A. Jansen, L. Zhang, D. P. Ellis, R. A. Saurous, and J. Anderson, 'Large-scale weaklysupervised content embeddings for music recommendation and tagging,' in Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2020. -
  12. [31][31] M. Won, J. Salamon, N. J. Bryan, G. J. Mysore, and X. Serra, 'Emotion embedding spaces for matching music to stories,' in Proc. of International Conference on Music Information Retrieval (ISMIR) , 2021. -
  13. [35][35] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, 'A simple framework for contrastive learning of visual representations,' in Proc. of International Conference on Machine Learning (ICML) , 2020. -
  14. [38][38] M.-W. C. Kenton, Jacob Devlin and L. K. Toutanova, 'BERT: Pre-training of deep bidirectional transformers for language understanding,' in Proceedings of NAACL-HLT , 2019, pp. 4171-4186. -
  15. [41][41] R. Typke, F. Wiering, R. C. Veltkamp, J. D. Reiss, G. A. Wiggins et al. , 'A survey of music information retrieval systems,' in Proc. of International Conference on Music Information Retrieval (ISMIR) , 2005. -
  16. [44][44] G. N. C. Tejada, J. Scholtes, and G. Spanakis, 'A study of BERT's processing of negations to determine sentiment,' in Proc. of the Benelux Conference on Artificial Intelligence and Belgian-Dutch Conference on Machine Learning , 2021. -

Cited by

Sorted by citations

See all 151

Related

Metadata

Authors
Huang, Q., Jansen, A., Lee, J., Ganti, R., Li, J. Y., & Ellis, D. P. W.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2208.12415
arXiv
2208.12415
OpenAlex
W4293575120
Cited by
49
References
0
Citations by year
2025: 14 · 2024: 22 · 2023: 13
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
  • Social Sciences › Arts and Humanities › Music › Music Technology and Sound Studies
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
Mulan, music tagging, cross-modal retrieval, ontology of music, two-tower model, transfer learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review