CoChat
Sign Up
PreprintCited by 17Music and Audio ProcessingMultimodal Machine Learning ApplicationsTopic Modeling

Contrastive Audio-Language Learning for Music

Open access
Manco, Ilaria, Benetos, Emmanouil, Quinton, Elio, Fazekas, György·2022·arXiv
DOI 10.48550/arxiv.2208.12208arXiv 2208.12208
Open this paper in CoChat — free accountOpen PDF

Abstract

As one of the most intuitive interfaces known to humans, natural language has the potential to mediate many tasks that involve human-computer interaction, especially in application-focused fields like Music Information Retrieval. In this work, we explore cross-modal learning in an attempt to bridge audio and language in the music domain. To this end, we propose MusCALL, a framework for Music Contrastive Audio-Language Learning. Our approach consists of a dual-encoder architecture that learns the alignment between pairs of music audio and descriptive sentences, producing multimodal embeddings that can be used for text-to-audio and audio-to-text retrieval out-of-the-box. Thanks to this property, MusCALL can be transferred to virtually any task that can be cast as text-based retrieval. Our experiments show that our method performs significantly better than the baselines at retrieving audio that matches a textual description and, conversely, text that matches an audio query. We also demonstrate that the multimodal alignment capability of our model can be successfully extended to the zero-shot transfer scenario for genre classification and auto-tagging on two public datasets.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2208.12208

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 17

Research works printed but not linked: 66

  1. [1][1] M. Müller, F. Kurth, D. Damm, C. Fremerey, and M. Clausen, 'Lyrics-Based Audio Retrieval and Multimodal Navigation in Music Collections,' Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , vol. 4675 LNCS, pp. 112-123, 2007. -
  2. [2][2] K. Tsukuda, 'Lyric Jumper: A Lyrics-Based Music Exploratory Web Service by Modeling Lyrics Generative Process.' in ISMIR , 2017, pp. 544-551. -
  3. [3][3] J. Lee, N. J. Bryan, J. Salamon, Z. Jin, and J. Nam, 'Disentangled Multidimensional Metric Learning for Music Similarity,' in ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing , 8 2020. -
  4. [4][4] B. Li and A. Kumar, 'Query by Video: Cross-Modal Music Retrieval,' in 20th International Society for Music Information Retrieval Conference (ISMIR) , 2019, pp. 604-611. -
  5. [5][5] A. Ghias, J. Logan, D. Chamberlin, and B. C. Smith, 'Query by humming,' in Proceedings of the third ACM international conference on Multimedia . Association for Computing Machinery (ACM), 1995, pp. 231-236. -
  6. [6][6] M. Müller, A. Arzt, S. Balke, M. Dorfer, and G. Widmer, 'Cross-Modal Music Retrieval and Applications: An Overview of Key Methodologies,' IEEE Signal Processing Magazine , vol. 36, no. 1, pp. 52-62, 1 2019. -
  7. [7][7] P. Knees, M. Schedl, and M. Goto, 'Intelligent User Interfaces for Music Discovery,' Transactions of the International Society for Music Information Retrieval , vol. 3, no. 1, pp. 165-179, 10 2020. -
  8. [8][8] K. Watanabe and M. Goto, 'Query-by-Blending: a Music Exploration System Blending Latent Vector Representations of Lyric Word, Song Audio, and Artist,' in Proceedings of the 20th International Society for Music Information Retrieval Conference , 2019. -
  9. [9][9] C. Hosey, L. Vujovi´ c, B. St. Thomas, J. GarciaGathright, and J. Thom, 'Just give me what I want: How people use and evaluate music search,' in Conference on Human Factors in Computing Systems - Proceedings . Association for Computing Machinery, 5 2019. -
  10. [11][11] C. C. Liem, M. Müller, D. Eck, G. Tzanetakis, and A. Hanjalic, 'The need for music information retrieval with user-centered and multimodal strategies,' in MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops - MIRUM 2011 Workshop, MIRUM'11 , 2011, pp. 1-6. -
  11. [13][13] B. Whitman and R. Rifkin, 'Musical query-bydescription as a multiclass learning problem,' in Proceedings of 2002 IEEE Workshop on Multimedia Signal Processing, MMSP 2002 . Institute of Electrical and Electronics Engineers Inc., 2002, pp. 153-156. -
  12. [14][14] A. van den Oord, S. Dieleman, and B. Schrauwen, 'Deep content-based music recommendation,' in Advances in Neural Information Processing Systems , 2013. -
  13. [15][15] I. Manco, E. Benetos, E. Quinton, and G. Fazekas, 'Learning music audio representations via weak language supervision,' in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022. -
  14. [18][18] S. Oramas, L. Espinosa-Anke, A. Lawlor, X. Serra, and H. Saggion, 'Exploring Customer Reviews for Music Genre Classification and Evolutionary Studies,' in 17th International Society for Music Information Retrieval Conference , 2016. -
  15. [19][19] S. Oramas, F. Barbieri, O. Nieto, and X. Serra, 'Multimodal Deep Learning for Music Genre Classification,' Transactions of the International Society for Music Information Retrieval , vol. 1, no. 1, pp. 4-21, 9 2018. -
  16. [20][20] R. Delbouys, R. Hennequin, F. Piccoli, J. Royo-Letelier, and M. Moussallam, 'Music mood detection based on audio and lyrics with deep neural net,' in Proceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018 , 2018. -
  17. [21][21] B. Jeon, C. Kim, A. Kim, D. Kim, J. Park, and J. W. Ha, 'Music emotion recognition via end-To-end multimodal neural networks,' in CEUR Workshop Proceedings , 2017. -
  18. [22][22] K. Choi, G. Fazekas, M. Sandler, B. Mcfee, and K. Cho, 'Towards Music Captioning: Generating Music Playlist Descriptions,' arXiv preprint arXiv:1608.04868 , 2016. - · arXiv
  19. [23][23] C. Tian, M. Michael, and H. Di, 'Music autotagging as captioning,' in Proceedings of the 1st Workshop on NLP for Music and Audio (NLP4MusA) . Association for Computational Linguistics, 2020, pp. 67-72. -
  20. [24][24] I. Manco, E. Benetos, E. Quinton, and G. Fazekas, 'MusCaps: Generating Captions for Music Audio,' in 2021 International Joint Conference on Neural Networks (IJCNN) . IEEE, 2021. -
  21. [25][25] X. Favory, K. Drossos, V. Tuomas, and X. Serra, 'COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representations,' in International Conference on Machine Learning (ICML), Workshop on Self-supervised learning in Audio and Speech , 2020. -
  22. [26][26] X. Favory, K. Drossos, T. Virtanen, and X. Serra, 'Learning Contextual Tag Embeddings for Cross-Modal Alignment of Audio and Tags,' in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2021. -
  23. [27][27] A. Ferraro, X. Favory, K. Drossos, Y. Kim, and D. Bogdanov, 'Enriched Music Representations with Multiple Cross-modal Contrastive Learning,' IEEE Signal Processing Letters , vol. 28, pp. 733-737, 4 2021. -
  24. [28][28] J. Choi, J. Lee, J. Park, and J. Nam, 'Zero-shot learning for audio-based music classification and tagging,' in Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019 , 2019. -
  25. [29][29] M. Won, S. Oramas, O. Nieto, F. Gouyon, and X. Serra, 'Multimodal Metric Learning for Tag-based Music Retrieval,' in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2021. -
  26. [30][30] M. Won, J. Salamon, N. J. Bryan, G. J. Mysore, and X. Serra, 'Emotion Embedding Spaces for Matching Music to Stories,' in International Society for Music Information Retrieval Conference (ISMIR) , 11 2021. -
  27. [31][31] J. Lu, D. Batra, D. Parikh, and S. Lee, 'ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks,' in Advances in Neural Information Processing Systems , 2019, pp. 13-23. -
  28. [32][32] Y. C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, 'UNITER: UNiversal Image-TExt Representation Learning,' in European Conference on Computer Vision , 2020. -
  29. [33][33] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, 'Learning Transferable Visual Models From Natural Language Supervision,' in International Conference on Machine Learning . PMLR, 2021. -
  30. [34][34] C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig, 'Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision,' in International Conference on Machine Learning . PMLR, 2021, pp. 49044916. -
  31. [35][35] J. Andreas, D. Klein, and S. Levine, 'Learning with Latent Language,' in NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , vol. 1. Association for Computational Linguistics (ACL), 11 2018, pp. 2166-2179. -
  32. [36][36] X. Liu, X. Mei, Q. Huang, J. Sun, J. Zhao, H. Liu, M. D. Plumbley, V. Kılıç, and W. Wang, 'Leveraging Pretrained BERT for Audio Captioning,' arXiv preprint arXiv:2203.02838 , 2022. - · arXiv
  33. [37][37] A.-M. Oncescu, A. S. Koepke, J. F. Henriques, Z. Akata, and S. Albanie, 'Audio Retrieval with Natural Language Queries,' in Interspeech , 5 2021. -
  34. [38][38] H.-H. Wu, P. Seetharaman, K. Kumar, and J. P. Bello, 'Wav2CLIP: Learning Robust Audio Representations From CLIP,' in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 10 2022. -
  35. [39][39] Y. Zhao, J. Hessel, Y. Yu, X. Lu, R. Zellers, and Y . Choi, 'Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer,' arXiv preprint arXiv:2112.08995 , 2021. - · arXiv
  36. [40][40] A. Guzhov, F. Raue, J. Hees, and A. Dengel, 'AudioCLIP: Extending CLIP to Image, Text and Audio,' in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022. -
  37. [41][41] C. Chen, D. Han, and J. Wang, 'Multimodal EncoderDecoder Attention Networks for Visual Question Answering,' IEEE Access , pp. 1-1, 2 2020. -
  38. [42][42] D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada, and K. Kashino, 'BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation,' in 2021 International Joint Conference on Neural Networks (IJCNN) . IEEE, 3 2021. -
  39. [43][43] A. Saeed, D. Grangier, and N. Zeghidour, 'Contrastive Learning of General-Purpose Audio Representations,' in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 10 2021. -
  40. [44][44] H. Al-Tahan, Y. M. I. C. On, and U. 2021, 'CLAR: Contrastive Learning of Auditory Representations,' in International Conference on Artificial Intelligence and Statistics . PMLR, 2021. -
  41. [46][46] E. Fonseca, D. Ortego, K. McGuinness, N. E. O'Connor, and X. Serra, 'Unsupervised Contrastive Learning of Sound Event Representations,' in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 11 2021. -
  42. [47][47] P. H. Le-Khac, G. Healy, and A. F. Smeaton, 'Contrastive Representation Learning: A Framework and Review,' IEEE Access , vol. 8, pp. 193 907-193 934, 10 2020. -
  43. [48][48] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, 'A Simple Framework for Contrastive Learning of Visual Representations,' in International conference on machine learning . PMLR, 2020. -
  44. [49][49] J.-B. Alayrac, A. Recasens, R. Schneider, R. Arandjelovi´ c, J. Ramapuram, J. De Fauw, L. Smaira, S. Dieleman, and A. Zisserman, 'Self-Supervised MultiModal Versatile Networks,' Advances in Neural Information Processing Systems , vol. 33, pp. 25-37, 6 2020. -
  45. [50][50] M. Patrick, Y. M. Asano, P. Kuznetsova, R. Fong, J. F. Henriques, G. Zweig, and A. Vedaldi, 'Multi-modal Self-Supervision from Generalized Data Transformations,' arXiv preprint arXiv:2003.04298v2 , 2020. - · arXiv
  46. [51][51] A. van den Oord, Y. Li, and O. Vinyals, 'Representation learning with contrastive predictive coding,' arXiv preprint arXiv:1807.03748 , 2018. - · arXiv
  47. [52][52] N. Saunshi, P. Orestis, S. Arora, K. Mikhail, and K. Hrishikesh, 'A Theoretical Analysis of Contrastive Unsupervised Representation Learning,' in International Conference on Machine Learning . PMLR, 2019. -
  48. [53][53] M. Zolfaghari, Y. Zhu, P. Gehler, and T. Brox, 'CrossCLR: Cross-modal Contrastive Learning For Multimodal Video Representations,' in International Conference on Computer Vision (ICCV) . Institute of Electrical and Electronics Engineers (IEEE), 9 2021, pp. 1430-1439. -
  49. [54][54] L. Ericsson, H. Gouk, C. C. Loy, and T. M. Hospedales, 'Self-Supervised Representation Learning: Introduction, Advances and Challenges,' IEEE Signal Processing Magazine , vol. 39, no. 3, pp. 42-62, 10 2021. -
  50. [55][55] N. Mu, A. Kirillov, D. Wagner, and S. Xie, 'SLIP: Selfsupervision meets Language-Image Pre-training,' arXiv preprint arXiv:2112.12750 , 2021. - · arXiv
  51. [56][56] H. Xie, O. Räsänen, K. Drossos, and T. Virtanen, 'Unsupervised Audio-Caption Aligning Learns Correspondences between Individual Sound Events and Textual Phrases,' in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022. -
  52. [57][57] K. He, Y. Wang, and J. Hopcroft, 'A powerful generative model using random weights for the deep image representation,' in Advances in Neural Information Processing Systems , 2016. -
  53. [58][58] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, A. Kaiser, and I. Polosukhin, 'Attention is all you need,' in Advances in Neural Information Processing Systems , vol. 2017-Decem. Neural information processing systems foundation, 6 2017, pp. 5999-6009. -
  54. [60][60] M. Won, K. Choi, and X. Serra, 'Semi-Supervised Music Tagging Transformer,' in International Society for Music Information Retrieval Conference (ISMIR) , 11 2021. -
  55. [61][61] R. Sennrich, B. Haddow, and A. Birch, 'Neural Machine Translation of Rare Words with Subword Units,' in 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 - Long Papers , vol. 3. Association for Computational Linguistics (ACL), 2016, pp. 1715-1725. -
  56. [62][62] N. Reimers and I. Gurevych, 'Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,' in EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference . Association for Computational Linguistics, 8 2019, pp. 3982-3992. -
  57. [63][63] K. Drossos, S. Lipping, and T. Virtanen, 'Clotho: An Audio Captioning Dataset,' in ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings , vol. 2020-May. IEEE, 5 2020, pp. 736-740. -
  58. [64][64] C. D. Kim, B. Kim, H. Lee, and G. Kim, 'AudioCaps: Generating captions for audios in the wild,' in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) . Association for Computational Linguistics (ACL), 2019, p. 119-132. -
  59. [65][65] T. Mikolov, K. Chen, G. Corrado, and J. Dean, 'Efficient estimation of word representations in vector space,' in 1st International Conference on Learning Representations, ICLR 2013 - Workshop Track Proceedings . International Conference on Learning Representations, ICLR, 2013. -
  60. [66][66] J. Bromley, I. Guyon, Y. Lecun, E. Sickinger, R. Shah, A. Bell, and L. Holmdel, 'Signature verification using a" siamese" time delay neural network,' Advances in neural information processing systems , vol. 6, 1993. -
  61. [67][67] X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer, 'LiT: Zero-Shot Transfer with Locked-image Text Tuning,' arXiv preprint arXiv:2111.07991 , 2021. - · arXiv
  62. [68][68] G. Tzanetakis and P. Cook, 'Musical genre classification of audio signals,' IEEE Transactions on Speech and Audio Processing , vol. 10, no. 5, pp. 293-302, 7 2002. -
  63. [69][69] E. Law, K. West, M. Mandel, M. Bay, and J. Stephen Downie, 'Evaluation of algorithms using games: The case of music tagging,' in Proceedings of the 10th ISMIR Conference , 2009. -
  64. [70][70] L. Wang, P. Luc, Y. Wu, A. Recasens, L. Smaira, A. Brock, A. Jaegle, J.-B. Alayrac, S. Dieleman, J. Carreira, and A. van den Oord, 'Towards Learning Universal Audio Representations,' in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 5 2022, pp. 4593-4597. -
  65. [71][71] L. A. Hendricks, J. Mellor, R. Schneider, J.-B. Alayrac, and A. Nematzadeh, 'Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers,' Transactions of the Association for Computational Linguistics , vol. 9, pp. 570-585, 1 2021. -
  66. [72][72] B. Lester, R. Al-Rfou, N. Constant, and G. Research, 'The Power of Scale for Parameter-Efficient Prompt Tuning,' in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics (ACL), 12 2021, pp. 3045-3059.

Authors
Manco, I., Benetos, E., Quinton, E., & Fazekas, G.
Affiliations
Queen Mary University of London (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2208.12208
arXiv
2208.12208
OpenAlex
W4293342670
Cited by
17
References
0
Citations by year
2025: 4 · 2024: 8 · 2023: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Alan Turing Institute
  • UK Research and Innovation (EP/ S022694/1)
Topics
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
contrastive learning, music information retrieval, cross-modal learning, dual-encoder architecture, multimodal embeddings, zero-shot transfer, music genre classification, music auto-tagging

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review