Learning Music Sequence Representation From Text Supervision
Open accessAbstract
Music representation learning is notoriously difficult for its complex human-related concepts contained in the sequence of numerical signals. To excavate better MUsic SEquence Representation from labeled audio, we propose a novel text-supervision pre-training method, namely MUSER. MUSER adopts an audio-spectrum-text tri-modal contrastive learning framework, where the text input could be any form of meta-data with the help of text templates while the spectrum is derived from an audio sequence. Our experiments reveal that MUSER could be more flexibly adapted to downstream tasks compared with the current data-hungry pre-training method, and it only requires 0.056% of pre-training data to achieve the state-of-the-art performance.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596 · see citing works
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · see citing works
CNN architectures for large-scale audio classification
Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F., Jansen, A., Moore, R. C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., et al. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952132 · Cited by 2,605 · see citing works
Computational Signal Processing with Wavelets
Teolis, A. (1998) · Birkhäuser Boston eBooks · doi:10.1007/978-1-4612-4142-3 · Cited by 389 · see citing works
Audioclip: Extending Clip to Image, Text and Audio
Guzhov, A., Raue, F., Hees, J., & Dengel, A. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9747631 · Cited by 330 · see citing works
Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings
Cramer, A. L., Wu, H.-H., Salamon, J., & Bello, J. P. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682475 · Cited by 289 · see citing works
Automatic tagging using deep convolutional neural networksPreprint
Choi, K., Fazekas, G., & Sandler, M. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00298 · Cited by 222 · see citing works
Evaluation Of Algorithms Using Games: The Case Of Music Tagging.
Law, E., West, K., Mandel, M. I., Bay, M., & Downie, J. S. (2009) · Zenodo · doi:10.5281/zenodo.1417646 · Cited by 168 · see citing works
The GTZAN dataset: Its contents, its faults, their effects on evaluation, and its future usePreprint
Sturm, B. L. T. (2013) · VBN Forskningsportal (Aalborg Universitet) · doi:10.48550/arxiv.1306.1461 · Cited by 156 · see citing works
Deep Learning and Music Adversaries
Kereliuk, C., Sturm, B. L. T., & Larsen, J. O. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2478068 · Cited by 132 · see citing works
End-to-end learning for music audio tagging at scalePreprint
Pons, J., Nieto, O., Prockup, M., Schmidt, E. M., Ehmann, A. F., & Serra, X. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02520 · Cited by 111 · see citing works
Music Genre Classification using Machine Learning TechniquesPreprint
Bahuleyan, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.01149 · Cited by 102 · see citing works
Contrastive Learning of Musical Representations
Spijkervet, J., & Burgoyne, J. (2021) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.2103.09410 · Cited by 84 · see citing works
musicnn: Pre-trained convolutional neural networks for music audio tagging
Pons, J., & Serra, X. (2019) · arXiv · doi:10.48550/arxiv.1909.06654 · Cited by 67 · cited 1× · see citing works
“Inspired by the success of deep learning on computer vision and natural language processing, researchers introduce large-scale pre-training techniques into music sequence representation learning [5, 6].”
BackgroundTransfer Learning By Supervised Pre-Training For Audio-Based Music Classification.
van den Oord, A., Dieleman, S., & Schrauwen, B. (2014) · Ghent University Academic Bibliography (Ghent University) · doi:10.5281/zenodo.1415889 · Cited by 60 · see citing works
Codified audio language modeling learns useful representations for music information retrievalPreprint
Castellon, R., Donahue, C., & Liang, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05677 · Cited by 47 · see citing works
Toward Interpretable Music Tagging with Self-AttentionPreprint
Won, M., Chun, S., & Serra, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04972 · Cited by 41 · see citing works
Computational Signal Processing with Wavelets
Teolis, A. (2017) · Modern Birkhäuser classics · doi:10.1007/978-3-319-65747-9 · Cited by 38 · see citing works
ESResNe(X)t-fbsp: Learning Robust Time-Frequency Transformation of Audio
Guzhov, A., Raue, F., Hees, J., & Dengel, A. R. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533654 · Cited by 36 · see citing works
A Simple Long-Tailed Recognition Baseline via Vision-Language ModelPreprint
Ma, T., Geng, S., Wang, M., Shao, J., Lu, J., Li, H., Gao, P., & Qiao, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.14745 · Cited by 34 · see citing works
Large-Scale Weakly-Supervised Content Embeddings for Music Recommendation and Tagging
Huang, Q., Jansen, A., Zhang, L., Ellis, D. P. W., Saurous, R. A., & Anderson, J. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053240 · Cited by 20 · see citing works
Representation Learning of Music Using Artist, Album, and Track InformationPreprint
Lee, J., Park, J., & Nam, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.11783 · Cited by 11 · see citing works
Codified audio language modeling learns useful representations for music information retrieval
Castellon, R., Donahue, C., & Liang, P. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5624604 · Cited by 9 · see citing works
Flagged first, then by citations
2 of the 28 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever, "Learning trans- ferable visual models from natural language supervi- sion," in ICML, 2021.
Cited by
Sorted by citations
Xie, Y., Ren, J., & Ji, X. (2022) · The Journal of the Acoustical Society of America · doi:10.1121/10.0015053 · Cited by 46 · see citing works
Self-citationMusic genre classification based on fusing audio and lyric information
Li, Y., Zhang, Z., Ding, H., & Chang, L. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-14252-6 · Cited by 24 · see citing works
Heterogeneous Contrastive Learning for Foundation Models and Beyond
Zheng, L., Jing, B., Li, Z., Tong, H., & He, J. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671454 · Cited by 20 · see citing works
Guiding the underwater acoustic target recognition with interpretable contrastive learning
Xie, Y., Ren, J., & Ji, X. (2023) · MTS/IEEE OCEANS Conference (OCEANS) · doi:10.1109/oceanslimerick52467.2023.10244447 · Cited by 16 · see citing works
Self-citationZhang, X., & Cheng, X. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.953501 · Cited by 13 · see citing works
LP-MusicCaps: LLM-Based Pseudo Music Captioning
Doh, S., Choi, K., Lee, J., & Nam, J. (2023) · arXiv · doi:10.48550/arxiv.2307.16372 · Cited by 16 · see citing works
“A workaround for this situation is to use music tagging datasets and generate sentences with tag concatenation [2,13] or prompt template [14].”
MethodMa, X. (2024) · Current Psychology · doi:10.1007/s12144-024-06498-0 · Cited by 3 · see citing works
Aligned Contrastive Learning for Text-to-Music Retrieval
Komatsu, T., Munakata, H., Hasumi, T., & Fujita, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890072 · Cited by 2 · see citing works
Data-Efficient Music Captioning Via Contrastive and Semantic Alignment
Kim, L., & Park, J. (2025) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc65261.2025.11249411
- Authors
- Chen, T., Xie, Y., Zhang, S., Huang, S., Zhou, H., & Li, J.
- Affiliations
- Beihang University (CN)
- Chinese Academy of Sciences (CN)
- Microsoft Research Asia (China) (CN)
- Venue
- ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2022-05-23
- Pages
- 4583-4587
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W4225328971
- Cited by
- 8
- References
- 28
- Citations by year
- 2025: 2 · 2024: 2 · 2023: 1 · 2022: 3
- FWCI
- 1.42
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Music Technology and Sound Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- music representation learning, multimodal contrastive learning, music metadata, text templates
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
