COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representations
Open accessAbstract
Audio representation learning based on deep neural networks (DNNs) emerged as an alternative approach to hand-crafted features. For achieving high performance, DNNs often need a large amount of annotated data which can be difficult and costly to obtain. In this paper, we propose a method for learning audio representations, aligning the learned latent representations of audio and associated tags. Aligning is done by maximizing the agreement of the latent representations of audio and tags, using a contrastive loss. The result is an audio embedding model which reflects acoustic and semantic characteristics of sounds. We evaluate the quality of our embedding model, measuring its performance as a feature extractor on three different tasks (namely, sound event recognition, and music genre and musical instrument classification), and investigate what type of characteristics the model captures. Our results are promising, sometimes in par with the state-of-the-art in the considered tasks and the embeddings produced with our method are well correlated with some acoustic descriptors.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prior work on audio-text representation learning makes use of tags.
“Similarly, prior work on audio-text representation learning also makes use of tags [26]”
From MusCaps: Generating Captions for Music Audio (2021) · cites this paper as [26]
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · see citing works
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235 · see citing works
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · see citing works
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · see citing works
Canonical Correlation Analysis: An Overview with Application to Learning Methods
Hardoon, D. R., Szedmák, S., & Shawe-Taylor, J. S. (2004) · Neural Computation · doi:10.1162/0899766042321814 · Cited by 3,297 · see citing works
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · see citing works
librosa: Audio and Music Signal Analysis in Python
McFee, B., Raffel, C., Liang, D., Ellis, D. P. W., McVicar, M., Battenberg, E., & Nieto, O. (2015) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-7b98e3ed-003 · Cited by 3,099 · see citing works
Musical genre classification of audio signals
Tzanetakis, G., & Cook, P. (2002) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2002.800560 · Cited by 2,832 · see citing works
Neural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729 · see citing works
CNN architectures for large-scale audio classification
Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F., Jansen, A., Moore, R. C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., et al. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952132 · Cited by 2,605 · see citing works
Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification
Salamon, J., & Bello, J. P. (2017) · IEEE Signal Processing Letters · doi:10.1109/lsp.2017.2657381 · Cited by 1,602 · see citing works
(2016) · neural information processing systems · Cited by 1,572 · see citing works
A Dataset and Taxonomy for Urban Sound Research
Salamon, J., Jacoby, C., & Bello, J. P. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2655045 · Cited by 1,355 · see citing works
A guide to convolution arithmetic for deep learningPreprint
Dumoulin, V., & Visin, F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.07285 · Cited by 1,223 · see citing works
YouTube-8M: A Large-Scale Video Classification BenchmarkPreprint
Abu-El-Haija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G. D., Varadarajan, B., & Vijayanarasimhan, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.08675 · Cited by 1,135 · see citing works
Bertin-Mahieux, T., Ellis, D. P. W., Whitman, B., & Lamere, P. (2011) · Columbia University · doi:10.7916/d8nz8j07 · Cited by 944 · see citing works
Arandjelović, R., & Zisserman, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.73 · Cited by 836 · see citing works
Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders
Schönfeld, E., Ebrahimi, S., Sinha, S., Darrell, T., & Akata, Z. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00844 · Cited by 747 · see citing works
Cross-modal Retrieval with Correspondence Autoencoder
Feng, F., Wang, X., & Li, R. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654902 · Cited by 572 · see citing works
Neural Audio Synthesis of Musical Notes with WaveNet AutoencodersPreprint
Engel, J., Resnick, C., Roberts, A., Dieleman, S., Douglas Eck, C., Simonyan, K., & Norouzi, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.01279 · Cited by 428 · see citing works
Raghu, M., Gilmer, J., Yosinski, J., & Sohl-Dickstein, J. (2017) · arXiv · doi:10.48550/arxiv.1706.05806 · Cited by 335 · see citing works
Font, F., Roma, G., & Serra, X. (2013) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2502081.2502245 · Cited by 305 · see citing works
Self-citationLook, Listen, and Learn More: Design Choices for Deep Audio Embeddings
Cramer, A. L., Wu, H.-H., Salamon, J., & Bello, J. P. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682475 · Cited by 289 · see citing works
Learning Grounded Meaning Representations with Autoencoders
Silberer, C., & Lapata, M. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-1068 · Cited by 221 · see citing works
Deep Learning and Music Adversaries
Kereliuk, C., Sturm, B. L. T., & Larsen, J. O. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2478068 · Cited by 132 · see citing works
Lee, J., Park, J., Kim, K., & Nam, J. (2018) · Applied Sciences · doi:10.3390/app8010150 · Cited by 131 · see citing works
Timbre analysis of music audio signals with convolutional neural networks
Pons, J., Slizovskaia, O., Gong, R., Gómez, E., & Serra, X. (2017) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2017.8081710 · Cited by 129 · see citing works
Self-citationHolden, B. (2002) · English teaching professional · Cited by 74 · see citing works
End-to-end Learning for Music Audio Tagging at Scale
(2018) · International Symposium/Conference on Music Information Retrieval · doi:10.5072/zenodo.251644 · Cited by 68 · see citing works
Self-citationmusicnn: Pre-trained convolutional neural networks for music audio tagging
Pons, J., & Serra, X. (2019) · arXiv · doi:10.48550/arxiv.1909.06654 · Cited by 67 · cited 2× · see citing works
“When trained with data augmentation on the US8K dataset, it achieved 79.0% accuracy. rVGG (Pons & Serra, 2019b) corresponds to a VGGish non-trained model (randomly weighted).”
MethodInfluentialSelf-citationTransfer Learning By Supervised Pre-Training For Audio-Based Music Classification.
van den Oord, A., Dieleman, S., & Schrauwen, B. (2014) · Ghent University Academic Bibliography (Ghent University) · doi:10.5281/zenodo.1415889 · Cited by 60 · see citing works
Transfer Learning For Music Classification And Regression Tasks.
Choi, K., Fazekas, G., Sandler, M. B., & Cho, K. (2017) · Queen Mary Research Online (Queen Mary University of London) · doi:10.5281/zenodo.1418014 · Cited by 50 · see citing works
Representation Learning of Music Using Artist LabelsPreprint
Park, J. Y., Lee, J., Park, J., Ha, J.-W., & Nam, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.06648 · Cited by 45 · see citing works
Mimilakis, S. I., Drossos, K., Santos, J. F., Schuller, G., Virtanen, T. I., & Bengio, Y. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461822 · Cited by 40 · see citing works
Self-citationMarchand, U., & Peeters, G. G. (2016) · HAL · Cited by 35 · see citing works
MaD TwinNet: Masker-Denoiser Architecture with Twin Networks for Monaural Sound Source Separation
Drossos, K., Mimilakis, S. I., Serdyuk, D., Schuller, G., Virtanen, T. I., & Bengio, Y. (2018) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489565 · Cited by 21 · see citing works
Self-citationRandomly Weighted CNNs for (Music) Audio Classification
Pons, J., & Serra, X. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682912 · Cited by 10 · see citing works
Self-citationData Augmentation for Instrument Classification Robust to Audio Effects
Ramires, A., & Serra, X. (2019) · RECERCAT (Consorci de Serveis Universitaris de Catalunya) · doi:10.48550/arxiv.1907.08520 · Cited by 6 · see citing works
Self-citationFacilitating the Manual Annotation of Sounds When Using Large Taxonomies
Favory, X., Fonseca, E., Font, F., & Serra, X. (2018) · RECERCAT (Consorci de Serveis Universitaris de Catalunya) · doi:10.5281/zenodo.4553681 · Cited by 3 · see citing works
Self-citationTensorflow Audio Models in Essentia
Alonso-Jiménez, P., Bogdanov, D., Pons, J., & Serra, X. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054688 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
1 of the 42 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Contrastive Representation Learning: A Framework and Review
Le-Khac, P. H., Healy, G., & Smeaton, A. F. (2020) · IEEE Access · doi:10.1109/access.2020.3031549 · Cited by 862 · see citing works
FSD50K: An Open Dataset of Human-Labeled Sound Events
Fonseca, E., Favory, X., Pons, J., Font, F., & Serra, X. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3133208 · Cited by 462 · see citing works
Self-citationBYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation
Niizumi, D., Takeuchi, D., Ohishi, Y., Harada, N., & Kashino, K. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534474 · Cited by 148 · see citing works
Audio self-supervised learning: A survey
Liu, S., Mallol-Ragolta, A., Parada‐Cabaleiro, E., Qian, K., Jing, X., Kathan, A., Hu, B., & Schuller, B. W. (2022) · Patterns · doi:10.1016/j.patter.2022.100616 · Cited by 116 · see citing works
BYOL for Audio: Exploring Pre-Trained General-Purpose Audio Representations
Niizumi, D., Takeuchi, D., Ohishi, Y., Harada, N., & Kashino, K. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3221007 · Cited by 68 · see citing works
MusCaps: Generating Captions for Music Audio
Manco, I., Benetos, E., Quinton, E., & Fazekas, G. (2021) · 2021 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533461 · Cited by 27 · see citing works
“Similarly, prior work on audio-text representation learning also makes use of tags [26], while we”
BackgroundEnriched music representations with multiple cross-modal contrastive learning
Ferraro, A., Favory, X., Drossos, K., Kim, Y., & Bogdanov, D. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3071082 · Cited by 23 · see citing works
Self-citationLearning Music Audio Representations Via Weak Language Supervision
Manco, I., Benetos, E., Quinton, E., & Fazekas, G. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746996 · Cited by 21 · see citing works
Koizumi, Y., Ohishi, Y., Niizumi, D., Takeuchi, D., & Yasuda, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.07331 · Cited by 32 · see citing works
One Billion Audio Sounds from GPU-Enabled Modular Synthesis
Turian, J. P., Shier, J., Tzanetakis, G., McNally, K., & Henry, M. (2021) · International Conference on Digital Audio Effects (DAFx) · doi:10.23919/dafx51585.2021.9768246 · Cited by 20 · see citing works
Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learning
Wu, H., Lei, C., Sun, X., Wang, P., Chen, Q., Cheng, K.-T. T., Lin, S. C.-F., & Wu, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01494 · Cited by 11 · see citing works
Learning Contextual Tag Embeddings for Cross-Modal Alignment of Audio and Tags
Favory, X., Drossos, K., Virtanen, T. I., & Serra, X. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414638 · Cited by 9 · see citing works
Self-citationCOCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representations
Ciranni, R., Mariani, G., Mancusi, M., Postolache, E., Fabbro, G., Rodolà, E., & Cosmo, L. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888146 · Cited by 8 · see citing works
Alonso-Jiménez, P., Favory, X., Aarabi, H. F., Bourdalas, G., Serra, X., Lidy, T., & Bogdanov, D. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095058 · Cited by 6 · see citing works
Self-citationWeck, B., Favory, X., Drossos, K., & Serra, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07410 · Cited by 5 · see citing works
Self-citationImproving Self-Supervised Learning for Audio Representations by Feature Diversity and Decorrelation
Nguyen, B., Uhlich, S., & Cardinaux, F. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097100 · Cited by 2 · see citing works
Evaluation of Deep Audio Representations for Semantic Sound Similarity
Araz, R. O., Bogdanov, D., Alonso-Jiménez, P., & Font, F. (2024) · International Workshop on Content-Based Multimedia Indexing (CBMI) · doi:10.1109/cbmi62980.2024.10859250 · Cited by 2 · see citing works
Evaluating Contrastive Methodologies for Music Representation Learning Using Playlist Data
Meehan, G., & Pauwels, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888157 · Cited by 1 · see citing works
An Empirical Study of Weakly Supervised Audio Tagging Embeddings for General Audio Representations
Dinkel, H., Yan, Z., Wang, Y., Zhang, J., & Wang, Y. (2022) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2022-54
Acoustics-Text Dual-Modal Joint Representation Learning for Cover Song Identification
Gu, Y., Jingli, JiayiZhou, Wang, Z., & Zhu, H. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389754
- Authors
- Favory, X., Drossos, K., Virtanen, T., & Serra, X.
- Affiliations
- Universitat Pompeu Fabra (ES)
- Tampere University (FI)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.08386
- OpenAlex
- W3035312337
- Cited by
- 20
- References
- 42
- Citations by year
- 2025: 2 · 2024: 1 · 2023: 4 · 2022: 4 · 2021: 7 · 2020: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Tampere University Institutional Repository (Tampere University) (submittedVersion)
- Repositori digital de la UPF (Universitat Pompeu Fabra) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Nvidia
- China Scholarship Council
- Topics
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- audio representation learning, autoencoder, semantic alignment, contrastive loss, audio embeddings, sound event recognition, music genre classification, musical instrument classification, acoustic descriptors, feature extraction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
