The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings
Abstract
A considerable challenge in applying deep learning to audio classification is the scarcity of labeled data. An increasingly popular solution is to learn deep audio embeddings from large audio collections and use them to train shallow classifiers using small labeled datasets. Look, Listen, and Learn (L3-Net) is an embedding trained through self-supervised learning of audio-visual correspondence in videos as opposed to other embeddings requiring labeled data. This framework has the potential to produce powerful out-of-the-box embeddings for downstream audio classification tasks, but has a number of unexplained design choices that may impact the embeddings’ behavior. In this paper we investigate how L3-Net design choices impact the performance of downstream audio classifiers trained with these embeddings. We show that audio-informed choices of input representation are important, and that using sufficient data for training the embedding is key. Surprisingly, we find that matching the content for training the embedding to the downstream task is not beneficial. Finally, we show that our best variant of the L3-Net embedding outperforms both the VGGish and SoundNet embeddings, while having fewer parameters and being trained on less data. Our implementation of the L3-Net embedding model as well as pre-trained models are made freely available online.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Individual Comparisons by Ranking Methods
Wilcoxon, F. (1945) · Biometrics Bulletin · doi:10.2307/3001968 · Cited by 15,354
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · cited 1×
“Recently the community has turned to transfer learning [19] as a solution to the issue of data scarcity.”
BackgroundAudio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134
Deep Metric Learning Using Triplet Network
Hoffer, E., & Ailon, N. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-24261-3_7 · Cited by 2,134
CNN architectures for large-scale audio classification
Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F., Jansen, A., Moore, R. C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., et al. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952132 · Cited by 2,023
Young, S. J., Jansen, J., Odell, J. J., Ollason, D., & Woodland, P. (1995) · Cited by 1,427
Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification
Salamon, J., & Bello, J. P. (2017) · IEEE Signal Processing Letters · doi:10.1109/lsp.2017.2657381 · Cited by 1,364
Self-citationA Dataset and Taxonomy for Urban Sound Research
Salamon, J., Jacoby, C., & Bello, J. P. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2655045 · Cited by 1,309
Self-citationA Scale for the Measurement of the Psychological Magnitude Pitch
Stevens, S. S., Volkmann, J., & Newman, E. B. (1937) · The Journal of the Acoustical Society of America · doi:10.1121/1.1915893 · Cited by 1,107 · cited 1×
“Mel spectrograms are designed to capture perceptually relevant information more efficiently with less frequency bands compared to a linear spectrogram [29].”
BackgroundEnvironmental sound classification with convolutional neural networks
Piczak, K. J. (2015) · 2015 IEEE 25th International Workshop on Machine Learning for Signal Processing (MLSP) · doi:10.1109/mlsp.2015.7324337 · Cited by 931
YouTube-8M: A Large-Scale Video Classification Benchmark
Abu-El-Haija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G. D., Varadarajan, B., & Vijayanarasimhan, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.08675 · Cited by 848
Arandjelovic, R., & Zisserman, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.73 · Cited by 782
Detection and Classification of Acoustic Scenes and Events
Stowell, D., Giannoulis, D., Benetos, E., Lagrange, M., & Plumbley, M. D. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2428998 · Cited by 600
Arandjelović, R., & Zisserman, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_27 · Cited by 403
ESC: Dataset for Environmental Sound Classification
Piczak, K. J. (2015) · Harvard Dataverse · doi:10.7910/dvn/ydeput · Cited by 257
Environmental Sound Classification Using Deep Convolutional Neural Networks and Data Augmentation
Davis, N., & Suresh, K. (2018) · 2018 IEEE Recent Advances in Intelligent Computational Systems (RAICS) · doi:10.1109/raics.2018.8635051 · Cited by 175
Multimodal Multi-task Learning for Dimensional and Continuous Emotion Recognition
Chen, S., Jin, Q., Zhao, J., & Wang, S. (2017) · Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge · doi:10.1145/3133944.3133949 · Cited by 147
Unsupervised Learning of Semantic Audio Representations
Jansen, A., Plakal, M., Pandya, R., Ellis, D. P. W., Hershey, S., Liu, J., Moore, R. C., & Saurous, R. A. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461684 · Cited by 119
Cartwright, M., Seals, A., Salamon, J., Williams, A. C., Mikloska, S., MacConnell, D., Law, E., Bello, J. P., & Nov, O. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134664 · Cited by 83
Self-citationImproved Audio Scene Classification Based on Label-Tree Embeddings and Convolutional Neural Networks
Phan, H., Hertel, L., Maaß, M., Koch, P., Mazur, R., & Mertins, A. (2017) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2017.2690564 · Cited by 69
A Human-in-the-Loop System for Sound Event Detection and Annotation
Kim, B., & Pardo, B. (2018) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3214366 · Cited by 60
Modeling multimodal cues in a deep learning-based framework for emotion recognition in the wild
Pini, S., Ahmed, O. B., Cornia, M., Baraldi, L., Cucchiara, R., & Huet, B. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3143006 · Cited by 44
OpenMIC-2018: An Open Data-set for Multiple Instrument Recognition
Humphrey, E. J., Durand, S., & McFee, B. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1492444 · Cited by 44
Deep CNN Framework for Audio Event Recognition using Weakly Labeled Web DataPreprint
Kumar, A., & Raj, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.02530 · Cited by 40
Generative Modeling of Audible Shapes for Object Perception
Zhang, Z., Wu, J., Li, Q., Huang, Z., Traer, J. A., McDermott, J. H., Tenenbaum, J. B., & Freeman, W. T. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.141 · Cited by 39
Large-scale audio event discovery in one million YouTube videos
Jansen, A., Gemmeke, J. F., Ellis, D. P. W., Liu, X., Lawrence, W., & Freedman, D. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952263 · Cited by 32
I-SED: An Interactive Sound Event Detector
Kim, B., & Pardo, B. (2017) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3025171.3025231 · Cited by 21
Cited by
Sound Event Detection: A tutorial
Mesaros, A., Heittola, T., Virtanen, T., & Plumbley, M. D. (2021) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3090678 · Cited by 268
Self-supervised learning for videos: A survey
Schiappa, M. C., Rawat, Y. S., & Shah, M. (2023) · ACM Computing Surveys · doi:10.1145/3577925 · Cited by 155
Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning
Liu, S., Hussain, A. S., Sun, C., & Shan, Y. (2024) · ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp48485.2024.10447027 · Cited by 38
“3) OpenL3 [18] is a framework for learning deep audio embeddings through unsupervised audio-visual correspondence, which can be used for various downstream tasks such as audio classification and analysis.”
BackgroundEnriched music representations with multiple cross-modal contrastive learning
Ferraro, A., Favory, X., Drossos, K., Kim, Y., & Bogdanov, D. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3071082 · Cited by 23
COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representations
Favory, X., Drossos, K., Virtanen, T., & Serra, X. (2020) · arXiv · doi:10.48550/arxiv.2006.08386 · Cited by 20
Metadata
- Authors
- Cramer, A. L., Wu, H.-H., Salamon, J., & Bello, J. P.
- Venue
- ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2019-05
- Pages
- 3852-3856
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2939574508
- Cited by
- 281
- References
- 31
- FWCI
- 28.92
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.