CoChat
Sign Up
PreprintCited by 56Music and Audio ProcessingSpeech Recognition and SynthesisSpeech and Audio Processing

HEAR: Holistic Evaluation of Audio Representations

Open access
Turian, Joseph, Shier, Jordie, Khan, Humair Raj, Raj, Bhiksha, Schuller, Björn W., Steinmetz, Christian J., Malloy, Colin, Tzanetakis, George +15 more·2022·arXiv
DOI 10.48550/arxiv.2203.03022arXiv 2203.03022
Open this paper in CoChat — free accountOpen PDF

Abstract

What audio embedding approach generalizes best to a wide range of downstream tasks across a variety of everyday domains without fine-tuning? The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios. HEAR evaluates audio representations using a benchmark suite across a variety of domains, including speech, environmental sound, and music. HEAR was launched as a NeurIPS 2021 shared challenge. In the spirit of shared exchange, each participant submitted an audio embedding model following a common API that is general-purpose, open-source, and freely available to use. Twenty-nine models by thirteen external teams were evaluated on nineteen diverse downstream tasks derived from sixteen datasets. Open evaluation code, submitted models and datasets are key contributions, enabling comprehensive and reproducible evaluation, as well as previously impossible longitudinal studies. It still remains an open question whether one single general-purpose audio representation can perform as holistically as the human ear.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2203.03022

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • There are a few recent efforts on joint audio and speech recognition but none of them exhibit advanced joint reasoning ability.

    “There are a few recent efforts on joint audio and speech recogntion [7,8,9,10] but none of them exhibit advanced joint reasoning ability.”

    From Joint Audio and Speech Understanding (2023) · cites this paper as [10]

    Table 1 (p. 7) shows that HEAR evaluates both general-audio and speech tasks, so it supports identifying the paper as a joint audio-and-speech evaluation effort. The Introduction (p. 2) describes its evaluations as classification and sound-event detection, while Related Audio Shared Tasks (p. 5) distinguishes HEAR from a benchmark that includes speech recognition. The paper does not assess advanced joint reasoning or establish that none of the cited efforts has that ability. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 59

  1. Shahin Amiriparian, Maurice Gerczuk, Sandra Ottl, Nicholas Cummins, Michael Freitag, Sergey Pugachevskiy, and Bj¨ orn Schuller. Snore Sound Classification Using Image-based Deep Spectrum Features. In Proceedings of the 18th Annual Conference of the International Speech Communication Association (INTERSPEECH) , pages 3512-3516, Stockholm, Sweden, 2017.
  2. Akshay Anantapadmanabhan, Ashwin Bellur, and Hema A Murthy. Modal analysis and transcription of strokes of the Mridangam using non-negative matrix factorization. In Proceedings of the 38th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 181-185, 2013.
  3. Joakim And´ en and St´ ephane Mallat. Deep Scattering Spectrum. IEEE Transactions on Signal Processing , 62(16):4114-4128, 2014.
  4. Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 609-617, 2017.
  5. Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In Advances in Neural Information Processing Systems , volume 29, 2016.
  6. Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems , volume 33, 2020.
  7. Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning. In to appear at International Conference on Learning Representations (ICLR) , 2022. URL http://arxiv.org/abs/2105.04906 . · arXiv
  8. Yoshua Bengio, R´ ejean Ducharme, and Pascal Vincent. A Neural Probabilistic Language Model. In T. Leen, T. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems , volume 13, 2001.
  9. Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems , volume 33, pages 1877-1901, 2020.
  10. Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. CREMA-D: Crowd-sourced Emotional Multimodal Actors Dataset. IEEE Transactions on Affective Computing , 5(4):377-390, 2014.
  11. Andrew N Carr, Quentin Berthet, Mathieu Blondel, Olivier Teboul, and Neil Zeghidour. Self-Supervised Learning of Audio Representations From Permutations With Differentiable Ranking. IEEE Signal Processing Letters , 28:708-712, 2021.
  12. Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. VGGSound: A Largescale Audio-Visual Dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 721-725, 2020a.
  13. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning (ICML) , Proceedings of Machine Learning Research, 2020b.
  14. Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.
  15. Kin Wai Cheuk, Yin-Jyun Luo, Emmanouil Benetos, and Dorien Herremans. Revisiting the Onsets and Frames Model with Additive Attention. In International Joint Conference on Neural Networks, (IJCNN) , pages 1-8, 2021.
  16. Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , volume 1, pages 539-546, 2005.
  17. Ronan Collobert, Jason Weston, L´ eon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of Machine Learning Research , 12:2493-2537, 2011.
  18. Jason Cramer, Ho-Hsiang Wu, Justin Salamon, and Juan Pablo Bello. Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3852-3856, 2019.
  19. Steven Davis and Paul Mermelstein. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Transactions on Acoustics, Speech, and Signal Processing , 28(4):357-366, 1980.
  20. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A largescale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 248-255, 2009.
  21. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , pages 4171-4186, 2019.
  22. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. ERASER: A Benchmark to Evaluate Rationalized NLP Models. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL) , pages 4443-4458, 2020.
  23. J Stephen Downie, Xiao Hu, Jin Ha Lee, Kahyun Choi, Sally Jo Cunningham, and Yun Hao. Ten years of MIREX: reflections, challenges and opportunities. In Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR) , pages 657-662. ISMIR, 2014.
  24. Gasser Elbanna, Neil Scheidwasser-Clow, Mikolaj Kegler, Pierre Beckmann, and Milos Cernak. Byol-s: Learning self-supervised speech representations by bootstrapping. In Joseph Turian, Bj¨ orn W. Schuller, Dorien Herremans, Katrin Kirchhoff, Paola Garcia Perera, and Philippe Esling, editors, Proceedings of HEAR: Holistic Evaluation of Audio Representations , volume 166 of Proceedings of Machine Learning Research . PMLR, 2022. In submission.
  25. Jesse Engel, Lamtharn (Hanoi) Hantrakul, Chenjie Gu, and Adam Roberts. DDSP: Differentiable Digital Signal Processing. In Proceedings of the 8th International Conference on Learning Representations (ICLR) , 2020.
  26. Jesse H. Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan. Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning (ICML) , volume 70 of Proceedings of Machine Learning Research , pages 1068-1077, 2017.
  27. Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. FSD50K: an Open Dataset of Human-Labeled Sound Events. CoRR arXiv , 2020. URL http://arxiv.org/abs/2010.00475 . · arXiv
  28. Eduardo Fonseca, Andres Ferraro, and Xavier Serra. Improving sound event classification by increasing shift invariance in convolutional neural networks. CoRR arXiv , 2021a. URL https://arxiv.org/abs/2107.00623 . · arXiv
  29. Eduardo Fonseca, Diego Ortego, Kevin McGuinness, Noel E O'Connor, and Xavier Serra. Unsupervised contrastive learning of sound event representations. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 371-375, 2021b.
  30. Jort F Gemmeke, Daniel P W Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio Set: An ontology and human-labeled dataset for audio events. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 776-780, 2017.
  31. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 9 of Proceedings of Machine Learning Research , pages 249-256, Chia Laguna Resort, Sardinia, Italy, 2010. PMLR.
  32. Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. In Proceedings of the 22nd Annual Conference of the International Speech Communication Association (INTERSPEECH) , 2021a.
  33. Yuan Gong, Yu-An Chung, and James Glass. PSLA: Improving Audio Tagging With Pretraining, Sampling, Labeling, and Aggregation. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 29:3292-3306, 2021b.
  34. Alicja Gosiewska, Katarzyna Woznica, and Przemyslaw Biecek. Interpretable Meta-Measure for Model Performance. CoRR arXiv , 2020. URL http://arxiv.org/abs/2006.02293 . · arXiv
  35. Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and Benchmarking Self-Supervised Visual Representation Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6390-6399, 2019. URL https://github.com/facebookresearch/ .
  36. Priya Goyal, Quentin Duval, Isaac Seessel, Mathilde Caron, Ishan Misra, Levent Sagun, Armand Joulin, and Piotr Bojanowski. Vision models are more robust and fair when pretrained on uncurated images without supervision. CoRR arXiv , 2022. URL https://arxiv.org/abs/2202.08360 . · arXiv
  37. Jean-Bastien Grill, Florian Strub, Florent Altch´ e, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R´ emi Munos, and Michal Valko. Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and HsuanTien Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 21271-21284, 2020.
  38. Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, and Douglas Eck. Onsets and Frames: Dual-Objective Piano Transcription. In Emilia G´ omez, Xiao Hu, Eric Humphrey, and Emmanouil Benetos, editors, Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR) , pages 50-57, 2018.
  39. Music Modeling and Generation with the MAESTRO Dataset. In Proceedings of the 7th International Conference on Learning Representations (ICLR) , 2019.
  40. Shawn Hershey, Sourish Chaudhuri, Daniel P W Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, Malcolm Slaney, Ron J Weiss, and Kevin Wilson. CNN Architectures for Large-Scale Audio Classification. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 131-135, 2017.
  41. Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. CoRR arXiv , 2017. URL http://arxiv.org/abs/1704.04861 . · arXiv
  42. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 29:3451-3460, 2021.
  43. Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning (ICML) , pages 2410-2419. PMLR, 2018.
  44. Rainer Kelz, Matthias Dorfer, Filip Korzeniowski, Sebastian B¨ ock, Andreas Arzt, and Gerhard Widmer. On the Potential of Simple Framewise Approaches to Piano Transcription. In Michael I. Mandel, Johanna Devaney, Douglas Turnbull, and George Tzanetakis, editors, Proceedings of the 17th International Society for Music Information Retrieval Conference (ISMIR) , pages 475-481, 2016.
  45. Bongjun Kim, Madhav Ghei, Bryan Pardo, and Zhiyao Duan. Vocal imitation set: a dataset of vocally imitated sound events using the AudioSet ontology. In Mark D. Plumbley, Christian Kroos, Juan P. Bello, Ga¨ el Richard, Daniel P. W. Ellis, and Annamaria Mesaros, editors, Proceedings of the Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE) , volume 1, pages 148-152, 2018a.
  46. Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello. Crepe: A Convolutional Representation for Pitch Estimation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 161-165, 2018b.
  47. Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the 9th International Conference on Learning Representations (ICLR) , 2021.
  48. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 28:2880-2894, 2020.
  49. Khaled Koutini, Jan Schl¨ uter, Hamid Eghbal-zadeh, and Gerhard Widmer. Efficient Training of Audio Transformers with Patchout. CoRR arXiv , 2021. URL http://arxiv.org/abs/2110.05069 . · arXiv
  50. Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. On Generative Spoken Language Modeling from Raw Audio. Transactions of the Association for Computational Linguistics , 9:1336-1354, 2021.
  51. Hanxiao Liu, Zihang Dai, David So, and Quoc Le. Pay Attention to MLPs. In Advances in Neural Information Processing Systems , volume 34, 2021.
  52. Beth Logan. Mel frequency cepstral coefficients for music modeling. In Proceedings of the 1st International Society for Music Information Retrieval Conference (ISMIR) , Plymouth, Mass, USA, 2000.
  53. Soroush Merhi, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron C. Courville, and Yoshua Bengio. SampleRNN: An Unconditional End-toEnd Neural Audio Generation Model. In Proceedings of the 5th International Conference on Learning Representations (ICLR) , 2017.
  54. Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. Metrics for polyphonic sound event detection. Applied Sciences , 6(6), 2016.
  55. Annamaria Mesaros, Toni Heittola, Emmanouil Benetos, Peter Foster, Mathieu Lagrange, Tuomas Virtanen, and Mark D Plumbley. Detection and classification of acoustic scenes and events: Outcome of the dcase 2016 challenge. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 26(2):379-393, 2017.
  56. Annamaria Mesaros, Toni Heittola, Emmanouil Benetos, Peter Foster, Mathieu Lagrange, Tuomas Virtanen, and Mark D Plumbley. Detection and Classification of Acoustic Scenes and Events: Outcome of the DCASE 2016 Challenge. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 26(2):379-393, 2018.
  57. Mashrur M Morshed, Ahmad Omar Ahsan, Hasan Mahmud, and Md. Kamrul Hasan. Learning Audio Representations With MLPs. In Joseph Turian, Bj¨ orn W. Schuller, Dorien Herremans, Katrin Kirchhoff, Paola Garcia Perera, and Philippe Esling, editors, Proceedings of HEAR: Holistic Evaluation of Audio Representations , volume 166 of Proceedings of Machine Learning Research . PMLR, 2022. In submission to HEAR-2021.
  58. Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation. In International Joint Conference on Neural Networks (IJCNN) , pages 1-8, 2021.
  59. Inˆ es Nolasco, Alessandro Terenzi, Stefania Cecchi, Simone Orcioni, Helen L Bear, and Emmanouil Benetos. Audio-based Identification of Beehive States. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 8256-8260, 2019.

Cited by

Sorted by citations

Authors
Turian, J., Shier, J., Khan, H. R., Raj, B., Schuller, B. W., Steinmetz, C. J., Malloy, C., Tzanetakis, G., Velarde, G., McNally, K., Henry, M., Pinto, N., Noufi, C., Clough, C., Herremans, D., Fonseca, E., Engel, J., Salamon, J., Esling, P., Manocha, P., Watanabe, S., Jin, Z., & Bisk, Y.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Volume
176
Pages
125-145
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2203.03022
arXiv
2203.03022
OpenAlex
W4221150524
Cited by
56
References
0
Citations by year
2025: 15 · 2024: 13 · 2023: 21 · 2022: 7
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
Keywords
sound representation, audio embeddings, benchmark, domain generalization, speech, environmental sounds, downstream tasks, open-source models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review