CoChat
Sign Up
PreprintCited by 13Music and Audio ProcessingSpeech Recognition and SynthesisSpeech and Audio Processing

The DOI matches its DataCite record.

Effective Pre-Training of Audio Transformers for Sound Event Detection

Open access
Schmid, Florian, Morocutti, Tobias, Foscarin, Francesco, Schlüter, Jan, Primus, Paul, Widmer, Gerhard·2024·arXiv
DOI 10.48550/arxiv.2409.09546arXiv 2409.09546
Open this paper in CoChat — free accountOpen PDF

Abstract

We propose a pre-training pipeline for audio spectrogram transformers for frame-level sound event detection tasks. On top of common pre-training steps, we add a meticulously designed training routine on AudioSet frame-level annotations. This includes a balanced sampler, aggressive data augmentation, and ensemble knowledge distillation. For five transformers, we obtain a substantial performance improvement over previously available checkpoints both on AudioSet frame-level predictions and on frame-level sound event detection downstream tasks, confirming our pipeline's effectiveness. We publish the resulting checkpoints that researchers can directly fine-tune to build high-performance models for sound event detection tasks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2409.09546

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 39

  1. Y. Gong, Y. Chung, and J. R. Glass, "AST: audio spectrogram transformer," in Proceedings of the Interspeech Conference, 2021.
  2. K. Koutini, J. Schlüter, H. Eghbal-zadeh, and G. Widmer, "Efficient training of audio transformers with patchout," in Proceedings of the Interspeech Conference, 2022.
  3. K. Chen, X. Du, B. Zhu, Z. Ma, T. Berg-Kirkpatrick, and S. Dubnov, "HTS-AT: A hierarchical token-semantic audio transformer for sound clas- sification and detection," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
  4. X. Liu, H. Lu, J. Yuan, and X. Li, "CAT: causal audio transformer for audio classification," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
  5. Y. Gong, S. Khurana, A. Rouditchenko, and J. R. Glass, "CMKD: cnn/transformer-based cross-model knowledge distillation for audio classification," CoRR, vol. abs/2203.06760, 2022, unpublished. · arXiv
  6. H. Dinkel, Y. Wang, Z. Yan, J. Zhang, and Y. Wang, "CED: consistent ensemble distillation for audio tagging," in Proceedings of the Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024.
  7. S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei, "BEATs: Audio pre-training with acoustic tokenizers," in Proceedings of the International Conference on Machine Learning (ICML), 2023.
  8. D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada, and K. Kashino, "Masked modeling duo: Towards a universal audio pre-training framework," Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 2391-2406, 2024.
  9. D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada, M. Yasuda, S. Tsubaki, and K. Imoto, "M2D-CLAP: masked modeling duo meets CLAP for learning general-purpose audio-language representation," in Proceedings of the Interspeech Conference, 2024.
  10. P. Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer, "Masked autoencoders that listen," in Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2022.
  11. S. Atito, M. Awais, W. Wang, M. D. Plumbley, and J. Kittler, "ASiT: Local-global audio spectrogram vision transformer for event classifi- cation," IEEE ACM Trans. Audio Speech Lang. Process., vol. 32, pp. 3684-3693, 2024.
  12. Y. Gong, C. Lai, Y. Chung, and J. R. Glass, "SSAST: self-supervised audio spectrogram transformer," in Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
  13. D. Chong, H. Wang, P. Zhou, and Q. Zeng, "Masked spectrogram prediction for self-supervised audio pre-training," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
  14. X. Li, N. Shao, and X. Li, "Self-supervised audio teacher-student transformer for both clip-level and frame-level tasks," Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 1336-1351, 2024.
  15. N. Shao, X. Li, and X. Li, "Fine-tune the pretrained ATST model for sound event detection," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024.
  16. F. Schmid, P. Primus, T. Morocutti, J. Greif, and G. Widmer, "Multi- iteration multi-stage fine-tuning of transformers for sound event detection with heterogeneous datasets," in Workshop on Detection and Classifica- tion of Acoustic Scenes and Events (DCASE), 2024, in press.
  17. J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "Audio Set: An ontology and human-labeled dataset for audio events," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.
  18. S. Hershey, D. P. Ellis, E. Fonseca, A. Jansen, C. Liu, R. C. Moore, and M. Plakal, "The benefit of temporally-strong labels in audio event classification," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
  19. Y. Li, Z. Guo, X. Wang, and H. Liu, "Advancing multi-grained alignment for contrastive language-audio pre-training," in ACM Multimedia, 2024.
  20. H. Dinkel, Z. Yan, Y. Wang, J. Zhang, and Y. Wang, "Streaming audio transformers for online audio tagging," in Proceedings of the Interspeech Conference, 2024.
  21. X. Xu, M. Wu, and K. Yu, "Investigating pooling strategies and loss functions for weakly-supervised text-to-audio grounding via contrastive learning," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
  22. Z. Xie, X. Xu, M. Wu, and K. Yu, "Enhance temporal relations in audio captioning with sound event detection," in Proceedings of the Interspeech Conference, 2023.
  23. Y. Zhang, Y. Gu, Y. Zeng, Z. Xing, Y. Wang, Z. Wu, and K. Chen, "FoleyCrafter: Bring silent videos to life with lifelike and synchronized sounds," CoRR, vol. abs/2407.01494, 2024, unpublished. · arXiv
  24. Z. Allen-Zhu and Y. Li, "Towards understanding ensemble, knowledge distillation and self-distillation in deep learning," in Proceedings of the International Conference on Learning Representations (ICLR) , 2023.
  25. C ¸. Bilen, G. Ferroni, F. Tuveri, J. Azcarreta, and S. Krstulović, "A framework for the robust evaluation of sound event detection," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020.
  26. J. Ebbers, R. Haeb-Umbach, and R. Serizel, "Threshold independent evaluation of sound event detection scores," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
  27. S. Cornell, J. Ebbers, C. Douwes, I. Martín-Morató, M. Harju, A. Mesaros, and R. Serizel, "DCASE 2024 task 4: Sound event detection with heterogeneous data and missing labels," in Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2024, in press.
  28. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An image is worth 16x16 words: Transformers for image recognition at scale," in Proceedings of the International Conference on Learning Representations (ICLR) , 2021.
  29. H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, "Training data-efficient image transformers & distillation through attention," in Proceedings of the International Conference on Machine Learning (ICML), 2021.
  30. F. Schmid, K. Koutini, and G. Widmer, "Efficient large-scale audio tagging via transformer-to-CNN knowledge distillation," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
  31. D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in Proceedings of the International Conference on Learning Representations (ICLR) , 2015.
  32. R. Dror, S. Shlomov, and R. Reichart, "Deep dominance -how to properly compare deep neural models," in Proceedings of the Conference of the Association for Computational Linguistics (ACL), 2019.
  33. H. Nam, S. Kim, and Y. Park, "Filteraugment: An acoustic environ- mental data augmentation method," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
  34. B. Kim, S. Yang, J. Kim, H. Park, J. Lee, and S. Chang, "Domain generalization with relaxed instance frequency-wise normalization for multi-device acoustic scene classification," in Proceedings of the Inter- speech Conference, 2022.
  35. F. Schmid, S. Masoudian, K. Koutini, and G. Widmer, "Knowledge distil- lation from transformers for low-complexity acoustic scene classification," in Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2022.
  36. H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz, "mixup: Beyond empirical risk minimization," in Proceedings of the International Conference on Learning Representations (ICLR) , 2018.
  37. Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, "PANNs: Large-scale pretrained audio neural networks for audio pattern recognition," Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2880-2894, 2020.
  38. N. Turpault, R. Serizel, A. P. Shah, and J. Salamon, "Sound event detection in domestic environments with weakly labeled data and soundscape synthesis," in Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2019.
  39. J. Turian, J. Shier, H. R. Khan, B. Raj, B. W. Schuller, C. J. Steinmetz, C. Malloy, G. Tzanetakis, G. Velarde, K. McNally, M. Henry, N. Pinto, C. Noufi, C. Clough, D. Herremans, E. Fonseca, J. H. Engel, J. Salamon, P. Esling, P. Manocha, S. Watanabe, Z. Jin, and Y. Bisk, "HEAR: holistic evaluation of audio representations," in NeurIPS Competitions and Demonstrations Track, 2021.

Cited by

Sorted by citations

See all 13

Metadata

Authors
Schmid, F., Morocutti, T., Foscarin, F., Schlüter, J., Primus, P., & Widmer, G.
Affiliations
Johannes Kepler University of Linz (AT)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
1-5
ISSN
1520-6149 (ISSN-L)
DOI
10.48550/arxiv.2409.09546
arXiv
2409.09546
OpenAlex
W4408353596
Cited by
13
References
0
Citations by year
2026: 7 · 2025: 6
FWCI
12.25
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
European Research Council
Topics
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
Keywords
sound event detection, audio transformers, pre-training, AudioSet, frame-level annotation, data augmentation, balanced sampling, knowledge distillation, ensemble learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Effective Pre-Training of Audio Transformers for Sound Event Detection. https://app.cochat.ai/papers/10.48550/arxiv.2409.09546
The paper
Schmid, F., Morocutti, T., Foscarin, F., Schlüter, J., Primus, P., & Widmer, G. (2024). Effective Pre-Training of Audio Transformers for Sound Event Detection. arXiv. https://doi.org/10.48550/arxiv.2409.09546