PreprintCited by 13Music and Audio ProcessingSpeech Recognition and SynthesisSpeech and Audio Processing
The DOI matches its DataCite record.
Effective Pre-Training of Audio Transformers for Sound Event Detection
Open accessSchmid, Florian, Morocutti, Tobias, Foscarin, Francesco, Schlüter, Jan, Primus, Paul, Widmer, Gerhard·2024·arXiv
DOI 10.48550/arxiv.2409.09546arXiv 2409.09546
Abstract
We propose a pre-training pipeline for audio spectrogram transformers for frame-level sound event detection tasks. On top of common pre-training steps, we add a meticulously designed training routine on AudioSet frame-level annotations. This includes a balanced sampler, aggressive data augmentation, and ensemble knowledge distillation. For five transformers, we obtain a substantial performance improvement over previously available checkpoints both on AudioSet frame-level predictions and on frame-level sound event detection downstream tasks, confirming our pipeline's effectiveness. We publish the resulting checkpoints that researchers can directly fine-tune to build high-performance models for sound event detection tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
DataCite · 10.48550/arxiv.2409.09546
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“We first run a frame-level sound event detection model on each audio file to obtain timestamped event labels under the AudioSet taxonomy, using a BEATs [24] backbone trained within the PretrainedSED framework [25].”
From MOSS-Audio Technical Report (2026) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 39
- Y. Gong, Y. Chung, and J. R. Glass, "AST: audio spectrogram transformer," in Proceedings of the Interspeech Conference, 2021.
- K. Koutini, J. Schlüter, H. Eghbal-zadeh, and G. Widmer, "Efficient training of audio transformers with patchout," in Proceedings of the Interspeech Conference, 2022.
- K. Chen, X. Du, B. Zhu, Z. Ma, T. Berg-Kirkpatrick, and S. Dubnov, "HTS-AT: A hierarchical token-semantic audio transformer for sound clas- sification and detection," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
- X. Liu, H. Lu, J. Yuan, and X. Li, "CAT: causal audio transformer for audio classification," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
- Y. Gong, S. Khurana, A. Rouditchenko, and J. R. Glass, "CMKD: cnn/transformer-based cross-model knowledge distillation for audio classification," CoRR, vol. abs/2203.06760, 2022, unpublished. · arXiv
- H. Dinkel, Y. Wang, Z. Yan, J. Zhang, and Y. Wang, "CED: consistent ensemble distillation for audio tagging," in Proceedings of the Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024.
- S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei, "BEATs: Audio pre-training with acoustic tokenizers," in Proceedings of the International Conference on Machine Learning (ICML), 2023.
- D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada, and K. Kashino, "Masked modeling duo: Towards a universal audio pre-training framework," Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 2391-2406, 2024.
- D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada, M. Yasuda, S. Tsubaki, and K. Imoto, "M2D-CLAP: masked modeling duo meets CLAP for learning general-purpose audio-language representation," in Proceedings of the Interspeech Conference, 2024.
- P. Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer, "Masked autoencoders that listen," in Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2022.
- S. Atito, M. Awais, W. Wang, M. D. Plumbley, and J. Kittler, "ASiT: Local-global audio spectrogram vision transformer for event classifi- cation," IEEE ACM Trans. Audio Speech Lang. Process., vol. 32, pp. 3684-3693, 2024.
- Y. Gong, C. Lai, Y. Chung, and J. R. Glass, "SSAST: self-supervised audio spectrogram transformer," in Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
- D. Chong, H. Wang, P. Zhou, and Q. Zeng, "Masked spectrogram prediction for self-supervised audio pre-training," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
- X. Li, N. Shao, and X. Li, "Self-supervised audio teacher-student transformer for both clip-level and frame-level tasks," Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 1336-1351, 2024.
- N. Shao, X. Li, and X. Li, "Fine-tune the pretrained ATST model for sound event detection," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024.
- F. Schmid, P. Primus, T. Morocutti, J. Greif, and G. Widmer, "Multi- iteration multi-stage fine-tuning of transformers for sound event detection with heterogeneous datasets," in Workshop on Detection and Classifica- tion of Acoustic Scenes and Events (DCASE), 2024, in press.
- J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "Audio Set: An ontology and human-labeled dataset for audio events," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.
- S. Hershey, D. P. Ellis, E. Fonseca, A. Jansen, C. Liu, R. C. Moore, and M. Plakal, "The benefit of temporally-strong labels in audio event classification," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
- Y. Li, Z. Guo, X. Wang, and H. Liu, "Advancing multi-grained alignment for contrastive language-audio pre-training," in ACM Multimedia, 2024.
- H. Dinkel, Z. Yan, Y. Wang, J. Zhang, and Y. Wang, "Streaming audio transformers for online audio tagging," in Proceedings of the Interspeech Conference, 2024.
- X. Xu, M. Wu, and K. Yu, "Investigating pooling strategies and loss functions for weakly-supervised text-to-audio grounding via contrastive learning," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
- Z. Xie, X. Xu, M. Wu, and K. Yu, "Enhance temporal relations in audio captioning with sound event detection," in Proceedings of the Interspeech Conference, 2023.
- Y. Zhang, Y. Gu, Y. Zeng, Z. Xing, Y. Wang, Z. Wu, and K. Chen, "FoleyCrafter: Bring silent videos to life with lifelike and synchronized sounds," CoRR, vol. abs/2407.01494, 2024, unpublished. · arXiv
- Z. Allen-Zhu and Y. Li, "Towards understanding ensemble, knowledge distillation and self-distillation in deep learning," in Proceedings of the International Conference on Learning Representations (ICLR) , 2023.
- C ¸. Bilen, G. Ferroni, F. Tuveri, J. Azcarreta, and S. Krstulović, "A framework for the robust evaluation of sound event detection," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020.
- J. Ebbers, R. Haeb-Umbach, and R. Serizel, "Threshold independent evaluation of sound event detection scores," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
- S. Cornell, J. Ebbers, C. Douwes, I. Martín-Morató, M. Harju, A. Mesaros, and R. Serizel, "DCASE 2024 task 4: Sound event detection with heterogeneous data and missing labels," in Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2024, in press.
- A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An image is worth 16x16 words: Transformers for image recognition at scale," in Proceedings of the International Conference on Learning Representations (ICLR) , 2021.
- H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, "Training data-efficient image transformers & distillation through attention," in Proceedings of the International Conference on Machine Learning (ICML), 2021.
- F. Schmid, K. Koutini, and G. Widmer, "Efficient large-scale audio tagging via transformer-to-CNN knowledge distillation," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
- D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in Proceedings of the International Conference on Learning Representations (ICLR) , 2015.
- R. Dror, S. Shlomov, and R. Reichart, "Deep dominance -how to properly compare deep neural models," in Proceedings of the Conference of the Association for Computational Linguistics (ACL), 2019.
- H. Nam, S. Kim, and Y. Park, "Filteraugment: An acoustic environ- mental data augmentation method," in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
- B. Kim, S. Yang, J. Kim, H. Park, J. Lee, and S. Chang, "Domain generalization with relaxed instance frequency-wise normalization for multi-device acoustic scene classification," in Proceedings of the Inter- speech Conference, 2022.
- F. Schmid, S. Masoudian, K. Koutini, and G. Widmer, "Knowledge distil- lation from transformers for low-complexity acoustic scene classification," in Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2022.
- H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz, "mixup: Beyond empirical risk minimization," in Proceedings of the International Conference on Learning Representations (ICLR) , 2018.
- Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, "PANNs: Large-scale pretrained audio neural networks for audio pattern recognition," Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2880-2894, 2020.
- N. Turpault, R. Serizel, A. P. Shah, and J. Salamon, "Sound event detection in domestic environments with weakly labeled data and soundscape synthesis," in Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2019.
- J. Turian, J. Shier, H. R. Khan, B. Raj, B. W. Schuller, C. J. Steinmetz, C. Malloy, G. Tzanetakis, G. Velarde, K. McNally, M. Henry, N. Pinto, C. Noufi, C. Clough, D. Herremans, E. Fonseca, J. H. Engel, J. Salamon, P. Esling, P. Manocha, S. Watanabe, Z. Jin, and Y. Bisk, "HEAR: holistic evaluation of audio representations," in NeurIPS Competitions and Demonstrations Track, 2021.
Cited by
Sorted by citations
Yang, C., Yu, C., Chen, H., Zhu, J., Chen, J., Chen, K., Wang, W., Wang, Y., Jiang, Y., Jiang, Y., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.01802
Metadata
- Authors
- Schmid, F., Morocutti, T., Foscarin, F., Schlüter, J., Primus, P., & Widmer, G.
- Affiliations
- Johannes Kepler University of Linz (AT)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 1-5
- ISSN
- 1520-6149 (ISSN-L)
- arXiv
- 2409.09546
- OpenAlex
- W4408353596
- Cited by
- 13
- References
- 0
- Citations by year
- 2026: 7 · 2025: 6
- FWCI
- 12.25
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- European Research Council
- Topics
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- sound event detection, audio transformers, pre-training, AudioSet, frame-level annotation, data augmentation, balanced sampling, knowledge distillation, ensemble learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
