HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection
Open accessAbstract
Audio classification is an important task of mapping audio samples into their corresponding labels. Recently, the transformer model with self-attention mechanisms has been adopted in this field. However, existing audio transformers require large GPU memories and long training time, meanwhile relying on pretrained vision models to achieve high performance, which limits the model’s scalability in audio tasks. To combat these problems, we introduce HTS-AT: an audio transformer with a hierarchical structure to reduce the model size and training time. It is further combined with a token-semantic module to map final outputs into class featuremaps, thus enabling the model for the audio event detection (i.e. localization in time). We evaluate HTS-AT on three datasets of audio classification where it achieves new state-of-the-art (SOTA) results on AudioSet and ESC50, and equals the SOTA on Speech Command V2. It also achieves better performance in event localization than the previous CNN-based models. Moreover, HTS-AT requires only 35% model parameters and 15% training time of the previous audio transformer. These results demonstrate the high performance and high efficiency of HTS-AT.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
HTSAT achieves SOTAs on three audio classification datasets.
“HTSAT is a transformer-based model with 4 groups of swintransformer blocks [20], which achieves SOTAs on three audio classification datasets.”
From Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation (2023) · cites this paper as [12]
Cites
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · see citing works
mixup: Beyond Empirical Risk Minimization
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017) · arXiv · doi:10.48550/arxiv.1710.09412 · Cited by 5,831 · see citing works
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., & Le, Q. V. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2680 · Cited by 3,558 · see citing works
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · see citing works
ESC: Dataset for Environmental Sound Classification
Piczak, K. J. (2015) · ACM Multimedia · doi:10.1145/2733373.2806390 · Cited by 2,195 · see citing works
PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition
Kong, Q., Cao, Y., Iqbal, T., Wang, Y., Wang, W., & Plumbley, M. D. (2020) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2020.3030497 · Cited by 1,235 · see citing works
AST: Audio Spectrogram Transformer
Gong, Y., Chung, Y.-A., & Glass, J. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-698 · Cited by 1,159 · see citing works
Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
Warden, P. (2018) · arXiv · doi:10.48550/arxiv.1804.03209 · Cited by 1,033 · see citing works
Sound Event Detection: A tutorial
Mesaros, A., Heittola, T., Virtanen, T. I., & Plumbley, M. D. (2021) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3090678 · Cited by 274 · see citing works
TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization
Gao, W., Wan, F., Pan, X., Peng, Z., Tian, Q. C., Han, Z., Zhou, B., & Ye, Q. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00288 · Cited by 236 · see citing works
Automatic Classification of Musical Instrument Sounds
Herrera, P., Peeters, G. G., & Dubnov, S. (2003) · Journal of New Music Research · doi:10.1076/jnmr.32.1.3.16798 · Cited by 179 · see citing works
Self-citationWang, Y., Li, J., & Metze, F. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682847 · Cited by 164 · see citing works
PSLA: Improving Audio Tagging With Pretraining, Sampling, Labeling, and Aggregation
Gong, Y., Chung, Y.-A., & Glass, J. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3120633 · Cited by 164 · see citing works
Keyword Transformer: A Self-Attention Model for Keyword Spotting
Berg, A., O’Connor, M., & Cruz, M. T. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1286 · Cited by 151 · see citing works
Sound Event Detection in Synthetic Domestic Environments
Serizel, R., Turpault, N., Shah, A., et al. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054478 · Cited by 99 · see citing works
The Benefit of Temporally-Strong Labels in Audio Event Classification
Hershey, S., Ellis, D. P. W., Fonseca, E., Jansen, A., Liu, C., Moore, R. C., & Plakal, M. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414579 · Cited by 97 · see citing works
Learning Efficient Representations for Keyword Spotting with Triplet Loss
Vygon, R., & Mikhaylovskiy, N. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87802-3_69 · Cited by 55 · see citing works
A Deep Residual Network for Large-Scale Acoustic Scene Analysis
Ford, L., Tang, H., Grondin, F., & Glass, J. R. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2731 · Cited by 51 · see citing works
Learning Audio Embeddings with User Listening Data for Content-Based Music Recommendation
Chen, K., Liang, B., Ma, X., et al. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414458 · Cited by 40 · see citing works
Self-citationZero-Shot Audio Source Separation through Query-Based Learning from Weakly-Labeled Data
Chen, K., Du, X., Zhu, B., Ma, Z., Berg-Kirkpatrick, T., & Dubnov, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i4.20366 · Cited by 32 · see citing works
Self-citationMusic SketchNet: Controllable Music Generation via Factorized Representations of Pitch and RhythmPreprint
Chen, K., Wang, C.-I., Berg-Kirkpatrick, T., & Dubnov, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.01291 · Cited by 27 · see citing works
Self-citationMusPy: A Toolkit for Symbolic Music GenerationPreprint
Dong, H., Chen, K., McAuley, J., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.01951 · Cited by 24 · see citing works
Self-citationContinuous Melody Generation via Disentangled Short-Term Representations and Structural Conditions
Chen, K., Xia, G., & Dubnov, S. (2020) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc.2020.00025 · Cited by 16 · see citing works
Self-citationAutomatic classification of musical instrument sounds
Kostek, B. (2000) · The Journal of the Acoustical Society of America · doi:10.1121/1.429090 · Cited by 12 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
ImageBind One Embedding Space to Bind Them All
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01457 · Cited by 863 · see citing works
Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., & Dubnov, S. (2023) · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49357.2023.10095969 · Cited by 459 · see citing works
Self-citationMei, X., Meng, C., Liu, H., Kong, Q., Ko, T., Zhao, C., Plumbley, M. D., Zou, Y., & Wang, W. (2024) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2024.3419446 · Cited by 150 · see citing works
Sparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17 · see citing works
“This model harnesses the capabilities of an audio transformer HTSAT [243] as an audio encoder, along with CLIP’s [244] text encoder and GPT2-base as a language model.”
MethodBootstrapping Language-Audio Pre-training for Music Captioning
Lanzendörfer, L. A., Pinkl, C., Perraudin, N., & Wattenhofer, R. (2025) · ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49660.2025.10887618 · Cited by 3 · see citing works
Joint Audio and Speech Understanding
Gong, Y., Liu, A. H., Luo, H., Karlinsky, L., & Glass, J. (2023) · arXiv · doi:10.48550/arxiv.2309.14405
“The TLTR and LoRA adapters are unfrozen in the second and third stages to stabilize training.”
Background
- Authors
- Chen, K., Du, X., Zhu, B., Ma, Z., Berg-Kirkpatrick, T., & Dubnov, S.
- Affiliations
- University of California San Diego (US)
- ByteDance (China) (CN)
- Venue
- ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2022-05-23
- Pages
- 646-650
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W4226442948
- Cited by
- 294
- References
- 27
- Citations by year
- 2026: 51 · 2025: 99 · 2024: 94 · 2023: 44 · 2022: 6
- FWCI
- 43.20
- Type
- conference paper
- Language
- en
- Open access
- green
- hal.science
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- HAL (submittedVersion, other-oa)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- audio transformers, audio classification, sound event detection, hierarchical model structure, self-attention, model efficiency, parameter reduction, training time reduction, AudioSet, voice commands
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
