CoChat
Sign Up
Cited by 164Music and Audio ProcessingSpeech Recognition and SynthesisDomain Adaptation and Few-Shot Learning

PSLA: Improving Audio Tagging With Pretraining, Sampling, Labeling, and Aggregation

Open access
Gong, Yuan, Chung, Yu-An, Glass, James·2021·IEEE/ACM Transactions on Audio, Speech, and Language Processing
DOI 10.1109/taslp.2021.3120633
Open this paper in CoChat — free accountOpen PDF

Abstract

Audio tagging is an active research area and has a wide range of applications. Since the release of AudioSet, great progress has been made in advancing model performance, which mostly comes from the development of novel model architectures and attention modules. However, we find that appropriate training techniques are equally important for building audio tagging models with AudioSet, but have not received the attention they deserve. To fill the gap, in this work, we present PSLA, a collection of training techniques that can noticeably boost the model accuracy including ImageNet pretraining, balanced sampling, data augmentation, label enhancement, model aggregation and their design choices. By training an EfficientNet with these techniques, we obtain a single model (with 13.6M parameters) and an ensemble model that achieve mean average precision (mAP) scores of 0.444 and 0.474 on AudioSet, respectively, outperforming the previous best system of 0.439 with 81M parameters. In addition, our model also achieves a new state-of-the-art mAP of 0.567 on FSD50K.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/taslp.2021.3120633

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing datasets for human vocal sounds have a relatively small number of vocal sound samples or noisy labels.

    “However, existing datasets have a relatively small number of vocal sound samples or noisy labels.”

    From Vocalsound: A Dataset for Improving Human Vocal Sounds Recognition (2022) · cites this paper as [6]

    Section V, “Label Enhancement” (pp. 7–8), describes noisy and missing labels in AudioSet, including the male, female, and child speech classes. It does not establish that existing human-vocal-sound datasets generally have few vocal samples; for those AudioSet speech classes, it specifically suggests that poor performance is not due to a small number of samples. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • State-of-the-art audio event classification models may not perform well in detecting human vocal sounds.

    “As a consequence, state-of-the-art audio event classification models may not perform well in detecting human vocal sounds.”

    From Vocalsound: A Dataset for Improving Human Vocal Sounds Recognition (2022) · cites this paper as [6]

    In V. Label Enhancement (p. 7), the paper reports low performance for male, female, and child speech classes and says class-specific performance patterns are similar across model architectures. But the Fig. 7 caption (p. 7) says the broader “Speech” class has much higher average precision, so the evidence does not extend to human vocal sounds generally. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • A balanced sampling strategy was used to make the model see roughly the same number of samples of each class during training.

    “Therefore, we use a balanced sampling strategy [6] to make the model see roughly the same number of samples of each class during training”

    From Vocalsound: A Dataset for Improving Human Vocal Sounds Recognition (2022) · cites this paper as [6]

    The Balanced Sampling section (p. 5) describes weighting samples so rare sound events are more likely to be seen during training. It says low- and high-frequency class samples are roughly equally seen, but also states that the sampled frequency of each class remains imbalanced. Thus, the source supports the strategy’s intent, not the claim that each class is seen roughly equally often. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • The FSD50K evaluation set has more accurate labels while AudioSet labels are relatively noisy.

    “The reason why we use FSD50K rather than AudioSet as the base dataset is because FSD50K, especially its evaluation set, has more accurate labels [8] while labels of AudioSet are relatively noisy [6, 9, 10].”

    From Vocalsound: A Dataset for Improving Human Vocal Sounds Recognition (2022) · cites this paper as [6]

  • Noisy label issue has been found in AudioSet.

    “Due to the difficulty of annotating YouTube videos with a large sound vocabulary, the noisy label issue has been found in AudioSet [6, 9, 10]”

    From Vocalsound: A Dataset for Improving Human Vocal Sounds Recognition (2022) · cites this paper as [6]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 56

  1. [1][1] K. J. Piczak, 'Esc: Dataset for environmental sound classification,' in Proceedings of the 23rd ACM international conference on Multimedia , 2015, pp. 1015-1018. -
  2. [2][2] P. Foster, S. Sigtia, S. Krstulovic, J. Barker, and M. D. Plumbley, 'Chime-home: A dataset for sound source recognition in a domestic environment,' in 2015 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) , 2015, pp. 1-5. -
  3. [3][3] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, 'Audio set: An ontology and humanlabeled dataset for audio events,' in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2017, pp. 776-780. -
  4. [4][4] L. Ford, H. Tang, F. Grondin, and J. R. Glass, 'A deep residual network for large-scale acoustic scene analysis.' in INTERSPEECH , 2019, pp. 2568-2572. -
  5. [5][5] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, 'Panns: Large-scale pretrained audio neural networks for audio pattern recognition,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 28, pp. 2880-2894, 2020. -
  6. [6][6] Q. Kong, Y. Xu, W. Wang, and M. D. Plumbley, 'Audio set classification with attention model: A probabilistic perspective,' in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2018, pp. 316-320. -
  7. [7][7] C. Yu, K. S. Barsim, Q. Kong, and B. Yang, 'Multi-level attention model for weakly supervised audio classification,' in DCASE Workshop , 2018. -
  8. [8][8] Q. Kong, C. Yu, Y. Xu, T. Iqbal, W. Wang, and M. D. Plumbley, 'Weakly labelled audioset tagging with attention neural networks,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 27, no. 11, pp. 1791-1802, 2019. -
  9. [9][9] S. Chen, J. Chen, Q. Jin, and A. Hauptmann, 'Class-aware self-attention for audio event recognition,' in Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval , 2018, pp. 28-36. -
  10. [11][11] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, 'Fsd50k: an open dataset of human-labeled sound events,' arXiv preprint arXiv:2010.00475 , 2020. - · arXiv
  11. [12][12] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 'Imagenet: A large-scale hierarchical image database,' in 2009 IEEE conference on computer vision and pattern recognition , 2009, pp. 248-255. -
  12. [13][13] K. Palanisamy, D. Singhania, and A. Yao, 'Rethinking cnn models for audio classification,' arXiv preprint arXiv:2007.11154 , 2020. - · arXiv
  13. [15][15] D. Lee, S. Lee, Y. Han, and K. Lee, 'Ensemble of convolutional neural networks for weakly-supervised sound event detection using multiple scale input,' in DCASE Workshop , 2017. -
  14. [16][16] P. Lopez-Meyer, J. A. del Hoyo Ontiveros, G. Stemmer, L. Nachman, and J. Huang, 'Ensemble of convolutional neural networks for the DCASE 2020 acoustic scene classification challenge,' in DCASE Workshop , 2020. -
  15. [17][17] D. P. Kingma and J. Ba, 'Adam: A method for stochastic optimization,' in International Conference on Learning Representations , 2015. -
  16. [18][18] A. P. Bradley, 'The use of the area under the roc curve in the evaluation of machine learning algorithms,' Pattern recognition , vol. 30, no. 7, pp. 1145-1159, 1997. -
  17. [19][19] J. Davis and M. Goadrich, 'The relationship between precision-recall and roc curves,' in Proceedings of the 23rd international conference on Machine learning , 2006. -
  18. [21][21] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, 'Mobilenetv2: Inverted residuals and linear bottlenecks,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 4510-4520. -
  19. [24][24] K. He, R. Girshick, and P. Doll´ ar, 'Rethinking imagenet pre-training,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 4918-4927. -
  20. [25][25] Y.-A. Chung, W.-N. Hsu, H. Tang, and J. Glass, 'An unsupervised autoregressive model for speech representation learning,' Interspeech , pp. 146-150, 2019. -
  21. [26][26] A. Saeed, D. Grangier, and N. Zeghidour, 'Contrastive learning of general-purpose audio representations,' arXiv preprint arXiv:2010.10915 , 2020. - · arXiv
  22. [27][27] J. Shor, A. Jansen, R. Maor, O. Lang, O. Tuval, F. de Chaumont Quitry, M. Tagliasacchi, I. Shavitt, D. Emanuel, and Y. Haviv, 'Towards learning a universal non-semantic representation of speech,' Interspeech , pp. 140-144, 2020. -
  23. [28][28] M. Tagliasacchi, B. Gfeller, F. de Chaumont Quitry, and D. Roblek, 'Pre-training audio representations with self-supervision,' IEEE Signal Processing Letters , vol. 27, pp. 600-604, 2020. -
  24. [29][29] A. Jansen, M. Plakal, R. Pandya, D. P. Ellis, S. Hershey, J. Liu, R. C. Moore, and R. A. Saurous, 'Unsupervised learning of semantic audio representations,' in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2018, pp. 126-130. -
  25. [30][30] L. Wang, K. Kawakami, and A. van den Oord, 'Contrastive predictive coding of audio with an adversary,' Interspeech , pp. 826-830, 2020. -
  26. [31][31] J. Salamon, C. Jacoby, and J. P. Bello, 'A dataset and taxonomy for urban sound research,' in Proceedings of the 22nd ACM international conference on Multimedia , 2014, pp. 1041-1044. -
  27. [32][32] K. He, X. Zhang, S. Ren, and J. Sun, 'Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,' in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1026-1034. -
  28. [33][33] G. Gwardys and D. M. Grzywczak, 'Deep image features in music information retrieval,' International Journal of Electronics and Telecommunications , vol. 60, no. 4, pp. 321-326, 2014. -
  29. [34][34] A. Guzhov, F. Raue, J. Hees, and A. Dengel, 'Esresnet: Environmental sound classification based on visual domain models,' in 2020 25th International Conference on Pattern Recognition (ICPR) , 2021, pp. 4933-4940. -
  30. [35][35] S. Adapa, 'Urban sound tagging using convolutional neural networks,' in DCASE Workshop , 2019, p. 5. -
  31. [36][36] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, 'Going deeper with convolutions,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 1-9. -
  32. [37][37] H. He and E. A. Garcia, 'Learning from imbalanced data,' IEEE Transactions on knowledge and data engineering , vol. 21, no. 9, pp. 1263-1284, 2009. -
  33. [38][38] B. Efron and R. J. Tibshirani, An introduction to the bootstrap . CRC press, 1994. -
  34. [39][39] L. Breiman, 'Bagging predictors,' Machine Learning , vol. 24, no. 2, pp. 123-140, 1996. -
  35. [40][40] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, 'SpecAugment: A simple data augmentation method for automatic speech recognition,' Interspeech , pp. 2613-2617, 2019. -
  36. [41][41] Y. Tokozume, Y. Ushiku, and T. Harada, 'Between-class learning for image classification,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 5486-5494. -
  37. [42][42] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, 'mixup: Beyond empirical risk minimization,' in International Conference on Learning Representations , 2018. -
  38. [43][43] Y. Tokozume, Y. Ushiku, and T. Harada, 'Learning from between-class examples for deep sound recognition,' in International Conference on Learning Representations , 2018. -
  39. [44][44] A. Mesaros, T. Heittola, A. Diment, B. Elizalde, A. Shah, E. Vincent, B. Raj, and T. Virtanen, 'Dcase 2017 challenge setup: Tasks, datasets and baseline system,' in DCASE Workshop , 2017. -
  40. [45][45] A. Shah, A. Kumar, A. G. Hauptmann, and B. Raj, 'A closer look at weak label learning for audio events,' arXiv preprint arXiv:1804.09288 , 2018. - · arXiv
  41. [46][46] L. H. Ford, 'Large-scale acoustic scene analysis with deep residual networks,' Master's thesis, Massachusetts Institute of Technology, 2019. -
  42. [47][47] K. Wu and D. G. Childers, 'Gender recognition from speech. part i: Coarse analysis,' JASA , vol. 90, no. 4, pp. 1828-1840, 1991. -
  43. [48][48] D. G. Childers and K. Wu, 'Gender recognition from speech. part ii: Fine analysis,' JASA , vol. 90, no. 4, pp. 1841-1856, 1991. -
  44. [49][49] Z.-Q. Wang and I. Tashev, 'Learning utterance-level representations for speech emotion and age/gender recognition using deep neural networks,' in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2017, pp. 5150-5154. -
  45. [50][50] M. Meire, L. Vuegen, and P. Karsmakers, 'The impact of missing labels and overlapping sound events on multi-label multi-instance learning for sound event classification,' in DCASE Workshop , 2019, pp. 159-163. -
  46. [51][51] P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson, 'Averaging weights leads to wider optima and better generalization,' in 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018 , 2018, pp. 876-885. -
  47. [53][53] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, 'Densely connected convolutional networks,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 4700-4708. -
  48. [54][54] B. Athiwaratkun, M. Finzi, P. Izmailov, and A. G. Wilson, 'There are many consistent explanations of unlabeled data: Why you should average,' in International Conference on Learning Representations , 2019. -
  49. [55][55] Z. Shi, L. Liu, H. Lin, R. Liu, A. Shi, and S. First, 'Hodgepodge: Sound event detection based on ensemble of semi-supervised learning methods,' in DCASE Workshop , 2019, p. 224. -
  50. [56][56] Y. Guo, M. Xu, Z. Wu, J. Wu, and B. Su, 'Multi-scale convolutional recurrent neural network with ensemble method for weakly labeled sound event detection,' in 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) , 2019, pp. 1-5. -
  51. [57][57] W. Lim, S. Suh, S. Park, and Y. Jeong, 'Sound event detection in domestic environments using ensemble of convolutional recurrent neural networks,' in DCASE Workshop , 2019. -
  52. [58][58] A. Chandra, H. Chen, and X. Yao, 'Trade-off between diversity and accuracy in ensemble generation,' in Multi-objective machine learning . Springer, 2006, pp. 429-464. -
  53. [59][59] P. Verma and J. Berger, 'Audio transformers: Transformer architectures for large scale audio understanding. adieu convolutions,' arXiv preprint arXiv:2105.00335 , 2021. - · arXiv
  54. [60][60] Y. Wu and T. Lee, 'Reducing model complexity for dnn based largescale audio classification,' in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2018, pp. 331-335. -
  55. [61][61] A. Kumar, M. Khadkevich, and C. F¨ ugen, 'Knowledge transfer from weakly labeled audio using convolutional neural network for sound events and scenes,' in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2018, pp. 326-330. -
  56. [62][62] Y. Wang, J. Li, and F. Metze, 'A comparison of five multiple instance learning pooling functions for sound event detection with weak labeling,' in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2019, pp. 31-35.

Cited by

Sorted by citations

Authors
Gong, Y., Chung, Y.-A., & Glass, J.
Affiliations
Massachusetts Institute of Technology (US)
ORCID
Venue
IEEE/ACM Transactions on Audio, Speech, and Language Processing
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2021
Volume
29
Pages
3292-3306
ISSN
  • 2329-9290 (ISSN-L)
  • 2329-9304
DOI
10.1109/taslp.2021.3120633
OpenAlex
W3164279099
Cited by
164
References
5
Citations by year
2026: 16 · 2025: 39 · 2024: 42 · 2023: 34 · 2022: 28 · 2021: 5
FWCI
12.73
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE/ACM Transactions on Audio Speech and Language Processing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • IEEE RESOURCE CENTERS
  • TIB Data Manager
Funders
Signify
Topics
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
audio tagging, AudioSet, ImageNet pre-training, balanced sampling, data augmentation, label enhancement, model aggregation, ensemble learning, EfficientNet, training methods

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review