TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
Open accessThe DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
Abstract
We introduce a self-supervised speech pre-training method called TERA, which stands for Transformer Encoder Representations from Alteration. Recent approaches often learn by using a single auxiliary task like contrastive prediction, autoregressive prediction, or masked reconstruction. Unlike previous methods, we use alteration along three orthogonal axes to pre-train Transformer Encoders on a large amount of unlabeled speech. The model learns through the reconstruction of acoustic frames from their altered counterpart, where we use a stochastic policy to alter along various dimensions: time, frequency, and magnitude. TERA can be used for speech representations extraction or fine-tuning with downstream models. We evaluate TERA on several downstream tasks, including phoneme classification, keyword spotting, speaker recognition, and speech recognition. We present a large-scale comparison of various self-supervised models. TERA achieves strong performance in the comparison by improving upon surface features and outperforming previous models. In our experiments, we study the effect of applying different alteration techniques, pre-training on more data, and pre-training on various features. We analyze different model sizes and find that smaller models are strong representation learners than larger models, while larger models are more effective for downstream fine-tuning than smaller models. Furthermore, we show the proposed method is transferable to downstream datasets not used in pre-training.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 1×
“The APC approach uses an autoregressive model to encode temporal information of past acoustic sequence; the model then predicts future frames like a recurrent-based LM [32] while conditioning on past frames.”
MethodGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
Kaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 4,609
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., & Le, Q. V. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2680 · Cited by 3,558
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369
How to Fine-Tune BERT for Text Classification?
Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32381-3_16 · Cited by 1,534
Maximum likelihood linear transformations for HMM-based speech recognition
Gales, M. J. F. (1998) · Computer Speech & Language · doi:10.1006/csla.1998.0043 · Cited by 1,482
wav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290
DARPA TIMIT: acoustic-phonetic continuous speech corpus CD-ROM, NIST speech disc 1-1.1
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., Pallett, D. S., & Dahlgren, N. L. (1993) · doi:10.6028/nist.ir.4930 · Cited by 1,267
Speaker-independent phone recognition using hidden Markov models
Lee, K.-F., & Hon, H.-W. (1989) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/29.46546 · Cited by 927
Linguistic Knowledge and Transferability of Contextual Representations
Liu, N. F., Gardner, M., Belinkov, Y., Peters, M. E., & Smith, N. A. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1112 · Cited by 621
Libri-Light: A Benchmark for ASR with Limited or No Supervision
Kahn, J., Riviere, M., Zheng, W., Kharitonov, E., Xu, Q., Mazare, P. E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9052942 · Cited by 458
Light Gated Recurrent Units for Speech Recognition
Ravanelli, M., Brakel, P., Omologo, M., & Bengio, Y. (2018) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2017.2762739 · Cited by 411
vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
Baevski, A., Schneider, S., & Auli, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05453 · Cited by 388
An Unsupervised Autoregressive Model for Speech Representation Learning
Chung, Y.-A., Hsu, W.-N., Tang, H., & Glass, J. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1473 · Cited by 357
Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
Liu, A. T., Yang, S.-W., Chi, P.-H., Hsu, P.-C., & Lee, H.-Y. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054458 · Cited by 337
Self-citationUnsupervised speech representation learning using WaveNet autoencoders
Chorowski, J., Weiss, R. J., Bengio, S., & van den Oord, A. (2019) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2019.2938863 · Cited by 325 · cited 2×
“Previous work explored mostly for reconstruction on the temporal axis, for example unidirectional reconstruction of magnitude or phase from past frames [8]–[10], [15]–[18], [23], or bidirectional reconstruction of a temporal frame from both past and future”
BackgroundThe Pytorch-kaldi Speech Recognition Toolkit
Ravanelli, M., Parcollet, T., & Bengio, Y. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683713 · Cited by 236
Learning Problem-Agnostic Speech Representations from Multiple Self-Supervised Tasks
Pascual, S., Ravanelli, M., Serrà, J., Bonafonte, A., & Bengio, Y. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2605 · Cited by 219
Unsupervised Pretraining Transfers Well Across Languages
Riviere, M., Joulin, A., Mazare, P.-E., & Dupoux, E. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054548 · Cited by 186 · cited 8×
“Besides, speech representations also help transfer learning and adaptation across different data distributions [6], [7], [9], [20], [21].”
BackgroundMethodInfluentialVery Deep Self-Attention Networks for End-to-End Speech Recognition
Pham, N.-Q., Nguyen, T.-S., Niehues, J., Müller, M., & Waibel, A. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2702 · Cited by 173
Audio Albert: A Lite Bert for Self-Supervised Learning of Audio Representation
Chi, P.-H., Chung, P.-H., Wu, T.-H., Hsieh, C.-C., Chen, Y.-H., Li, S.-W., & Lee, H.-Y. (2021) · 2021 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383575 · Cited by 168
Self-citationGenerative Pre-Training for Speech with Autoregressive Predictive Coding
Chung, Y.-A., & Glass, J. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054438 · Cited by 157
Deep Contextualized Acoustic Representations for Semi-Supervised Speech Recognition
Ling, S., Liu, Y., Salazar, J., & Kirchhoff, K. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053176 · Cited by 145
An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models
Chronopoulou, A., Baziotis, C., & Potamianos, A. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1213 · Cited by 106
Phone recognition with hierarchical convolutional deep maxout networks
Tóth, L. (2015) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-015-0068-3 · Cited by 101
Learning Filterbanks from Raw Speech for Phone Recognition
Zeghidour, N., Usunier, N., Kokkinos, I., Schaiz, T., Synnaeve, G., & Dupoux, E. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462015 · Cited by 98
Effectiveness of Self-Supervised Pre-Training for ASR
Baevski, A., & Mohamed, A. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054224 · Cited by 98
Vector-Quantized Autoregressive Predictive Coding
Chung, Y.-A., Tang, H., & Glass, J. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1228 · Cited by 97
Effectiveness of self-supervised pre-training for speech recognition
Baevski, A., Auli, M., & Mohamed, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03912 · Cited by 93
Improving Transformer-based Speech Recognition Using Unsupervised Pre-training
Jiang, D., Lei, X., Li, W., Luo, N., Hu, Y., Zou, W., & Li, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.09932 · Cited by 92
Learning Robust and Multilingual Speech Representations
Kawakami, K., Wang, L., Dyer, C., Blunsom, P., & van den Oord, A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.106 · Cited by 86
Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction
Wang, W., Tang, Q., & Livescu, K. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053541 · Cited by 79
Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies
Liu, A. H., Chung, Y.-A., & Glass, J. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-349 · Cited by 75
Self-citationPre-Training Audio Representations With Self-Supervision
Tagliasacchi, M., Gfeller, B., Quitry, F. D. C., & Roblek, D. (2020) · IEEE Signal Processing Letters · doi:10.1109/lsp.2020.2985586 · Cited by 62
Speech-XLNet: Unsupervised Acoustic Model Pretraining for Self-Attention Networks
Song, X., Wang, G., Huang, Y., Wu, Z., Su, D., & Meng, H. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1511 · Cited by 62
Improved Speech Representations with Multi-Target Autoregressive Predictive Coding
Chung, Y.-A., & Glass, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.213 · Cited by 58
Self-supervised audio representation learning for mobile devicesPreprint
Tagliasacchi, M., Gfeller, B., Quitry, F. D. C., & Roblek, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11796 · Cited by 50
Large-Scale Unsupervised Pre-Training for End-to-End Spoken Language Understanding
Wang, P., Wei, L., Cao, Y., Xie, J., & Nie, Z. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053163 · Cited by 46
Understanding Self-Attention of Self-Supervised Audio Transformers
Yang, S.-W., Liu, A. T., & Lee, H.-Y. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2231 · Cited by 42
Self-citationA Further Study of Unsupervised Pretraining for Transformer Based Speech Recognition
Jiang, D., Li, W., Zhang, R., Cao, M. Y., Luo, N., Yang, H., Zou, W., Han, K., & Li, X. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414539 · Cited by 42
Defense for Black-Box Attacks on Anti-Spoofing Models by Self-Supervised Learning
Wu, H., Liu, A. T., & Lee, H.-Y. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2026 · Cited by 38
Self-citationLi, C.-Y., Yuan, P.-C., & Lee, H.-Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054675 · Cited by 37
Self-citationUnsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion
Liu, A. T., Hsu, P.-C., & Lee, H.-Y. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2048 · Cited by 36
Self-citationLing, S., Salazar, J., Liu, Y., & Kirchhoff, K. (2020) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2020-2 · Cited by 34
A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning
Khurana, S., Laurent, A., Hsu, W.-N., Chorowski, J., Łańcucki, A., Marxer, R., & Glass, J. R. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-3084 · Cited by 34
Learning audio representations via phase predictionPreprint
Quitry, F. D. C., Tagliasacchi, M., & Roblek, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.11910 · Cited by 13
Song, L., Li, L., Hong, Q., & Liu, L. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2007 · Cited by 12
Masked Pre-trained Encoder base on Joint CTC-TransformerPreprint
Liu, L., & Huang, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.11978 · Cited by 9
A Further Study of Unsupervised Pre-training for Transformer Based Speech RecognitionPreprint
Jiang, D., Li, W., Zhang, R., Cao, M. Y., Luo, N., Han, Y., Zou, W., & Li, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.09862 · Cited by 8
Adversarial Contrastive Predictive Coding for Unsupervised Learning of Disentangled Representations.Preprint
Ebbers, J., Kuhlmann, M. T., & Haeb‐Umbach, R. (2020) · arXiv (Cornell University) · Cited by 5
(2020) · neural information processing systems
Research works printed but not linked: 2
- lopes, 2011, Phone recognition on the timit database, Speech Technol /Book, 1, 285
- mikolov, 2013, Proc 1st Int Conf Learn Representations
Cited by
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3188113 · Cited by 1,926
SUPERB: Speech Processing Universal PERformance Benchmark
Yang, S.-W., Chi, P.-H., Chuang, Y.-S., Lai, C.-I. J., Lakhotia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T., et al. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1775 · Cited by 681
Self-citationFLAVA: A Foundational Language And Vision Alignment Model
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., & Kiela, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01519 · Cited by 554
A review of deep learning techniques for speech processing
Mehrish, A., Majumder, N., Bharadwaj, R., Mihalcea, R., & Poria, S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101869 · Cited by 389
Self-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372
“Whereas most studies use masking along the temporal dimension of the input, speech can also be masked along the frequency dimension when spectral input features are used [103], [106].”
BackgroundMethodSelf-citation
Metadata
- Authors
- Liu, A. T., Li, S.-W., & Lee, H.-Y.
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2021
- Volume
- 29
- Pages
- 2351-2366
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W3041561163
- Cited by
- 312
- References
- 64
- FWCI
- 22.23
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- National Science Council (110-2628-E-002-001)
- National Taiwan University
- ASUS AICS
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.