The DOI matches its Crossref record; 107 of the 114 references it cites were found in a registry or an index.
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
Open accessAbstract
We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio. We find that the combination of pre-training, self-training and scaling up model size greatly increases data efficiency, even for extremely large tasks with tens of thousands of hours of labeled data. In particular, on an ASR task with 34 k hours of labeled data, by fine-tuning an 8 billion parameter pre-trained Conformer model we can match state-of-the-art (SoTA) performance with only 3% of the training data and significantly improve SoTA with the full training set. We also report on the universal benefits gained from using big pre-trained and self-trained models for a large set of downstream tasks that cover a wide range of speech domains and span multiple orders of magnitudes of dataset sizes, including obtaining SoTA performance on many public benchmarks. In addition, we utilize the learned representation of pre-trained networks to achieve SoTA results on non-ASR tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some of these tests produce very low error rates that even outperform those achieved through manual transcription [Zhang et al. 2022].”
From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Zhang et al. 2022
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
Kaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 5,270
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., & Le, Q. V. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2680 · Cited by 3,558
Self-citationTransformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169
Self-citation
Research works printed but not linked: 2
- Zavaliagkos, 1998, Utilizing untranscribed training data to improve performance, 301
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
Cited by
Sorted by citations
Items 101–161 of 161
Training Large ASR Encoders With Differential Privacy
Chauhan, G., Chien, S., Thakkar, O. D., Thakurta, A., & Narayanan, A. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832249 · Cited by 2
Echo: A crowd-sourced Romanian speech dataset.
Ungureanu, R.-D., & Dascălu, M. (2024) · Interaction design & architecture(s)/ID&A Interaction design & architecture(s) · doi:10.55612/s-5002-062-009 · Cited by 2
AdaT-DFL: A Semi-Supervised Framework for Speech Depression Detection with Limited Reliable Labels
Hao, Y., Zhou, Y., & Lan, L. (2024) · IEEE International Conference on Automation, Electronics and Electrical Engineering (AUTEEE) · doi:10.1109/auteee62881.2024.10869812 · Cited by 2
Matějů, L., Nouza, J., Červa, P., & Žďánský, J. (2025) · Speech Communication · doi:10.1016/j.specom.2025.103221 · Cited by 2
Chen, X., Ma, Z., Tang, C., Wang, Y., & Zheng, Z. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095130 · Cited by 2
Self-citationG-Augment: Searching for the Meta-Structure of Data Augmentation Policies for ASR
Wang, G., Cubuk, E. D., Rosenberg, A., Cheng, S., Weiss, R. J., Ramabhadran, B., Moreno, P. J., Le, Q. V., & Park, D. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022748 · Cited by 3
Self-citationEditorial Editorial of Special Issue on Self-Supervised Learning for Speech and Audio Processing
Lee, H.-Y., Watanabe, S., Livescu, K., Mohamed, A., & Sainath, T. N. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3205434 · Cited by 2
Self-citationAnalysis of Medical Image Processing using Machine Learning Applications - A Review
(2024) · International Journal of Advanced Trends in Computer Science and Engineering · doi:10.30534/ijatcse/2024/061362024 · Cited by 1
Yang, Z., Yeo, Y. H., Jiang, R., Fu, X., Chen, W., Xi, W., & Zhao, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888331 · Cited by 1
The Transforter - A Transformer Model to Predict Delays in Multimodal Transportation Networks
Kasthurirajan, P., Regikumar, R., Ramanujam, A., & Jayaprakash, R. (2024) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc58415.2024.10920213 · Cited by 1
Multimodal Audio-Visual Emotion Recognition for Social Robotics
Simone, G. D., Greco, L., Saggese, A., & Vento, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05060-1_21 · Cited by 1
Park, K. H., Kim, J., & Kang, U. (2025) · PLoS ONE · doi:10.1371/journal.pone.0333915 · Cited by 1
Leveraging broadcast media subtitle transcripts for automatic speech recognition and subtitling
Poncelet, J., & hamme, H. V. (2026) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-026-00450-9 · Cited by 6
A HiFi-GAN method-based algorithm for expanding and classifying Parkinson's vocal samples
Qin, J., Jiang, Z., Wang, Z., & Ji, C. (2022) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic57696.2022.10075112 · Cited by 1
Self-citationUsing Commercial ASR Solutions to Assess Reading Skills in Children: A Case Report
Piton, T., Hermann, E., Pasqualotto, A., Cohen, M., Magimai.-Doss, M., & Bavelier, D. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-928 · Cited by 1
Improved Long-Form Speech Recognition By Jointly Modeling The Primary And Non-Primary Speakers
Arumugam, G. P., Chang, S.-Y., Sainath, T. N., Wang, Q., & Bijwadia, S. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389736 · Cited by 1
Self-citationExploration of Pretrained Models used in Automatic Speech Recognition System: A Review and Analysis
Suhasini, S., Pandiaraj, A., & Ramshankar, N. (2023) · International Conference on Recent Trends in Advance Computing (ICRTAC) · doi:10.1109/icrtac59277.2023.10480839 · Cited by 1
SOA: Reducing Domain Mismatch in SSL Pipeline by Speech Only Adaptation for Low Resource ASR
Shankar, N. B., Fan, R., & Alwan, A. A. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10625884 · Cited by 1
Hearing emotions: fine-tuning speech emotion recognition models
Mohammed, P., Hadžić, B., Alkostantini, M. E., Kubota, N., Shiban, Y., & Rätsch, M. (2025) · Symposium on Pattern Recognition and Applications (SPRA) · doi:10.1117/12.3057659 · Cited by 1
UME: Upcycling Mixture-of-Experts for Scalable and Efficient Automatic Speech Recognition
Fu, L., Yu, S., Li, S., Fan, L., Wu, Y., & He, X. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890533 · Cited by 1
Self-citationDesign of college English oral training system based on collaborative filtering algorithm
Hou, P. (2025) · International Conference on Big Data, Computational Intelligence, and Applications (BDCIA) · doi:10.1117/12.3059572 · Cited by 1
Joint Audio and Speech Understanding
Gong, Y., Liu, A. H., Luo, H., Karlinsky, L., & Glass, J. (2023) · arXiv · doi:10.48550/arxiv.2309.14405 · Cited by 67
“The TLTR and LoRA adapters are unfrozen in the second and third stages to stabilize training.”
BackgroundSíntese de fala aplicada à geração de conjunto de dados para reconhecimento automático de fala
Casanova, E. (2022) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2022.tde-02092022-142539
2022 Index IEEE Journal of Selected Topics in Signal Processing Vol. 16
(2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3219225
(2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3193339
Jin-yuan, C., Yu-Jun, Q., Guo-Peng, M., & Xiao-Ning, M. (2022) · International Conference on Smart Electronics and Communication (ICOSEC) · doi:10.1109/icosec54921.2022.9952064
Singing Voice Detection via Similarity-Based Semi-Supervised Learning
Chen, X., Gao, Y., & Li, W. (2022) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3551626.3564963
A Multimodal Activation Detection Model for Wake-Free Robots
Zhang, H., Wang, J., Yang, S., & Sun, Y. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-1354-1_10
Dong, Z., & Ma, J. (2023) · Soft Computing · doi:10.1007/s00500-023-08982-4
RetractedHubert, R., Sokolov, A., & Riezler, S. (2023) · International Conference on Spoken Language Translation (IWSLT) · doi:10.18653/v1/2023.iwslt-1.4
Monte Carlo Self-Training for Speech Recognition
Tripathi, A., Khorram, S., Han, L., Kim, J., Zhang, Q., & Sak, H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446250
Mathematical analysis of AMRes: unlocking enhanced recognition across audio-visual domains
Zoughi, T., & Deypir, M. (2024) · International Journal of Information Technology · doi:10.1007/s41870-024-01739-8 · Cited by 1
Cruise English Intelligent Dialogue System Considering Novel Audio Denoising Algorithm
Tang, Z. (2024) · International Conference on Expert Clouds and Applications (ICOECA) · doi:10.1109/icoeca62351.2024.00024
disentanglement using pre-trained features
Liu, S., & Zhao, Y. (2024) · Jisuanji shenghuojia. · doi:10.54097/tk7x3833
Gan, L., Xie, Z., & Wang, B. (2024) · International Conference on Intelligent Algorithms for Computational Intelligence Systems (IACIS) · doi:10.1109/iacis61494.2024.10721621
Yuan, X., & Li, D. (2024) · International Conference on Artificial Intelligence and Computer Information Technology (AICIT) · doi:10.1109/aicit62434.2024.10730296
Design of Japanese Speech Recognition and Real-Time Translation System Based on Deep Learning
Zhang, X. (2024) · Lecture notes in electrical engineering · doi:10.1007/978-981-97-6465-5_18
Artificial Intelligence Technology in the Field of Broadcasting and Hosting
Wang, C. (2024) · IEEE International Conference on Advances in Electrical Engineering and Computer Applications (AEECA) · doi:10.1109/aeeca62331.2024.00068
Xie, P., & Chen, K. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890166 · Cited by 1
A New Framework Toward Accurate Emotions Recognition for Human Speeches
Fouad, M. (2024) · International Conference on Computer and Applications (ICCA) · doi:10.1109/icca62237.2024.10928160
Mehra, S. (2025) · International Journal of Computer Applications · doi:10.5120/ijca2025925146
Lin, S., & Feng, Y. (2025) · Journal of Circuits Systems and Computers · doi:10.1142/s0218126625504262
Li, H., Tang, C., Yue, X., & Li, X. (2025) · Frontiers in Communications and Networks · doi:10.3389/frcmn.2025.1662788
Parameter Transfer for Single-Task Reinforcement Learning
Landers, M., & Doryab, A. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228810
Mongolian Speech Recognition Based on Semi-supervised Learning and Syllable Subword Modeling Units
Li, Y., Wang, Y., Gao, Z., & Bao, F. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3352-7_18
Self-citationBalancing Specialization and Generalization Trade-Off for Speech Recognition Models
Cygert, S., Despot-Mładanowicz, P., & Czyżewski, A. (2025) · Electronics · doi:10.3390/electronics14244792
Fan, P., Wang, D., Zhang, Y., Jiang, R., & Gu, H. (2025) · Energy Conversion and Management X · doi:10.1016/j.ecmx.2025.101289
Self-citationLanguage Adaptation Wake Word Spotting via Latent Space from Pre-Trained Speech Models
Xiong, S., Zhou, H., Shen, K., Cheng, S., Chen, H., Wan, G., Li, K., Du, J., & Dai, L. (2025) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc65261.2025.11249276
UniTrain: A universal iterative semi-supervised training framework for graph representation learning
Chen, X., Li, J., Huang, Y., Zhuang, J., Shi, C., Xu, Z., & Fu, Y.-G. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.108623
Self-citationWagner, D., Baumann, I., Engert, N., Nöth, E., Riedhammer, K., & Bocklet, T. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434680
Improving Streaming ASR via Differentially Private Fusion of Data from Multiple Sources
Shejwalkar, V., Thakkar, O. D., Chien, S., Rafidi, N., & Narayanan, A. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434648
Few-Shot Learning Model With Small Scale Pre-Training for Cross-Lingual E2E ASR
Eledath, D., Das, A., & Ramasubramanian, V. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3684414
Decoder-Only Conformer with Modality-Aware Sparse Mixtures of Experts for ASR
Lee, J., & Mimura, M. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464341
Trilevel Supervised, Unsupervised, and Distilled Learning for Speech Recognition
Chien, J., Lin, Y., & Cui, X. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3693975
Chen, P.-Y., Lin, C., Pai, N.-S., Huang, P., Kuo, C., Li, C., & Lin, C. (2026) · Sensors · doi:10.3390/s26123955
Learning Speech Representations with Variational Predictive Coding
Yeh, S.-L., Bell, P., & Tang, H. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.738
Maltby, H., Wall, J., Glackin, C., Moniri, M., Salami, I., Li, L., & Cannings, N. (2026) · Computer Speech & Language · doi:10.1016/j.csl.2026.102037
Cai, H., Zhang, P., Xiao, Y., Ji, S., & Xiao, M. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3724421
Toward Extremely Low-Bit and Multi-Precision Conformer and Speech Foundation Model Quantization
Li, Z., Xu, H., Jin, Z., Wang, T., Meng, L., Wang, H., Chen, Y., Cui, M., Hu, S., Geng, M., et al. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3724410
Bilingual End-to-End Neural Diarization for Kazakh-Russian Code-Switched Speech
Ilyassov, A., Syzdykova, Z., Igbayev, S., Khikmetov, A., Davydov, D., & Feldman, G. (2026) · International Conference on Cybersecurity, Digital Forensics, and AI Applications (ICCSDFAI) · doi:10.1109/iccsdfai70505.2026.11647883
Metadata
- Authors
- Zhang, Y., Park, D. S., Han, W., Qin, J., Gulati, A., Shor, J., Jansen, A., Xu, Y., Huang, Y., Wang, S., Zhou, Z., Li, B., Ma, M., Chan, W., Yu, J., Wang, Y., Cao, L., Sim, K. C., Ramabhadran, B., Sainath, T. N., Beaufays, F., Chen, Z., Le, Q. V., Chiu, C.-C., Pang, R., & Wu, Y.
- Affiliations
- Google (United States) (US)
- Google, Inc., USA
- ORCID
- Venue
- IEEE Journal of Selected Topics in Signal Processing
- Published
- 2022-10
- Volume
- 16
- Issue
- 6
- Pages
- 1519-1532
- ISSN
- 1932-4553 (ISSN-L)
- 1941-0484
- OpenAlex
- W3204696009
- Cited by
- 161
- References
- 114
- Citations by year
- 2026: 11 · 2025: 28 · 2024: 40 · 2023: 51 · 2022: 28
- FWCI
- 12.09
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Journal of Selected Topics in Signal Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- speech recognition, pre-training, self-training, data efficiency, conformational isomers, model scaling, transfer learning, representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
