The DOI matches its Crossref record; 107 of the 114 references it cites were found in a registry or an index.
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
Open accessAbstract
We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio. We find that the combination of pre-training, self-training and scaling up model size greatly increases data efficiency, even for extremely large tasks with tens of thousands of hours of labeled data. In particular, on an ASR task with 34 k hours of labeled data, by fine-tuning an 8 billion parameter pre-trained Conformer model we can match state-of-the-art (SoTA) performance with only 3% of the training data and significantly improve SoTA with the full training set. We also report on the universal benefits gained from using big pre-trained and self-trained models for a large set of downstream tasks that cover a wide range of speech domains and span multiple orders of magnitudes of dataset sizes, including obtaining SoTA performance on many public benchmarks. In addition, we utilize the learned representation of pre-trained networks to achieve SoTA results on non-ASR tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some of these tests produce very low error rates that even outperform those achieved through manual transcription [Zhang et al. 2022].”
From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Zhang et al. 2022
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
Kaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 5,270
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., & Le, Q. V. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2680 · Cited by 3,558
Self-citationTransformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169
Self-citationConformer: Convolution-augmented Transformer for Speech Recognition
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-3015 · Cited by 3,150
Self-citationAudio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134
Self-citationSupervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120
Unsupervised word sense disambiguation rivaling supervised methods
Yarowsky, D. (1995) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/981658.981684 · Cited by 2,411
Self-Training With Noisy Student Improves ImageNet Classification
Xie, Q., Luong, M.-T., Hovy, E., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01070 · Cited by 2,393
Self-citationwav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369
VoxCeleb: A Large-Scale Speaker Identification Dataset
Nagrani, A., Chung, J. S., & Zisserman, A. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-950 · Cited by 2,210
Sequence Transduction with Recurrent Neural NetworksPreprint
Graves, A. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1211.3711 · Cited by 1,842
Piczak, K. J. (2015) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2733373.2806390 · Cited by 1,572
Transformer-XL: Attentive Language Models Beyond a Fixed-Length ContextPreprint
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019) · arXiv (Cornell University) · doi:10.4230/lipics.ecrts.2023.7 · Cited by 1,476
Self-citationAdafactor: Adaptive Learning Rates with Sublinear Memory CostPreprint
Shazeer, N., & Stern, M. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1804.04235 · Cited by 1,410
wav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290
Learning extraction patterns for subjective expressions
Riloff, E., & Wiebe, J. (2003) · Proceedings of the 2003 conference on Empirical methods in natural language processing - · doi:10.3115/1119355.1119369 · Cited by 1,059
Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
Warden, P. (2018) · arXiv · doi:10.48550/arxiv.1804.03209 · Cited by 1,033
CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset
Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., & Verma, R. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2336244 · Cited by 954
The AMI Meeting Corpus: A Pre-announcement
Carletta, J., Ashby, S., Bourban, S., Flynn, M., Guillemot, M., Hain, T., Kadlec, J., Vasilis Karaiskos, B., Kraaij, W., Kronenthal, M., et al. (2006) · Lecture notes in computer science · doi:10.1007/11677482_3 · Cited by 939
(2020) · doi:10.1109/sc41405.2020.00024 · Cited by 928
Common Voice: A Massively-Multilingual Speech Corpus
Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F. M., & Weber, G. (2019) · arXiv · doi:10.48550/arxiv.1912.06670 · Cited by 905
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893
Japanese and Korean voice search
Schuster, M., & Nakajima, K. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6289079 · Cited by 862
On Using Monolingual Corpora in Neural Machine TranslationPreprint
Gülçehre, Ç., Fırat, O., Xu, K., Cho, K., Barrault, L., Lin, H.-C., Bougares, F., Schwenk, H., & Bengio, Y. (2015) · HAL · doi:10.48550/arxiv.1503.03535 · Cited by 602
Streaming End-to-end Speech Recognition for Mobile Devices
He, Y., Sainath, T. N., Prabhavalkar, R., McGraw, I., Alvarez, R., Zhao, D., Rybach, D., Kannan, A., Wu, Y., Pang, R., et al. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682336 · Cited by 601
Self-citationLibri-Light: A Benchmark for ASR with Limited or No Supervision
Kahn, J., Rivière, M., Zheng, W., Kharitonov, E., Xu, Q., Mazaré, P.-E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9052942 · Cited by 537
Probability of error of some adaptive pattern-recognition machines
Scudder, H. J. (1965) · IEEE Transactions on Information Theory · doi:10.1109/tit.1965.1053799 · Cited by 510
vq-wav2vec: Self-Supervised Learning of Discrete Speech RepresentationsPreprint
Baevski, A., Schneider, S., & Auli, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05453 · Cited by 500
Zhang, Q., Han, L., Sak, H., Tripathi, A., McDermott, E., Koo, S., & Kumar, S. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053896 · Cited by 462
Mesh-TensorFlow: Deep Learning for SupercomputersPreprint
Shazeer, N., Cheng, Y.-L., Parmar, N., Tran, D., Vaswani, A., Koanantakool, P., Hawkins, P. G., Lee, H., Hong, M., Young, C., et al. (2018) · Neural Information Processing Systems · doi:10.48550/arxiv.1811.02084 · Cited by 452
Fast and accurate recurrent neural network acoustic models for speech recognition
Sak, H., Senior, A., Rao, K., & Beaufays, F. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-350 · Cited by 426
Self-citationGShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., & Chen, Z. (2020) · arXiv · doi:10.48550/arxiv.2006.16668 · Cited by 412
Self-citationAn Unsupervised Autoregressive Model for Speech Representation Learning
Chung, Y.-A., Hsu, W.-N., Tang, H., & Glass, J. R. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1473 · Cited by 410
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020) · neural information processing systems · Cited by 386
Unsupervised speech representation learning using WaveNet autoencoders
Chorowski, J., Weiss, R. J., Bengio, S., & van den Oord, A. (2019) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2019.2938863 · Cited by 325
Pushing the Limits of Semi-Supervised Learning for Automatic Speech RecognitionPreprint
Zhang, Y., Qin, J., Park, D., Han, W., Chiu, C., Pang, R., Le, Q. V., & Wu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10504 · Cited by 323
Self-citationCHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings
Watanabe, S., Mandel, M., Barker, J., Emmanuel Vincent, A., Arora, A., Chang, X., Khudanpur, S. P., Manohar, V., Povey, D., Raj, D., et al. (2020) · International Workshop on Speech Processing in Everyday Environments (CHiME) · doi:10.21437/chime.2020-1 · Cited by 322
TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation
Hernandez, F., Nguyen, V., Ghannay, S., Tomashenko, N., & Estève, Y. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99579-3_21 · Cited by 280
ESC: Dataset for Environmental Sound Classification
Piczak, K. J. (2015) · Harvard Dataverse · doi:10.7910/dvn/ydeput · Cited by 257
TED-LIUM: an Automatic Speech Recognition dedicated corpus
Rousseau, A., Deléglise, P., & Estève, Y. (2012) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/48xafxbrk3qa · Cited by 224
Improved Noisy Student Training for Automatic Speech Recognition
Park, D., Zhang, Y., Ye, J., Han, W., Chiu, C., Li, B., Wu, Y., & Le, Q. V. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1470 · Cited by 214
Self-citationSelf-Training for End-to-End Speech Recognition
Kahn, J., Lee, A., & Hannun, A. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054295 · Cited by 205
GSPMD: General and Scalable Parallelization for ML Computation GraphsPreprint
Xu, Y., Lee, H., Chen, D., Hechtman, B. A., Huang, Y., Joshi, R., Krikun, M., Lepikhin, D., Ly, A., Maggioni, M., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04663 · Cited by 191
Self-citationLiao, H., McDermott, E., & Senior, A. (2013) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2013.6707758 · Cited by 188
Unsupervised Pretraining Transfers Well Across Languages
Riviere, M., Joulin, A., Mazare, P.-E., & Dupoux, E. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054548 · Cited by 186
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech
Chung, Y.-A., & Glass, J. R. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2341 · Cited by 182
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern ArchitecturesPreprint
Synnaeve, G., Xu, Q., Kahn, J., Likhomanenko, T., Grave, É., Pratap, V., Sriram, A., Liptchinsky, V., & Collobert, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08460 · Cited by 165
Self-Training and Pre-Training are Complementary for Speech Recognition
Xu, Q., Baevski, A., Likhomanenko, T., Tomasello, P., Conneau, A., Collobert, R., Synnaeve, G., & Auli, M. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414641 · Cited by 152
Deep Contextualized Acoustic Representations for Semi-Supervised Speech Recognition
Ling, S., Liu, Y., Salazar, J., & Kirchhoff, K. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053176 · Cited by 149
Specaugment on Large Scale Datasets
Park, D., Zhang, Y., Chiu, C., Chen, Y., Li, B., Chan, W., Le, Q. V., & Wu, Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053205 · Cited by 145
Self-citationEffectiveness of self-supervised pre-training for speech recognitionPreprint
Baevski, A., Auli, M., & Mohamed, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03912 · Cited by 142
Speaker-dependent audio-visual emotion recognition
Haq, S., & Jackson, P. J. B. (2009) · Open Research (University of Surrey) · Cited by 135
Deep neural network features and semi-supervised training for low resource speech recognition
Thomas, S., Seltzer, M. L., Church, K., & Hynek Hermansky, B. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6638959 · Cited by 133
Front-end processing for the CHiME-5 dinner party scenario
Boeddeker, C., Heitkaemper, J., Schmalenstroeer, J., Drude, L., Heymann, J., & Haeb‐Umbach, R. (2018) · International Workshop on Speech Processing in Everyday Environments (CHiME) · doi:10.21437/chime.2018-8 · Cited by 132
Towards Learning a Universal Non-Semantic Representation of Speech
Shor, J., Jansen, A., Maor, R., Lang, O., Tuval, O., Quitry, F. D. C., Tagliasacchi, M., Shavitt, I., Emanuel, D., & Haviv, Y. A. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1242 · Cited by 128
Self-citationUnsupervised Learning of Semantic Audio Representations
Jansen, A., Plakal, M., Pandya, R., Ellis, D. P. W., Hershey, S., Liu, J., Moore, R. C., & Saurous, R. A. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461684 · Cited by 121
Self-citationIterative Pseudo-Labeling for Speech Recognition
Xu, Q., Likhomanenko, T., Kahn, J., Hannun, A., Synnaeve, G., & Collobert, R. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1800 · Cited by 112
Feature Learning with Raw-Waveform CLDNNs for Voice Activity Detection
Zazo, R., Sainath, T. N., Simko, G., & Parada, C. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-268 · Cited by 108
Self-citationThe INTERSPEECH 2020 Computational Paralinguistics Challenge: Elderly Emotion, Breathing & Masks
Schuller, B. W., Batliner, A. M., Bergler, C., Meßner, E.-M., Hamilton, A. F. D. C., Amiriparian, S., Baird, A., Rizos, G., Schmitt, M., Stappen, L., et al. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-0032 · Cited by 108
Multimodal and Temporal Perception of Audio-visual Cues for Emotion Recognition
Ghaleb, E., Popa, M. C., & Asteriadis, S. (2019) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2019.8925444 · Cited by 106
Learning Robust and Multilingual Speech Representations
Kawakami, K., Wang, L., Dyer, C., Blunsom, P., & van den Oord, A. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.106 · Cited by 102
Toward Domain-Invariant Speech Recognition via Large Scale Training
Narayanan, A., Misra, A., Sim, K. C., Pundak, G., Tripathi, A., Elfeky, M. G., Haghani, P., Strohman, T., & Bacchiani, M. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639610 · Cited by 101
Self-citationBroaden Your Views for Self-Supervised Video Learning
Recasens, A., Luc, P., Alayrac, J.-B., Wang, L., Strub, F., Tallec, C., Malinowski, M., Pătrăucean, V., Altché, F., Valko, M., et al. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00129 · Cited by 98
A Better and Faster end-to-end Model for Streaming ASR
Li, B., Gulati, A., Yu, J., Sainath, T. N., Chiu, C., Narayanan, A., Chang, S.-Y., Pang, R., He, Y., Qin, J., et al. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413899 · Cited by 94
Self-citationSpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network
Chan, W., Park, D., Lee, C., Zhang, Y., Le, Q., & Norouzi, M. (2021) · arXiv · doi:10.48550/arxiv.2104.02133 · Cited by 79
Self-citationLessons from Building Acoustic Models with a Million Hours of Speech
Parthasarathi, S. H. K., & Ström, N. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683690 · Cited by 72
Scaling End-to-End Models for Large-Scale Multilingual ASR
Li, B., Pang, R., Sainath, T. N., Gulati, A., Zhang, Y., Qin, J., Haghani, P., Huang, W., Ma, M., & Bai, J. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9687871 · Cited by 69
Self-citationWav2KWS: Transfer Learning From Speech Representations for Keyword Spotting
Seo, D., Oh, H., & Jung, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3078715 · Cited by 65
Knowledge Distillation from Offline to Streaming RNN Transducer for End-to-End Speech Recognition
Kurata, G., & Saon, G. (2020) · Interspeech · doi:10.21437/interspeech.2020-2442 · Cited by 57
Multi-Format Contrastive Learning of Audio RepresentationsPreprint
Wang, L., & van den Oord, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06508 · Cited by 57
Rethinking Evaluation in ASR: Are Our Models Robust Enough?
Likhomanenko, T., Xu, Q., Pratap, V., Tomasello, P., Kahn, J., Avidov, G., Collobert, R., & Synnaeve, G. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1758 · Cited by 56
An Investigation of a Knowledge Distillation Method for CTC Acoustic Models
Takashima, R., Li, S., & Kawai, H. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461995 · Cited by 53
Kurata, G., & Audhkhasi, K. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639629 · Cited by 53
Automatic Cross-Replica Sharding of Weight Update in Data-Parallel TrainingPreprint
Xu, Y.-Z., Lee, H., Chen, D., Choi, H., Hechtman, B. A., & Wang, S.-B. (2020) · arXiv.org · doi:10.48550/arxiv.2004.13336 · Cited by 53
Self-citationTwin Networks: Matching the Future for Sequence GenerationPreprint
Serdyuk, D., Ke, N. R., Sordoni, A., Trischler, A., Pal, C., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.06742 · Cited by 48
Semi-supervised Training for End-to-end Models via Weak Distillation
Li, B., Sainath, T. N., Pang, R., & Wu, Z. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682172 · Cited by 47
Self-citationImproved Training for Online End-to-end Speech Recognition Systems
Kim, S., Seltzer, M. L., Li, J., & Zhao, R. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2517 · Cited by 43
On the Limit of English Conversational Speech Recognition
Tüske, Z., Saon, G., & Kingsbury, B. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-211 · Cited by 41
Lightly Supervised Acoustic Model Training
Lamel, L. F., Gauvain, J., & Adda, G. (2000) · Cited by 38
Hsu, W.-N., & Glass, J. R. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462037 · Cited by 37
The STC System for the CHiME-6 Challenge
Medennikov, I., Korenevsky, M. L., Prisyach, T., Khokhlov, Y., Korenevskaya, M., Sorokin, I., Timofeeva, T., Mitrofanov, A., Andrusenko, A., Podluzhny, I., et al. (2020) · International Workshop on Speech Processing in Everyday Environments (CHiME) · doi:10.21437/chime.2020-9 · Cited by 35
Semi-Supervised Speech Recognition via Local Prior MatchingPreprint
Hsu, W.-N., Lee, A., Synnaeve, G., & Hannun, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10336 · Cited by 34
RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions
Chiu, C., Narayanan, A., Han, W., Prabhavalkar, R., Zhang, Y., Jaitly, N., Pang, R., Sainath, T. N., Nguyen, P., Cao, L., et al. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383518 · Cited by 34
Self-citationSpoken Language Identification Using ConvNets
Sarthak, Shukla, S., & Mittal, G. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-34255-5_17 · Cited by 32
The STC System for the CHiME 2018 Challenge
Medennikov, I., Sorokin, I., Romanenko, A., Popov, D., Khokhlov, Y., Prisyach, T., Malkovskiy, N., Bataev, V., Astapov, S., Korenevsky, M. L., et al. (2018) · International Workshop on Speech Processing in Everyday Environments (CHiME) · doi:10.21437/chime.2018-1 · Cited by 32
Kanda, N., Ye, G., Wu, Y., Gaur, Y., Wang, X., Meng, Z., Chen, Z., & Yoshioka, T. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-102 · Cited by 31
Self-citationInjecting Text in Self-Supervised Speech Pretraining
Chen, Z., Zhang, Y., Rosenberg, A., Ramabhadran, B., Wang, G., & Moreno, P. J. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688018 · Cited by 28
Self-citationFRILL: A Non-Semantic Speech Embedding for Mobile Devices
Peplinski, J., Shor, J., Joglekar, S., Garrison, J., & Patel, S. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-2070 · Cited by 25
Self-citationOn the importance of normative data in speech-based assessmentPreprint
Noorian, Z., Pou-Prom, C., & Rudzicz, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.00069 · Cited by 24
Paralinguistic Classification of Mask Wearing by Image Classifiers and Fusion
Szep, J., & Hariri, S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2857 · Cited by 21
Analysis of low-resource acoustic model self-training
Novotney, S., & Schwartz, R. L. (2009) · Interspeech (USB) · doi:10.21437/interspeech.2009-86 · Cited by 20
Venugopalan, S., Shor, J., Plakal, M., Tobin, J., Tomanek, K., Green, J. R., & Brenner, M. P. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1913 · Cited by 18
Self-citationDoutre, T., Han, W., Ma, M., Lu, Z., Chiu, C., Pang, R., Narayanan, A., Misra, A., Zhang, Y., & Cao, L. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413692 · Cited by 16
Self-citationTwin Regularization for Online Speech Recognition
Ravanelli, M., Serdyuk, D., & Bengio, Y. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1407 · Cited by 15
Universal ASR: Unify and Improve Streaming ASR with Full-context Modeling.Preprint
Yu, J., Han, W., Gulati, A., Chiu, C., Li, B., Sainath, T. N., Wu, Y., & Pang, R. (2020) · arXiv (Cornell University) · Cited by 14
Self-citationSemi-Supervision in ASR: Sequential MixMatch and Factorized TTS-Based Augmentation
Chen, Z., Rosenberg, A., Zhang, Y., Zen, H., Ghodsi, M., Huang, Y., Emond, J., Wang, G., Ramabhadran, B., & Moreno, P. J. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-677 · Cited by 12
Self-citationSemi-Supervised end-to-end Speech Recognition via Local Prior Matching
Hsu, W.-N., Lee, A., Synnaeve, G., & Hannun, A. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383552 · Cited by 6
Doutre, T., Han, W., Chiu, C., Pang, R., Siohan, O., & Cao, L. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-637 · Cited by 3
Self-citationKanda, N., Ye, G., Wu, Y., Gaur, Y., Wang, X., Meng, Z., Chen, Z., & Yoshioka, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.16776 · Cited by 3
Self-citationDoutre, T., Han, W., Chiu, C., Pang, R., Siohan, O., & Cao, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.14346 · Cited by 2
Self-citation
Research works printed but not linked: 2
- Zavaliagkos, 1998, Utilizing untranscribed training data to improve performance, 301
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
Cited by
Sorted by citations
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3188113 · Cited by 1,926
Self-citationXLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
Babu, A., Wang, C., Tjandra, A., Lakhotia, K., Xu, Q., Goyal, N., Singh, K., von Platen, P., Saraf, Y., Pino, J., et al. (2022) · Interspeech 2022 · doi:10.21437/interspeech.2022-143 · Cited by 586
Machine Learning Methods for Small Data Challenges in Molecular Science
Dou, B., Zhu, Z., Merkurjev, E., Ke, L., Chen, L., Jian, J., Zhu, Y., Liu, J., Zhang, B., & Wei, G. (2023) · Chemical Reviews · doi:10.1021/acs.chemrev.3c00189 · Cited by 548
CLAP Learning Audio Concepts from Natural Language Supervision
Elizalde, B., Deshmukh, S., Ismail, M. A., & Wang, H. (2023) · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49357.2023.10095889 · Cited by 449
Recent Advances in End-to-End Automatic Speech Recognition
Li, J. (2022) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000050 · Cited by 451
Metadata
- Authors
- Zhang, Y., Park, D. S., Han, W., Qin, J., Gulati, A., Shor, J., Jansen, A., Xu, Y., Huang, Y., Wang, S., Zhou, Z., Li, B., Ma, M., Chan, W., Yu, J., Wang, Y., Cao, L., Sim, K. C., Ramabhadran, B., Sainath, T. N., Beaufays, F., Chen, Z., Le, Q. V., Chiu, C.-C., Pang, R., & Wu, Y.
- Affiliations
- Google (United States) (US)
- Google, Inc., USA
- ORCID
- Venue
- IEEE Journal of Selected Topics in Signal Processing
- Published
- 2022-10
- Volume
- 16
- Issue
- 6
- Pages
- 1519-1532
- ISSN
- 1932-4553 (ISSN-L)
- 1941-0484
- OpenAlex
- W3204696009
- Cited by
- 161
- References
- 114
- Citations by year
- 2026: 11 · 2025: 28 · 2024: 40 · 2023: 51 · 2022: 28
- FWCI
- 12.09
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Journal of Selected Topics in Signal Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- speech recognition, pre-training, self-training, data efficiency, conformational isomers, model scaling, transfer learning, representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
