HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Open accessAbstract
Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training phase, and (3) sound units have variable lengths with no explicit segmentation. To deal with these three problems, we propose the Hidden-Unit BERT (HuBERT) approach for self-supervised speech representation learning, which utilizes an offline clustering step to provide aligned target labels for a BERT-like prediction loss. A key ingredient of our approach is applying the prediction loss over the masked regions only, which forces the model to learn a combined acoustic and language model over the continuous inputs. HuBERT relies primarily on the consistency of the unsupervised clustering step rather than the intrinsic quality of the assigned cluster labels. Starting with a simple k-means teacher of 100 clusters, and using two iterations of clustering, the HuBERT model either matches or improves upon the state-of-the-art wav2vec 2.0 performance on the Librispeech (960h) and Libri-light (60,000h) benchmarks with 10min, 1h, 10h, 100h, and 960h fine-tuning subsets. Using a 1B parameter model, HuBERT shows up to 19% and 13% relative WER reduction on the more challenging dev-other and test-other evaluation subsets.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The performance of the HuBERT model is better than that of the wav2vec 2.0 model.
“The results show that the performance of the HuBERT model is better than that of the wav2vec 2.0 model [9].”
From Voice Deepfake Detection Using the Self-Supervised Pre-Training Model HuBERT (2023) · cites this paper as [9]
The Introduction says HuBERT “either matches or improves upon” wav2vec 2.0 across the tested fine-tuning subsets. The Main Results section also says HuBERT is “on par” with two leading wav2vec 2.0-based pre-training results. Thus, the source supports comparable or sometimes better performance, but not an unqualified claim that HuBERT performs better. (LLM, conf=0.88)
Experimental — accuracy not yet measuredSelf-supervised speech models can extract robust acoustic features for unknown domains.
“Current research shows that self-supervised speech models can extract robust acoustic features for unknown domains.”
From Voice Deepfake Detection Using the Self-Supervised Pre-Training Model HuBERT (2023) · cites this paper as [9]
The Introduction (page unavailable) says self-supervised learning can produce universal speech representations and that its features generalize to multiple downstream applications. It also identifies new domains as a motivation, but the cited passages do not establish that the features are robust in unknown domains. (LLM, conf=0.82)
Experimental — accuracy not yet measuredHuBERT is a speech-representation learning model that predicts acoustic k-means-cluster identities for masked time points from unlabelled input waveforms.
“HuBERT is a speech-representation learning model that is trained to predict acoustic k-means-cluster identities corresponding to masked time points from unlabelled input waveforms.”
From A high-performance neuroprosthesis for speech decoding and avatar control (2023) · cites this paper as [12]
The sequence over time of discrete speech units derived by HuBERT captures the content of the original speech waveform.
“We refer to these cluster identities as discrete speech units, and the temporal sequence of these speech units represents the content of the original waveform.”
From A high-performance neuroprosthesis for speech decoding and avatar control (2023) · cites this paper as [12]
Hsu et al. introduced HuBERT, a novel self-supervised learning framework for speech representation learning that employs a more direct predictive loss by separating the acoustic unit discovery step from the masked prediction representation learning phase than wav2vec 2.0.
“Hsu et al. [56] introduced a novel self-supervised learning framework for speech representation learning named HuBERT, which employs a more direct predictive loss by separating the acoustic unit discovery step from the masked prediction representation learning phase than wav2vec 2.0.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [56]
On several speech recognition tasks, the searched LightHuBERT achieves comparable predictive performance as the baseline teacher model HuBERT, even though it has 29% fewer model parameters.
“On several speech recognition tasks, the searched LightHuBERT achieves comparable predictive performance as the baseline teacher model HuBERT [231], even though it has 29% fewer model parameters.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [231]
Audio codec contains abundant speaker information and acoustic information, which could maintain speaker identity in reconstruction compared to HuBERT codes.
“It contains abundant speaker information and acoustic information, which could maintain speaker identity in reconstruction compared to HuBERT codes [Hsu et al., 2021].”
From Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023) · cites this paper as Hsu et al., 2021
The HuBERT paper’s Introduction discusses speaker identity as an aspect of high-fidelity speech representation, and its Implementation section describes HuBERT codewords. Neither section compares HuBERT codes with audio codecs or assesses speaker-identity preservation in reconstruction; searches for those terms found no such comparison in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measuredRecent FER technologies have been implemented in educational institutions, facilitating applications such as monitoring student engagement and assessing learning outcomes through sentiment analysis.
“Recent FER technologies have been implemented in educational institutions [25, 26], facilitating applications such as the monitoring of student engagement and the assessment of learning outcomes through sentiment analysis.”
From Multimodal Dynamic Face Emotion Verification Model based on Priori Prompt Words for Dynamic Facial Expression Recognition (2025) · cites this paper as [26]
The Introduction and Conclusion (pages unavailable) describe self-supervised speech representations and speech-recognition results, not facial expression recognition in educational institutions. Searches for classroom use, student engagement, learning outcomes, and sentiment analysis found no passage addressing the attributed claim. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Least squares quantization in PCM
Lloyd, S. (1982) · IEEE Transactions on Information Theory · doi:10.1109/tit.1982.1056489 · Cited by 15,836 · see citing works
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Self-citationLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · see citing works
Self-citation(2007) · doi:10.5555/1283383.1283494 · Cited by 7,364 · see citing works
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · see citing works
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370 · see citing works
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · see citing works
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-4009 · Cited by 3,538 · see citing works
Conformer: Convolution-augmented Transformer for Speech Recognition
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.08100 · Cited by 3,150 · see citing works
Conformer: Convolution-augmented Transformer for Speech Recognition
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-3015 · Cited by 3,150 · see citing works
Neural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729 · see citing works
Deep Clustering for Unsupervised Learning of Visual Features
Caron, M., Bojanowski, P., Joulin, A., & Douze, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_9 · Cited by 2,674 · see citing works
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., & Joulin, A. (2020) · arXiv · doi:10.48550/arxiv.2006.09882 · Cited by 2,499 · see citing works
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369 · see citing works
Self-citationSpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827 · see citing works
Gray, R. M., & Neuhoff, D. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.720541 · Cited by 1,647 · see citing works
wav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290 · see citing works
Connectionist Speech Recognition: A Hybrid Approach
Bourlard, H. A., & Morgan, N. H. (1993) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,115 · see citing works
Applying Convolutional Neural Networks concepts to hybrid NN-HMM model for speech recognition
Abdel‐Hamid, O., Mohamed, A., Jiang, H., & Penn, G. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288864 · Cited by 892 · see citing works
Self-citationLibri-Light: A Benchmark for ASR with Limited or No Supervision
Kahn, J., Rivière, M., Zheng, W., Kharitonov, E., Xu, Q., Mazaré, P.-E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9052942 · Cited by 475 · see citing works
Self-citationvq-wav2vec: Self-Supervised Learning of Discrete Speech RepresentationsPreprint
Baevski, A., Schneider, S., & Auli, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05453 · Cited by 431 · see citing works
A word graph algorithm for large vocabulary continuous speech recognition
Ortmanns, S., Ney, H., & Aubert, X. L. (1997) · Computer Speech & Language · doi:10.1006/csla.1996.0022 · Cited by 400 · see citing works
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020) · neural information processing systems · Cited by 386 · see citing works
Self-citationExploring Simple Siamese Representation Learning
Chen, X., & He, K. (2020) · arXiv · doi:10.48550/arxiv.2011.10566 · Cited by 381 · see citing works
An Unsupervised Autoregressive Model for Speech Representation Learning
Chung, Y.-A., Hsu, W.-N., Tang, H., & Glass, J. R. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1473 · Cited by 356 · see citing works
Self-citationMockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
Liu, A. T., Yang, S.-W., Chi, P.-H., Hsu, P.-C., & Lee, H.-Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054458 · Cited by 335 · see citing works
Pushing the Limits of Semi-Supervised Learning for Automatic Speech RecognitionPreprint
Zhang, Y., Qin, J., Park, D., Han, W., Chiu, C., Pang, R., Le, Q. V., & Wu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10504 · Cited by 233 · see citing works
Learning Problem-Agnostic Speech Representations from Multiple Self-Supervised Tasks
Pascual, S., Ravanelli, M., Serrà, J., Bonafonte, A., & Bengio, Y. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2605 · Cited by 222 · see citing works
A review of large-vocabulary continuous-speech
Young, S. J. (1996) · IEEE Signal Processing Magazine · doi:10.1109/79.536824 · Cited by 216 · see citing works
Improved Noisy Student Training for Automatic Speech Recognition
Park, D., Zhang, Y., Ye, J., Han, W., Chiu, C., Li, B., Wu, Y., & Le, Q. V. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1470 · Cited by 214 · see citing works
A Nonparametric Bayesian Approach to Acoustic Model Discovery
Lee, C.-Y., & Glass, J. R. (2012) · Cited by 210 · see citing works
Self-Training for End-to-End Speech Recognition
Kahn, J., Lee, A., & Hannun, A. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054295 · Cited by 205 · see citing works
Wav2Letter++: A Fast Open-source Speech Recognition System
Pratap, V., Hannun, A., Xu, Q., Cai, J., Kahn, J., Synnaeve, G., Liptchinsky, V., & Collobert, R. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683535 · Cited by 174 · see citing works
Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
Hsu, W.-N., Zhang, Y., & Glass, J. (2017) · neural information processing systems · Cited by 166 · see citing works
Self-citationGenerative Pre-Training for Speech with Autoregressive Predictive Coding
Chung, Y.-A., & Glass, J. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054438 · Cited by 157 · see citing works
Deep Contextualized Acoustic Representations for Semi-Supervised Speech Recognition
Ling, S., Liu, Y., Salazar, J., & Kirchhoff, K. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053176 · Cited by 145 · see citing works
Learning Latent Representations for Speech Generation and Transformation
Hsu, W.-N., Zhang, Y., & Glass, J. R. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-349 · Cited by 141 · see citing works
Self-citationHubert: How Much Can a Bad Teacher Benefit ASR Pre-Training?
Hsu, W.-N., Tsai, Y.-H. H., Bolte, B., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414460 · Cited by 123 · see citing works
Self-citationIterative Pseudo-Labeling for Speech Recognition
Xu, Q., Likhomanenko, T., Kahn, J., Hannun, A., Synnaeve, G., & Collobert, R. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1800 · Cited by 112 · see citing works
Effectiveness of self-supervised pre-training for speech recognitionPreprint
Baevski, A., Auli, M., & Mohamed, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03912 · Cited by 105 · see citing works
Self-citationVariational Inference for Acoustic Unit Discovery
Ondel, L., Burget, L., & Černocký, J. (2016) · Procedia Computer Science · doi:10.1016/j.procs.2016.04.033 · Cited by 94 · see citing works
Self-supervised Learning from a Multi-view PerspectivePreprint
Tsai, Y.-H. H., Wu, Y., Salakhutdinov, R., & Morency, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05576 · Cited by 91 · see citing works
Self-citationUnsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction
Wang, W., Tang, Q., & Livescu, K. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053541 · Cited by 81 · see citing works
From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition
Le, D., Zhang, X., Zheng, W., Fügen, C., Zweig, G., & Seltzer, M. L. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003972 · Cited by 71 · see citing works
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector QuantizationPreprint
Ling, S., & Liu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.06659 · Cited by 60 · see citing works
Improved Speech Representations with Multi-Target Autoregressive Predictive Coding
Chung, Y.-A., & Glass, J. R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.213 · Cited by 55 · see citing works
Unsupervised Training on Large Amounts of Broadcast News Data
Ma, J., Matsoukas, S., Kimball, O., et al. (2006) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2006.1660839 · Cited by 51 · see citing works
slimIPL: Language-Model-Free Iterative Pseudo-Labeling
Likhomanenko, T., Xu, Q., Kahn, J., Synnaeve, G., & Collobert, R. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-740 · Cited by 46 · see citing works
An Unsupervised Autoregressive Model for Speech Representation LearningPreprint
Chung, Y.-A., Hsu, W.-N., Tang, H., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03240 · Cited by 45 · see citing works
Self-citationHidden Markov Model Variational Autoencoder for Acoustic Unit Discovery
Ebbers, J., Heymann, J., Drude, L., Glarner, T., Haeb‐Umbach, R., & Raj, B. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1160 · Cited by 44 · see citing works
Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio RepresentationPreprint
Chi, P.-H., Chung, P.-H., Wu, T.-H., Hsieh, C.-C., Chen, Y.-H., Li, S.-W. D., & Lee, H.-Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.08575 · Cited by 41 · see citing works
A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning
Khurana, S., Laurent, A., Hsu, W.-N., Chorowski, J., Łańcucki, A., Marxer, R., & Glass, J. R. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-3084 · Cited by 28 · see citing works
Self-citationIterative Pseudo-Labeling for Speech RecognitionPreprint
Xu, Q., Likhomanenko, T., Kahn, J., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.09267 · Cited by 27 · see citing works
Semi-Supervised Speech Recognition via Local Prior MatchingPreprint
Hsu, W.-N., Lee, A., Synnaeve, G., & Hannun, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10336 · Cited by 26 · see citing works
Self-citationFull Bayesian Hidden Markov Model Variational Autoencoder for Acoustic Unit Discovery
Glarner, T., Hanebrink, P., Ebbers, J., et al. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2148 · Cited by 23 · see citing works
Contrastive Semi-Supervised Learning for ASR
Xiao, A., Fuegen, C., & Mohamed, A. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414079 · Cited by 16 · see citing works
Self-citationA Factorial Deep Markov Model for Unsupervised Disentangled Representation Learning from Speech
Khurana, S., Joty, S., Ali, A. M. A., et al. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683131 · Cited by 14 · see citing works
Self-training and Pre-training are Complementary for Speech RecognitionPreprint
Xu, Q., Baevski, A., Likhomanenko, T., Tomasello, P., Conneau, A., Collobert, R., Synnaeve, G., & Auli, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11430 · Cited by 11 · see citing works
Contrastive Semi-supervised Learning for ASRPreprint
Xiao, A., Fuegen, C., & Mohamed, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.05149 · Cited by 2 · see citing works
Self-citationImproved Speech Representations with Multi-Target Autoregressive Predictive CodingPreprint
Chung, Y.-A., & Glass, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05274 · Cited by 1 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 6
- [3]3. E. Kharitonov, M. Rivière, G. Synnaeve, L. Wolf, P.-E. Mazaré, M. Douze, and E. Dupoux, "Data augmenting contrastive learn- ing of speech representations in the time domain," arXiv preprint arXiv:2007.00991, 2020. · arXiv
- [7]7. G. Zavaliagkos and T. Colthurst, "Utilizing untranscribed training data to improve performance," in DARPA Broadcast News Transcription and Understanding Workshop, 1998.
- [20]20. K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, "Electra: Pre- training text encoders as discriminators rather than generators," arXiv preprint arXiv:2003.10555, 2020. · arXiv
- [31]31. J. Chorowski, R. J. Weiss, S. Bengio, and A. van den Oord, "Unsu- pervised speech representation learning using wavenet autoencoders," IEEE/ACM transactions on audio, speech, and language processing, vol. 27, no. 12, pp. 2041-2053, 2019.
- [37]37. D. Povey, "Discriminative training for large vocabulary speech recogni- tion," Ph.D. dissertation, University of Cambridge, 2005.
- [59]59. V. Pratap et al., "wav2letter++: The fastest open-source speech recog- nition system," arXiv preprint arXiv:1812.07625, 2018. · arXiv
Cited by
Sorted by citations
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3188113 · Cited by 1,926 · see citing works
SUPERB: Speech Processing Universal PERformance Benchmark
Yang, S.-W., Chi, P.-H., Chuang, Y.-S., Lai, C.-I. J., Lakhotia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T., et al. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1775 · Cited by 681 · see citing works
Self-citationThe rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970 · see citing works
A high-performance neuroprosthesis for speech decoding and avatar control
Metzger, S. L., Littlejohn, K. T., Silva, A. B., Moses, D. A., Seaton, M. P., Wang, R., Dougherty, M. E., Liu, J. R., Wu, P., Berger, M. A., et al. (2023) · Nature · doi:10.1038/s41586-023-06443-4 · Cited by 522 · see citing works
A survey on LLM-based multi-agent systems: Workflow, infrastructure, and challenges
Li, X., Wang, S., Zeng, S., Wu, Y., & Yang, Y. (2024) · Vicinagearth · doi:10.1007/s44336-024-00009-2 · Cited by 404 · see citing works
Self-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372 · see citing works
“3) Challenges: The iterative nature of pre-training for the HuBERT and wavLM could present a practical inconvenience when working with large volumes of data.”
MethodInfluentialSelf-citationCompute Trends Across Three Eras of Machine Learning
Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P. (2022) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9891914 · Cited by 363 · see citing works
WhisperX: Time-Accurate Speech Transcription of Long-Form Audio
Bain, M., Huh, J., Han, T., & Zisserman, A. (2023) · INTERSPEECH 2023 · doi:10.21437/interspeech.2023-78 · Cited by 337 · see citing works
Chung, Y.-A., Zhang, Y., Han, W., Chiu, C.-C., Qin, J., Pang, R., & Wu, Y. (2021) · 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru51503.2021.9688253 · Cited by 293 · see citing works
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.02111 · Cited by 250 · see citing works
An analog-AI chip for energy-efficient speech recognition and transcription
Ambrogio, S., Narayanan, P., Okazaki, A., Fasoli, A., Mackin, C., Hosokawa, K., Nomura, A., Yasuda, T., Chen, A., Friz, A., et al. (2023) · Nature · doi:10.1038/s41586-023-06337-5 · Cited by 242 · see citing works
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224 · see citing works
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154 · see citing works
Investigating Self-Supervised Front Ends for Speech Spoofing Countermeasures
Wang, X., & Yamagishi, J. (2022) · The Speaker and Language Recognition Workshop (Odyssey 2022) · doi:10.21437/odyssey.2022-14 · Cited by 138 · see citing works
Zhang, L., Wang, X., Cooper, E., Evans, N., & Yamagishi, J. (2023) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2022.3233236 · Cited by 100 · see citing works
Large-Scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification
Chen, Z., Chen, S., Wu, Y., Qian, Y., Wang, C., Liu, S., Qian, Y., & Zeng, M. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9747814 · Cited by 112 · see citing works
COCOA: Cross Modality Contrastive Learning for Sensor Data
Deldari, S., Xue, H., Saeed, A., Smith, D. V., & Salim, F. D. (2022) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3550316 · Cited by 92 · see citing works
Speech based detection of Alzheimer's disease: a survey of AI techniques, datasets and challenges
Ding, K., Chetty, M., Noori Hoshyar, A., Bhattacharya, T., & Klein, B. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10961-6 · Cited by 82 · see citing works
Pre-trained models for detection and severity level classification of dysarthria from speech
Javanmardi, F., Kadiri, S. R., & Alku, P. (2024) · Speech Communication · doi:10.1016/j.specom.2024.103047 · Cited by 74 · see citing works
Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers
Gong, Y., Khurana, S., Karlinsky, L., & Glass, J. (2023) · INTERSPEECH 2023 · doi:10.21437/interspeech.2023-2193 · Cited by 65 · see citing works
Deep Multimodal Emotion Recognition on Human Speech: A Review
Koromilas, P., & Giannakopoulos, T. (2021) · Applied Sciences · doi:10.3390/app11177962 · Cited by 58 · see citing works
A Robust Audio Deepfake Detection System via Multi-View Feature
Yang, Y., Qin, H., Zhou, H., Wang, C., Guo, T., Han, K., & Wang, Y. (2024) · ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp48485.2024.10446560 · Cited by 57 · see citing works
SUPERB: Speech processing Universal PERformance Benchmark
Yang, S.-W., Chi, P.-H., Chuang, Y.-S., Lai, C.-I. J., Lakhotia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.01051 · Cited by 52 · see citing works
Self-citationA conformer-based classifier for variable-length utterance processing in anti-spoofing
Rosello, E., Gomez-Alanis, A., Gomez, A. M., & Peinado, A. (2023) · INTERSPEECH 2023 · doi:10.21437/interspeech.2023-1820 · Cited by 47 · see citing works
SafeEar: Content Privacy-Preserving Audio Deepfake Detection
Li, X., Li, K., Zheng, Y., Yan, C., Ji, X., & Xu, W. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670285 · Cited by 39 · see citing works
Chumachenko, K., Iosifidis, A., & Gabbouj, M. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00470 · Cited by 34 · see citing works
Feng, T., & Narayanan, S. (2023) · 2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII) · doi:10.1109/acii59096.2023.10388152 · Cited by 34 · see citing works
Voice Deepfake Detection Using the Self-Supervised Pre-Training Model HuBERT
Li, L., Lu, T., Ma, X., Yuan, M., & Wan, D. (2023) · Applied Sciences · doi:10.3390/app13148488 · Cited by 33 · see citing works
Word Segmentation on Discovered Phone Units With Dynamic Programming and Self-Supervised Scoring
Kamper, H. (2023) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2022.3229264 · Cited by 24 · see citing works
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
Chu, Y., Xu, J., Zhou, X., Yang, Q., Zhang, S., Yan, Z., Zhou, C., & Zhou, J. (2023) · arXiv · doi:10.48550/arxiv.2311.07919 · Cited by 61 · see citing works
“SpeechGPT (Zhang et al., 2023a) first converts human speech into discrete HuBERT tokens (Hsu et al., 2021), and then designs a three-stage training pipeline on paired speech data, speech instruction data and chain-of-modality instruction data accordingly.”
MethodLeveraging Self-Supervised Learning for Speaker Diarization
Han, J., Landini, F., Rohdin, J., Silnova, A., Diez, M., & Burget, L. (2025) · ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49660.2025.10889475 · Cited by 23 · see citing works
Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speech
Cho, C. J., Wu, P., Mohamed, A., & Anumanchipalli, G. K. (2023) · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49357.2023.10094711 · Cited by 21 · see citing works
Self-citationChou, H.-C., Goncalves, L., Leem, S.-G., Salman, A. N., Lee, C.-C., & Busso, C. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3411290 · Cited by 19 · see citing works
Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptation
Huang, K. P., Fu, Y.-K., Zhang, Y., & Lee, H.-Y. (2022) · arXiv · doi:10.48550/arxiv.2203.16104 · Cited by 19 · see citing works
Text-Free Prosody-Aware Generative Spoken Language Modeling
Kharitonov, E., Lee, A., Polyak, A., Adi, Y., Copet, J., Lakhotia, K., Nguyen, T.-A., Rivière, M., Mohamed, A., Dupoux, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2109.03264 · Cited by 16 · see citing works
Self-citationWavLM model ensemble for audio deepfake detection
Combei, D., Stan, A., Oneata, D., & Cucu, H. (2024) · The Automatic Speaker Verification Spoofing Countermeasures Workshop (ASVspoof 2024) · doi:10.21437/asvspoof.2024-25 · Cited by 15 · see citing works
“• HuBERT [11] is a Hidden-Unit BERT approach for self-supervised speech representation learning.”
MethodHaresamudram, H., Essa, I., & Plötz, T. (2024) · Sensors · doi:10.3390/s24041238 · Cited by 14 · see citing works
Zhu, Y., Powar, S., & Falk, T. H. (2024) · 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10626010 · Cited by 11 · see citing works
Sparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17 · see citing works
“They utilised Hidden-unit BERT (HuBERT) [92] as a discrete unit extractor for the transformation of continuous speech to discrete units, the Meta AI LLaMA [93] model as LLM, and HiFi-GAN as a unit vocoder.”
MethodTahaoglu, G. (2025) · Applied Sciences · doi:10.3390/app15179685 · Cited by 11 · see citing works
AWaveFormer: Audio Wavelet Transformer Network for Generalized Audio Deepfake Detection
Wang, R., Chen, Z., Wang, B., Ba, Z., & Ren, K. (2025) · IEEE Transactions on Audio, Speech and Language Processing · doi:10.1109/taslpro.2025.3611229 · Cited by 10 · see citing works
Schäfer, K., Choi, J.-E., & Neu, M. (2024) · The Automatic Speaker Verification Spoofing Countermeasures Workshop (ASVspoof 2024) · doi:10.21437/asvspoof.2024-9 · Cited by 8 · see citing works
“As only front-ends not pre-trained on MLS or other databases sourced from LibriVox audiobooks [10] were allowed, we choose to use HuBERT base [11], HuBERT-MGR [12] (hubert base robust mgr) and Wav2Vec2.”
Harnessing machine learning in diagnosing complex hoarseness cases
Roitman, A., Edelstain, Y., Katzir, C., Ofir, H., Peleg, N., Doweck, I., & Yanir, Y. (2025) · American Journal of Otolaryngology · doi:10.1016/j.amjoto.2024.104533 · Cited by 6 · see citing works
Yang, C., Yu, C., Chen, H., Zhu, J., Chen, J., Chen, K., Wang, W., Wang, Y., Jiang, Y., Jiang, Y., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.01802
“Self-supervised speech encoders such as HuBERT [54] and WavLM [20] learn robust representations for speech recognition and full-stack speech processing.”
Xie, H. (2025) · Proceedings of the 2025 International Conference on Artificial Intelligence and Educational Systems · doi:10.1145/3744367.3744415
Measuring the Robustness of Audio Deepfake Detection under Real-World Corruption
Li, X., Chen, P.-Y., & Wei, W. (2026) · Proceedings of the Sixteenth ACM Conference on Data and Application Security and Privacy · doi:10.1145/3800506.3803494
- Authors
- Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A.
- Affiliations
- Meta (United States) (US)
- Carnegie Mellon University (US)
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2021
- Volume
- 29
- Pages
- 3451-3460
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W3169320628
- Cited by
- 2,983
- References
- 69
- Citations by year
- 2026: 661 · 2025: 904 · 2024: 651 · 2023: 544 · 2022: 199 · 2021: 24
- FWCI
- 125.27
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/ACM Transactions on Audio Speech and Language Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- HuBERT, self-supervised speech representation learning, K-means clustering, wav2vec 2.0, LibriSpeech, speech recognition, unsupervised pre-training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
