Self-Supervised Speech Representation Learning: A Review
Open accessThe DOI matches its Crossref record; 361 of the 362 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Although supervised deep learning has revolutionized speech and audio processing, it has necessitated the building of specialist models for individual tasks and application scenarios. It is likewise difficult to apply this to dialects and languages for which only limited labeled data is available. Self-supervised representation learning methods promise a single universal model that would benefit a wide variety of tasks and domains. Such methods have shown success in natural language processing and computer vision domains, achieving new levels of performance while reducing the number of labels required for many downstream scenarios. Speech representation learning is experiencing similar progress in three main categories: generative, contrastive, and predictive methods. Other approaches rely on multi-modal data for pre-training, mixing text or visual data streams with speech. Although self-supervised speech representation is still a nascent research area, it is closely related to acoustic word embedding and learning with zero lexical resources, both of which have seen active research for many years. This review presents approaches for self-supervised speech representation learning and their connection to other research areas. Since many current methods focus solely on automatic speech recognition as a downstream task, we review recent efforts on benchmarking learned representations to extend the application beyond speech recognition.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Many popular self-supervised learning (SSL) models (Mohamed et al., 2022, Table IV) are trained using the Librispeech database”
From ASVspoof 5: Design, collection and validation of resources for spoofing, deepfake, and adversarial attack detection using crowdsourced speech (2026) · cites this paper as Mohamed et al., 2022
In the citing paper
“A previous study [34] already proposed a method to adapt SimCLR to the HAR domain, finding that random 3D rotation was an effective transformation for accelerometer data.”
From Comparing self-supervised learning techniques for wearable human activity recognition (2025) · cites this paper as [34]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–362 of 362
Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition
Thomas, B., Kessler, S., & Karout, S. A. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746223 · Cited by 106
A Unified Energy-Based Framework for Unsupervised Learning
Ranzato, M., Boureau, Y.-L., Chopra, S., & LeCun, Y. (2007) · International Conference on Artificial Intelligence and Statistics · Cited by 105
State clustering in hidden Markov model-based continuous speech recognition
Young, S. J., & Woodland, P. C. (1994) · Computer Speech & Language · doi:10.1006/csla.1994.1019 · Cited by 103 · cited 1×
“phonemes; state-clustering algorithms [29] then resulted in GMMs for context-dependent phonemes.”
BackgroundA neural implementation of canonical correlation analysis
Lai, P., & Fyfe, C. (1999) · Neural Networks · doi:10.1016/s0893-6080(99)00075-1 · Cited by 103 · cited 1×
“Some typical approaches in this category include Multi-view autoencoders and variations [154], [155], Multi-modal deep Boltzmann machines [156], Canonical correlation analysis (CCA) [157] and its nonlinear extensions [158], [159], [160], [161], [162], [163], [164], [165], [166], Multi-view contrastive losses [167], [1…”
MethodMultilingual representations for low resource speech recognition and keyword search
Cui, J., Kingsbury, B. E. D., Ramabhadran, B., Sethy, A., Audhkhasi, K., Cui, X., Kislal, E. E., Mangu, L., Nußbaum-Thom, M., Picheny, M., et al. (2015) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2015.7404803 · Cited by 101 · cited 1×
“Multilingual and cross-lingual supervised models have proven superior in low-resource speech recognition tasks [66].”
BackgroundAn Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition
Chang, X., Maekaku, T., Guo, P., Shi, J. Y., Lu, Y.-J., Subramanian, A. S., Wang, T., Yang, S.-W., Tsao, Y., Lee, H.-Y., et al. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688137 · Cited by 101 · cited 1×
“Similar problems were also investigated using HuBERT and wav2vec 2.0 models in [280].”
MethodInfluentialSelf-citationTowards Learning Universal Audio Representations
Wang, L., Luc, P., Wu, Y., Recasens, A., Smaira, L., Brock, A., Jaegle, A., Alayrac, J.-B., Dieleman, S., Carreira, J., et al. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746790 · Cited by 101 · cited 4×
“Apart from the speech community, researchers have also established HEAR (holistic evaluation of audio representations) [262], NOSS (non-semantic speech benchmark) [263], and HARES (holistic audio representation evaluation suite) [264] to benchmark audio representations.”
BackgroundInfluentialKocabiyikoglu, A. C., Besacier, L., & Kraif, O. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4tgpdx9xax4j · Cited by 101 · cited 3×
“VoxCeleb1 [236] for SID/ASV (automatic speaker verification) tasks, FSC (Fluent Speech Commands) [237] for IC (intent classification), QUESST (QUESST 2014) [238] for QbE (query by example), LS En-Fr [239] and CoVoST-2 [240] for ST (speech translation), and ALFFA and OpenSLR-multi for multilingual ASR.”
MethodTextless Speech-to-Speech Translation on Real Data
Lee, A., Gong, H., Duquenne, P.-A., Schwenk, H., Chen, P.-J., Wang, C., Popuri, S., Adi, Y., Pino, J., Gu, J., et al. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.63 · Cited by 100 · cited 1×
“tions such as spoken question answering [333], speech separation [334], TTS [335], and speech-to-speech translation [336].”
BackgroundRenshaw, D., Kamper, H., Jansen, A., & Goldwater, S. J. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-644 · Cited by 99 · cited 1×
“In addition to the approaches discussed above, several other works on speech representation learning take inspiration from the AE framework [86], [87], [88], [89], [90], [91].”
BackgroundTowards End-to-End Unsupervised Speech Recognition
Liu, A. H., Hsu, W.-N., Auli, M., & Baevski, A. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10023187 · Cited by 99 · cited 2×
“0 [296] further removes the requirement of the segmentation stage, so the unsupervised ASR model can be learned in an end-”
BackgroundMethodEffectiveness of Self-Supervised Pre-Training for ASR
Baevski, A., & Mohamed, A. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054224 · Cited by 98 · cited 1×
“The Discrete BERT approach [129] uses a pre-trained vq-wav2vec”
MethodSelf-citationVector-Quantized Autoregressive Predictive Coding
Chung, Y.-A., Tang, H., & Glass, J. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1228 · Cited by 97 · cited 2×
“In VQ-APC [98], quantization is used with the APC”
MethodLearning Speaker Representations with Mutual Information
Ravanelli, M., & Bengio, Y. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2380 · Cited by 96 · cited 3×
“ConvDMM [235] LS 50 [234]/360/ 960 hrs PC/PR WSJ si284 WSJ eval92 X 5(12)/50(12)/100(12) mins, 4(12)/8(12)/40(12) hrs”
BackgroundInfluentialUnsupervised Cross-Modal Alignment of Speech and Text Embedding SpacesPreprint
Chung, Y.-A., Weng, W., Tong, S., & Glass, J. R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.07467 · Cited by 95 · cited 1×
“[286], [288] use adversarial training to map the word-level speech embedding”
BackgroundA segmental framework for fully-unsupervised large-vocabulary speech recognition
Kamper, H., Jansen, A., & Goldwater, S. (2017) · Computer Speech & Language · doi:10.1016/j.csl.2017.04.008 · Cited by 94 · cited 1×
“For example, searching within a corpus of recorded speech for segments that match a given (written or spoken) query can be seen as finding segments whose representations are most similar to that of the query [118], [192], [193], [194]; word embeddings can be defined by pooling representations of instances of a given w…”
BackgroundQuery-by-Example Search with Discriminative Neural Acoustic Word Embeddings
Settle, S., Levin, K., Kamper, H., & Livescu, K. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1592 · Cited by 93 · cited 1×
“For example, searching within a corpus of recorded speech for segments that match a given (written or spoken) query can be seen as finding segments whose representations are most similar to that of the query [118], [192], [193], [194]; word embeddings can be defined by pooling representations of instances of a given w…”
BackgroundSelf-citationThe Zero Resource Speech Challenge 2019: TTS Without T
Dunbar, E., Algayres, R., Karadayi, J., Bernard, M., Benjumea, J., Cao, X.-N., Miskic, L., Dugrain, C., Ondel, L., Black, A. W., et al. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2904 · Cited by 93 · cited 1×
“The Zero Resource Speech Challenge 2019 [325] extended”
BackgroundWang, Y., Boumadane, A., & Heba, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.02735 · Cited by 93
Effectiveness of self-supervised pre-training for speech recognition
Baevski, A., Auli, M., & Mohamed, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03912 · Cited by 93 · cited 3×
“The Discrete BERT approach [129] uses a pre-trained vq-wav2vec model to derive a discrete vocabulary [124].”
BackgroundMethodInfluentialSelf-citationCommunity Evaluation and Exchange of Word Vectors at wordvectors.org
Faruqui, M., & Dyer, C. (2014) · Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.3115/v1/p14-5004 · Cited by 92
Improving Transformer-based Speech Recognition Using Unsupervised Pre-training
Jiang, D., Lei, X., Li, W., Luo, N., Hu, Y., Zou, W., & Li, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.09932 · Cited by 92 · cited 11×
“We use the following abbreviations: EN : English; Multi : multilingual; ZH : Chinese; Fr : French; ASR : automatic speech recognition; PR : phoneme recognition; PC : phoneme classification; SID : speaker identification; ASV : automatic speaker verification; Sentiment : sentiment analysis; ST : speech translation; QbE…”
BackgroundMethodInfluentialInfants use multimodal information to imitate speech sounds
Legerstee, M. (1990) · Infant Behavior and Development · doi:10.1016/0163-6383(90)90039-b · Cited by 91 · cited 1×
“provides a proof of concept for this, as it is believed that infants benefit from the visual modality when learning language [146].”
BackgroundVariational Inference for Acoustic Unit Discovery
Ondel, L., Burget, L., & Černocký, J. (2016) · Procedia Computer Science · doi:10.1016/j.procs.2016.04.033 · Cited by 91 · cited 1×
“Following the success of representation learning, baseline and challenge techniques have shifted from purely generative models [327], [328], deep autoencoders [83], [329], and incorporation of neural-network-”
BackgroundCycle-consistency Training for End-to-end Speech Recognition
Hori, T., Astudillo, R. F., Hayashi, T., Zhang, Y., Watanabe, S., & Roux, J. L. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683307 · Cited by 91
Self-citationDeep Variational Canonical Correlation Analysis
Wang, W., Yan, X., Lee, H., & Livescu, K. (2016) · arXiv · doi:10.48550/arxiv.1610.03454 · Cited by 89 · cited 1×
“Some typical approaches in this category include Multi-view autoencoders and variations [154], [155], Multi-modal deep Boltzmann machines [156], Canonical correlation analysis (CCA) [157] and its nonlinear extensions [158], [159], [160], [161], [162], [163], [164], [165], [166], Multi-view contrastive losses [167], [1…”
MethodSelf-citationRapid evaluation of speech representations for spoken term discovery
Carlin, M. A., Thomas, S., Jansen, A., & Hermansky, H. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-304 · Cited by 88 · cited 1×
“These are often evaluated on word discrimination, that is the task of determining whether two word segments correspond to the same word or not [203].”
BackgroundLaptev, A., Korostik, R., Svischev, A., Andrusenko, A., Medennikov, I., & Rybin, S. (2020) · International Congress on Image and Signal Processing (CISP) · doi:10.1109/cisp-bmei51763.2020.9263564 · Cited by 87 · cited 1×
“straightforward approach is to simply use multi-speaker TTS to generate the waveform with various acoustic variations [314], [315], [316], [317], [318].”
MethodLearning Hierarchical Discrete Linguistic Units from Visually-Grounded SpeechPreprint
Harwath, D. F., Hsu, W.-N., & Glass, J. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09602 · Cited by 87 · cited 1×
“other contextual modality), such models often learn multiple levels of linguistic representations from the shallowest to the deepest model layers [181], [182], [183].”
BackgroundLearning Robust and Multilingual Speech Representations
Kawakami, K., Wang, L., Dyer, C., Blunsom, P., & van den Oord, A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.106 · Cited by 86 · cited 10×
“and conventionally used for evaluation, while their audio streams are also aggregated to build diversified and large-scale corpora for unsupervised pre-training [109], [123], [218].”
BackgroundMethodResultInfluentialUnispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training
Chen, S., Wu, Y., Wang, C., Chen, Z., Chen, Z., Liu, S., Wu, J., Qian, Y., Wei, F., Li, J., et al. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9747077 · Cited by 85 · cited 4×
“At the same time, we observe a growing industry investment in pre-training with larger datasets, e.g., CPC-8k (8k hours) for Bidir-CPC [123], LL (60k hours) for CPC modified [122], wav2vec 2.0 [125], and HuBERT [130], Alexa internal datasets (10k hours) for wav2vec-c [126], Didi internal datasets (10k hours) for MPC […”
BackgroundMethodInfluentialPARP: Prune, Adjust and Re-Prune for Self-Supervised Speech RecognitionPreprint
Lai, C.-I., Zhang, Y., Liu, A. H., Chang, S., Liao, Y.-L., Chuang, Y.-S., Qian, K., Khurana, S., Cox, D., & Glass, J. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05933 · Cited by 81 · cited 1×
“representation models through network pruning [342] or knowledge distillation [343].”
BackgroundThe Speech Accent Archive: towards a typology of English accents
Weinberger, S. H., & Kunath, S. A. (2011) · BRILL eBooks · doi:10.1163/9789401206884_014 · Cited by 80 · cited 1×
“18 A collection of AudioSet, AVSpeech, CV-dataset, LS, WSJ, TIMIT, Speech Accent Archive (SSA) [253], TED3, and SWB. SSA is a growing annotated corpus of English speech with various accents.”
BackgroundUnsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction
Wang, W., Tang, Q., & Livescu, K. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053541 · Cited by 79 · cited 2×
“Whereas most studies use masking along the temporal dimension of the input, speech can also be masked along the frequency dimension when spectral input features are used [103], [106].”
BackgroundMethodSelf-citationAn auto-encoder based approach to unsupervised learning of subword units
Badino, L., Canevari, C., Fadiga, L., et al. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6855085 · Cited by 77 · cited 1×
“In addition to the approaches discussed above, several other works on speech representation learning take inspiration from the AE framework [86], [87], [88], [89], [90], [91].”
BackgroundLarge-Scale Representation Learning from Visually Grounded Untranscribed Speech
Ilharco, G., Zhang, Y., & Baldridge, J. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1006 · Cited by 77 · cited 1×
“Other choices include training with a masked margin softmax loss [177], [178] or a masked prediction loss [179].”
BackgroundA multimodal real-time MRI articulatory corpus for speech research
Narayanan, S. S., Bresch, E., Ghosh, P., et al. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-308 · Cited by 76 · cited 1×
“face [148], [149], lip-movement [150], articulatory flesh point measurements [151], [152], or simultaneous magnetic resonance imaging (MRI) scans [153].”
BackgroundWhich *BERT? A Survey Organizing Contextualized Encoders
Xia, P., Wu, S., & Durme, B. V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.608 · Cited by 76 · cited 1×
“[21], [22] and computer vision (CV) [23].”
Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies
Liu, A. H., Chung, Y.-A., & Glass, J. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-349 · Cited by 75 · cited 1×
“In non-autoregressive predictive coding (NPC) [107], time masking is introduced through masked convolution blocks.”
BackgroundOn Generative Spoken Language Modeling from Raw Audio
Lakhotia, K., Kharitonov, E., Hsu, W.-N., Adi, Y., Polyak, A., Bolte, B., Nguyen, T.-A., Copet, J., Baevski, A., Mohamed, A., et al. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00430 · Cited by 75 · cited 7×
“The generative spoken language model (GSLM) [78] utilizes discrete representations from wav2vec 2.0, HuBERT, and CPC algorithms as inputs to an autoregressive language model trained by using the cross-entropy function to maximize the probability of predicting the next discrete speech token.”
BackgroundMethodInfluentialSelf-citationNonparametric Canonical Correlation AnalysisPreprint
Michaeli, T., Wang, W., & Livescu, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04839 · Cited by 74 · cited 1×
“Some typical approaches in this category include Multi-view autoencoders and variations [154], [155], Multi-modal deep Boltzmann machines [156], Canonical correlation analysis (CCA) [157] and its nonlinear extensions [158], [159], [160], [161], [162], [163], [164], [165], [166], Multi-view contrastive losses [167], [1…”
MethodSelf-citationSparse coding for speech recognition
Sivaram, G. S. V. S., Nemala, S. K., Elhilali, M., Tran, T. D., & Heřmanský, H. (2010) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2010.5495649 · Cited by 73 · cited 1×
“trastive estimation (NCE) [37], sparse coding [38], [39], [40], and energy-based methods [41].”
Backgroundwav2vec-C: A Self-Supervised Model for Speech Representation Learning
Sadhu, S., He, D., Huang, C.-W., Mallidi, S. H., Wu, M., Rastrow, A., Stolcke, A., Droppo, J., & Maas, R. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-717 · Cited by 72 · cited 5×
“For instance, 10 k hours of real-world far-field English voice commands for self-supervised pre-training have been collected at Amazon [126].”
BackgroundMethodInfluentialSegmental acoustic indexing for zero resource keyword search
Levin, K., Jansen, A., & Durme, B. V. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7179089 · Cited by 70 · cited 1×
“For example, searching within a corpus of recorded speech for segments that match a given (written or spoken) query can be seen as finding segments whose representations are most similar to that of the query [118], [192], [193], [194]; word embeddings can be defined by pooling representations of instances of a given w…”
BackgroundLessons from Building Acoustic Models with a Million Hours of Speech
Krishnan Parthasarathi, S. H., & Strom, N. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683690 · Cited by 70 · cited 1×
“relies on a supervised teacher model to label a large volume of speech-only data, which is then used to augment the initial labeled data to train a student model [4], [5], [6], [61].”
MethodVision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech
Harwath, D. F., Chuang, G., & Glass, J. R. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462396 · Cited by 69 · cited 1×
“be used for cross-lingual retrieval, by considering the visual signal as an “interlingua” [169]–[171].”
MethodSemi-Supervised Sequence-to-Sequence ASR Using Unpaired Speech and Text
Baskar, M. K., Watanabe, S., Astudillo, R. F., Hori, T., Burget, L., & Černocký, J. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3167 · Cited by 69
Self-citationUnified Hypersphere Embedding for Speaker Recognition
Hajibabaei, M., & Dai, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.08312 · Cited by 69
Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
Latif, S., Rana, R. K., Khalifa, S., Jurdak, R., Qadir, J., & Schuller, B. W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.00378 · Cited by 66 · cited 1×
“One review addresses speech representation learning based on deep learning models [25],”
BackgroundInterface Databases: Design and Collection of a Multilingual Emotional Speech Database
Hozjan, V., Kačič, Z., Moreno, A., Bonafonte, A., & Rodríguez, A. N. (2002) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/35qmchms4f2w · Cited by 66 · cited 4×
“INTERFACE is an emotional speech database for Slovenian, English, Spanish, and French, and contains six emotions: anger, sadness, joy, fear, disgust, and surprise, plus neutral.”
MethodInfluentialKamper, H. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683639 · Cited by 65 · cited 1×
“Later work used variants of neural autoencoders [118], [200], [201], [202].”
MethodVQVAE Unsupervised Unit Discovery and Multi-Scale Code2Spec Inverter for Zerospeech Challenge 2019
Tjandra, A., Sisman, B., Zhang, M., Sakti, S., Li, H., & Nakamura, S. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-3232 · Cited by 65 · cited 1×
“Following the success of representation learning, baseline and challenge techniques have shifted from purely generative models [327], [328], deep autoencoders [83], [329], and incorporation of neural-network-”
BackgroundImproving Speech Recognition Using Consistent Predictions on Synthesized Speech
Wang, G., Rosenberg, A., Chen, Z., Zhang, Y., Ramabhadran, B., Wu, Y., & Moreno, P. J. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053831 · Cited by 64 · cited 1×
“TTS without using paired data [302], [303], [304].”
MethodSTCN: Stochastic Temporal Convolutional NetworksPreprint
Aksan, E., & Hilliges, O. (2019) · arXiv (Cornell University) · doi:10.3929/ethz-b-000586490 · Cited by 63
Speech-XLNet: Unsupervised Acoustic Model Pretraining for Self-Attention Networks
Song, X., Wang, G., Huang, Y., Wu, Z., Su, D., & Meng, H. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1511 · Cited by 62 · cited 2×
“and conventionally used for evaluation, while their audio streams are also aggregated to build diversified and large-scale corpora for unsupervised pre-training [109], [123], [218].”
BackgroundMethodPre-Training Audio Representations With Self-Supervision
Tagliasacchi, M., Gfeller, B., Quitry, F. D. C., & Roblek, D. (2020) · IEEE Signal Processing Letters · doi:10.1109/lsp.2020.2985586 · Cited by 61 · cited 2×
“Last, [116] puts together five datasets (MUSAN [245], Bird Audio Detection [246], Speech Commands [247], Spoken Language Identification [248], and TUT Urban Acoustic Scenes”
BackgroundWord Discovery in Visually Grounded, Self-Supervised Speech Models
Peng, P., & Harwath, D. F. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10652 · Cited by 61
Multi-Modal Data Augmentation for End-to-end ASR
Renduchintala, A., Ding, S., Wiesner, M., & Watanabe, S. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2456 · Cited by 59 · cited 1×
“, encoder features [319] and phoneme features [320], [321].”
BackgroundSelf-citationImproved Speech Representations with Multi-Target Autoregressive Predictive Coding
Chung, Y.-A., & Glass, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.213 · Cited by 58 · cited 1×
“In [97], the APC objective is extended to multi-target training.”
BackgroundNguyen, T. A., de Seyssel, M., Rozé, P., Rivière, M., Kharitonov, E., Baevski, A., Dunbar, E., & Emmanuel Dupoux, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.11588 · Cited by 58 · cited 1×
“The Zero Resource Speech Challenge 2021 [326], the latest challenge, expanded the scope to include language modeling tasks.”
BackgroundThe DIRHA-ENGLISH corpus and related tasks for distant-speech recognition in domestic environments
Ravanelli, M., Cristoforetti, L., Gretter, R., Pellin, M., Sosi, A., & Omologo, M. (2015) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2015.7404805 · Cited by 57 · cited 3×
“Studies leverage Hub5, DIRHA, and CHiME-5 to measure the efficacy of representations in ASR.”
BackgroundMachine Speech Chain with One-shot Speaker Adaptation
Tjandra, A., Sakti, S., & Nakamura, S. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1558 · Cited by 56 · cited 1×
“A speech chain [302], [305] is one successful way to utilize audio-only and text-only data to train both end-to-end ASR/TTS models.”
MethodSimilarity Analysis of Self-Supervised Speech Representations
Chung, Y.-A., Belinkov, Y., & Glass, J. R. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414321 · Cited by 56 · cited 1×
“In [275], representations based on different training criteria (masked predictive coding, contrastive predictive coding, and autoregressive predictive coding) were compared and analyzed with respect to the correlation between their training loss and performance on both phone discrimination and…”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization
Ling, S., & Liu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.06659 · Cited by 56 · cited 2×
“0 [92] LS 960 hrs ASR LS 100 hrs LS test-clean, LS test-other 1 (12)/10(12)/100 hrs”
BackgroundTowards better decoding and language model integration in sequence to sequence modelsPreprint
Chorowski, J., & Jaitly, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.02695 · Cited by 56
HEAR: Holistic Evaluation of Audio Representations
Turian, J., Shier, J., Khan, H. R., Raj, B., Schuller, B. W., Steinmetz, C. J., Malloy, C., Tzanetakis, G., Velarde, G., McNally, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.03022 · Cited by 56 · cited 4×
“Apart from the speech community, researchers have also established HEAR (holistic evaluation of audio representations) [262], NOSS (non-semantic speech benchmark) [263], and HARES (holistic audio representation evaluation suite) [264] to benchmark audio representations.”
BackgroundInfluentialSelf-citationTachbelie, M. Y., Abate, S. T., & Besacier, L. (2013) · Speech Communication · doi:10.1016/j.specom.2013.01.008 · Cited by 55 · cited 2×
“The ALFFA project3 collects speech of African languages to promote the development of speech technologies in Africa, and [123] leverages four African languages collected in the project for evaluation: Amharic [241], Fongbe [242], Swahili [243], and”
BackgroundWang, Y., Lee, H.-Y., & Lee, L.-S. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462002 · Cited by 55 · cited 1×
“[264] Audio word2vec [265] Oracle Phoneme GAN TIMIT (PER): 63.”
BackgroundSelf-citationChrupała, G. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12967 · Cited by 55 · cited 1×
“A recent review of datasets, as well as methods, in this category is provided by Chrupała [170].”
MethodSVMSVM: support vector machine speaker verification methodology
Wan, V., & Renals, S. J. (2003) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2003.1202334 · Cited by 54 · cited 1×
“use in speech processing include speech recognition [31], [32] and speaker recognition [33].”
BackgroundHeck, M., Sakti, S., & Nakamura, S. (2017) · 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru.2017.8269011 · Cited by 54 · cited 1×
“Following the success of representation learning, baseline and challenge techniques have shifted from purely generative models [327], [328], deep autoencoders [83], [329], and incorporation of neural-network-”
BackgroundTowards Unsupervised Speech-to-text Translation
Chung, Y.-A., Weng, W., Tong, S., & Glass, J. R. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683550 · Cited by 54 · cited 1×
“[286], [288] use adversarial training to map the word-level speech embedding”
BackgroundEloff, R., Nortje, A., Niekerk, B. V., Govender, A., Nortje, L., Pretorius, A., Biljon, E. V., Westhuizen, E. V. D., Staden, L. V., & Kamper, H. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1518 · Cited by 52 · cited 8×
“As described above, AEs consist of an encoder and decoder; the pretext task is to reconstruct the given input.”
MethodInfluentialLanguage Learning Using Speech to Image Retrieval
Merkx, D., Frank, S. L., & Ernestus, M. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3067 · Cited by 52 · cited 1×
“Typical approaches involve learning a neural representation model for each modality, with a multi-modal contrastive loss that encourages paired examples in the two modalities to have similar representations while unpaired examples remain different [171], [172], [173], [174], [175], [176].”
BackgroundX-ray microbeam speech production database
Westbury, J., Milenkovic, P., Weismer, G., & Kent, R. (1990) · The Journal of the Acoustical Society of America · doi:10.1121/1.2029064 · Cited by 50 · cited 1×
“face [148], [149], lip-movement [150], articulatory flesh point measurements [151], [152], or simultaneous magnetic resonance imaging (MRI) scans [153].”
BackgroundThe Zero Resource Speech Challenge 2020: Discovering Discrete Subword and Word Units
Dunbar, E., Karadayi, J., Bernard, M., Cao, X.-N., Algayres, R., Ondel, L., Besacier, L., Sakti, S., & Emmanuel Dupoux, A. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2743 · Cited by 50 · cited 8×
“Such models are typically evaluated on cross-modal retrieval, although some work has also used the models for other downstream tasks such as the ZeroSpeech and SUPERB benchmark tasks [180].”
BackgroundMethodInfluentialLiu, D.-R., Chen, K., Lee, H.-Y., & Lee, L.-S. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1800 · Cited by 49 · cited 2×
“use a phoneme segmentation module before the generator to segment utterances into phoneme-level segments [283], [290], [293].”
BackgroundMethodSelf-citationUnsupervised Training on Large Amounts of Broadcast News Data
Ma, J., Matsoukas, S., Kimball, O., & Schwartz, R. (2006) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2006.1660839 · Cited by 47 · cited 1×
“relies on a supervised teacher model to label a large volume of speech-only data, which is then used to augment the initial labeled data to train a student model [4], [5], [6], [61].”
MethodTransformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis: ZeroSpeech 2020 Challenge
Tjandra, A., Sakti, S., & Nakamura, S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-3033 · Cited by 46 · cited 1×
“SSL techniques are also benchmarked with the ZeroSpeech challenge [265], [266].”
MethodIntegrating articulatory data in deep neural network-based acoustic modeling
Badino, L., Canevari, C., Fadiga, L., & Metta, G. (2015) · Computer Speech & Language · doi:10.1016/j.csl.2015.05.005 · Cited by 45 · cited 1×
“Some typical approaches in this category include Multi-view autoencoders and variations [154], [155], Multi-modal deep Boltzmann machines [156], Canonical correlation analysis (CCA) [157] and its nonlinear extensions [158], [159], [160], [161], [162], [163], [164], [165], [166], Multi-view contrastive losses [167], [1…”
MethodEnd-to-end Feedback Loss in Speech Chain Framework via Straight-through Estimator
Tjandra, A., Sakti, S., & Nakamura, S. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683480 · Cited by 45 · cited 1×
“initially freezes either the ASR or TTS model, extensions of this study [303], [306], [307] implement the joint training of”
MethodA Cross-Task Analysis of Text Span Representations
Toshniwal, S., Shi, H., Shi, B., Gao, L., Livescu, K., & Gimpel, K. (2020) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2020.repl4nlp-1.20 · Cited by 45 · cited 1×
“Since most work on acoustic word embeddings preceded the very recent wave of new self-supervised frame-level representations, one question is whether word (or more generally segment) embeddings could be derived more simply by pooling self-supervised frame-level representations, as has been done for text span embedding…”
BackgroundSelf-citationslimIPL: Language-Model-Free Iterative Pseudo-Labeling
Likhomanenko, T., Xu, Q., Kahn, J., Synnaeve, G., & Collobert, R. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-740 · Cited by 45 · cited 2×
“For comparison, we also show performance from such methods (iterative pseudo labeling (IPL) [63], slimIPL [254], noisy student [62]), as well as the current state of the art—conformer XXL + noisy student [255]—which augments SSL with various advanced techniques including self-training.”
MethodResultAn Unsupervised Autoregressive Model for Speech Representation LearningPreprint
Chung, Y.-A., Hsu, W.-N., Tang, H., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03240 · Cited by 45 · cited 8×
“APC is formulated as In text-based autoregressive LMs, c is set to 1 to enable autoregressive generation.”
InfluentialTextless Speech Emotion Conversion using Discrete and Decomposed RepresentationsPreprint
Kreuk, F., Polyak, A., Copet, J., Kharitonov, E., Nguyen, T.-A., Rivière, M., Hsu, W.-N., Mohamed, A., Dupoux, E., & Adi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07402 · Cited by 45
Self-citationDUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answering
Lin, G.-T., Chuang, Y.-S., Chung, H.-L., Yang, S., Chen, H.-J., Dong, S. A., Li, S.-W. D., Mohamed, A., Lee, H.-Y., & Lee, L.-S. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-612 · Cited by 44
Self-citationSelf-supervised audio representation learning for mobile devicesPreprint
Tagliasacchi, M., Gfeller, B., Quitry, F. D. C., & Roblek, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11796 · Cited by 44 · cited 2×
“[116], phase information [117], and the temporal gap between two segments [115], [116].”
BackgroundA Further Study of Unsupervised Pretraining for Transformer Based Speech Recognition
Jiang, D., Li, W., Zhang, R., Cao, M. Y., Luo, N., Yang, H., Zou, W., Han, K., & Li, X. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414539 · Cited by 42 · cited 6×
“For the purpose of pre-training and evaluating Mandarin speech representations, the authors of [101], [104] also compiled Open Mandarin, an open-source Mandarin dataset of 1.”
BackgroundMethodInfluentialDEVELOPMENTS OF SWAHILI RESOURCES FOR AN AUTOMATIC SPEECH RECOGNITION SYSTEM
Gelas, H., Besacier, L., & Pellegrino, F. (2012) · Cited by 42
Fast-Slow Transformer for Visually Grounding Speech
Peng, P., & Harwath, D. F. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747103 · Cited by 41 · cited 1×
“Typical approaches involve learning a neural representation model for each modality, with a multi-modal contrastive loss that encourages paired examples in the two modalities to have similar representations while unpaired examples remain different [171], [172], [173], [174], [175], [176].”
BackgroundTraining Neural Speech Recognition Systems with Synthetic Speech AugmentationPreprint
Li, J., Gadde, R. T., Ginsburg, B., & Lavrukhin, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00707 · Cited by 40 · cited 1×
“straightforward approach is to simply use multi-speaker TTS to generate the waveform with various acoustic variations [314], [315], [316], [317], [318].”
MethodA generalization of isolated word recognition using vector quantization
Burton, D., Shore, J. E., & Buck, J. (2005) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1983.1171915 · Cited by 39 · cited 1×
“Codebook learning is facilitated by a two-term auxiliary loss similar to classical vector quantization dictionary learning [81], [82].”
MethodDiscovering discrete subword units with binarized autoencoders and hidden-Markov-model encoders
Badino, L., Mereta, A., & Rosasco, L. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-639 · Cited by 39 · cited 1×
“In addition to the approaches discussed above, several other works on speech representation learning take inspiration from the AE framework [86], [87], [88], [89], [90], [91].”
BackgroundUeno, S., Mimura, M., Sakai, S., & Kawahara, T. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682816 · Cited by 38 · cited 1×
“straightforward approach is to simply use multi-speaker TTS to generate the waveform with various acoustic variations [314], [315], [316], [317], [318].”
MethodDiscreTalk: Text-to-Speech as a Machine Translation ProblemPreprint
Hayashi, T., & Watanabe, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.05525 · Cited by 38 · cited 16×
“Thus, we design a loss function L asr → tts ( X, ˆ X ) , where ˆ X is generated by Thus, we train the ASR model (or both ASR and TTS models) using only the acoustic input by minimizing L asr → tts .”
BackgroundInfluentialSelf-citationLearning Words from Images and Speech
Synnaeve, G., Maarten, V., & Emmanuel Dupoux, A. (2014) · doi:10.6084/m9.figshare.1277822 · Cited by 38
Zhu, Q., Zhang, J., Zhang, Z., Wu, M., Fang, X., & Dai, L.-R. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747379 · Cited by 37 · cited 1×
“per, there is already some related research about enhancing the robustness of SSL models [281], [352], [353], [354] and identifying their vulnerability to adversarial attack [351].”
BackgroundWang, Y.-H., Chung, C.-T., & Lee, H.-Y. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-877 · Cited by 36 · cited 1×
“[266] MFCC GAS [267] Phoneme Empirical-ODM [268] Self-training TIMIT (PER): 41.”
Self-citationHolzenberger, N., Du, M., Karadayi, J., Riad, R., & Emmanuel Dupoux, A. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2364 · Cited by 35 · cited 1×
“Later work used variants of neural autoencoders [118], [200], [201], [202].”
MethodScharenborg, O., Besacier, L., Black, A. W., et al. (2018) · HAL AMU · Cited by 34
A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning
Khurana, S., Laurent, A., Hsu, W.-N., Chorowski, J., Łańcucki, A., Marxer, R., & Glass, J. R. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-3084 · Cited by 34
An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks
Chang, K.-W., Tseng, W.-C., Li, S.-W. D., & Lee, H.-Y. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10610 · Cited by 34
Self-citationSupport vector machines for segmental minimum Bayes risk decoding of continuous speech
Venkataramani, V., Chakrabartty, S., & Byrne, B. (2004) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2003.1318396 · Cited by 33 · cited 1×
“use in speech processing include speech recognition [31], [32] and speaker recognition [33].”
BackgroundLing, S., Salazar, J., Liu, Y., & Kirchhoff, K. (2020) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2020-2 · Cited by 33 · cited 1×
“language classes was analyzed in [272] for a 12-layer BERTphone model.”
MethodSelf-citationDo Self-Supervised and Supervised Methods Learn Similar Visual Representations?Preprint
Grigg, T. G., Busbridge, D., Ramapuram, J., & Webb, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.00528 · Cited by 32 · cited 1×
“Interestingly, similar network behavior was also recently described for self-supervised learners in computer vision: using a contrastive self-supervised learner (SimCLR) that optimizes for augmentation invariance, [270] show that it is the intermediate representations that most closely approximate information learned…”
BackgroundUnspeech: Unsupervised Speech Context Embeddings
Milde, B., & Biemann, C. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2194 · Cited by 31 · cited 1×
“Unspeech [101] X[t1,t2] X[t0,t1],X[t2,t3]”
Acoustically Grounded Word Embeddings for Improved Acoustics-to-word Speech Recognition
Settle, S., Audhkhasi, K., Livescu, K., & Picheny, M. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682903 · Cited by 31 · cited 2×
“In addition to the approaches discussed above, several other works on speech representation learning take inspiration from the AE framework [86], [87], [88], [89], [90], [91].”
BackgroundSelf-citationAnalyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processing
van Niekerk, B., Nortje, L., Baas, M., & Kamper, H. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1182 · Cited by 31 · cited 1×
“It is shown that the per-utterance mean of CPC features captures speaker information to a large extent [271].”
BackgroundUnsupervised Speech Recognition via Segmental Empirical Output Distribution MatchingPreprint
Yeh, C., Chen, J., Yu, C., & Yu, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.09323 · Cited by 31 · cited 2×
“Other ways to map speech and text include via segmental empirical output distribution matching (segmental empirical-ODM) [290] and decipherment algorithm [294].”
MethodAnguera, X., Rodríguez-Fuentes, L. J., Buzo, A., Metze, F., Szöke, I., & Peñagarikano, M. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7179090 · Cited by 30
Understanding Self-Attention of Self-Supervised Audio Transformers
Yang, S.-W., Liu, A. T., & Lee, H.-Y. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2231 · Cited by 29 · cited 1×
“In [274] the attention patterns in generatively trained Transformer represen-”
BackgroundSelf-citationImproving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction
Wang, H., Qian, Y., Wang, X., Wang, Y., Wang, C., Liu, S., Yoshioka, T., Li, J., & Wang, D. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746220 · Cited by 29 · cited 1×
“per, there is already some related research about enhancing the robustness of SSL models [281], [352], [353], [354] and identifying their vulnerability to adversarial attack [351].”
BackgroundAre Discrete Units Necessary for Spoken Language Modeling?
Nguyen, T. A., Sagot, B., & Emmanuel Dupoux, A. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3200909 · Cited by 29
Colorful Image ColorizationPreprint
Zhang, R., Isola, P., & Efros, A. A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.08511 · Cited by 27 · cited 1×
“Early breakthroughs included end-to-end learning of deep neural architectures via pretext tasks for restoring the true color of black-and-white images [45], joint learning of latent rep-”
BackgroundUsing Personalized Speech Synthesis and Neural Language Generator for Rapid Speaker Adaptation
Huang, Y., He, L., Wei, W., Gale, W., Li, J., & Gong, Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053104 · Cited by 26 · cited 1×
“straightforward approach is to simply use multi-speaker TTS to generate the waveform with various acoustic variations [314], [315], [316], [317], [318].”
MethodPeng, P., & Harwath, D. F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.03543 · Cited by 25 · cited 1×
“ZeroSpeech and SUPERB benchmark tasks [180].”
Improving Streaming Transformer Based ASR Under a Framework of Self-Supervised Learning
Cao, S., Kang, Y., Fu, Y., Xu, X., Sun, S., Zhang, Y., & Ma, L. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1454 · Cited by 23 · cited 1×
“bidirectional Transformer have hindered the representation model used in streaming scenarios, leading to studies that address these problems [347].”
BackgroundHavard, W. N., Chevrot, J., & Besacier, L. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683069 · Cited by 22
Improving Self-Supervised Learning for Speech Recognition with Intermediate Layer Supervision
Wang, C., Wu, Y., Chen, S., Liu, S., Li, J., Qian, Y., & Yang, Z. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747022 · Cited by 22
Talk, Don’t Write: A Study of Direct Speech-Based Image Retrieval
Sanabria, R., Waters, A., & Baldridge, J. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-96 · Cited by 21 · cited 1×
“Other choices include training with a masked margin softmax loss [177], [178] or a masked prediction loss [179].”
BackgroundDiscretization and Re-synthesis: an alternative method to solve the Cocktail Party ProblemPreprint
Shi, J., Chang, X., Hayashi, T., Lu, Y., Watanabe, S., & Xu, B. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.09382 · Cited by 20
Self-citationParameter-Efficient Transfer Learning with Diff Pruning
Guo, D., Rush, A. M., & Kim, Y. (2020) · arXiv · doi:10.48550/arxiv.2012.07463 · Cited by 20
Siamese Capsule Network for End-to-End Speaker Recognition in the Wild
Hajavi, A., & Etemad, A. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414722 · Cited by 19
Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptation
Huang, K. P., Fu, Y.-K., Zhang, Y., et al. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-519 · Cited by 19
Self-citationImproving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptation
Huang, K. P., Fu, Y.-K., Zhang, Y., & Lee, H.-Y. (2022) · arXiv · doi:10.48550/arxiv.2203.16104 · Cited by 19
Self-citationDeciphering Speech: a Zero-Resource Approach to Cross-Lingual Transfer in ASR
Klejch, O., Wallington, E., & Bell, P. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10170 · Cited by 18 · cited 2×
“Other ways to map speech and text include via segmental empirical output distribution matching (segmental empirical-ODM) [290] and decipherment algorithm [294].”
BackgroundMethodvan Staden, L., & Kamper, H. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383625 · Cited by 17 · cited 1×
“Some initial results suggest that at least very simple pooling approaches like downsampling and mean or max pooling are not successful [202], [207], but more work is needed to reach conclusive results.”
BackgroundChen, K., Tsai, C.-P., Liu, D.-R., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04100 · Cited by 17
Self-citationVoita, E., Sennrich, R., & Titov, I. (2019) · arXiv · doi:10.48550/arxiv.1909.01380 · Cited by 17
Text-Free Prosody-Aware Generative Spoken Language Modeling
Kharitonov, E., Lee, A., Polyak, A., Adi, Y., Copet, J., Lakhotia, K., Nguyen, T.-A., Rivière, M., Mohamed, A., Dupoux, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2109.03264 · Cited by 16 · cited 4×
“One workaround is two-stage training [77], [332];”
BackgroundInfluentialSelf-citationA Brief Overview of Unsupervised Neural Speech Representation LearningPreprint
Borgholt, L., Havtorn, J. D., Edin, J., Maaløe, L., & Igel, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.01829 · Cited by 16 · cited 1×
“Recently, a brief overview with a general focus on speech representation learning was published [24].”
BackgroundSelf-citationMasumura, R., Makishima, N., Ihori, M., Takashima, A., Tanaka, T., & Orihashi, S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1930 · Cited by 15 · cited 1×
“, encoder features [319] and phoneme features [320], [321].”
BackgroundAnalyzing The Robustness of Unsupervised Speech Recognition
Lin, G.-T., Hsu, C.-J., Liu, D.-R., Lee, H.-Y., & Tsao, Y. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747357 · Cited by 15 · cited 1×
“The robustness of wav2vec-U was further analyzed with respect to domain-mismatch scenarios in which the domains of unpaired speech and text were different [301].”
MethodSelf-citationLaleye, F. A. A., Besacier, L., Ezin, E. C., et al. (2016) · Annals of Computer Science and Information Systems · doi:10.15439/2016f153 · Cited by 15 · cited 2×
“The ALFFA project3 collects speech of African languages to promote the development of speech technologies in Africa, and [123] leverages four African languages collected in the project for evaluation: Amharic [241], Fongbe [242], Swahili [243], and”
BackgroundUnsupervised Sequence Classification using Sequential Output StatisticsPreprint
Liu, Y., Chen, J., & Deng, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.07817 · Cited by 15
Maekaku, T., Chang, X., Fujita, Y., Chen, L., Watanabe, S., & Rudnicky, A. I. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1503 · Cited by 14 · cited 1×
“based TTS/VC techniques [266] to self-supervised learning [330].”
MethodSelf-citationMulti-Modal Pre-Training for Automated Speech Recognition
Chan, D. M., Ghosh, S., Chakrabarty, D., & Hoffmeister, B. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746449 · Cited by 14 · cited 1×
“Other choices include training with a masked margin softmax loss [177], [178] or a masked prediction loss [179].”
BackgroundConsiderations in applying clustering techniques to speaker independent word recognition
Rabiner, L. R., & Wilpon, J. (2005) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1979.1170822 · Cited by 13 · cited 1×
“work such as [26], [27], word patterns were clustered semiautomatically using techniques such as k-means, after which isolated words were recognized by finding the training cluster”
MethodMachine Speech Chain with One-shot Speaker AdaptationPreprint
Tjandra, A., Sakti, S., & Nakamura, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.10525 · Cited by 13 · cited 1×
“A speech chain [302], [305] is one successful way to utilize audio-only and text-only data to train both end-to-end ASR/TTS models.”
MethodOn the Contributions of Visual and Textual Supervision in Low-Resource Semantic Speech Retrieval
Pasad, A., Shi, B., Kamper, H., & Livescu, K. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3051 · Cited by 12 · cited 1×
“, the speech is transcribed), visual grounding still helps learn a better representation for retrieval [172].”
BackgroundSelf-citationLearning audio representations via phase predictionPreprint
Quitry, F. D. C., Tagliasacchi, M., & Roblek, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.11910 · Cited by 12 · cited 2×
“[116], phase information [117], and the temporal gap between two segments [115], [116].”
BackgroundMethodA Comparison of Discrete Latent Variable Models for Speech Representation Learning
Zhou, H., Baevski, A., & Auli, M. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413680 · Cited by 12 · cited 1×
“A similar insight was obtained in [276], which compared vq-vae and vq-wav2vec with respect to their ability to discover”
ResultTextless Speech Emotion Conversion using Decomposed and Discrete RepresentationsPreprint
Kreuk, F., Polyak, A., Copet, J., Kharitonov, E., Nguyen, T. A., Rivière, M., Hsu, W.-N., Mohamed, A., Dupoux, E., & Adi, Y. (2021) · arXiv (Cornell University) · Cited by 12
Self-citationContent-Context Factorized Representations for Automated Speech Recognition
Chan, D., & Ghosh, S. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-390 · Cited by 12
Scaling ASR Improves Zero and Few Shot LearningPreprint
Xiao, A., Zheng, W., Keren, G., Le, D., Zhang, F., Fuegen, C., Kalinli, O., Saraf, Y., & Mohamed, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05948 · Cited by 12 · cited 1×
“original teachers to capture the complexities in vastly larger speech-only data [62], [63], [64].”
BackgroundSelf-citationA Correspondence Variational Autoencoder for Unsupervised Acoustic Word EmbeddingsPreprint
Peng, P., Kamper, H., & Livescu, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.02221 · Cited by 12 · cited 2×
“Some initial results suggest that at least very simple pooling approaches like downsampling and mean or max pooling are not successful [202], [207], but more work is needed to reach conclusive results.”
BackgroundMethodSelf-citationA DNN-HMM-DNN Hybrid Model for Discovering Word-Like Units from Spoken Captions and Image Regions
Wang, L., & Hasegawa‐Johnson, M. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1148 · Cited by 10 · cited 1×
“to learn word-like units [184], [185], [186] and can be used for cross-lingual retrieval, by considering the visual signal as”
Phonetically Motivated Self-Supervised Speech Representation Learning
Yue, X., & Li, H. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-905 · Cited by 10 · cited 1×
“Instead of just masking a fixed number of consecutive frames, pMPC [105] selects masked”
BackgroundPredicting the Features of World Atlas of Language Structures from Speech
Gutkin, A., Merkulova, T., & Jansche, M. (2018) · doi:10.21437/sltu.2018-52 · Cited by 9 · cited 1×
“be used for cross-lingual retrieval, by considering the visual signal as an “interlingua” [169]–[171].”
MethodMasked Pre-trained Encoder base on Joint CTC-TransformerPreprint
Liu, L., & Huang, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.11978 · Cited by 9 · cited 1×
“The function f(·) is typically a Transformer encoder [100], [101], [102], but recurrent neural networks have also been used [103].”
MethodA NEW RESOURCE FOR PRODUCTION MODELLING IN SPEECH TECHNOLOGY
Wrench, A. (2023) · doi:10.25144/18431 · Cited by 9 · cited 1×
“face [148], [149], lip-movement [150], articulatory flesh point measurements [151], [152], or simultaneous magnetic resonance imaging (MRI) scans [153].”
BackgroundAn Exploration of Self-Supervised Pretrained Representations for End-to-End Speech RecognitionPreprint
Chang, X., Maekaku, T., Guo, P., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04590 · Cited by 8 · cited 1×
“Similar problems were also investigated using HuBERT and wav2vec 2.0 models in [280].”
MethodInfluentialSelf-Supervised Learning for speech recognition with Intermediate layer supervisionPreprint
Wang, C., Wu, Y., Chen, S., Liu, S., Li, J., Qian, Y., & Yang, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.08778 · Cited by 6 · cited 1×
“Another study [273] used explicit self-supervised loss at the intermediate layers rather than just the output layer of a”
BackgroundDropout Regularization for Self-Supervised Learning of Transformer Encoder Speech Representation
Luo, J. J., Wang, J., Cheng, N., & Xiao, J. Z. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1066 · Cited by 4 · cited 1×
“Furthermore, two dropout regularization methods—attention dropout and layer dropout—are introduced with the TERA model [106], [111].”
MethodCharacterizing the Adversarial Vulnerability of Speech self-Supervised Learning
Wu, H., Zheng, B., Li, X., Wu, X., Lee, H.-Y., & Meng, H. M. L. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747242 · Cited by 4 · cited 2×
“per, there is already some related research about enhancing the robustness of SSL models [281], [352], [353], [354] and identifying their vulnerability to adversarial attack [351].”
BackgroundSelf-citationTowards Learning Universal Audio RepresentationsPreprint
Wang, L., Luc, P., Wu, Y. X., Recasens, A., Smaira, L., Brock, A., Jaegle, A., Alayrac, J.-B., Dieleman, S., Carreira, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12124 · Cited by 2 · cited 4×
“Apart from the speech community, researchers have also established HEAR (holistic evaluation of audio representations) [262], NOSS (non-semantic speech benchmark) [263], and HARES (holistic audio representation evaluation suite) [264] to benchmark audio representations.”
BackgroundInfluentialA Comparison of Discrete Latent Variable Models for Speech Representation LearningPreprint
Zhou, H., Baevski, A., & Auli, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14230 · cited 1×
“A similar insight was obtained in [276], which compared vq-vae and vq-wav2vec with respect to their ability to discover”
ResultDeciphering Speech: a Zero-Resource Approach to Cross-Lingual Transfer in ASRPreprint
Klejch, O., Wallington, E., & Bell, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.06799 · cited 2×
“Other ways to map speech and text include via segmental empirical output distribution matching (segmental empirical-ODM) [290] and decipherment algorithm [294].”
BackgroundMethodThe Zero Resource Speech Challenge 2019: TTS without T
Dunbar, E., Algayres, R., Karadayi, J., et al. (2019) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.48550/arxiv.1904.11469 · cited 1×
“The Zero Resource Speech Challenge 2019 [325] extended”
BackgroundWord Discovery in Visually Grounded, Self-Supervised Speech ModelsPreprint
Peng, P., & Harwath, D. F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.15081 · cited 1×
“to learn word-like units [184], [185], [186] and can be used for cross-lingual retrieval, by considering the visual signal as”
Towards End-to-end Unsupervised Speech RecognitionPreprint
Liu, A. H., Hsu, W.-N., Auli, M., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.02492 · cited 2×
“0 [296] further removes the requirement of the segmentation stage, so the unsupervised ASR model can be learned in an end-”
BackgroundMethodUnsupervised Speech Recognition
Baevski, A., Hsu, W.-N., Conneau, A., & Auli, M. (2021) · arXiv · doi:10.48550/arxiv.2105.11084 · cited 3×
“Wav2vec-U [268] achieved state-of-the-art results at the time,”
Background
Research works printed but not linked: 18
- 10.1109/massp.1984.1162229 · DOI
- Radford, 2019, Language models are unsupervised multitask learners, 1, 9
- 10.5555/3524938.3525087 · doi:10.5555/3524938.3525087
- Garofolo, 1993, TIMIT acoustic-phonetic continuous speech corpus LDC93S1
- Yang, 2019, Generalized Autoregressive Pretraining for Language Understanding, 32
- 10.1109/icassp40776.2020.9053176 · DOI · Paper page
- Baevski, 2020, vq-wav2vec: Self-supervised learning of discrete speech representations
- Baevski, 2020, wav2vec 2.0: A framework for self-supervised learning of speech representations, 33, 12449
- 10.1007/978-3-030-01264-9_9 · DOI · Paper page
- 10.5555/3495724.3497510 · doi:10.5555/3495724.3497510
- 10.1109/icassp.2018.8461761 · DOI
- 10.1109/asru.2017.8269008 · doi:10.1109/asru.2017.8269008
- Maas, 2012, Word-level acoustic modeling with convolutional vector regression
- Rodrguez-Fuentes, 2014, GTTS-EHU systems for QUESST at MediaEval 2014
- 10.1109/icassp40776.2020.9054548 · DOI · Paper page
- Pu, 2021, Scaling effect of self-supervised models, 1084
- Goodfellow, 2014, Generative adversarial nets, 2672
- Arjovsky, 2017, GAN wasserstein, 214
Cited by
Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects
Zhang, K., Wen, Q., Zhang, C., Cai, R., Jin, M., Liu, Y., Zhang, J. Y., Liang, Y., Pang, G., Song, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387317 · Cited by 295
“Using the learning paradigm as a taxonomy is arguably the most popular among the existing SSL surveys, including [20], [22], [23], [24], [25], [26], [27].”
MethodTen deep learning techniques to address small data problems with remote sensing
Safonova, A., Ghazaryan, G., Stiller, S., Main-Knorn, M., Nendel, C., & Ryo, M. (2023) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2023.103569 · Cited by 196
“, 2022), physics (Ma and Liu, 2020), speech representation (Mohamed et al., 2022), RS (S.”
BackgroundComparative Layer-Wise Analysis of Self-Supervised Speech Models
Pasad, A., Shi, B., & Livescu, K. (2023) · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49357.2023.10096149 · Cited by 105
“Self-supervised models have become a nearly ubiquitous approach for learning speech representations and improving performance on downstream tasks [1–5], but our understanding of their properties and strategies for their use is still limited.”
BackgroundSelf-citationGalić, I., Habijan, M., Leventić, H., & Romić, K. (2023) · Electronics · doi:10.3390/electronics12214411 · Cited by 150
Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study
Edin, J., Junge, A., Havtorn, J. D., Borgholt, L., Maistro, M., Ruotsalo, T., & Maaløe, L. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591918 · Cited by 137
“Finally, while PLM-ICD outperforms the other models in this paper, the improvements are limited compared to the effect of pre-training in other domains [1, 6, 9, 23, 28].”
Self-citation
Metadata
- Authors
- Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., Sainath, T. N., & Watanabe, S.
- Affiliations
- Meta (United States) (US)
- National Taiwan University (TW)
- University of Copenhagen (DK)
- Technical University of Denmark (DK)
- Amazon (United States) (US)
- Toyota Technological Institute at Chicago (US)
- Google (United States) (US)
- Carnegie Mellon University (US)
- ORCID
- Venue
- IEEE Journal of Selected Topics in Signal Processing
- Published
- 2022-10
- Volume
- 16
- Issue
- 6
- Pages
- 1179-1210
- ISSN
- 1932-4553 (ISSN-L)
- 1941-0484
- OpenAlex
- W4281492411
- Semantic Scholar
- f76a5b176f3435214eb87dd105f730f0b53672c3
- Cited by
- 372
- References
- 345
- Citations by year
- 2026: 66 · 2025: 91 · 2024: 110 · 2023: 94 · 2022: 11
- FWCI
- 5.35
- Type
- journal article
- Language
- en
- Open access
- Licence
- Locations
- IEEE Journal of Selected Topics in Signal Processing (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion)
- Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) (submittedVersion, other-oa)
- arXiv (Cornell University) (cc-by)
- Funders
- Innovation Fund Denmark through Industrial Ph.D. Programme (0153-00167B)
- Innovation Fund Denmark thorugh Industrial Ph.D. Programme (8053-00184B)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Keywords
- self-supervised learning, speech representation learning, generative models, contrastive learning, prediction model, multimodal pre-training, acoustic word embeddings, low-resource languages, speech recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.