Self-Supervised Speech Representation Learning: A Review
Open accessThe DOI matches its Crossref record; 361 of the 362 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Although supervised deep learning has revolutionized speech and audio processing, it has necessitated the building of specialist models for individual tasks and application scenarios. It is likewise difficult to apply this to dialects and languages for which only limited labeled data is available. Self-supervised representation learning methods promise a single universal model that would benefit a wide variety of tasks and domains. Such methods have shown success in natural language processing and computer vision domains, achieving new levels of performance while reducing the number of labels required for many downstream scenarios. Speech representation learning is experiencing similar progress in three main categories: generative, contrastive, and predictive methods. Other approaches rely on multi-modal data for pre-training, mixing text or visual data streams with speech. Although self-supervised speech representation is still a nascent research area, it is closely related to acoustic word embedding and learning with zero lexical resources, both of which have seen active research for many years. This review presents approaches for self-supervised speech representation learning and their connection to other research areas. Since many current methods focus solely on automatic speech recognition as a downstream task, we review recent efforts on benchmarking learned representations to extend the application beyond speech recognition.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Many popular self-supervised learning (SSL) models (Mohamed et al., 2022, Table IV) are trained using the Librispeech database”
From ASVspoof 5: Design, collection and validation of resources for spoofing, deepfake, and adversarial attack detection using crowdsourced speech (2026) · cites this paper as Mohamed et al., 2022
In the citing paper
“A previous study [34] already proposed a method to adapt SimCLR to the HAR domain, finding that random 3D rotation was an effective transformation for accelerometer data.”
From Comparing self-supervised learning techniques for wearable human activity recognition (2025) · cites this paper as [34]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 4×
“c) Masked Reconstruction: Masked reconstruction is largely inspired by the masked language model (MLM) task from BERT [54].”
BackgroundMethodInfluential(2014) · Cited by 33,419
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 1×
“improve continuously via the EMA teacher network; however, student-teacher approaches inflate the existing computational challenges of very large models and may necessitate the use of methods that decrease instantaneous memory utilization such as mixed precision training, model parallelism and model sharding [142].”
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning the parts of objects by non-negative matrix factorization
Lee, D. D., & Seung, H. S. (1999) · Nature · doi:10.1038/44565 · Cited by 13,868 · cited 1×
“niques include k-means clustering [11], mixture models [12], autoencoders [13], and non-negative matrix factorization [14].”
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · cited 1×
“This approach has become especially popular in the CV context [57], [58], [59], [60].”
BackgroundRepresentation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · cited 2×
“the underlying explanatory factors for the observed input [9].”
BackgroundDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 4×
“DeCoAR [99] combines the bidirectionality of the popular NLP model ELMo [51] and the reconstruction objective of APC to alleviate this issue”
BackgroundMethodInfluentialHinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260 · cited 1×
“of this transformation, offering impressive gains for scenarios rich in labeled data [1], [2], [3].”
BackgroundSelf-citationMachine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955 · cited 1×
“patterns in training samples for which there are no preassigned labels or scores [10].”
BackgroundImproved training of Wasserstein GANs
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. (2017) · arXiv · doi:10.48550/arxiv.1704.00028 · Cited by 6,800 · cited 2×
“There is a large amount of work using gradient penalty in the objective of training discriminators [268], [283], [293], [296], which is inspired by Improved Wasserstein Generative Adversarial Network (WGAN) [285].”
BackgroundCaruana, R. (1997) · Machine Learning · doi:10.1023/a:1007379606734 · Cited by 6,636 · cited 1×
“captured by models trained on one task to different but related tasks [65].”
MethodRepresentation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · cited 18×
“As the CPC model, it uses the InfoNCE loss [56] to maximize the similarity between a contextualized representation and a localized representation.”
BackgroundMethodInfluential“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · cited 1×
“One of the most commonly used datasets in this category is LibriSpeech (LS) [212], a labeled corpus containing 960 hours of read English speech, which is also derived from an open-source audiobooks project8.”
MethodEmergence of simple-cell receptive field properties by learning a sparse code for natural images
Olshausen, B. A., & Field, D. J. (1996) · Nature · doi:10.1038/381607a0 · Cited by 5,694 · cited 1×
“trastive estimation (NCE) [37], sparse coding [38], [39], [40], and energy-based methods [41].”
BackgroundEmerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 1×
“ponential moving average (EMA) teacher for self-supervised visual representations [139], [140], the data2vec model [132] computes targets Y using an EMA of its own parameters.”
MethodWaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · cited 2×
“However, the progress of TTS systems boosted by deep learning [80], [313] has inspired another interesting and straightforward research direction: artificially creating paired text and audio data {X̂, Y }with only text data Y by generating the corresponding audio data X̂ with TTS.”
BackgroundMethodTraining Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 1×
“For example, approximate training methods such as contrastive divergence [42] were a”
MethodLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 2×
“Due to the popularity of SSL, reviews have been published about the technology in general [16], [17], [18] as well as its application to natural language processing (NLP) [19], [20],”
BackgroundLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 2×
“Due to the popularity of SSL, reviews have been published about the technology in general [16], [17], [18] as well as its application to natural language processing (NLP) [19], [20],”
BackgroundGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 1×
“For example, in NLP, a representation model is usually assessed on GLUE, which comprises nine tasks [43], whereas in speech, a representation model can be evaluated on SUPERB, which comprises ten tasks [44], as described in detail in Section IV-E.”
Method(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1953039 · Cited by 4,051 · cited 2×
“Examples of early techniques include restricted Boltzmann machines (RBM) [15], denoising autoencoders [36], noise con-”
BackgroundMethodBootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · cited 1×
“ponential moving average (EMA) teacher for self-supervised visual representations [139], [140], the data2vec model [132] computes targets Y using an EMA of its own parameters.”
MethodCategorical Reparameterization with Gumbel-Softmax
Jang, E., Gu, S., & Poole, B. (2016) · arXiv · doi:10.48550/arxiv.1611.01144 · Cited by 3,616 · cited 2×
“The quantization module g(·) uses a Gumbel softmax [84] with”
MethodFront-End Factor Analysis for Speaker Verification
Dehak, N., Kenny, P. J., Dehak, R., Dumouchel, P., & Ouellet, P. (2011) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2010.2064307 · Cited by 3,549 · cited 1×
“Subsequent techniques in speaker and language verification [34], [35] similarly extracted parameters (concatenated means) from trained background GMMs as representations that were then combined with low-rank projections of speaker/session-or language-specific vectors.”
MethodSimple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Zhang, R., Isola, P., & Efros, A. A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_40 · Cited by 3,178 · cited 1×
“Early breakthroughs included end-to-end learning of deep neural architectures via pretext tasks for restoring the true color of black-and-white images [45], joint learning of latent rep-”
BackgroundAudio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · cited 3×
“The speech examples in Audioset [209], which consists of over 2M 10-second YouTube video clips human-annotated with 632 audio events, have also been used for pre-training.”
MethodInfluentialNoise-contrastive estimation: A new estimation principle for unnormalized statistical models
Gutmann, M. U., & Hyvärinen, A. (2010) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 2,992
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3122291 · Cited by 2,983 · cited 23×
“3) Challenges: The iterative nature of pre-training for the HuBERT and wavLM could present a practical inconvenience when working with large volumes of data.”
MethodInfluentialSelf-citationProduct Quantization for Nearest Neighbor Search
Jégou, H., Douze, M., & Schmid, C. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.57 · Cited by 2,956 · cited 1×
“0 concatenates quantized representations from multiple codebooks at each timestep, so-called Product Quantization (PQ) [136].”
BackgroundGray, R. M. (1990) · Readings in Speech Recognition · doi:10.1016/b978-0-08-051584-7.50011-5 · Cited by 2,845
Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461368 · Cited by 2,791 · cited 1×
“However, the progress of TTS systems boosted by deep learning [80], [313] has inspired another interesting and straightforward research direction: artificially creating paired text and audio data {X̂, Y }with only text data Y by generating the corresponding audio data X̂ with TTS.”
Background(2011) · Cited by 2,752
Self-citationNeural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729 · cited 1×
“vector-quantized variational autoencoder (VQ-VAE) is another model in this category [75]; it extends the original VAE [48] with a novel parameterization of the posterior distribution for”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · cited 1×
“Due to the popularity of SSL, reviews have been published about the technology in general [16], [17], [18] as well as its application to natural language processing (NLP) [19], [20],”
BackgroundDeep Clustering for Unsupervised Learning of Visual Features
Caron, M., Bojanowski, P., Joulin, A., & Douze, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_9 · Cited by 2,674 · cited 1×
“resentations and their cluster assignments [46], and the prediction of the relative positions of image patches [47].”
BackgroundUnsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · cited 1×
“resentations and their cluster assignments [46], and the prediction of the relative positions of image patches [47].”
BackgroundUnsupervised Learning of Visual Features by Contrasting Cluster Assignments
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., & Joulin, A. (2020) · arXiv · doi:10.48550/arxiv.2006.09882 · Cited by 2,499 · cited 1×
“This approach has become especially popular in the CV context [57], [58], [59], [60].”
BackgroundStochastic Backpropagation and Approximate Inference in Deep Generative Models
Rezende, D. J., Mohamed, S., & Wierstra, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.4082 · Cited by 2,405
Deep Canonical Correlation Analysis
Andrew, G., Arora, R., Bilmes, J., & Livescu, K. (2013) · International Conference on Machine Learning · Cited by 2,401
Self-citationEfficient sparse coding algorithms
Lee, H., Battle, A., Raina, R., & Ng, A. Y. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0105 · Cited by 2,394 · cited 1×
“trastive estimation (NCE) [37], sparse coding [38], [39], [40], and energy-based methods [41].”
BackgroundSelf-citationParameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270 · cited 1×
“Are there other ways to use these models? In the NLP community, the adapter-based method [320]–[322] and prompt/instruction learning [20] yield competitive performance compared with fine-tuning, but these methods have yet to be fully exploited for speech.”
VoxCeleb: A Large-Scale Speaker Identification Dataset
Nagrani, A., Chung, J. S., & Zisserman, A. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-950 · Cited by 2,210 · cited 1×
“VoxCeleb1 [236] for SID/ASV (automatic speaker verification) tasks, FSC (Fluent Speech Commands) [237] for IC (intent classification), QUESST (QUESST 2014) [238] for QbE (query by example), LS En-Fr [239] and CoVoST-2 [240] for ST (speech translation), and ALFFA and OpenSLR-multi for multilingual ASR.”
MethodSWITCHBOARD: telephone speech corpus for research and development
Godfrey, J. J., Holliman, E. C., & McDaniel, J. (1992) · [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1992.225858 · Cited by 2,134 · cited 5×
“SWB is a 260-hour conversational speech recognition dataset containing two-sided telephone conversations.”
InfluentialMaximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains
Gauvain, J.-L., & Lee, C.-H. (1994) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.279278 · Cited by 2,108 · cited 1×
“(GMMs) [28], which facilitated the modeling of more variability in the input data.”
BackgroundSelf-Supervised Visual Feature Learning With Deep Neural Networks: A Survey
Jing, L., & Tian, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2992393 · Cited by 2,036 · cited 1×
“[21], [22] and computer vision (CV) [23].”
Learning deep structured semantic models for web search using clickthrough data
Huang, P.-S., He, X., Gao, J., Deng, L., Acero, A., & Heck, L. (2013) · Proceedings of the 22nd ACM international conference on Information & Knowledge Management · doi:10.1145/2505515.2505665 · Cited by 2,009 · cited 1×
“Some typical approaches in this category include Multi-view autoencoders and variations [154], [155], Multi-modal deep Boltzmann machines [156], Canonical correlation analysis (CCA) [157] and its nonlinear extensions [158], [159], [160], [161], [162], [163], [164], [165], [166], Multi-view contrastive losses [167], [1…”
MethodImproved Baselines with Momentum Contrastive Learning
Chen, X., Fan, H., Girshick, R., & He, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04297 · Cited by 1,956 · cited 1×
“This approach has become especially popular in the CV context [57], [58], [59], [60].”
BackgroundStatistical Methods for Speech Recognition
Sun, D. X., & Jelinek, F. (1999) · Journal of the American Statistical Association · doi:10.2307/2670189 · Cited by 1,946 · cited 1×
“We combine the ASR and language model via a noisy channel model [309], a weighted finite state transducer [310], or shallow”
MethodA Tutorial on Energy-Based Learning
LeCun, Y., Chopra, S., Hadsell, R. T., Ranzato, A., & Huang, F. J. (2006) · Cited by 1,931
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3188113 · Cited by 1,926 · cited 11×
“The WavLM model achieved substantial improvements on the speech separation, speaker verification and speaker diarization tasks in the SUPERB benchmark, while also performing well on many other tasks compared to HuBERT and wav2vec 2.0. d) data2vec: Motivated by the success of using an exponential moving average (EMA) t…”
BackgroundMethodInfluentialEstimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
Bengio, Y., Léonard, N., & Courville, A. (2013) · arXiv · doi:10.48550/arxiv.1308.3432 · Cited by 1,910 · cited 1×
“In the VQ-VAE this is done using the straightthrough estimator [79], i.”
MethodMultimodal Learning with Deep Boltzmann Machines
Srivastava, N., & Salakhutdinov, R. (2012) · Journal of Machine Learning Research · doi:10.5555/2627435.2697059 · Cited by 1,807
Relations Between Two Sets of Variates
Hotelling, H. (1936) · Biometrika · doi:10.2307/2333955 · Cited by 1,645
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591 · cited 1×
“[21], [22] and computer vision (CV) [23].”
A primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552 · cited 17×
“Due to the popularity of SSL, reviews have been published about the technology in general [16], [17], [18] as well as its application to natural language processing (NLP) [19], [20],”
BackgroundInfluentialMultimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433 · cited 2×
“Researchers also evaluate representations for sentiment analysis with the INTERFACE [233] and MOSEI (CMU Multimodal Opinion Sentiment and Emotion Intensity) [234] datasets.”
MethodSlow Feature Analysis: Unsupervised Learning of Invariances
Wiskott, L., & Sejnowski, T. J. (2002) · Neural Computation · doi:10.1162/089976602317318938 · Cited by 1,350 · cited 1×
“Depending on the downstream task, we are often interested in learning so-called slow features that typically span multiple input frames [95].”
Backgroundwav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290
Efficient transformers: A survey
Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2023) · ACM Computing Surveys · doi:10.1145/3530811 · Cited by 1,267 · cited 1×
“with the focus on self-attention [346], but these technology has not been widely used in speech SSL.”
MethodEfficient Transformers: A Survey
Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2020) · arXiv · doi:10.48550/arxiv.2009.06732 · Cited by 1,267
A Survey of Quantization Methods for Efficient Neural Network Inference
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2022) · Low-Power Computer Vision · doi:10.1201/9781003162810-13 · Cited by 1,219 · cited 1×
“els via conditional computing [344] and neural network quantization [345] as well as extensive work on improving the specific efficiency of Transformer models, especially”
Method(1993) · Cited by 1,173
The design for the wall street journal-based CSR corpus
Paul, D. B., & Baker, J. M. (1992) · Proceedings of the workshop on Speech and Natural Language - HLT '91 · doi:10.3115/1075527.1075614 · Cited by 1,099
Self-supervised Learning: Generative or Contrastive
Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., & Tang, J. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3090866 · Cited by 1,087 · cited 1×
“Due to the popularity of SSL, reviews have been published about the technology in general [16], [17], [18] as well as its application to natural language processing (NLP) [19], [20],”
BackgroundSpeech Commands: A Dataset for Limited-Vocabulary Speech Recognition
Warden, P. (2018) · arXiv · doi:10.48550/arxiv.1804.03209 · Cited by 1,033 · cited 4×
“Last, [116] puts together five datasets (MUSAN [245], Bird Audio Detection [246], Speech Commands [247], Spoken Language Identification [248], and TUT Urban Acoustic Scenes”
BackgroundInfluentialLearning multiple layers of representation
Hinton, G. E. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.09.004 · Cited by 1,011 · cited 1×
“target downstream applications requiring a minimal number of labeled data [7], [8], [9].”
BackgroundCommon Voice: A Massively-Multilingual Speech Corpus
Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F. M., & Weber, G. (2019) · arXiv · doi:10.48550/arxiv.1912.06670 · Cited by 905 · cited 1×
“These corpora include Common Voice (CV) [214], Multilingual LibriSpeech”
BackgroundMegatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · cited 2×
“Such tasks are widely used in NLP, for example, using the previous tokens in a sentence to predict the next token such as in ELMo [51], the GPT series [52], and Megatron [53], or predicting the masked tokens in a sentence such as with the bidirectional encoder representations from Transformers (BERT) series [54], [55].”
BackgroundWeighted finite-state transducers in speech recognition
Mohri, M., Pereira, F., & Riley, M. (2002) · Computer Speech & Language · doi:10.1006/csla.2001.0184 · Cited by 885 · cited 1×
“We combine the ASR and language model via a noisy channel model [309], a weighted finite state transducer [310], or shallow”
MethodOn Deep Multi-View Representation Learning
Wang, W., Livescu, K., & Bilmes, J. (2015) · Cited by 871
Self-citationPerceptual Signal Analysis and Eigenvalue-Weighted Metrics for Musical Audio Quality Assessment
Majumder, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-27827-2_35 · Cited by 845
Maximum mutual information estimation of hidden Markov model parameters for speech recognition
Bahl, L., Brown, P., de Souza, P., & Mercer, R. (2005) · ICASSP '86. IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1986.1169179 · Cited by 817 · cited 1×
“Furthermore, dynamical models like hidden Markov models (HMMs) [30] allowed for the processing of continuous speech rather than just isolated word recognition.”
BackgroundSimple statistical gradient-following algorithms for connectionist reinforcement learning
Williams, R. J. (1992) · Machine Learning · doi:10.1023/a:1022672621406 · Cited by 817
AISHELL-1: An open-source Mandarin speech corpus and a speech recognition baseline
Bu, H., Du, J., Na, X., Wu, B., & Zheng, H. (2017) · 2017 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA) · doi:10.1109/icsda.2017.8384449 · Cited by 791 · cited 1×
“1 [226] (178 hours of read speech), aidatatang 200zh (200 hours, read speech) [227], MAGICDATA Mandarin Chinese Read Speech Corpus (755 hours, read speech) [228], Free”
BackgroundBitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Ben Zaken, E., Goldberg, Y., & Ravfogel, S. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.1 · Cited by 765
Recent advances in the automatic recognition of audiovisual speech
Pomianos, G., Neti, C., Gravier, G., Garg, A., & Senior, A. W. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817150 · Cited by 752 · cited 1×
“Multi-modal work includes supervised settings, such as audio-visual ASR [143], [144] and person identification [145] which have been studied for decades.”
BackgroundA robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
Artetxe, M., Labaka, G., & Agirre, E. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1073 · Cited by 751 · cited 1×
“[271] Speech2vec (English) Oracle Word2Vec (French) VecMap [272] LM rescore, sequence DAE ST (BLEU): 10.”
BackgroundGenerating diverse high-fidelity images with vq-vae-2
Razavi, A., Oord, A. V. D., & Vinyals, O. (2019) · arXiv · doi:10.48550/arxiv.1906.00446 · Cited by 709
SUPERB: Speech Processing Universal PERformance Benchmark
Yang, S.-W., Chi, P.-H., Chuang, Y.-S., Lai, C.-I. J., Lakhotia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T., et al. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1775 · Cited by 681 · cited 14×
“The WavLM model achieved substantial improvements on the speech separation, speaker verification and speaker diarization tasks in the SUPERB benchmark, while also performing well on many other tasks compared to HuBERT and wav2vec 2.0. d) data2vec: Motivated by the success of using an exponential moving average (EMA) t…”
BackgroundMethodInfluentialSelf-citationUnsupervised Cross-Lingual Representation Learning for Speech Recognition
Conneau, A., Baevski, A., Collobert, R., Mohamed, A., & Auli, M. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-329 · Cited by 637 · cited 1×
“In [282], a wav2vec 2.0 model was trained on data from multiple different languages and different corpora (Babel, Common Voice, and multilingual LS) jointly, followed by fine-tuning for each individual language.”
MethodInfluentialSelf-citationA Recurrent Latent Variable Model for Sequential Data
Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A. C., & Bengio, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.02216 · Cited by 635 · cited 1×
“VAEs have been applied to speech in numerous works [70], [71], [72], [73], [74].”
MethodUnsupervised Neural Machine Translation
Artetxe, M., Labaka, G., Agirre, E., & Cho, K. (2017) · arXiv · doi:10.48550/arxiv.1710.11041 · Cited by 632 · cited 1×
“Success in unsupervised neural machine translation (MT) [287], [299], [300] has inspired innovative exploration of various unsupervised ASR algorithms.”
BackgroundEphrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Hassidim, A., Freeman, W. T., & Rubinstein, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201357 · Cited by 620 · cited 1×
“AVSpeech [210] is another large-scale audio-visual dataset used in SSL research, comprising 4.”
Learning a Distance Metric from Relative Comparisons
Schultz, M., & Joachims, T. (2003) · Cited by 609
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
Babu, A., Wang, C., Tjandra, A., Lakhotia, K., Xu, Q., Goyal, N., Singh, K., von Platen, P., Saraf, Y., Pino, J., et al. (2022) · Interspeech 2022 · doi:10.21437/interspeech.2022-143 · Cited by 586 · cited 5×
“and conventionally used for evaluation, while their audio streams are also aggregated to build diversified and large-scale corpora for unsupervised pre-training [109], [123], [218].”
BackgroundInfluentialConnectionist Speech Recognition
Bourlard, H. A., & Morgan, N. (1994) · Springer eBooks · doi:10.1007/978-1-4615-3210-1 · Cited by 585
Unsupervised Machine Translation Using Monolingual Corpora Only
Lample, G., Conneau, A., Denoyer, L., & Ranzato, M. (2017) · arXiv · doi:10.48550/arxiv.1711.00043 · Cited by 584 · cited 1×
“Success in unsupervised neural machine translation (MT) [287], [299], [300] has inspired innovative exploration of various unsupervised ASR algorithms.”
BackgroundGutmann, M. U., & Hyvärinen, A. (2012) · Edinburgh Research Explorer (University of Edinburgh) · doi:10.5555/2503308.2188396 · Cited by 581 · cited 5×
“The approach is similar to previous work on Noise-Contrastive Estimation (NCE) [135].”
BackgroundInfluentialChung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54184-6_6 · Cited by 537 · cited 1×
“face [148], [149], lip-movement [150], articulatory flesh point measurements [151], [152], or simultaneous magnetic resonance imaging (MRI) scans [153].”
BackgroundDeep Clustering for Unsupervised Learning of Visual Features
Caron, M., Bojanowski, P., Joulin, A., & Douze, M. (2018) · arXiv · doi:10.48550/arxiv.1807.05520 · Cited by 533 · cited 1×
“resentations and their cluster assignments [46], and the prediction of the relative positions of image patches [47].”
Background(2007) · The MIT Press eBooks · doi:10.7551/mitpress/7443.001.0001 · Cited by 525
Learning words from sights and sounds: A computational model
Roy, D. K., & Pentland, A. P. (2002) · Cognitive Science · doi:10.1207/s15516709cog2601_4 · Cited by 521
KERNEL AND NONLINEAR CANONICAL CORRELATION ANALYSIS
LAI, P. L., & FYFE, C. (2000) · International Journal of Neural Systems · doi:10.1142/s012906570000034x · Cited by 509
A Probabilistic Interpretation of Canonical Correlation Analysis
Bach, F., & Jordan, M. I. (2005) · Cited by 474
On Using Monolingual Corpora in Neural Machine Translation
Gülçehre, Ç., Fırat, O., Xu, K., Cho, K., Barrault, L., Lin, H.-C., Bougares, F., Schwenk, H., & Bengio, Y. (2015) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.48550/arxiv.1503.03535 · Cited by 466 · cited 1×
“We combine the ASR and language model via a noisy channel model [287], a weighted finite state transducer [288], or shallow fusion [289], [290].”
MethodThe Fisher Corpus: a Resource for the Next Generations of Speech-to-Text
Cieri, C., Miller, D. A. B., & Walker, K. P. (2004) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/5q862povzthr · Cited by 464 · cited 1×
“corpus [194] collects over 2k hours of conversational telephone speech, 1k hours of which is utilized for pre-training [86].”
BackgroundLibri-Light: A Benchmark for ASR with Limited or No Supervision
Kahn, J., Riviere, M., Zheng, W., Kharitonov, E., Xu, Q., Mazare, P. E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9052942 · Cited by 458 · cited 1×
“Libri-light (LL) [208], one of these datasets,”
Self-citationA vector quantization approach to speaker recognition
Soong, F., Rosenberg, A., Rabiner, L., & Juang, B. (2005) · ICASSP '85. IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1985.1168412 · Cited by 433
Self-Supervised Representation Learning: Introduction, advances, and challenges
Ericsson, L., Gouk, H., Loy, C. C., & Hospedales, T. M. (2022) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3134634 · Cited by 433 · cited 1×
“Due to the popularity of SSL, reviews have been published about the technology in general [16], [17], [18] as well as its application to natural language processing (NLP) [19], [20],”
BackgroundAutomatic lipreading to enhance speech recognition (speech reading)
Petajan, E. D. (1984) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 420
On rectified linear units for speech processing
Zeiler, M. D., Ranzato, M., Monga, R., Mao, M., Yang, K., Le, Q. V., Nguyen, P., Senior, A., Vanhoucke, V., Dean, J., et al. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638312 · Cited by 420 · cited 1×
“In addition to the approaches discussed above, several other works on speech representation learning take inspiration from the AE framework [86], [87], [88], [89], [90], [91].”
BackgroundLanguage recognition via i-vectors and dimensionality reduction
Dehak, N., Torres-Carrasquillo, P. A., Reynolds, D., & Dehak, R. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-328 · Cited by 419 · cited 1×
“Subsequent techniques in speaker and language verification [34], [35] similarly extracted param-”
MethodStowell, D., Wood, M. D., Pamuła, H., Stylianou, Y., & Glotin, H. (2019) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13103 · Cited by 414 · cited 3×
“Last, [116] puts together five datasets (MUSAN [245], Bird Audio Detection [246], Speech Commands [247], Spoken Language Identification [248], and TUT Urban Acoustic Scenes”
BackgroundNeural Audio Synthesis of Musical Notes with WaveNet Autoencoders
Engel, J., Resnick, C., Roberts, A., Dieleman, S., Eck, D., Simonyan, K., & Norouzi, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.01279 · Cited by 370 · cited 1×
“[117] employs the NSynth dataset [250] on top”
MethodMLS: A Large-Scale Multilingual Dataset for Speech Research
Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., & Collobert, R. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-2826 · Cited by 361 · cited 4×
“These corpora include Common Voice (CV) [214], Multilingual LibriSpeech (MLS) [215], VoxPopuli (VP) [216], and BABEL (BBL) [217].”
InfluentialTowards Better Decoding and Language Model Integration in Sequence to Sequence Models
Chorowski, J., & Jaitly, N. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-343 · Cited by 358
An Unsupervised Autoregressive Model for Speech Representation Learning
Chung, Y.-A., Hsu, W.-N., Tang, H., & Glass, J. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1473 · Cited by 357 · cited 8×
“APC is formulated as In text-based autoregressive LMs, c is set to 1 to enable autoregressive generation.”
InfluentialMockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
Liu, A. T., Yang, S.-W., Chi, P.-H., Hsu, P.-C., & Lee, H.-Y. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054458 · Cited by 337 · cited 4×
“Therefore it is common to mask chunks of consecutive frames [100], [104].”
BackgroundMethodInfluentialSelf-citationWang, C., Riviere, M., Lee, A., Wu, A., Talnikar, C., Haziza, D., Williamson, M., Pino, J., & Dupoux, E. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.80 · Cited by 329 · cited 1×
“(MLS) [215], VoxPopuli (VP) [216], and BABEL (BBL) [217].”
Unsupervised speech representation learning using WaveNet autoencoders
Chorowski, J., Weiss, R. J., Bengio, S., & van den Oord, A. (2019) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2019.2938863 · Cited by 325 · cited 1×
“Following the success of representation learning, baseline and challenge techniques have shifted from purely generative models [327], [328], deep autoencoders [83], [329], and incorporation of neural-network-”
BackgroundTERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
Liu, A. T., Li, S.-W., & Lee, H.-Y. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3095662 · Cited by 312 · cited 3×
“Whereas most studies use masking along the temporal dimension of the input, speech can also be masked along the frequency dimension when spectral input features are used [103], [106].”
BackgroundMethodSelf-citationDetection and Classification of Acoustic Scenes and Events: Outcome of the DCASE 2016 Challenge
Mesaros, A., Heittola, T., Benetos, E., Foster, P., Lagrange, M., Virtanen, T., & Plumbley, M. D. (2018) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2017.2778423 · Cited by 298
Improving Neural Machine Translation Models with Monolingual Data
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1009 · Cited by 296 · cited 1×
“This approach is similar to the back-translation technique developed in neural machine translation [322].”
MethodChung, Y.-A., Zhang, Y., Han, W., Chiu, C.-C., Qin, J., Pang, R., & Wu, Y. (2021) · 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru51503.2021.9688253 · Cited by 293 · cited 1×
“0 approach, the w2v-BERT model has managed to improve results even further [127].”
MethodTED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation
Hernandez, F., Nguyen, V., Ghannay, S., Tomashenko, N., & Estève, Y. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-99579-3_21 · Cited by 279 · cited 4×
“Several datasets, including GigaSpeech [220], TED-LIUM 3 (TED3) [221], TED-LIUM 2 (TED2) [222], Switchboard (SWB) [223], TIMIT [96], and VoxLingua107 [224], are labeled”
InfluentialThe Fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, Task and Baselines
Barker, J., Watanabe, S., Vincent, E., & Trmal, J. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1768 · Cited by 279 · cited 1×
“CHiME-5 [232] is a challenge that aims to advance robust ASR and presents a dataset of natural conversational speech collected under a dinner party scenario”
BackgroundSelf-citationSpeech Model Pre-Training for End-to-End Spoken Language Understanding
Lugosch, L., Ravanelli, M., Ignoto, P., Tomar, V. S., & Bengio, Y. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2396 · Cited by 272 · cited 4×
“VoxCeleb1 [236] for SID/ASV (automatic speaker verification) tasks, FSC (Fluent Speech Commands) [237] for IC (intent classification), QUESST (QUESST 2014) [238] for QbE (query by example), LS En-Fr [239] and CoVoST-2 [240] for ST (speech translation), and ALFFA and OpenSLR-multi for multilingual ASR.”
MethodInfluentialUnsupervised learning of spoken language with visual context
Harwath, D. F., Torralba, A. B., & Glass, J. R. (2016) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 271
Self-supervised Learning with Random-projection Quantizer for Speech RecognitionPreprint
Chiu, C., Qin, J., Zhang, Y., Yu, J., & Wu, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.01855 · Cited by 270
VOXLINGUA107: A Dataset for Spoken Language Recognition
Valk, J., & Alumäe, T. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383459 · Cited by 268 · cited 3×
“Several datasets, including GigaSpeech [220], TED-LIUM 3 (TED3) [221], TED-LIUM 2 (TED2) [222], Switchboard (SWB) [223], TIMIT [96], and VoxLingua107 [224], are labeled”
Layer-Wise Analysis of a Self-Supervised Speech Representation Model
Pasad, A., Chou, J.-C., & Livescu, K. (2021) · 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru51503.2021.9688093 · Cited by 262 · cited 2×
“[68] further show that fine-tuning the pre-trained model with a character-level CTC training criterion changes the behavior of the last layers (especially the final two layers), breaking”
BackgroundSelf-citationShparberg, A. L. (2023) · The Charleston Advisor · doi:10.5260/chara.24.3.41 · Cited by 260
Lightly supervised and unsupervised acoustic model training
Lamel, L., Gauvain, J.-L., & Adda, G. (2002) · Computer Speech & Language · doi:10.1006/csla.2001.0186 · Cited by 258 · cited 1×
“relies on a supervised teacher model to label a large volume of speech-only data, which is then used to augment the initial labeled data to train a student model [4], [5], [6], [61].”
MethodMulti-Task Self-Supervised Learning for Robust Speech Recognition
Ravanelli, M., Zhong, J., Pascual, S., Swietojanski, P., Monteiro, J., Trmal, J., & Bengio, Y. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053569 · Cited by 258 · cited 1×
“PASE and PASE+ [69], [113] use multiple targets, including the waveform, log power spectrum, mel”
MethodGenerative Spoken Dialogue Language Modeling
Nguyen, T. A., Kharitonov, E., Copet, J., Adi, Y., Hsu, W.-N., Elkahky, A. M., Tomasello, P., Algayres, R., Sagot, B., Mohamed, A., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00545 · Cited by 253 · cited 2×
“] → POWER) and continues using words in the same general mood (dark → BLACKNESS)6 and has been extended to model and generate dialogues [332].”
BackgroundSelf-citationDistilhubert: Speech Representation Learning by Layer-Wise Distillation of Hidden-Unit Bert
Chang, H.-J., Yang, S.-W., & Lee, H.-Y. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747490 · Cited by 245 · cited 1×
“representation models through network pruning [342] or knowledge distillation [343].”
BackgroundSelf-citationUnsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.05192 · Cited by 242 · cited 1×
“resentations and their cluster assignments [46], and the prediction of the relative positions of image patches [47].”
Backgrounddata2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., & Auli, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.03555 · Cited by 231 · cited 1×
“ponential moving average (EMA) teacher for self-supervised visual representations [139], [140], the data2vec model [132] computes targets Y using an EMA of its own parameters.”
MethodGigaSpeech: An Evolving, Multi-Domain ASR Corpus with 10,000 Hours of Transcribed Audio
Chen, G., Chai, S., Wang, G.-B., Du, J., Zhang, W.-Q., Weng, C., Su, D., Povey, D., Trmal, J., Zhang, J., et al. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1965 · Cited by 229 · cited 3×
“Several datasets, including GigaSpeech [220], TED-LIUM 3 (TED3) [221], TED-LIUM 2 (TED2) [222], Switchboard (SWB) [223], TIMIT [96], and VoxLingua107 [224], are labeled”
Self-citationCoVoST 2 and Massively Multilingual Speech-to-Text TranslationPreprint
Wang, C., Wu, A., & Pino, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.10310 · Cited by 222
O'Shaughnessy, D. (1988) · IEEE Potentials · doi:10.1109/45.1890 · Cited by 221 · cited 1×
“tion [94] and autoregressive language models (LM) for text, where the model learns to predict future information from past.”
BackgroundParameter-Efficient Transfer Learning with Diff Pruning
Guo, D., Rush, A., & Kim, Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.378 · Cited by 220 · cited 1×
“[339] are lightweight modules inserted into SSL models, and in downstream tasks, the parameters of SSL models are frozen, and only the adapters are trained.”
BackgroundLearning Problem-Agnostic Speech Representations from Multiple Self-Supervised Tasks
Pascual, S., Ravanelli, M., Serrà, J., Bonafonte, A., & Bengio, Y. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2605 · Cited by 219 · cited 4×
“PASE and PASE+ [69], [113] use multiple targets, including the waveform, log power spectrum, mel”
BackgroundMethodInfluentialWord Translation Without Parallel Data
Conneau, A., Lample, G., Ranzato, M., Denoyer, L., & Jeǵou, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.04087 · Cited by 219 · cited 1×
“Success in unsupervised neural machine translation (MT) [287], [299], [300] has inspired innovative exploration of various unsupervised ASR algorithms.”
BackgroundImproved Noisy Student Training for Automatic Speech Recognition
Park, D. S., Zhang, Y., Jia, Y., Han, W., Chiu, C.-C., Li, B., Wu, Y., & Le, Q. V. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1470 · Cited by 218 · cited 3×
“For comparison, we also show performance from such methods (iterative pseudo labeling (IPL) [63], slimIPL [254], noisy student [62]), as well as the current state of the art—conformer XXL + noisy student [255]—which augments SSL with various advanced techniques including self-training.”
BackgroundMethodResultSpeech Resynthesis from Discrete Disentangled Self-Supervised Representations
Polyak, A., Adi, Y., Copet, J., Kharitonov, E., Lakhotia, K., Hsu, W.-N., Mohamed, A., & Dupoux, E. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-475 · Cited by 211 · cited 2×
“speech representation learning to model language directly from audio, bypassing the need for text or labels [76], [77], [78], [331].”
BackgroundSelf-citationSpeech recognition and keyword spotting for low-resource languages : Babel project research at CUED
Gales, M., Knill, K., Ragni, A., & Rath, S. P. (2014) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 207
TED-LIUM: an Automatic Speech Recognition dedicated corpus
Rousseau, A., Deléglise, P., & Estève, Y. (2012) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/48xafxbrk3qa · Cited by 204
Chung, Y.-A., Wu, C.-C., Shen, C.-H., Lee, H.-Y., & Lee, L.-S. (2016) · Interspeech 2016 · doi:10.21437/interspeech.2016-82 · Cited by 202 · cited 2×
“For example, searching within a corpus of recorded speech for segments that match a given (written or spoken) query can be seen as finding segments whose representations are most similar to that of the query [118], [192], [193], [194]; word embeddings can be defined by pooling representations of instances of a given w…”
BackgroundMethodSelf-citationDeep multimodal semantic embeddings for speech and images
Harwath, D. F., & Glass, J. R. (2015) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2015.7404800 · Cited by 192 · cited 1×
“Typical approaches involve learning a neural representation model for each modality, with a multi-modal contrastive loss that encourages paired examples in the two modalities to have similar representations while unpaired examples remain different [171], [172], [173], [174], [175], [176].”
BackgroundA modified K-means clustering algorithm for use in isolated work recognition
Wilpon, J., & Rabiner, L. (1985) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1985.1164581 · Cited by 190 · cited 1×
“work such as [26], [27], word patterns were clustered semiautomatically using techniques such as k-means, after which isolated words were recognized by finding the training cluster”
MethodPushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
Zhang, Y., Qin, J., Park, D., Han, W., Chiu, C., Pang, R., Le, Q. V., & Wu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10504 · Cited by 190 · cited 3×
“For comparison, we also show performance from such methods (iterative pseudo labeling (IPL) [63], slimIPL [254], noisy student [62]), as well as the current state of the art—conformer XXL + noisy student [255]—which augments SSL with various advanced techniques including self-training.”
BackgroundMethodUnsupervised Pretraining Transfers Well Across Languages
Riviere, M., Joulin, A., Mazare, P.-E., & Dupoux, E. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054548 · Cited by 186 · cited 2×
“the size (number of parameters) of the model? Several studies have demonstrated better downstream performance when using larger datasets [123], [131], [277].”
BackgroundMethodRobust Self-Supervised Audio-Visual Speech Recognition
Shi, B., Hsu, W.-N., & Mohamed, A. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-99 · Cited by 185
Self-citationRobust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training
Hsu, W.-N., Sriram, A., Baevski, A., Likhomanenko, T., Xu, Q., Pratap, V., Kahn, J., Lee, A., Collobert, R., Synnaeve, G., et al. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-236 · Cited by 182 · cited 2×
“per, there is already some related research about enhancing the robustness of SSL models [281], [352], [353], [354] and identifying their vulnerability to adversarial attack [351].”
BackgroundAVICAR: audio-visual speech corpus in a car environment
Lee, B., Hasegawa-Johnson, M., Goudeseune, C., Kamdar, S., Borys, S., Liu, M., & Huang, T. (2004) · Interspeech 2004 · doi:10.21437/interspeech.2004-424 · Cited by 181
Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised RepresentationsPreprint
Choi, H.-S., Lee, J., Kim, W., Lee, J. H., Heo, H., & Lee, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.14513 · Cited by 180
Dynamical Variational Autoencoders: A Comprehensive Review
Girin, L., Leglaive, S., Bie, X., Diard, J., Hueber, T., & Alameda-Pineda, X. (2022) · Foundations and Trends in Machine Learning · doi:10.1561/2200000089 · Cited by 176 · cited 4×
“While typical autoencoders learn data representations using unsupervised objectives by reconstructing the input after passing it through an information bottleneck, VAEs estimate a neural model of a probability density function (pdf) that approximates the unknown “true” distribution of the observed data, for which we o…”
BackgroundInfluentialContentVec: An Improved Self-Supervised Speech Representation by Disentangling SpeakersPreprint
Qian, K., Zhang, Y., Gao, H., Ni, J., Lai, C.-I., Cox, D., Hasegawa‐Johnson, M., & Chang, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.09224 · Cited by 174 · cited 1×
“Some work has been in this direction [348], [349], [350].”
BackgroundThe zero resource speech challenge 2017
Dunbar, E., Cao, X. N., Benjumea, J., Karadayi, J., Bernard, M., Besacier, L., Anguera, X., & Dupoux, E. (2017) · 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru.2017.8268953 · Cited by 169 · cited 1×
“The Zero Resource Speech Challenge 2017 [324] focused on unseen language and speaker aspects from the previous challenge.”
BackgroundAudio Albert: A Lite Bert for Self-Supervised Learning of Audio Representation
Chi, P.-H., Chung, P.-H., Wu, T.-H., Hsieh, C.-C., Chen, Y.-H., Li, S.-W., & Lee, H.-Y. (2021) · 2021 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383575 · Cited by 168 · cited 2×
“AALBERT [100] LS 360 hrs PC LS 100 hrs(13) LS 100 hrs(13) 80(12) hrs”
Self-citationSmith, N. D., & Gales, M. (2001) · Cited by 168
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input
Harwath, D., Recasens, A., Surís, D., Chuang, G., Torralba, A., & Glass, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_40 · Cited by 165
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech
Chung, Y.-A., & Glass, J. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-2341 · Cited by 160 · cited 3×
“For example, searching within a corpus of recorded speech for segments that match a given (written or spoken) query can be seen as finding segments whose representations are most similar to that of the query [118], [192], [193], [194]; word embeddings can be defined by pooling representations of instances of a given w…”
BackgroundHierarchical mixtures of experts and the EM algorithm
Jordan, M. I., & Jacobs, R. A. (2005) · Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan) · doi:10.1109/ijcnn.1993.716791 · Cited by 159 · cited 1×
“niques include k-means clustering [11], mixture models [12], autoencoders [13], and non-negative matrix factorization [14].”
Speech Recognition with Augmented Synthesized Speech
Rosenberg, A., Zhang, Y., Ramabhadran, B., Ye, J., Moreno, P. J., Wu, Y., & Wu, Z. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003990 · Cited by 158 · cited 1×
“straightforward approach is to simply use multi-speaker TTS to generate the waveform with various acoustic variations [314], [315], [316], [317], [318].”
MethodGenerative Pre-Training for Speech with Autoregressive Predictive Coding
Chung, Y.-A., & Glass, J. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054438 · Cited by 157 · cited 1×
“b) Autoregressive prediction: Autoregressive predictive coding (APC) [92], [93] takes inspiration from the classic Linear Predictive Coding (LPC) approach for speech feature extrac-”
Multilingual Distributed Representations without Word AlignmentPreprint
Hermann, K. M., & Blunsom, P. (2013) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.48550/arxiv.1312.6173 · Cited by 156 · cited 1×
“Some typical approaches in this category include Multi-view autoencoders and variations [154], [155], Multi-modal deep Boltzmann machines [156], Canonical correlation analysis (CCA) [157] and its nonlinear extensions [158], [159], [160], [161], [162], [163], [164], [165], [166], Multi-view contrastive losses [167], [1…”
MethodQuery-by-example keyword spotting using long short-term memory networks
Chen, G., Parada, C., & Sainath, T. N. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178970 · Cited by 154 · cited 1×
“For example, searching within a corpus of recorded speech for segments that match a given (written or spoken) query can be seen as finding segments whose representations are most similar to that of the query [118], [192], [193], [194]; word embeddings can be defined by pooling representations of instances of a given w…”
BackgroundSelf-citationDeep convolutional acoustic word embeddings using word-pair side information
Kamper, H., Wang, W., & Livescu, K. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472619 · Cited by 153 · cited 1×
“, [89]) and word-level [199], [204] representations.”
Self-citationSequential Neural Models with Stochastic Layers
Fraccaro, M., Sønderby, S. K., Paquet, U., & Winther, O. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07571 · Cited by 153 · cited 1×
“VAEs have been applied to speech in numerous works [70], [71], [72], [73], [74].”
MethodCSTR VCTK Corpus: English Multi-Speaker Corpus for CSTR Voice Cloning Toolkit
Veaux, C. (2025) · TIB Data Manager · doi:10.57702/c0brh9iz · Cited by 152
Jansen, A., Dupoux, E., Goldwater, S. J., Johnson, M., Khudanpur, S. P., Church, K., Feldman, N. H., Heřmanský, H., Metze, F., Rose, R. C., et al. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639245 · Cited by 148 · cited 1×
“, transcriptions and other annotations [302].”
Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
Hsu, W.-N., Zhang, Y., & Glass, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.07902 · Cited by 148
Text-Free Prosody-Aware Generative Spoken Language Modeling
Kharitonov, E., Lee, A., Polyak, A., Adi, Y., Copet, J., Lakhotia, K., Nguyen, T. A., Rivière, M., Mohamed, A., Dupoux, E., et al. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.593 · Cited by 147 · cited 4×
“One workaround is two-stage training [77], [332];”
BackgroundInfluentialSelf-citationRepresentations of language in a model of visually grounded speech signal
Chrupała, G., Gelderloos, L., & Alishahi, A. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1057 · Cited by 146 · cited 1×
“other contextual modality), such models often learn multiple levels of linguistic representations from the shallowest to the deepest model layers [181], [182], [183].”
BackgroundDeep Contextualized Acoustic Representations for Semi-Supervised Speech Recognition
Ling, S., Liu, Y., Salazar, J., & Kirchhoff, K. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053176 · Cited by 145 · cited 1×
“DeCoAR [99] combines the bidirectionality of the popular NLP model ELMo [51] and the reconstruction objective of APC to alleviate this issue”
MethodSelf-citationLearning Latent Representations for Speech Generation and Transformation
Hsu, W.-N., Zhang, Y., & Glass, J. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-349 · Cited by 141 · cited 1×
“VAEs have been applied to speech in numerous works [70], [71], [72], [73], [74].”
MethodListening while speaking: Speech chain by deep learning
Tjandra, A., Sakti, S., & Nakamura, S. (2017) · 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru.2017.8268950 · Cited by 140 · cited 2×
“A speech chain [302], [305] is one successful way to utilize audio-only and text-only data to train both end-to-end ASR/TTS models.”
MethodUnsupervised learning of acoustic features via deep canonical correlation analysis
Wang, W., Arora, R., Livescu, K., & Bilmes, J. A. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178840 · Cited by 137 · cited 1×
“Some typical approaches in this category include Multi-view autoencoders and variations [154], [155], Multi-modal deep Boltzmann machines [156], Canonical correlation analysis (CCA) [157] and its nonlinear extensions [158], [159], [160], [161], [162], [163], [164], [165], [166], Multi-view contrastive losses [167], [1…”
MethodSelf-citationWord embeddings for speech recognition
Samy Bengio, A., & Heigold, G. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-273 · Cited by 136 · cited 1×
“For example, searching within a corpus of recorded speech for segments that match a given (written or spoken) query can be seen as finding segments whose representations are most similar to that of the query [118], [192], [193], [194]; word embeddings can be defined by pooling representations of instances of a given w…”
BackgroundThe zero resource speech challenge 2015
Versteegh, M., Thiollière, R., Schatz, T., Cao, X. N., Anguera, X., Jansen, A., & Dupoux, E. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-638 · Cited by 136 · cited 2×
“The relation between model sizes and downstream performances have also been investigated [278], [279].”
BackgroundUnsupervised neural network based feature extraction using weak top-down constraints
Kamper, H., Elsner, M., Jansen, A., & Goldwater, S. J. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7179087 · Cited by 133 · cited 2×
“In addition to the approaches discussed above, several other works on speech representation learning take inspiration from the AE framework [86], [87], [88], [89], [90], [91].”
BackgroundAdaptation Algorithms for Neural Network-Based Speech Recognition: An Overview
Bell, P., Fainberg, J., Klejch, O., Li, J., Renals, S. J., & Świętojański, P. (2020) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2020.3045349 · Cited by 131 · cited 1×
“One survey indeed addresses TL for speech processing [67] but does not include current SSL technologies for speech.”
BackgroundConditional Computation in Neural Networks for faster models
Bengio, E., Bacon, P., Pineau, J., & Precup, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06297 · Cited by 130 · cited 1×
“els via conditional computing [344] and neural network quantization [345] as well as extensive work on improving the specific efficiency of Transformer models, especially”
MethodVoita, E., Sennrich, R., & Titov, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1448 · Cited by 127 · cited 1×
“In an analysis of text-based Transformers trained with a masked language model criterion [269] observed a similar compression plus reconstruction pattern.”
MethodBack-Translation-Style Data Augmentation for end-to-end ASR
Hayashi, T., Watanabe, S., Zhang, Y., Toda, T., Hori, T., Astudillo, R. F., & Takeda, K. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639619 · Cited by 123 · cited 1×
“, encoder features [319] and phoneme features [320], [321].”
BackgroundSelf-citationTowards Learning a Universal Non-Semantic Representation of Speech
Shor, J., Jansen, A., Maor, R., Lang, O., Tuval, O., Quitry, F. D. C., Tagliasacchi, M., Shavitt, I., Emanuel, D., & Haviv, Y. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1242 · Cited by 123
HKUST/MTS: A Very Large Scale Mandarin Telephone Speech Corpus
Liu, Y., Fung, P., Yongsheng, Y., Cieri, C., Huang, S., & Graff, D. (2006) · Lecture notes in computer science · doi:10.1007/11939993_73 · Cited by 119 · cited 1×
“) [208], AISHELL-1 [209] (178 hours of read speech), aidatatang 200zh (200 hours, read speech) [210], MAGICDATA Mandarin Chinese Read Speech Corpus (755 hours, read speech) [211], Free ST Chinese Mandarin Corpus (ST-CMDS, 100 hours, read speech) [212], and Primewords”
BackgroundFixed-dimensional acoustic embeddings of variable-length segments in low-resource settings
Levin, K., Henry, K. E., Jansen, A., & Livescu, K. (2013) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2013.6707765 · Cited by 118 · cited 2×
“them as vectors of distances from the target segment to a number of pre-defined “template” segments [199].”
BackgroundSelf-citationAleksic, P. S., & Katsaggelos, A. K. (2006) · Proceedings of the IEEE · doi:10.1109/jproc.2006.886017 · Cited by 117 · cited 1×
“Multi-modal work includes supervised settings, such as audio-visual ASR [143], [144] and person identification [145] which have been studied for decades.”
BackgroundVector-Quantized Neural Networks for Acoustic Unit Discovery in the ZeroSpeech 2020 Challenge
Niekerk, B. V., Nortje, L., & Kamper, H. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1693 · Cited by 114 · cited 2×
“SSL techniques are also benchmarked with the ZeroSpeech challenge [265], [266].”
MethodJiang, D., Li, W., Cao, M. Y., Zou, W., & Li, X. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-391 · Cited by 113 · cited 2×
“Interestingly, similar network behavior was also recently described for self-supervised learners in computer vision: using a contrastive self-supervised learner (SimCLR) that optimizes for augmentation invariance, [270] show that it is the intermediate representations that most closely approximate information learned…”
BackgroundNatural-quality background noise coding using residual substitution
El-Maleh, K., & Kabal, P. (1999) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1999-599 · Cited by 112
Iterative Pseudo-Labeling for Speech Recognition
Xu, Q., Likhomanenko, T., Kahn, J., Hannun, A., Synnaeve, G., & Collobert, R. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1800 · Cited by 112 · cited 4×
“For comparison, we also show performance from such methods (iterative pseudo labeling (IPL) [63], slimIPL [254], noisy student [62]), as well as the current state of the art—conformer XXL + noisy student [255]—which augments SSL with various advanced techniques including self-training.”
BackgroundMethodResultInfluentialEvain, S. V., Nguyen, H., Le, H., Boito, M. Z., Mdhaffar, S., Alisamir, S., Tong, Z., Tomashenko, N., Dinarelli, M., Parcollet, T., et al. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-556 · Cited by 112 · cited 3×
“mark [260] shares a vision similar to SUPERB and provides a reproducible framework for assessing SSL in French with ASR, spoken language understanding, speech translation, and emo-”
MethodSLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-TrainingPreprint
Bapna, A., Chung, Y.-A., Wu, N., Gulati, A., Ye, J., Clark, J. H., Johnson, M., Riesa, J., Conneau, A., & Zhang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10329 · Cited by 111 · cited 1×
“The SLAM approach [191] is an example where speech and text are first represented using two separate pre-trained encoders followed by a multi-modal encoder to build the joint”
MethodCoVoST 2 and Massively Multilingual Speech Translation
Wang, C., Wu, A., Gu, J., et al. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-2027 · Cited by 110
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Shi, B., Hsu, W.-N., Lakhotia, K., & Mohamed, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.02184 · Cited by 110 · cited 2×
“Some typical approaches in this category include Multi-view autoencoders and variations [154], [155], Multi-modal deep Boltzmann machines [156], Canonical correlation analysis (CCA) [157] and its nonlinear extensions [158], [159], [160], [161], [162], [163], [164], [165], [166], Multi-view contrastive losses [167], [1…”
BackgroundMethodSelf-citationNonlinear Feature Extraction Using Generalized Canonical Correlation Analysis
Melzer, T., Reiter, M. J., & Bischof, H. (2001) · Lecture notes in computer science · doi:10.1007/3-540-44668-0_50 · Cited by 109 · cited 1×
“Some typical approaches in this category include Multi-view autoencoders and variations [154], [155], Multi-modal deep Boltzmann machines [156], Canonical correlation analysis (CCA) [157] and its nonlinear extensions [158], [159], [160], [161], [162], [163], [164], [165], [166], Multi-view contrastive losses [167], [1…”
MethodGauthier, E., Besacier, L., Voisin, S., Melese, M., & Elingui, U. P. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/5fsd4jpr7a4x · Cited by 108 · cited 3×
“For example, to demonstrate the robustness against unseen languages, the systems were developed with English, French, and Mandarin and tested on two “surprise” languages: German and Wolof.”
BackgroundRavi, S., & Knight, K. K. (2012) · Cited by 107
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos
Rouditchenko, A., Boggust, A., Harwath, D., Chen, B., Joshi, D., Thomas, S., Audhkhasi, K., Kuehne, H., Panda, R., Feris, R., et al. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1312 · Cited by 106 · cited 1×
“Typical approaches involve learning a neural representation model for each modality, with a multi-modal contrastive loss that encourages paired examples in the two modalities to have similar representations while unpaired examples remain different [171], [172], [173], [174], [175], [176].”
BackgroundPhrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration
Wang, S., Thompson, L., & Iyyer, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.846 · Cited by 106 · cited 1×
“Since most work on acoustic word embeddings preceded the very recent wave of new self-supervised frame-level representations, one question is whether word (or more generally segment) embeddings could be derived more simply by pooling self-supervised frame-level representations, as has been done for text span embedding…”
Background
Research works printed but not linked: 18
- 10.1109/massp.1984.1162229 · DOI
- Radford, 2019, Language models are unsupervised multitask learners, 1, 9
- 10.5555/3524938.3525087 · doi:10.5555/3524938.3525087
- Garofolo, 1993, TIMIT acoustic-phonetic continuous speech corpus LDC93S1
- Yang, 2019, Generalized Autoregressive Pretraining for Language Understanding, 32
- 10.1109/icassp40776.2020.9053176 · DOI · Paper page
- Baevski, 2020, vq-wav2vec: Self-supervised learning of discrete speech representations
- Baevski, 2020, wav2vec 2.0: A framework for self-supervised learning of speech representations, 33, 12449
- 10.1007/978-3-030-01264-9_9 · DOI · Paper page
- 10.5555/3495724.3497510 · doi:10.5555/3495724.3497510
- 10.1109/icassp.2018.8461761 · DOI
- 10.1109/asru.2017.8269008 · doi:10.1109/asru.2017.8269008
- Maas, 2012, Word-level acoustic modeling with convolutional vector regression
- Rodrguez-Fuentes, 2014, GTTS-EHU systems for QUESST at MediaEval 2014
- 10.1109/icassp40776.2020.9054548 · DOI · Paper page
- Pu, 2021, Scaling effect of self-supervised models, 1084
- Goodfellow, 2014, Generative adversarial nets, 2672
- Arjovsky, 2017, GAN wasserstein, 214
Cited by
Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects
Zhang, K., Wen, Q., Zhang, C., Cai, R., Jin, M., Liu, Y., Zhang, J. Y., Liang, Y., Pang, G., Song, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387317 · Cited by 295
“Using the learning paradigm as a taxonomy is arguably the most popular among the existing SSL surveys, including [20], [22], [23], [24], [25], [26], [27].”
MethodTen deep learning techniques to address small data problems with remote sensing
Safonova, A., Ghazaryan, G., Stiller, S., Main-Knorn, M., Nendel, C., & Ryo, M. (2023) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2023.103569 · Cited by 196
“, 2022), physics (Ma and Liu, 2020), speech representation (Mohamed et al., 2022), RS (S.”
BackgroundComparative Layer-Wise Analysis of Self-Supervised Speech Models
Pasad, A., Shi, B., & Livescu, K. (2023) · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49357.2023.10096149 · Cited by 105
“Self-supervised models have become a nearly ubiquitous approach for learning speech representations and improving performance on downstream tasks [1–5], but our understanding of their properties and strategies for their use is still limited.”
BackgroundSelf-citationGalić, I., Habijan, M., Leventić, H., & Romić, K. (2023) · Electronics · doi:10.3390/electronics12214411 · Cited by 150
Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study
Edin, J., Junge, A., Havtorn, J. D., Borgholt, L., Maistro, M., Ruotsalo, T., & Maaløe, L. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591918 · Cited by 137
“Finally, while PLM-ICD outperforms the other models in this paper, the improvements are limited compared to the effect of pre-training in other domains [1, 6, 9, 23, 28].”
Self-citation
Metadata
- Authors
- Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., Sainath, T. N., & Watanabe, S.
- Affiliations
- Meta (United States) (US)
- National Taiwan University (TW)
- University of Copenhagen (DK)
- Technical University of Denmark (DK)
- Amazon (United States) (US)
- Toyota Technological Institute at Chicago (US)
- Google (United States) (US)
- Carnegie Mellon University (US)
- ORCID
- Venue
- IEEE Journal of Selected Topics in Signal Processing
- Published
- 2022-10
- Volume
- 16
- Issue
- 6
- Pages
- 1179-1210
- ISSN
- 1932-4553 (ISSN-L)
- 1941-0484
- OpenAlex
- W4281492411
- Semantic Scholar
- f76a5b176f3435214eb87dd105f730f0b53672c3
- Cited by
- 372
- References
- 345
- Citations by year
- 2026: 66 · 2025: 91 · 2024: 110 · 2023: 94 · 2022: 11
- FWCI
- 5.35
- Type
- journal article
- Language
- en
- Open access
- Licence
- Locations
- IEEE Journal of Selected Topics in Signal Processing (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion)
- Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) (submittedVersion, other-oa)
- arXiv (Cornell University) (cc-by)
- Funders
- Innovation Fund Denmark through Industrial Ph.D. Programme (0153-00167B)
- Innovation Fund Denmark thorugh Industrial Ph.D. Programme (8053-00184B)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Keywords
- self-supervised learning, speech representation learning, generative models, contrastive learning, prediction model, multimodal pre-training, acoustic word embeddings, low-resource languages, speech recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.