Unsupervised Speech Representation Learning Using WaveNet Autoencoders
Open accessThe DOI matches its Crossref record; 78 of the 78 references it cites were found in a registry or an index.
Abstract
We consider the task of unsupervised extraction of meaningful latent representations of speech by applying autoencoding neural networks to speech waveforms. The goal is to learn a representation able to capture high level semantic content from the signal, e.g. phoneme identities, while being invariant to confounding low level details in the signal such as the underlying pitch contour or background noise. Since the learned representation is tuned to contain only phonetic content, we resort to using a high capacity WaveNet decoder to infer information discarded by the encoder from previous samples. Moreover, the behavior of autoencoder models depends on the kind of constraint that is applied to the latent representation. We compare three variants: a simple dimensionality reduction bottleneck, a Gaussian Variational Autoencoder (VAE), and a discrete Vector Quantized VAE (VQ-VAE). We analyze the quality of learned representations in terms of speaker independence, the ability to predict phonetic content, and the ability to accurately reconstruct individual spectrogram frames. Moreover, for discrete encodings extracted using the VQ-VAE, we measure the ease of mapping them to phonemes. We introduce a regularization scheme that forces the representations to focus on the phonetic content of the utterance and report performance comparable with the top entries in the ZeroSpeech 2017 unsupervised acoustic unit discovery task.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Thus far, there have been numerous methods proposed to detect deepfakes [25–29].”
From Deep learning for deepfakes creation and detection: A survey (2022) · cites this paper as [27]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“However, recent breakthroughs in computer vision [3], [4], machine translation [5], [6], speech recognition [7], [8], and language understanding [9], [10] rely on large”
BackgroundAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“We used the Adam optimizer [52] with initial learning rate 4× 10−4 which was halved after 400k, 600k, and 800k steps.”
MethodImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“For instance, in the visual domain, features discovered on ImageNet [27] are routinely used as input representations in other computer vision tasks [28].”
MethodGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
(2014) · Cited by 33,419
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · cited 1×
“Each layer can be interpreted as a feature extractor whose outputs are passed to upstream units [21].”
BackgroundReducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · cited 2×
“The Variational Autoencoder (VAE) [38] proposes a different interpretation of feature learning which follows a probabilistic”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 2×
“have been shown to create a hierarchy of visual atoms [11] that match some properties of the visual cortex [22].”
BackgroundLearning the parts of objects by non-negative matrix factorization
Lee, D. D., & Seung, H. S. (1999) · Nature · doi:10.1038/44565 · Cited by 13,868 · cited 2×
“such representations may be difficult to interpret because the reconstruction of an input depends on all latent features [36].”
BackgroundMethodSpeech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“However, recent breakthroughs in computer vision [3], [4], machine translation [5], [6], speech recognition [7], [8], and language understanding [9], [10] rely on large”
BackgroundDomain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318 · cited 1×
“enforce invariance to a known nuisance factor [65].”
BackgroundExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · cited 1×
“The deep learning revolution was triggered by the development of hierarchical representation learning algorithms, such as stacked Restricted Boltzman Machines [1] and Denoising Autoencoders [2].”
Method“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · cited 2×
“We focus on representations learned with autoencoders applied to raw waveforms and spectrogram features and investigate the quality of learned representations on LibriSpeech [17].”
MethodHow transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · cited 1×
“For instance, in the visual domain, features discovered on ImageNet [28] are routinely used as input representations in other computer vision tasks [29].”
MethodEmergence of simple-cell receptive field properties by learning a sparse code for natural images
Olshausen, B. A., & Field, D. J. (1996) · Nature · doi:10.1038/381607a0 · Cited by 5,694 · cited 1×
“In contrast, dictionary learning techniques, such as sparse [37] and non-negative [36] decompositions, express each input pattern using a combination of a small number of selected features out”
MethodGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 1×
“However, recent breakthroughs in computer vision [3], [4], machine translation [5], [6], speech recognition [7], [8], and language understanding [9], [10] rely on large”
BackgroundWaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · cited 6×
“We used 256 quantization levels after mu-law companding [18].”
BackgroundMethodInfluentialAn Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
Bai, S., Kolter, J. Z., & Koltun, V. (2018) · arXiv · doi:10.48550/arxiv.1803.01271 · Cited by 4,655 · cited 1×
“For text, these correspond to n-gram models [43] and convolutional neural language models [44], [45].”
MethodKaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 4,609
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 2×
“Likewise, in natural language processing, universal text representations can be extracted from networks trained for machine translation [32] or language inference [33], [34].”
BackgroundMethod(2017) · International Conference on Learning Representations · Cited by 3,248
Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461368 · Cited by 2,791
Self-citationNeural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729 · cited 7×
“through estimator to bring the gradient from the decoder to the encoder, and two VQ-related terms: the distance from each prototype to its assigned vectors and the commitment cost [19]: L = log px | zq(x) )”
BackgroundMethodInfluentialSelf-citationGenerating Sentences from a Continuous Space
Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., & Bengio, S. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1002 · Cited by 2,061 · cited 3×
“This is a common problem of sequential VAEs that use loss terms to regularize the latent encoding [49].”
BackgroundInfluentialSupervised Learning of Universal Sentence Representations from Natural Language Inference Data
Conneau, A., Kiela, D., Schwenk, H., Barrault, L., & Bordes, A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1070 · Cited by 2,049 · cited 1×
“Likewise, in natural language processing, universal text representations can be extracted from networks trained for machine translation [32] or language inference [33], [34].”
BackgroundEstimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
Bengio, Y., Léonard, N., & Courville, A. (2013) · arXiv · doi:10.48550/arxiv.1308.3432 · Cited by 1,910 · cited 1×
“During the backward pass, the gradient of the loss with respect to the pre-quantized embedding is approximated using the straight-through estimator [42], i.”
MethodAcceleration of Stochastic Approximation by Averaging
Polyak, B. T., & Juditsky, A. B. (1992) · SIAM Journal on Control and Optimization · doi:10.1137/0330046 · Cited by 1,827 · cited 1×
“Polyak averaging [53] was applied to all checkpoints used for model”
MethodSlow Feature Analysis: Unsupervised Learning of Invariances
Wiskott, L., & Sejnowski, T. J. (2002) · Neural Computation · doi:10.1162/089976602317318938 · Cited by 1,350
Language Modeling with Gated Convolutional Networks
Dauphin, Y., Fan, A., Auli, M., & Grangier, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08083 · Cited by 1,080 · cited 1×
“For text, these correspond to n-gram models [43] and convolutional neural language models [44], [45].”
MethodState-of-the-Art Speech Recognition with Sequence-to-Sequence Models
Chiu, C.-C., Sainath, T. N., Wu, Y., Prabhavalkar, R., Nguyen, P., Chen, Z., Kannan, A., Weiss, R. J., Rao, K., Gonina, E., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462105 · Cited by 1,043
Self-citationGated Self-Matching Networks for Reading Comprehension and Question Answering
Wang, W., Yang, N., Wei, F., Chang, B., & Zhou, M. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1018 · Cited by 666 · cited 1×
“However, recent breakthroughs in computer vision [3], [4], machine translation [5], [6], speech recognition [7], [8], and language understanding [9], [10] rely on large”
BackgroundUnsupervised Machine Translation Using Monolingual Corpora Only
Lample, G., Conneau, A., Denoyer, L., & Ranzato, M. (2017) · arXiv · doi:10.48550/arxiv.1711.00043 · Cited by 584
Learned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572 · cited 1×
“Likewise, in natural language processing, universal text representations can be extracted from networks trained for machine translation [32] or language inference [33], [34].”
BackgroundLearning the speech front-end with raw waveform CLDNNs
Sainath, T. N., Weiss, R. J., Senior, A., Wilson, K. W., & Vinyals, O. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-1 · Cited by 430
Self-citationNeural Audio Synthesis of Musical Notes with WaveNet Autoencoders
Engel, J., Resnick, C., Roberts, A., Dieleman, S., Eck, D., Simonyan, K., & Norouzi, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.01279 · Cited by 370 · cited 1×
“Latent collapse can also be avoided in deterministic autoencoders, such as [64], which coupled a convolutional encoder to a powerful autoregressive WaveNet decoder [18] to learn a latent representation of music audio consisting of isolated notes from a variety of instruments.”
BackgroundQANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
Yu, A. W., Dohan, D., Luong, M.-T., Zhao, R., Chen, K., Norouzi, M., & Le, Q. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.09541 · Cited by 368 · cited 1×
“However, recent breakthroughs in computer vision [3], [4], machine translation [5], [6], speech recognition [7], [8], and language understanding [9], [10] rely on large labeled datasets and make little to no use of unsupervised representation learning.”
BackgroundEnd-to-end learning for music audio
Dieleman, S., & Schrauwen, B. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6854950 · Cited by 364 · cited 1×
“to learn auditory-like frequency decompositions on music [23] and speech [24]–[27] in their lower layers.”
BackgroundPixelVAE: A Latent Variable Model for Natural ImagesPreprint
Gulrajani, I., Kumar, K., Ahmed, F., Taïga, A. A., Visin, F., Vázquez, D., & Courville, A. C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05013 · Cited by 355 · cited 1×
“Depending on the use case, the encoder can process the whole utterance, emit a single latent vector and feed it to an autoregressive decoder [33], [46] or the encoder can periodically emit vectors of latent features to be consumed by the decoder [19], [47].”
MethodUnsupervised Pattern Discovery in Speech
Park, A. S., & Glass, J. R. (2008) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2007.909282 · Cited by 351 · cited 1×
“Linguistic unit discovery models detect recurring speech patterns at a word-like level, finding commonly repeated segments with a constrained dynamic time warping [74].”
BackgroundZoneout: Regularizing RNNs by Randomly Preserving Hidden ActivationsPreprint
Krueger, D., Maharaj, T., Kramár, J., Pezeshki, M., Ballas, N., Ke, N. R., Goyal, A., Bengio, Y., Courville, A. C., & Pal, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01305 · Cited by 323 · cited 3×
“We therefore introduce a dropout-inspired [50] time-jitter regularizer, also reminiscent of Zoneout [51] regularization for recurrent networks.”
InfluentialDeep Variational Information Bottleneck
Alemi, A. A., Fischer, I., Dillon, J. V., & Murphy, K. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.00410 · Cited by 321
Analysis of CNN-based speech recognition system using raw speech as input
Palaz, D., Magimai-Doss, M., & Collobert, R. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-3 · Cited by 288 · cited 1×
“Similarly, when applied to audio waveforms, neural networks have been shown to learn auditory-like frequency decompositions on music [22] and speech [23], [24], [25], [26] in their lower layers.”
BackgroundUnsupervised Learning of Disentangled and Interpretable Representations from Sequential DataPreprint
Hsu, W.-N., Zhang, Y., & Glass, J. R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.07902 · Cited by 285 · cited 1×
“[66] explore a hierarchical latent space which separates sequence-dependent variables from those which are sequence-independent ones.”
BackgroundImproved bottleneck features using pretrained deep neural networks
Yu, D., & Seltzer, M. L. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-91 · Cited by 272 · cited 1×
“Similarly, the speech community has used bottleneck features extracted from networks trained on phoneme prediction tasks [30], [31] as feature representations for speech recognition systems.”
BackgroundDisentangled Sequential AutoencoderPreprint
Li, Y., & Mandt, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.02991 · Cited by 258 · cited 1×
“[67] introduce a stochastic latent variable model for sequential data which also yields disentangled representations and allows content swapping between generated sequences.”
BackgroundHierarchical Generative Modeling for Controllable Speech SynthesisPreprint
Hsu, W.-N., Zhang, Y., Weiss, R. J., Zen, H., Wu, Y., Wang, Y., Cao, Y. J., Ye, J., Chen, Z., Shen, J., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.07217 · Cited by 252 · cited 1×
“1In TensorFlow this can be conveniently implemented using zq(x) = ze(x) + stop_gradient(eq(x) − ze(x)) autoregressive decoder [33], [46] or the encoder can periodically emit vectors of latent features to be consumed by the decoder”
MethodSelf-citationInvariant Representations without Adversarial TrainingPreprint
Moyer, D., Gao, S., Brekelmans, R., Steeg, G. V., & Galstyan, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09458 · Cited by 241 · cited 1×
“since this information is explicitly made available to the decoder conditioning network, and therefore need not be preserved in the latent encoding [54].”
BackgroundLearning a better representation of speech soundwaves using restricted boltzmann machines
Jaitly, N., & Hinton, G. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947700 · Cited by 231 · cited 1×
“to learn auditory-like frequency decompositions on music [23] and speech [24]–[27] in their lower layers.”
BackgroundA Nonparametric Bayesian Approach to Acoustic Model Discovery
Lee, C.-Y., & Glass, J. R. (2012) · Cited by 208
The language-independent bottleneck features
Vesely, K., Karafiat, M., Grezl, F., Janda, M., & Egorova, E. (2012) · 2012 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2012.6424246 · Cited by 196 · cited 1×
“e routinely used as input representations in other computer vision tasks [29]. Similarly, the speech community has used bottleneck features extracted from networks trained on phoneme prediction tasks [30], [31] as feature representations for speech recognition systems. Likewise, in natural language processing, univers…”
MethodAcoustic modeling with deep neural networks using raw time signal for LVCSR
Tüske, Z., Golik, P., Schlüter, R., & Ney, H. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-223 · Cited by 178
The zero resource speech challenge 2017
Dunbar, E., Cao, X. N., Benjumea, J., Karadayi, J., Bernard, M., Besacier, L., Anguera, X., & Dupoux, E. (2017) · 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru.2017.8268953 · Cited by 169
Improving Variational Inference with Inverse Autoregressive Flow
Kingma, D. P., Salimans, T., Józefowicz, R., Chen, X., Sutskever, I., & Welling, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04934 · Cited by 164 · cited 1×
“An alternative formulation of the VAE objective explicitly constrains the amount of information contained in the latent representation [41]:”
BackgroundSpeech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech
Chung, Y.-A., & Glass, J. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-2341 · Cited by 160
Schatz, T., Peddinti, V., Bach, F., Jansen, A., Heřmanský, H., & Emmanuel Dupoux, A. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-441 · Cited by 150 · cited 1×
“The ZeroSpeech evaluation scheme uses the minimal pair ABX test [56], [57] which assesses the model’s ability to”
MethodEfficient spoken term discovery using randomized algorithms
Jansen, A., & Durme, B. V. (2011) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2011.6163965 · Cited by 141 · cited 1×
“[61] trains an autoencoder on pairs of speech segments found using an unsupervised term discovery system [79].”
MethodDeep neural network features and semi-supervised training for low resource speech recognition
Thomas, S., Seltzer, M. L., Church, K., & Heřmanský, H. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6638959 · Cited by 132 · cited 1×
“In such scenario, limited amounts of data may be sufficient to learn an acoustic model on the representation discovered without supervision, but insufficient to learn the acoustic model and a data representation in a fully supervised manner [15], [16].”
BackgroundUnsupervised cross-lingual knowledge transfer in DNN-based LVCSR
Swietojanski, P., Ghoshal, A., & Renals, S. (2012) · 2012 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2012.6424230 · Cited by 108
A segmental framework for fully-unsupervised large-vocabulary speech recognition
Kamper, H., Jansen, A., & Goldwater, S. J. (2017) · Computer Speech & Language · doi:10.1016/j.csl.2017.04.008 · Cited by 101 · cited 1×
“segments into a common vector space where they could be clustered into word types [75].”
BackgroundUnsupervised Cross-Modal Alignment of Speech and Text Embedding SpacesPreprint
Chung, Y.-A., Weng, W., Tong, S., & Glass, J. R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.07467 · Cited by 95 · cited 1×
“of word embeddings discovered on separate corpora [77] the approach yielded an ASR system trained on unpaired speech and text data [78].”
MethodVariational Inference for Acoustic Unit Discovery
Ondel, L., Burget, L., & Černocký, J. (2016) · Procedia Computer Science · doi:10.1016/j.procs.2016.04.033 · Cited by 91 · cited 1×
“Extending it with an HMM temporal structure for sub-phonetic units leads to the DP-HMM and the HDP-HMM [69], [70], [71].”
MethodChen, H., Leung, C.-C., Xie, L., Ma, B., & Li, H. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-642 · Cited by 77 · cited 2×
“DP-GMM [66] imposes a Dirichlet Process prior over a Gaussian Mixture Model.”
MethodHidden Markov Model Variational Autoencoder for Acoustic Unit Discovery
Ebbers, J., Heymann, J., Drude, L., Glarner, T., Haeb-Umbach, R., & Raj, B. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1160 · Cited by 45
Multilingual bottle-neck feature learning from untranscribed speech
Chen, H., Leung, C.-C., Xie, L., Ma, B., & Li, H. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8269009 · Cited by 35 · cited 2×
“This is also visible in the large degradation [57] AND THREE OTHER SUBMISSIONS THAT USED NEURAL NETWORK IN AN UNSUPERVISED SETTING [58], [59], [60].”
BackgroundHeck, M. C., Sakti, S., & Nakamura, S. (2016) · Procedia Computer Science · doi:10.1016/j.procs.2016.04.032 · Cited by 34 · cited 4×
“Nevertheless, the multilingual prequantized features yield accuracies comparable to [58].”
BackgroundMethodInfluentialEvaluating speech features with the minimal-pair ABX task (II): resistance to noise
Schatz, T., Peddinti, V., Cao, X.-N., Bach, F., Heřmanský, H., & Emmanuel Dupoux, A. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-228 · Cited by 31
Nagamine, T., & Mesgarani, N. (2017) · PubMed · Cited by 30
Full Bayesian Hidden Markov Model Variational Autoencoder for Acoustic Unit Discovery
Glarner, T., Hanebrink, P., Ebbers, J., & Haeb‐Umbach, R. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2148 · Cited by 22 · cited 2×
“Introducing a HMM prior as in [73] could promote a latent representation which better mimics the temporal phonetic structure of speech.”
MethodYuan, Y., Leung, C.-C., Xie, L., Chen, H., Ma, B., & Li, H. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8269010 · Cited by 21 · cited 2×
“[61] trains an autoencoder on pairs of speech segments found using an unsupervised term discovery system [79].”
BackgroundMethodOn Using Backpropagation for Speech Texture Generation and Voice Conversion
Chorowski, J., Weiss, R. J., Saurous, R. A., & Samy Bengio, A. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461282 · Cited by 19
Self-citationLearning Word Embeddings from SpeechPreprint
Chung, Y.-A., & Glass, J. R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.01515 · Cited by 19 · cited 1×
“[76] replace the segmental autoencoder with a model that instead predicts a nearby speech segment and demonstrate that the representation shares many properties with word embeddings.”
Ansari, T. K., Rajath Kumar, M. P., Singh, S., & Ganapathy, S. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8269013 · Cited by 16 · cited 1×
“[60] extended this scheme with features discovered by an autoencoder trained on”
BackgroundUnsupervised Incremental Online Learning and Prediction of Musical Audio Signals
Marxer, R., & Purwins, H. (2016) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2016.2530409 · Cited by 14
An 800 bps VQ-based LPC voice coder
Wang, X., & Kuo, C. ‐. J. (1998) · The Journal of the Acoustical Society of America · doi:10.1121/1.422247 · Cited by 3 · cited 1×
“largest inventory of tokens, the overall encoder bitrate is low: 14 bits at 50 Hz = 700 bps, which is on par with the lowest bitrate of classical speech codecs [63].”
BackgroundThe Zero Resource Speech Challenge 2019: TTS without T
Dunbar, E., Algayres, R., Karadayi, J., et al. (2019) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.48550/arxiv.1904.11469
Research works printed but not linked: 3
- bishop, 2006, Continuous latent variables, Pattern Recognition and Machine Learning
- lee, 0, Sparse deep belief net model for visual area V2, Proc Adv Neural Inf Process Syst, 873
- jurafsky, 2009, Speech and Language Processing
Cited by
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3122291 · Cited by 2,983
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3188113 · Cited by 1,926
“The research line of generative learning can be traced back to the auto-encoding model, which reconstructs the whole speech from latent variables, either continuous [25]– [27] or discrete [28].”
Backgroundwav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290
A Unifying Review of Deep and Shallow Anomaly Detection
Ruff, L., Kauffmann, J. R., Vandermeulen, R. A., Montavon, G., Samek, W., Kloft, M., Dietterich, T. G., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3052449 · Cited by 914
Masood, M., Nawaz, M., Malik, K. M., Javed, A., Irtaza, A., & Malik, H. (2023) · Applied Intelligence · doi:10.1007/s10489-022-03766-z · Cited by 543
“A novel conditional autoencoder, namely DF-VAE, is used to create manipulated videos.”
BackgroundInfluential
Metadata
- Authors
- Chorowski, J., Weiss, R. J., Bengio, S., & van den Oord, A.
- Affiliations
- University of Wrocław (PL)
- Google (United States) (US)
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2019-12
- Volume
- 27
- Issue
- 12
- Pages
- 2041-2053
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W3100270690
- Cited by
- 325
- References
- 78
- Citations by year
- 2026: 6 · 2025: 18 · 2024: 23 · 2023: 32 · 2022: 54 · 2021: 101 · 2020: 66 · 2019: 23
- FWCI
- 27.83
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/ACM Transactions on Audio Speech and Language Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Keywords
- latent representations, phoneme prediction, variational autoencoder, vector quantized variational autoencoder, speaker independence, regularization, spectrogram reconstruction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.