CoChat

On Generative Spoken Language Modeling from Raw Audio

Open access
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Adelrahman Mohamed, Emmanuel Dupoux A · 2021 · Transactions of the Association for Computational Linguistics
Cited by 75

The DOI matches its OpenAlex record; 16 of the 16 references it cites were found in a registry or an index.

Abstract

We introduce Generative Spoken Language Modeling, the task of learning the acoustic and linguistic characteristics of a language from raw audio (no text, no labels), and a set of metrics to automatically evaluate the learned representations at acoustic and linguistic levels for both encoding and generation. We set up baseline systems consisting of a discrete speech encoder (returning pseudo-text units), a generative language model (trained on pseudo- text), and a speech decoder (generating a waveform from pseudo-text) all trained without supervision and validate the proposed metrics with human evaluation. Across 3 speech encoders (CPC, wav2vec 2.0, HuBERT), we find that the number of discrete units (50, 100, or 200) matters in a task-dependent and encoder- dependent way, and that some combinations approach text-based systems.1

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

OpenAlex · 10.1162/tacl_a_00430

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 16 references

Research works printed but not linked: 58

  1. Baevski, 2019, Effectiveness of self-supervised pre-training for speech recognition, CoRR
  2. Baevski, 2021, Unsupervised speech recognition, arXiv preprint arXiv:2012.15454 · arXiv
  3. Baevski, 2020, vq-wav2vec: Self-supervised learning of discrete speech representations, International Conference on Learning Representations (ICLR)
  4. Baevski, 2020, wav2vec 2.0: A framework for self-supervised learning of speech representations, 12449
  5. Brown, 2020, Language models are few-shot learners
  6. Chen, 2020, Unsupervised acoustic unit representation learning for voice conversion using WaveNet auto- encoders, 4866, 10.21437/Interspeech.2020-1785 · DOI
  7. Chi, 2021, Audio ALBERT: A lite BERT for self-supervised learning of audio representation, 344, 10.1109/SLT48900.2021.9383575 · DOI · Paper page
  8. Chung, 2020, Improved speech representations with multi-target autoregressive predictive coding, 2353, 10.18653/v1/2020.acl-main.213 · DOI · Paper page
  9. Chung, 2019, An unsupervised autoregressive model for speech representation learning, 146, 10.21437/Interspeech.2019-1473 · DOI · Paper page
  10. Dunbar, 2019, The Zero Resource Speech Challenge 2019: TTS without T, 1088, 10.21437/Interspeech.2019-2904 · DOI · Paper page
  11. Dunbar, 2020, The Zero Resource Speech Challenge 2020: Discovering discrete subword and word units, 4831, 10.21437/Interspeech.2020-2743 · DOI
  12. Dupoux, 2018, Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language- learner, Cognition, 173, 43, 10.1016/j.cognition.2017.11.008 · DOI · Paper page
  13. Ebbers, 2017, Hidden Markov Model variational autoencoder for acoustic unit discovery, 488, 10.21437/Interspeech.2017-1160 · DOI · Paper page
  14. Eloff, 2019, Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks, 1103, 10.21437/Interspeech.2019-1518 · DOI · Paper page
  15. Feng, 2019, Combining adversarial training and disentangled speech representation for robust zero- resource subword modeling, 1093, 10.21437/Interspeech.2019-1337 · DOI
  16. Glarner, 2018, Full Bayesian Hidden Markov Model variational autoencoder for acoustic unit discovery, 2688, 10.21437/Interspeech.2018-2148 · DOI
  17. Hsu, 2020, Text-free image- to-speech synthesis using learned segmental units, arXiv preprint arXiv:2012.15454 · arXiv
  18. Hsu, 2021, HuBERT: How much can a bad teacher benefit ASR pre-training?, 6533, 10.1109/ICASSP39728.2021.9414460 · DOI · Paper page
  19. Hsu, 2017, Learning latent representations for speech generation and transformation, 1273, 10.21437/Interspeech.2017-349 · DOI · Paper page
  20. Hsu, 2017, Unsupervised learning of disentangled and interpretable representations from sequential data, 1878
  21. Jin, 2018, FFTNet: A real-time speaker-dependent neural vocoder, 2251, 10.1109/ICASSP.2018.8462431 · DOI
  22. Kahn, 2020, Libri-light: A benchmark for ASR with limited or no supervision, 7669, 10.1109/ICASSP40776.2020.9052942 · DOI · Paper page
  23. Kharitonov, 2021, Data augmenting contrastive learning of speech representations in the time domain, arXiv preprint arXiv:2007.00991, 215, 10.1109/SLT48900.2021.9383605 · DOI
  24. Khurana, 2019, A factorial deep Markov Model for unsupervised disentangled representation learning from speech, 6540
  25. Khurana, 2020, A convolutional deep Markov model for unsupervised speech representation learning, 3790, 10.21437/Interspeech.2020-3084 · DOI
  26. Kong, 2020, HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis, 17022
  27. Kumar, 2019, MelGAN: Generative adversarial networks for conditional waveform synthesis, 14910
  28. Lai, 2021, Semi- supervised spoken language understanding via self-supervised speech and language model pretraining, 7468, 10.1109/ICASSP39728.2021.9414922 · DOI
  29. Lee, 2012, A nonparametric Bayesian approach to acoustic model discovery, 40
  30. Ling, 2020, DeCoAR 2.0: Deep contextualized acoustic representations with vector quantization, arXiv preprint arXiv:2012.06659 · arXiv
  31. Ling, 2020, Deep contextualized acoustic representations for semi-supervised speech recognition, 6429
  32. Liu, 2020, Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders, 6419, 10.1109/ICASSP40776.2020.9054458 · DOI · Paper page
  33. Liu, 2020, Non-autoregressive predictive coding for learning speech representations from local dependencies, arXiv preprint arXiv:2011.00406 · arXiv
  34. Liu, 2019, Unsupervised end-to-end learning of discrete linguistic units for voice conversion, 1108, 10.21437/Interspeech.2019-2048 · DOI
  35. Morita, 2020, Exploring TTS without T using biologically/psychologically motivated neural network modules (ZeroSpeech 2020), 4856, 10.21437/Interspeech.2020-3127 · DOI
  36. Nayak, 2019, Virtual phone discovery for speech synthesis without text, 10.1109/GlobalSIP45357.2019.8969412 · DOI
  37. Nguyen, 2020, The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling
  38. Niekerk, 2020, Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 Challenge, 4836, 10.21437/Interspeech.2020-1693 · DOI · Paper page
  39. Ondel, 2016, Variational inference for acoustic unit discovery, Procedia Computer Science, 81, 80, 10.1016/j.procs.2016.04.033 · DOI · Paper page
  40. Ott, 2019, Fairseq: A fast, extensible toolkit for sequence modeling, 48
  41. Pascual, 2019, Learning problem-agnostic speech representations from multiple self-supervised tasks, 161, 10.21437/Interspeech.2019-2605 · DOI · Paper page
  42. Polyak, 2021, Speech resynthesis from discrete disentangled self-supervised representations, 10.21437/Interspeech.2021-475 · DOI · Paper page
  43. Polyak, 2021, High fidelity speech regeneration with application to speech enhancement, 7143, 10.1109/ICASSP39728.2021.9414853 · DOI
  44. Polyak, 2020, Unsupervised cross-domain singing voice conversion, 801, 10.21437/Interspeech.2020-1862 · DOI
  45. Polyak, 2020, TTS skins: Speaker conversion via ASR, 786, 10.21437/Interspeech.2020-1416 · DOI · Paper page
  46. Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
  47. Ravanelli, 2020, Multi-task self-supervised learning for robust speech recognition, 6989, 10.1109/ICASSP40776.2020.9053569 · DOI · Paper page
  48. Ribeiro, 2011, CROWDMOS: An approach for crowdsourcing mean opinion score studies, 2416, 10.1109/ICASSP.2011.5946971 · DOI · Paper page
  49. Rivière, 2020, Towards unsupervised learning of speech features in the wild, 156, 10.1109/SLT48900.2021.9383461 · DOI
  50. Schatz, 2021, Early phonetic learning without phonetic categories: Insights from large-scale simulations on realistic input, Proceedings of the National Academy of Sciences, 118, 10.1073/pnas.2001844118 · DOI
  51. Shen, 2018, Natural TTS synthesis by conditioning WaveNet on MEL spectrogram predictions, 4779, 10.1109/ICASSP.2018.8461368 · DOI · Paper page
  52. Tjandra, 2020, Transformer VQ-VAE for unsupervised unit discovery and speech synthesis: ZeroSpeech 2020 Challenge, 4851, 10.21437/Interspeech.2020-3033 · DOI
  53. Tjandra, 2019, VQVAE unsupervised unit discovery and multi-scale Code2Spec inverter for Zerospeech Challenge 2019, 1118, 10.21437/Interspeech.2019-3232 · DOI · Paper page
  54. Tobing, 2020, Cyclic spectral modeling for unsupervised unit discovery into voice conversion with excitation and waveform modeling, 4861, 10.21437/Interspeech.2020-2559 · DOI
  55. Vaswani, 2017, Attention is all you need, 5998
  56. Versteegh, 2016, The Zero Resource Speech Challenge 2015: Proposed approaches and results, Procedia Computer Science, 81, 67, 10.1016/j.procs.2016.04.031 · doi:10.1016/j.procs.2016.04.031
  57. Wang, 2020, Unsupervised pre-training of bidirectional speech encoders via masked reconstruction, 6889, 10.1109/ICASSP40776.2020.9053541 · DOI · Paper page
  58. Anne, 2020, Self-supervised representations improve end-to-end speech translation, 1491, 10.21437/Interspeech.2020-3094 · DOI

Cited by

See all 75

Metadata

Authors
Lakhotia, K., Kharitonov, E., Hsu, W.-N., Adi, Y., Polyak, A., Bolte, B., Nguyen, T.-A., Copet, J., Baevski, A., Mohamed, A., & Emmanuel Dupoux, A.
Venue
Transactions of the Association for Computational Linguistics
Publisher
MIT Press
Published
2021
Volume
9
Pages
1336-1354
ISSN
2307-387X (ISSN-L)
OpenAlex
W3129009457
Cited by
75
References
16
FWCI
4.41
Type
journal article
Language
en
Open access
diamond

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: On Generative Spoken Language Modeling from Raw Audio. https://app.cochat.ai/papers/10.1162/tacl_a_00430
The paper
Lakhotia, K., Kharitonov, E., Hsu, W.-N., Adi, Y., Polyak, A., Bolte, B., Nguyen, T.-A., Copet, J., Baevski, A., Mohamed, A., & Emmanuel Dupoux, A. (2021). On Generative Spoken Language Modeling from Raw Audio. Transactions of the Association for Computational Linguistics. https://doi.org/10.1162/tacl_a_00430

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.