Text-Free Prosody-Aware Generative Spoken Language Modeling
Open accessAbstract
Speech pre-training has primarily demonstrated efficacy on classification tasks, while its capability of generating novel speech, similar to how GPT-2 can generate coherent paragraphs, has barely been explored. Generative Spoken Language Modeling (GSLM) \cite{Lakhotia2021} is the only prior work addressing the generative aspects of speech pre-training, which replaces text with discovered phone-like units for language modeling and shows the ability to generate meaningful novel sentences. Unfortunately, despite eliminating the need of text, the units used in GSLM discard most of the prosodic information. Hence, GSLM fails to leverage prosody for better comprehension, and does not generate expressive speech. In this work, we present a prosody-aware generative spoken language model (pGSLM). It is composed of a multi-stream transformer language model (MS-TLM) of speech, represented as discovered unit and prosodic feature streams, and an adapted HiFi-GAN model converting MS-TLM outputs to waveforms. We devise a series of metrics for prosody modeling and generation, and re-use metrics from GSLM for content modeling. Experimental results show that the pGSLM can utilize prosody to improve both prosody and content modeling, and also generate natural, meaningful, and coherent speech given a spoken prompt. Audio samples can be found at https://speechbot.github.io/pgslm. Codes and models are available at https://github.com/pytorch/fairseq/tree/main/examples/textless_nlp/pgslm.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The delayed pattern approach enhances the model's ability to capture and reproduce long-range prosodic dependencies.
“For prosody target prediction, we employ a delayed pattern approach (Kharitonov et al., 2021), which enhances the model's ability to capture and reproduce long-range prosodic dependencies.”
From The Llama 3 Herd of Models (2024) · cites this paper as Kharitonov et al., 2021
Section 3.2.1 (p. 4) introduces delayed prosody prediction, so the citation supports identifying the approach. But the stated purpose of the delay is to let prosody prediction use the phonetic content of the same segment; the paper does not attribute improved capture or reproduction of long-range prosodic dependencies specifically to the delay. Section 2.2 (p. 2) discusses dependencies across time steps for the broader autoregressive model. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · see citing works
Variation across Speech and Writing
Biber, D. (1988) · doi:10.1017/cbo9780511621024 · Cited by 5,227 · see citing works
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · see citing works
Kaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 4,837 · see citing works
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3122291 · Cited by 2,983 · see citing works
Self-citationNeural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729 · see citing works
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369 · see citing works
Self-citationTacotron: towards end-to-end speech synthesis
Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., et al. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1452 · Cited by 1,746 · see citing works
wav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290 · see citing works
Prosody in the Comprehension of Spoken Language: A Literature Review
Cutler, A., Dahan, D., & van Donselaar, W. (1997) · Language and Speech · doi:10.1177/002383099704000203 · Cited by 1,070 · see citing works
SUPERB: Speech Processing Universal PERformance Benchmark
Yang, S.-W., Chi, P.-H., Chuang, Y.-S., Lai, C.-I. J., Lakhotia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T., et al. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1775 · Cited by 681 · see citing works
Self-citationFastSpeech 2: Fast and High-Quality End-to-End Text to Speech
Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T.-Y. (2020) · arXiv · doi:10.48550/arxiv.2006.04558 · Cited by 580 · see citing works
Libri-Light: A Benchmark for ASR with Limited or No Supervision
Kahn, J., Rivière, M., Zheng, W., Kharitonov, E., Xu, Q., Mazaré, P.-E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9052942 · Cited by 475 · see citing works
Self-citationSUPERSEDED - CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit
Veaux, C., Yamagishi, J., & MacDonald, K. (2016) · University of Edinburgh. The Centre for Speech Technology Research (CSTR) · doi:10.7488/ds/1495 · Cited by 434 · see citing works
Prosodic features which cue back-channel responses in English and Japanese
Ward, N., & Tsukahara, W. (2000) · Journal of Pragmatics · doi:10.1016/s0378-2166(99)00109-5 · Cited by 432 · see citing works
Prosody-based automatic segmentation of speech into sentences and topics
Shriberg, E. E., Stolcke, A., Hakkani-Tür, D. Z., & Tür, G. (2000) · Speech Communication · doi:10.1016/s0167-6393(00)00028-5 · Cited by 424 · see citing works
An Unsupervised Autoregressive Model for Speech Representation Learning
Chung, Y.-A., Hsu, W.-N., Tang, H., & Glass, J. R. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1473 · Cited by 356 · see citing works
Self-citationMockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
Liu, A. T., Yang, S.-W., Chi, P.-H., Hsu, P.-C., & Lee, H.-Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054458 · Cited by 335 · see citing works
Can Prosody Aid the Automatic Classification of Dialog Acts in Conversational Speech?
Shriberg, E. E., Stolcke, A., Jurafsky, D. S., Coccaro, N., Meteer, M. W., Bates, R. A., Taylor, P., Ries, K., Martin, R., & Ess-Dykema, C. V. (1998) · Language and Speech · doi:10.1177/002383099804100410 · Cited by 308 · see citing works
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations
Polyak, A., Adi, Y., Copet, J., Kharitonov, E., Lakhotia, K., Hsu, W.-N., Mohamed, A., & Emmanuel Dupoux, A. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-475 · Cited by 228 · see citing works
Self-citationCROWDMOS: An approach for crowdsourcing mean opinion score studies
Ribeiro, F. P., Florencio, D. A. F., Zhang, C., & Seltzer, M. L. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5946971 · Cited by 223 · see citing works
On Generative Spoken Language Modeling from Raw Audio
Lakhotia, K., Kharitonov, E., Hsu, W.-N., Adi, Y., Polyak, A., Bolte, B., Nguyen, T.-A., Copet, J., Baevski, A., Mohamed, A., et al. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00430 · Cited by 212 · see citing works
Self-citationRobust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training
Hsu, W.-N., Sriram, A., Baevski, A., Likhomanenko, T., Xu, Q., Pratap, V., Kahn, J., Lee, A., Collobert, R., Synnaeve, G., et al. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-236 · Cited by 191 · see citing works
Self-citationHubert: How Much Can a Bad Teacher Benefit ASR Pre-Training?
Hsu, W.-N., Tsai, Y.-H. H., Bolte, B., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414460 · Cited by 123 · see citing works
Self-citationData Augmenting Contrastive Learning of Speech Representations in the Time Domain
Kharitonov, E., Rivière, M., Synnaeve, G., Wolf, L., Mazaré, P.-E., Douze, M., & Emmanuel Dupoux, A. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383605 · Cited by 115 · see citing works
Self-citationChu, W., & Alwan, A. A. (2009) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2009.4960497 · Cited by 106 · see citing works
Prosody Modeling for Automatic Speech Recognition and Understanding
Shriberg, E. E., & Stolcke, A. (2004) · The IMA volumes in mathematics and its applications · doi:10.1007/978-1-4419-9017-4_5 · Cited by 82 · see citing works
Nakatani, T., Amano, S., Irino, T., Ishizuka, K., & Kondo, T. (2007) · Speech Communication · doi:10.1016/j.specom.2007.09.003 · Cited by 69 · see citing works
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector QuantizationPreprint
Ling, S., & Liu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.06659 · Cited by 60 · see citing works
The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systems
Adigwe, A., Tits, N., Haddad, K. E., Ostadabbas, S., & Dutoit, T. (2018) · arXiv · doi:10.48550/arxiv.1806.09514 · Cited by 52 · see citing works
Parsing Speech: a Neural Approach to Integrating Lexical and Acoustic-Prosodic Information
Tran, T., Toshniwal, S., Bansal, M., Gimpel, K., Livescu, K., & Mari Ostendorf, E. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1007 · Cited by 46 · see citing works
Unsupervised Cross-Domain Singing Voice Conversion
Polyak, A., Wolf, L., Adi, Y., & Taigman, Y. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1862 · Cited by 45 · see citing works
Self-citationPROSODY MODELS FOR CONVERSATIONAL SPEECH RECOGNITION
Mari Ostendorf, E., Izhak Shafran, J., & Bates, R. A. (2003) · Cited by 34 · see citing works
Prosodic and temporal features for language modeling for dialog
Ward, N., de la Vega, A. I., & Baumann, T. (2011) · Speech Communication · doi:10.1016/j.specom.2011.07.009 · Cited by 26 · see citing works
Towards Unsupervised Learning of Speech Features in the Wild
Rivière, M., & Emmanuel Dupoux, A. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383461 · Cited by 20 · see citing works
Self-citationUnsupervised Learning of Disentangled Speech Content and Style Representation
Tjandra, A., Pang, R., Zhang, Y., & Karita, S. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1936 · Cited by 13 · see citing works
Using Prosodic Features in Language Models for Meetings
Huang, S., & Renals, S. J. (2008) · Springer eBooks · doi:10.1007/978-3-540-78155-4_17 · Cited by 11 · see citing works
Exploiting prosodic breaks in language modeling with random forests
Su, Y., & Jelinek, F. (2008) · Speech prosody · doi:10.21437/speechprosody.2008-21 · Cited by 6 · see citing works
Predicting glottal closure insufficiency using fundamental frequency contour analysis
Cohen, J. T., Cohen, A., Benyamini, L., Adi, Y., & Keshet, J. (2019) · Head & Neck · doi:10.1002/hed.25709 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 4
- Adrian Ła´ ncucki. 2021. Fastpitch: Parallel text-tospeech with pitch prediction. In ICASSP 20212021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 6588-6592. IEEE.
- M. Paul Lewis, Gary F. Simon, and Charles D. Fennig. 2016. Ethnologue: Languages of the World, Nineteenth edition . SIL International. Online version: http://www.ethnologue.com.
- Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. Speech resynthesis from discrete disentangled selfsupervised representations. In Proc. INTERSPEECH .
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.
Cited by
Sorted by citations
Self-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372 · see citing works
“One workaround is two-stage training [77], [332];”
BackgroundInfluentialSelf-citationSpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
Ao, J., Wang, R., Zhou, L., Wang, C., Ren, S., Wu, Y., Liu, S., Ko, T., Li, Q., Zhang, Y., et al. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.393 · Cited by 148 · see citing works
Textless Speech-to-Speech Translation on Real Data
Lee, A., Gong, H., Duquenne, P.-A., Schwenk, H., Chen, P., Wang, C., Popuri, S., Adi, Y., Pino, J., Gu, J., et al. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.63 · Cited by 120 · see citing works
Self-citationTextless Speech Emotion Conversion using Discrete & Decomposed Representations
Kreuk, F., Polyak, A., Copet, J., Kharitonov, E., Nguyen, T. A., Rivière, M., Hsu, W.-N., Mohamed, A., Emmanuel Dupoux, A., & Adi, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.769 · Cited by 31 · see citing works
Self-citationtextless-lib: a Library for Textless Spoken Language Processing
Kharitonov, E., Copet, J., Lakhotia, K., Nguyen, T. A., Tomasello, P., Lee, A., Elkahky, A. M., Hsu, W.-N., Mohamed, A., Emmanuel Dupoux, A., et al. (2022) · doi:10.18653/v1/2022.naacl-demo.1 · Cited by 28 · see citing works
Self-citationSpeechlmscore: Evaluating Speech Generation Using Speech Language Model
Maiti, S., Peng, Y., Saeki, T., & Watanabe, S. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095710 · Cited by 25 · see citing works
DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answering
Lin, G.-T., Chuang, Y.-S., Chung, H.-L., Yang, S., Chen, H.-J., Dong, S. A., Li, S.-W. D., Mohamed, A., Lee, H.-Y., & Lee, L.-S. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-612 · Cited by 20 · see citing works
Self-citationCan statistical learning bootstrap early language acquisition? A modeling investigationPreprint
Lavechin, M., de Seyssel, M., Titeux, H., Bredin, H., Wisniewski, G., Cristià, A., & Emmanuel Dupoux, A. (2022) · doi:10.31234/osf.io/rx94d · Cited by 13 · see citing works
BabySLM: language-acquisition-friendly benchmark of self-supervised spoken language models
Lavechin, M., Sy, Y., Titeux, H., Blandón, M. A. C., Räsänen, O. J., Bredin, H., Emmanuel Dupoux, A., & Cristià, A. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-978 · Cited by 12 · see citing works
Toward Joint Language Modeling for Speech Units and Text
Chou, J.-C., Chien, C.-M., Hsu, W.-N., Livescu, K., Babu, A., Conneau, A., Baevski, A., & Auli, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.438 · Cited by 11 · see citing works
Self-citationProsAudit, a prosodic benchmark for self-supervised speech models
de Seyssel, M., Lavechin, M., Titeux, H., Thomas, A., Virlet, G., Revilla, A. S., Wisniewski, G., Ludusan, B., & Emmanuel Dupoux, A. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-438 · Cited by 8 · see citing works
Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling
Gat, I., Kreuk, F., Nguyen, T. A., Lee, A., Copet, J., Synnaeve, G., Emmanuel Dupoux, A., & Adi, Y. (2023) · International Conference on Spoken Language Translation (IWSLT) · doi:10.18653/v1/2023.iwslt-1.46 · Cited by 9 · see citing works
Self-citationUnraveling the thread: understanding and addressing sequential failures in human-robot interaction
Tisserand, L., Stephenson, B., Baldauf-Quilliatre, H., Lefort, M., & Armetta, F. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1359782 · Cited by 7 · see citing works
Generative Spoken Language Model based on continuous word-sized audio tokens
Algayres, R., Adi, Y., Nguyen, T. A., Copet, J., Synnaeve, G., Sagot, B., & Emmanuel Dupoux, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.182 · Cited by 3 · see citing works
Self-citationRWEN-TTS: Relation-Aware Word Encoding Network for Natural Text-to-Speech Synthesis
Oh, S., Noh, H., Hong, Y., & Oh, I. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26575 · Cited by 1 · see citing works
Fong, J., Lyth, D., Henter, G. E., Tang, H., & King, S. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10138 · Cited by 1 · see citing works
Artificial neural networks to analyze and simulate language acquisition in childrenPreprint
Lavechin, M. (2023) · doi:10.31234/osf.io/5p8ge
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · see citing works
Self-citation
- Authors
- Kharitonov, E., Lee, A., Polyak, A., Adi, Y., Copet, J., Lakhotia, K., Nguyen, T.-A., Rivière, M., Mohamed, A., Dupoux, E., & Hsu, W.-N.
- Affiliations
- Meta (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2109.03264
- OpenAlex
- W4286984129
- Cited by
- 16
- References
- 42
- Citations by year
- 2024: 1 · 2023: 8 · 2022: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- HiFi-GAN, speech synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
