CoChat
Sign Up
Cited by 11Speech Recognition and SynthesisSpeech and Audio ProcessingVoice and Speech Disorders

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speech

Open access
Lux, Florian, Koch, Julia, Vu, Ngoc Thang·2023·2022 IEEE Spoken Language Technology Workshop (SLT)
DOI 10.1109/slt54892.2023.10022433
Open this paper in CoChat — free accountOpen PDF

Abstract

The cloning of a speaker's voice using an untranscribed reference sample is one of the great advances of modern neural text-to-speech (TTS) methods. Approaches for mimicking the prosody of a transcribed reference audio have also been proposed recently. In this work, we bring these two tasks together for the first time through utterance level normalization in conjunction with an utterance level speaker embedding. We further introduce a lightweight aligner for extracting fine-grained prosodic features, that can be finetuned on individual samples within seconds. We show that it is possible to clone the voice of a speaker as well as the prosody of a spoken reference independently without any degradation in quality and high similarity to both original voice and prosody, as our objective evaluation and human study show. All of our code and trained models are available, alongside static and interactive demos.

Overview

Integrity signals

Registry

Crossref · 10.1109/slt54892.2023.10022433

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 5 references

Research works printed but not linked: 46

  1. Jia, 2018, Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis, NeurIPS, 31
  2. Arik, 2018, Neural voice cloning with a few samples, NeurIPS, 31
  3. Wang, 2018, Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis, 5180
  4. Ŀańricucki, 2021, FastPitch: Parallel text-to-speech with pitch prediction, 6588
  5. 10.1109/ICASSP40776.2020.9054535 · DOI · Paper page
  6. 10.21437/Odyssey.2018-15 · DOI
  7. 10.21437/Interspeech.2021-1774 · DOI
  8. 10.21437/SSW.2019-49 · DOI
  9. 10.21437/Interspeech.2020-2861 · DOI
  10. Skerry-Ryan, 2018, Towards end-to-end prosody transfer for expressive speech synthesis with tacotron, 4693
  11. 10.21437/Interspeech.2019-2571 · DOI
  12. 10.21437/Interspeech.2021-1583 · DOI
  13. 10.21437/Interspeech.2021-1610 · DOI
  14. Ren, 2019, FastSpeech: fast, robust and controllable text to speech, NeurIPS
  15. Ren, 2020, FastSpeech 2: Fast and High-Quality End to End Text to Speech, ICLR
  16. 10.21437/blizzard.2023-4 · DOI
  17. 10.1109/ICASSP40776.2020.9053512 · DOI
  18. Hayashi, 2021, ESPnet2-TTS: Extending the Edge of TTS Research, arXiv preprint
  19. 10.21437/Interspeech.2022-10703 · DOI
  20. 10.21437/Interspeech.2020-2650 · DOI · Paper page
  21. 10.1016/j.csl.2019.101027 · DOI
  22. Ravanelli, 2021, SpeechBrain: A General-Purpose Speech Toolkit
  23. Kong, 2020, HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis, NeurIPS, 33
  24. Boersma, 2001, Praat, a system for doing phonetics by computer, Glot. Int., 5, 341
  25. 10.1145/1143844.1143891 · DOI · Paper page
  26. Martos, VRAIN-UPV MLLPs system for the Blizzard Challenge 2021, 2021
  27. Kim, 2020, Glow-tts: A generative flow for text-to-speech via monotonic alignment search, Advances in Neural Information Processing Systems, 33, 8067
  28. 10.18653/v1/2022.acl-long.472 · DOI
  29. King, The Blizzard Chal-lenge 2011, 2011
  30. Ito, 2017, The LJ Speech Dataset
  31. 10.21437/interspeech.2019-2441 · DOI
  32. 10.21437/Interspeech.2021-1599 · DOI
  33. Veaux, 2017, Superseded-CSTR VCTK corpus: English multi-speaker corpus for CSTR voice cloning toolkit
  34. 10.1007/978-3-030-87626-5_15 · DOI
  35. Muller, 2021, Thorsten-Open German Voice (Neutral) Dataset
  36. Ling, The Blizzard Challenge 2021, 10.21437/Blizzard.2021-1 · DOI
  37. 10.21437/interspeech.2019-1500 · DOI
  38. 10.1109/ASRU46091.2019.9003750 · DOI
  39. Denisov, 2019, IMS-speech: A speech to text tool, Studientexte zur Sprachkommunikation: Elektronische Sprachsignalverarbeitung 2019, 170
  40. 10.1109/PACRIM.1993.407206 · DOI · Paper page
  41. Berndt, Using dynamic time warping to find patterns in time series, 359
  42. 10.1016/j.specom.2007.09.003 · DOI
  43. 10.21437/Odyssey.2020-58 · DOI
  44. 10.1109/TASL.2010.2064307 · DOI · Paper page
  45. Van der Maaten, 2008, Visual-izing data using t-SNE, Journal of machine learning research, 9
  46. 10.1016/j.csl.2022.101362 · DOI

Cited by

Sorted by citations

Metadata

Authors
Lux, F., Koch, J., & Vu, N. T.
Affiliations
University of Stuttgart (DE)
ORCID
Venue
2022 IEEE Spoken Language Technology Workshop (SLT)
Publisher
IEEE
Published
2023-01-09
Pages
962-969
DOI
10.1109/slt54892.2023.10022433
OpenAlex
W4319862723
Cited by
11
References
5
Citations by year
2026: 2 · 2025: 3 · 2024: 3 · 2023: 3
FWCI
1.95
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Spoken Language Technology Workshop (SLT) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
Keywords
fine-grained alignment, voice cloning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review