CoChat
Sign Up
PreprintCited by 205Speech Recognition and SynthesisSpeech and Audio ProcessingVoice and Speech Disorders

YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone

Open access
Casanova, Edresson, Weber, Julian, Shulby, Christopher, Junior, Arnaldo Candido, Gölge, Eren, Ponti, Moacir Antonelli·2021·arXiv
DOI 10.48550/arxiv.2112.02418arXiv 2112.02418
Open this paper in CoChat — free accountOpen PDF

Abstract

YourTTS brings the power of a multilingual approach to the task of zero-shot multi-speaker TTS. Our method builds upon the VITS model and adds several novel modifications for zero-shot multi-speaker and multilingual training. We achieved state-of-the-art (SOTA) results in zero-shot multi-speaker TTS and results comparable to SOTA in zero-shot voice conversion on the VCTK dataset. Additionally, our approach achieves promising results in a target language with a single-speaker dataset, opening possibilities for zero-shot multi-speaker TTS and zero-shot voice conversion systems in low-resource languages. Finally, it is possible to fine-tune the YourTTS model with less than 1 minute of speech and achieve state-of-the-art results in voice similarity and with reasonable quality. This is important to allow synthesis for speakers with a very different voice or recording characteristics from those seen during training.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2112.02418

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Voice-conversion models require as little as 3 s of recorded speech.

    “Voice-conversion models require as little as 3 s of recorded speech126; however, larger personalized training datasets can be leveraged if the individual has additional pre-injury audio recordings.”

    From The speech neuroprosthesis (2024) · cites this paper as [126]

    Section 5 (Zero-Shot Voice Conversion; page not provided) reports voice-conversion experiments using target-speaker reference recordings longer than 3 seconds. This supports the use of short recordings, but does not establish that exactly 3 seconds is sufficient or that voice-conversion models generally require only that amount. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • In the open condition on the ASVspoof5 evaluation set, attacks A28 (pre-trained YourTTS) and A31 (malacopula+) were the most challenging, with pooled min DCF of 0.470 and 0.269 respectively, and scores greater than 0.5 in combination with five and two codecs respectively.

    “we identified the attacks A28 (pre-trained YourTTS [30]) and A31 (malacopula+[31]) as the most challenging, having a pooled min DCF of 0.470 and 0.269 and having scores greater 0.5 in combination with five and two codecs.”

    From Robust audio deepfake detection: exploring front-/back-end combinations and data augmentation strategies for the ASVspoof5 Challenge (2024) · cites this paper as [30]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 39

  1. [1][1] Y. Jia, Y. Zhang, R. Weiss, Q. Wang, J. Shen, F. Ren, P. Nguyen, R. Pang, I. L. Moreno, Y. Wu et al., "Transfer learning from speaker verification to multispeaker text-to-speech synthesis," in Advances in neural information processing systems, 2018, pp. 4480–4490.
  2. [2][2] E. Cooper, C.-I. Lai, Y. Yasuda, F. Fang, X. Wang, N. Chen, and J. Yamagishi, "Zero-shot multi-speaker text-to-speech with state-of-the-art neural speaker embeddings," in ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6184–6188.
  3. [3][3] S. Choi, S. Han, D. Kim, and S. Ha, "Attentron: Few-shot text- to-speech utilizing attention-based variable-length embedding," arXiv preprint arXiv:2005.08484, 2020. · arXiv
  4. [4][4] E. Casanova, C. Shulby, E. Gölge, N. M. Müller, F. S. de Oliveira, A. Candido Jr., A. da Silva Soares, S. M. Aluisio, and M. A. Ponti, "SC-GlowTTS: An Efficient Zero-Shot Multi-Speaker Text-To- Speech Model," in Proc. Interspeech 2021, 2021, pp. 3645–3649.
  5. [5][5] S. Arik, J. Chen, K. Peng, W. Ping, and Y. Zhou, "Neural voice cloning with a few samples," in Advances in Neural Information Processing Systems, 2018, pp. 10 019–10 029.
  6. [6][6] W. Ping, K. Peng, A. Gibiansky, S. O. Arik, A. Kannan, S. Narang, J. Raiman, and J. Miller, "Deep voice 3: 2000-speaker neural text-to-speech," arXiv preprint arXiv:1710.07654, 2017. · arXiv
  7. [7][7] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan et al., "Natural tts synthesis by conditioning wavenet on mel spectrogram pre- dictions," in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 4779– 4783.
  8. [8][8] L. Wan, Q. Wang, A. Papir, and I. L. Moreno, "Generalized end-to-end loss for speaker verification," in 2018 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 4879–4883.
  9. [9][9] W. Cai, J. Chen, and M. Li, "Exploring the encoding layer and loss function in end-to-end speaker and language recognition system," arXiv preprint arXiv:1804.05160, 2018. · arXiv
  10. [10][10] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudan- pur, "X-vectors: Robust dnn embeddings for speaker recognition," in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5329–5333. 13https://github.com/coqui-ai/TTS http://centrodeia.org https://cyberlabs.ai https://www.defined.ai https://github.com/coqui-ai/TTS
  11. [11][11] N. Kumar, S. Goel, A. Narang, and B. Lall, "Normalization Driven Zero-Shot Multi-Speaker Speech Synthesis," in Proc. In- terspeech 2021, 2021, pp. 1354–1358.
  12. [12][12] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, "wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations," Advances in Neural Information Processing Systems, vol. 33, 2020.
  13. [13][13] X. Tan, T. Qin, F. Soong, and T.-Y. Liu, "A survey on neural speech synthesis," arXiv preprint arXiv:2106.15561, 2021. · arXiv
  14. [15][15] Y. Cao, X. Wu, S. Liu, J. Yu, X. Li, Z. Wu, X. Liu, and H. Meng, "End-to-end code-switched tts with mix of monolingual record- ings," in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 6935–6939.
  15. [16][16] Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Z. Chen, R. Skerry-Ryan, Y. Jia, A. Rosenberg, and B. Ramabhadran, "Learning to speak fluently in a foreign language: Multilingual speech synthesis and cross-language voice cloning," Proc. Interspeech 2019, pp. 2080– 2084, 2019.
  16. [17][17] T. Nekvinda and O. Dušek, "One model, many languages: Meta- learning for multilingual text-to-speech," Proc. Interspeech 2020, pp. 2972–2976, 2020.
  17. [18][18] S. Li, B. Ouyang, L. Li, and Q. Hong, "Light-tts: Lightweight multi-speaker multi-lingual text-to-speech," in ICASSP 2021- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 8383–8387.
  18. [19][19] J. Kim, J. Kong, and J. Son, "Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech," arXiv preprint arXiv:2106.06103, 2021. · arXiv
  19. [20][20] J. Kim, S. Kim, J. Kong, and S. Yoon, "Glow-tts: A generative flow for text-to-speech via monotonic alignment search," arXiv preprint arXiv:2005.11129, 2020. · arXiv
  20. [21][21] L. Dinh, J. Sohl-Dickstein, and S. Bengio, "Density estimation using real NVP," in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. [Online]. Available: https://openreview.net/forum?id=HkpbnH9lx
  21. [22][22] A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, "Wavenet: A generative model for raw audio," arXiv preprint arXiv:1609.03499, 2016. · arXiv
  22. [23][23] J. Kong, J. Kim, and J. Bae, "Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis," arXiv preprint arXiv:2010.05646, 2020. · arXiv
  23. [24][24] D. P. Kingma and M. Welling, "Auto-encoding variational bayes," arXiv preprint arXiv:1312.6114, 2013. · arXiv
  24. [25][25] R. Prenger, R. Valle, and B. Catanzaro, "Waveglow: A flow-based generative network for speech synthesis," in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 3617–3621.
  25. [26][26] D. Xin, Y. Saito, S. Takamichi, T. Koriyama, and H. Saruwatari, "Cross-Lingual Speaker Adaptation Using Domain Adaptation and Speaker Consistency Loss for Text-To-Speech Synthesis," in Proc. Interspeech 2021, 2021, pp. 1614–1618.
  26. [27][27] M. Bińkowski, J. Donahue, S. Dieleman, A. Clark, E. Elsen, N. Casagrande, L. C. Cobo, and K. Simonyan, "High fidelity speech synthesis with adversarial networks," in International Conference on Learning Representations, 2019.
  27. [28][28] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, "Fastspeech 2: Fast and high-quality end-to-end text to speech," in International Conference on Learning Representations, 2021.
  28. [29][29] H. S. Heo, B.-J. Lee, J. Huh, and J. S. Chung, "Clova baseline sys- tem for the voxceleb speaker recognition challenge 2020," arXiv preprint arXiv:2009.14153, 2020. · arXiv
  29. [30][30] J. S. Chung, J. Huh, S. Mun, M. Lee, H. S. Heo, S. Choe, C. Ham, S. Jung, B.-J. Lee, and I. Han, "In defence of metric learning for speaker recognition," in Interspeech, 2020.
  30. [31][31] J. S. Chung, A. Nagrani, and A. Zisserman, "Voxceleb2: Deep speaker recognition," in Proc. Interspeech 2018, 2018, pp. 1086–1090. [Online]. Available: http://dx.doi.org/10.21437/ Interspeech.2018-1929
  31. [32][32] A. Nagrani, J. S. Chung, and A. Zisserman, "Voxceleb: a large-scale speaker identification dataset," arXiv preprint arXiv:1706.08612, 2017. · arXiv
  32. [34][34] H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu, "Libritts: A corpus derived from librispeech for text- to-speech," arXiv preprint arXiv:1904.02882, 2019. · arXiv
  33. [36][36] X. Hao, X. Su, R. Horaud, and X. Li, "Fullsubnet: A full-band and sub-band fusion model for real-time single- channel speech enhancement," ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Jun 2021. [Online]. Available: http: //dx.doi.org/10.1109/ICASSP39728.2021.9414177 · DOI
  34. [38][38] R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Hen- retty, R. Morais, L. Saunders, F. Tyers, and G. Weber, "Com- mon voice: A massively-multilingual speech corpus," in Proceed- ings of the 12th Language Resources and Evaluation Conference, 2020, pp. 4218–4222.
  35. [41][41] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., "Pytorch: An imperative style, high-performance deep learning library," Ad- vances in neural information processing systems, vol. 32, pp. 8026–8037, 2019.
  36. [42][42] F. Ribeiro, D. Florêncio, C. Zhang, and M. Seltzer, "Crowdmos: An approach for crowdsourcing mean opinion score studies," in Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on. IEEE, 2011, pp. 2416–2419.
  37. [43][43] C. Jemine, "Master thesis: Real-time voice cloning," 2019.
  38. [44][44] S. Wang and D. Borth, "Noisevc: Towards high quality zero-shot voice conversion," arXiv preprint arXiv:2104.06074, 2021. · arXiv
  39. [45][45] K. Qian, Y. Zhang, S. Chang, X. Yang, and M. Hasegawa- Johnson, "Autovc: Zero-shot voice style transfer with only au- toencoder loss," in International Conference on Machine Learn- ing. PMLR, 2019, pp. 5210–5219.

Cited by

Sorted by citations

Related

Authors
Casanova, E., Weber, J., Shulby, C., Junior, A. C., Gölge, E., & Ponti, M. A.
Affiliations
  • Universidade de São Paulo (BR)
  • Sopra (France) (FR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2112.02418
arXiv
2112.02418
OpenAlex
W4200631896
Cited by
200
References
0
Citations by year
2026: 2 · 2025: 74 · 2024: 68 · 2023: 45 · 2022: 11
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Health Sciences › Medicine › Physiology › Voice and Speech Disorders
Keywords
zero-shot voice conversion, multilingual speech synthesis, VITS, low-resource languages, voice similarity, fine-tuning, VCTK

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review