CoChat
Sign Up
PreprintCited by 336Speech Recognition and SynthesisSpeech and Audio ProcessingGenerative Adversarial Networks and Image Synthesis

AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss

Open access
Qian, Kaizhi, Zhang, Yang, Chang, Shiyu, Yang, Xuesong, Hasegawa-Johnson, Mark·2019·arXiv
DOI 10.48550/arxiv.1905.05879arXiv 1905.05879
Open this paper in CoChat — free accountOpen PDF

Abstract

Non-parallel many-to-many voice conversion, as well as zero-shot voice conversion, remain under-explored areas. Deep style transfer algorithms, such as generative adversarial networks (GAN) and conditional variational autoencoder (CVAE), are being applied as new solutions in this field. However, GAN training is sophisticated and difficult, and there is no strong evidence that its generated speech is of good perceptual quality. On the other hand, CVAE training is simple but does not come with the distribution-matching property of a GAN. In this paper, we propose a new style transfer scheme that involves only an autoencoder with a carefully designed bottleneck. We formally show that this scheme can achieve distribution-matching style transfer by training only on a self-reconstruction loss. Based on this scheme, we proposed AUTOVC, which achieves state-of-the-art results in many-to-many voice conversion with non-parallel data, and which is the first to perform zero-shot voice conversion.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1905.05879

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent research in VC technologies has addressed long-standing challenges such as data scarcity, cross-lingual conversion, and real-time processing.

    “Recent research [59,60,61,62,63] in VC technologies has addressed long-standing challenges such as data scarcity, cross-lingual conversion, and real-time processing.”

    From Audio Deepfake Detection: What Has Been Achieved and What Lies Ahead (2025) · cites this paper as [60]

    The Introduction describes AUTOVC as using non-parallel data, and §5.3 reports zero-shot conversion with only 20 seconds of reference speech from an unseen target speaker. These findings support the data-scarcity part of the sentence. The paper does not report cross-lingual conversion or real-time processing; its mention of flexibility for low-resourced languages is not a cross-lingual result. No page numbers are available in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 27

  1. Arik, S. O., Chrzanowski, M., Coates, A., Diamos, G., Gib- iansky, A., Kang, Y., Li, X., Miller, J., Ng, A., Raiman, J., et al. Deep voice: Real-time neural text-to-speech. arXiv preprint arXiv:1702.07825, 2017. · arXiv
  2. Atalla, C., Tam, B., Song, A., and Cottrell, G. Look ma, no GANs! image transformation with modifAE, 2019. URL https://openreview.net/forum? id=B1ethsR9Ym.
  3. Biadsy, F., Weiss, R. J., Moreno, P. J., Kanvesky, D., and Jia, Y. Parrotron: An end-to-end speech-to-speech conversion model and its applications to hearing-impaired speech and speech separation. arXiv preprint arXiv:1904.04169, 2019. · arXiv
  4. Chou, J.-c., Yeh, C.-c., Lee, H.-y., and Lee, L.-s. Multi- target voice conversion without parallel data by adversar- ially learning disentangled audio representations. arXiv preprint arXiv:1804.02812, 2018. · arXiv
  5. Donahue, C., McAuley, J., and Puckette, M. Adversarial audio synthesis. arXiv preprint arXiv:1802.04208, 2018. · arXiv
  6. Fan, Z.-C., Lai, Y.-L., and Jang, J.-S. R. SVSGAN: Singing voice separation via generative adversarial net- work. In 2018 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pp. 726- 730. IEEE, 2018.
  7. Fang, F., Yamagishi, J., Echizen, I., and Lorenzo-Trueba, J. High-quality nonparallel voice conversion based on cycle-consistent adversarial network. arXiv preprint arXiv:1804.00425, 2018. · arXiv
  8. Gao, Y., Singh, R., and Raj, B. Voice impersonation using generative adversarial networks. arXiv preprint arXiv:1802.06840, 2018. · arXiv
  9. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672-2680, 2014.
  10. Heigold, G., Moreno, I., Bengio, S., and Shazeer, N. End- to-end text-dependent speaker verification. In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE In- ternational Conference on, pp. 5115-5119. IEEE, 2016.
  11. Hosseini-Asl, E., Zhou, Y., Xiong, C., and Socher, R. A multi-discriminator cyclegan for unsupervised non- parallel speech domain adaptation. arXiv preprint arXiv:1804.00522, 2018. · arXiv
  12. Hsu, C.-C., Hwang, H.-T., Wu, Y.-C., Tsao, Y., and Wang, H.-M. Voice conversion from non-parallel corpora us- ing variational auto-encoder. In Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2016 Asia-Pacific, pp. 1-6. IEEE, 2016.
  13. Hsu, C.-C., Hwang, H.-T., Wu, Y.-C., Tsao, Y., and Wang, H.-M. Voice conversion from unaligned corpora using variational autoencoding Wasserstein generative adversar- ial networks. arXiv preprint arXiv:1704.00849, 2017. · arXiv
  14. Huang, W.-C., Hwang, H.-T., Peng, Y.-H., Tsao, Y., and Wang, H.-M. Voice conversion based on cross-domain features using variational auto encoders. arXiv preprint arXiv:1808.09634, 2018. · arXiv
  15. Jia, Y., Zhang, Y., Weiss, R. J., Wang, Q., Shen, J., Ren, F., Chen, Z., Nguyen, P., Pang, R., Moreno, I. L., et al. Trans- fer learning from speaker verification to multispeaker text- to-speech synthesis. arXiv preprint arXiv:1806.04558, 2018. · arXiv
  16. Kameoka, H., Kaneko, T., Tanaka, K., and Hojo, N. Acvae- vc: Non-parallel many-to-many voice conversion with auxiliary classifier variational autoencoder. arXiv preprint arXiv:1808.05092, 2018a. · arXiv
  17. Kameoka, H., Kaneko, T., Tanaka, K., and Hojo, N. Stargan- vc: Non-parallel many-to-many voice conversion with star generative adversarial networks. arXiv preprint arXiv:1806.02169, 2018b. · arXiv
  18. Kaneko, T. and Kameoka, H. Parallel-data-free voice conver- sion using cycle-consistent adversarial networks. arXiv preprint arXiv:1711.11293, 2017. · arXiv
  19. Kingma, D. P., Mohamed, S., Rezende, D. J., and Welling, M. Semi-supervised learning with deep generative mod- els. In Advances in neural information processing sys- tems, pp. 3581-3589, 2014.
  20. Nachmani, E. and Wolf, L. Unsupervised singing voice conversion. arXiv preprint arXiv:1904.06590, 2019. · arXiv
  21. Pascual, S., Bonafonte, A., and Serra, J. Segan: Speech en- hancement generative adversarial network. arXiv preprint arXiv:1703.09452, 2017. · arXiv
  22. Saito, Y., Ijima, Y., Nishida, K., and Takamichi, S. Non- parallel voice conversion using variational autoencoders conditioned by phonetic posteriorgrams and d-vectors. In Proc. ICASSP, pp. 5274-5278, 2018.
  23. Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al. Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4779-4783. IEEE, 2018.
  24. Subakan, Y. C. and Smaragdis, P. Generative adversar- ial source separation. In 2018 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pp. 26-30. IEEE, 2018.
  25. Wan, L., Wang, Q., Papir, A., and Moreno, I. L. Generalized end-to-end loss for speaker verification. In 2018 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4879-4883. IEEE, 2018.
  26. Wester, M., Wu, Z., and Yamagishi, J. Analysis of the voice conversion challenge 2016 evaluation results. In INTERSPEECH, pp. 1637-1641, 2016.
  27. Xie, F.-L., Soong, F. K., and Li, H. A KL divergence and DNN-based approach to voice conversion without parallel training sentences. In INTERSPEECH, pp. 287- 291, 2016.

Cited by

Sorted by citations

Authors
Qian, K., Zhang, Y., Chang, S., Yang, X., & Hasegawa-Johnson, M.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • IBM (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
Pages
5210-5219
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1905.05879
arXiv
1905.05879
OpenAlex
W2945478979
Cited by
336
References
5
Citations by year
2026: 1 · 2025: 27 · 2024: 53 · 2023: 76 · 2022: 69 · 2021: 67 · 2020: 40 · 2019: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, public-domain)
  • arXiv (Cornell University)
  • International Conference on Machine Learning
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
zero-shot voice conversion, non-parallel data, autoencoder, distribution matching

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review