CoChat

Non-Parallel Sequence-to-Sequence Voice Conversion With Disentangled Linguistic and Speaker Representations

Open access
Zhang, Jing-Xuan, Ling, Zhen-Hua, Dai, Li-Rong · 2020 · IEEE/ACM Transactions on Audio, Speech, and Language Processing
Cited by 77

The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.

Abstract

This article presents a method of sequence-to-sequence (seq2seq) voice conversion using non-parallel training data. In this method, disentangled linguistic and speaker representations are extracted from acoustic features, and voice conversion is achieved by preserving the linguistic representations of source utterances while replacing the speaker representations with the target ones. Our model is built under the framework of encoder-decoder neural networks. A recognition encoder is designed to learn the disentangled linguistic representations with two strategies. First, phoneme transcriptions of training data are introduced to provide the references for leaning linguistic representations of audio signals. Second, an adversarial training strategy is employed to further wipe out speaker information from the linguistic representations. Meanwhile, speaker representations are extracted from audio signals by a speaker encoder. The model parameters are estimated by two-stage training, including a pre-training stage using a multi-speaker dataset and a fine-tuning stage using the dataset of a specific conversion pair. Since both the recognition encoder and the decoder for recovering acoustic features are seq2seq neural networks, there are no constrains of frame alignment and frame-by-frame conversion in our proposed method. Experimental results showed that our method obtained higher similarity and naturalness than the best non-parallel voice conversion method in Voice Conversion Challenge 2018. Besides, the performance of our proposed method was closed to the state-of-the-art parallel seq2seq voice conversion method.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/taslp.2019.2960721

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 10 references

Research works printed but not linked: 48

  1. 10.1109/ICASSP.2019.8682589 · DOI
  2. 10.1109/ICASSP.2018.8461384 · DOI · Paper page
  3. 10.21437/Interspeech.2018-1504 · DOI
  4. 10.21437/Interspeech.2018-1190 · DOI · Paper page
  5. miyoshi, 0, Voice conversion using sequence-to-sequence learning of context posterior probabilities, Proc Annu Conf Int Speech Commun Assoc, 1268
  6. oord, 0, WaveNet: A generative model for raw audio, Proc ISCA Speech Synthesis Workshop, 125
  7. 10.21437/Interspeech.2018-1830 · DOI · Paper page
  8. hsu, 0, Voice conversion from unaligned corpora using variational autoencoding Wasserstein generative adversarial networks, Proc Annu Conf Int Speech Commun Assoc, 3364
  9. 10.1109/APSIPA.2016.7820786 · DOI · Paper page
  10. maaten, 2008, Visualizing data using t-SNE, J Mach Learn Res, 9, 2579
  11. 10.1109/TASLP.2016.2593263 · DOI · Paper page
  12. 10.1109/ICME.2016.7552917 · DOI · Paper page
  13. 10.1016/0167-6393(89)90041-1 · DOI
  14. 10.1109/ICASSP.1985.1168479 · DOI
  15. zhang, 0, Improving sequence-to-sequence acoustic modeling by adding text-supervision, Proc IEEE Int Conf Acoust Speech Signal Process, 6785
  16. 10.1109/ICASSP.2006.1659962 · DOI · Paper page
  17. duxans, 0, Voice conversion of non-aligned data using unit selection, Proc TC-STAR Workshop Speech to Speech Translation, 238
  18. 10.1109/TASL.2009.2038669 · DOI · Paper page
  19. erro, 0, Frame alignment method for cross-lingual voice conversion, Proc Annu Conf Int Speech Commun Assoc, 1969
  20. 10.1016/S0167-6393(98)00085-5 · DOI · Paper page
  21. 10.1109/ICASSP.1998.675407 · doi:10.1109/icassp.1998.675407
  22. 10.1587/transinf.2015EDP7457 · DOI · Paper page
  23. wu, 0, Merlin: An open source neural network speech synthesis system, Proc ISCA Speech Synthesis Workshop, 1
  24. kingma, 0, Adam: A method for stochastic optimization, Proc Int Conf Learn Represent, 1
  25. 10.1109/ICASSP.2018.8462020 · DOI
  26. radford, 0, Unsupervised representation learning with deep convolutional generative adversarial networks, Proc Int Conf Learn Representations, 1
  27. chorowski, 0, Attention-based models for speech recognition, Proc Adv Neural Inf Process Syst, 577
  28. 10.1109/TASL.2010.2047683 · DOI · Paper page
  29. 10.1109/TASLP.2014.2353991 · DOI · Paper page
  30. 10.1109/ICASSP.2019.8683204 · DOI
  31. 10.1109/ICASSP.2015.7178896 · DOI · Paper page
  32. 10.1109/TASLP.2014.2379589 · doi:10.1109/taslp.2014.2379589
  33. sutskever, 0, Sequence to sequence learning with neural networks, Proc Adv Neural Inf Process Syst, 3104
  34. 10.3115/v1/D14-1179 · DOI · Paper page
  35. bahdanau, 0, Neural machine translation by jointly learning to align and translate, Proc Int Conf Learn Representations, 1
  36. 10.1109/TASLP.2019.2892235 · DOI · Paper page
  37. 10.1109/ICASSP.2019.8683282 · DOI · Paper page
  38. 10.1016/S0167-6393(99)00015-1 · doi:10.1016/s0167-6393(99)00015-1
  39. 10.1109/ICASSP.1998.674423 · DOI · Paper page
  40. 10.1016/j.specom.2017.01.008 · DOI · Paper page
  41. wu, 2006, Voice conversion using duration-embedded bi-HMMs for expressive speech synthesis, IEEE Trans Audio Speech Lang Process, 14, 1109, 10.1109/TASL.2006.876112 · DOI
  42. 10.1109/TASL.2007.907344 · DOI · Paper page
  43. 10.1007/978-3-540-74048-3_4 · DOI · Paper page
  44. 10.1109/ICASSP.2009.4960478 · DOI · Paper page
  45. 10.1109/ICASSP.2018.8461368 · DOI · Paper page
  46. 10.1109/CVPR.2005.202 · DOI · Paper page
  47. arik, 0, Neural voice cloning with a few samples, Proc Adv Neural Inf Process Syst, 10�040
  48. nachmani, 0, Fitting new speakers based on a short untranscribed sample, Proc Int Conf Mach Learn, 3683

Cited by

See all 77

Metadata

Authors
Zhang, J.-X., Ling, Z.-H., & Dai, L.-R.
Venue
IEEE/ACM Transactions on Audio, Speech, and Language Processing
Published
2020
Volume
28
Pages
540-552
ISSN
  • 2329-9290 (ISSN-L)
  • 2329-9304
OpenAlex
W2996414377
Cited by
77
References
10
FWCI
4.60
Type
journal article
Language
en
Open access
green
Funders
  • National Key R&D Program of China (2019YFF0303001)
  • National Natural Science Foundation of China (61871358)
  • Key Science and Technology Project of Anhui Province (18030901016)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Non-Parallel Sequence-to-Sequence Voice Conversion With Disentangled Linguistic and Speaker Representations. https://app.cochat.ai/papers/10.1109/taslp.2019.2960721
The paper
Zhang, J.-X., Ling, Z.-H., & Dai, L.-R. (2020). Non-Parallel Sequence-to-Sequence Voice Conversion With Disentangled Linguistic and Speaker Representations. IEEE/ACM Transactions on Audio, Speech, and Language Processing. https://doi.org/10.1109/taslp.2019.2960721

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.