Non-Parallel Sequence-to-Sequence Voice Conversion With Disentangled Linguistic and Speaker Representations
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
This article presents a method of sequence-to-sequence (seq2seq) voice conversion using non-parallel training data. In this method, disentangled linguistic and speaker representations are extracted from acoustic features, and voice conversion is achieved by preserving the linguistic representations of source utterances while replacing the speaker representations with the target ones. Our model is built under the framework of encoder-decoder neural networks. A recognition encoder is designed to learn the disentangled linguistic representations with two strategies. First, phoneme transcriptions of training data are introduced to provide the references for leaning linguistic representations of audio signals. Second, an adversarial training strategy is employed to further wipe out speaker information from the linguistic representations. Meanwhile, speaker representations are extracted from audio signals by a speaker encoder. The model parameters are estimated by two-stage training, including a pre-training stage using a multi-speaker dataset and a fine-tuning stage using the dataset of a specific conversion pair. Since both the recognition encoder and the decoder for recovering acoustic features are seq2seq neural networks, there are no constrains of frame alignment and frame-by-frame conversion in our proposed method. Experimental results showed that our method obtained higher similarity and naturalness than the best non-parallel voice conversion method in Voice Conversion Challenge 2018. Besides, the performance of our proposed method was closed to the state-of-the-art parallel seq2seq voice conversion method.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226
Tacotron: towards end-to-end speech synthesis
Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., et al. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1452 · Cited by 1,746 · cited 1×
“Here, the structure of the seq2seq decoder is similar to the Tacotron model [45], [46] for speech synthesis.”
MethodOut of Time: Automated Lip Sync in the Wild
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54427-4_19 · Cited by 689
Transfer learning from speaker verification to multispeaker text-to-speech synthesis
Jia, Y., Zhang, Y., Weiss, R. J., Wang, Q., Shen, J., Ren, F., Chen, Z., Nguyen, P., Pang, R., Moreno, I. L., et al. (2018) · arXiv · doi:10.48550/arxiv.1806.04558 · Cited by 519
Talking Face Generation by Adversarially Disentangled Audio-Visual Representation
Zhou, H., Liu, Y., Liu, Z., Luo, P., & Wang, X. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019299 · Cited by 416
CycleGAN-VC: Non-parallel Voice Conversion Using Cycle-Consistent Adversarial Networks
Kaneko, T., & Kameoka, H. (2018) · 2018 26th European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2018.8553236 · Cited by 358 · cited 1×
“nearest neighbor search step and a conversion step alignment (INCA) [23], [24] and CycleGAN-based VC [25]–[27].”
MethodCycleGAN-VC2: improved CycleGAN-based non-parallel voice conversion.
Kaneko, T., Kameoka, H., Tanaka, K., & Hojo, N. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682897 · Cited by 272
“High-quality nonparallel voice conversion based on cycle-consistent adversarial network,”
Fang, F., Yamagishi, J., Echizen, I., & Lorenzo-Trueba, J. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462342 · Cited by 128
Research works printed but not linked: 48
- 10.1109/ICASSP.2019.8682589 · DOI
- 10.1109/ICASSP.2018.8461384 · DOI · Paper page
- 10.21437/Interspeech.2018-1504 · DOI
- 10.21437/Interspeech.2018-1190 · DOI · Paper page
- miyoshi, 0, Voice conversion using sequence-to-sequence learning of context posterior probabilities, Proc Annu Conf Int Speech Commun Assoc, 1268
- oord, 0, WaveNet: A generative model for raw audio, Proc ISCA Speech Synthesis Workshop, 125
- 10.21437/Interspeech.2018-1830 · DOI · Paper page
- hsu, 0, Voice conversion from unaligned corpora using variational autoencoding Wasserstein generative adversarial networks, Proc Annu Conf Int Speech Commun Assoc, 3364
- 10.1109/APSIPA.2016.7820786 · DOI · Paper page
- maaten, 2008, Visualizing data using t-SNE, J Mach Learn Res, 9, 2579
- 10.1109/TASLP.2016.2593263 · DOI · Paper page
- 10.1109/ICME.2016.7552917 · DOI · Paper page
- 10.1016/0167-6393(89)90041-1 · DOI
- 10.1109/ICASSP.1985.1168479 · DOI
- zhang, 0, Improving sequence-to-sequence acoustic modeling by adding text-supervision, Proc IEEE Int Conf Acoust Speech Signal Process, 6785
- 10.1109/ICASSP.2006.1659962 · DOI · Paper page
- duxans, 0, Voice conversion of non-aligned data using unit selection, Proc TC-STAR Workshop Speech to Speech Translation, 238
- 10.1109/TASL.2009.2038669 · DOI · Paper page
- erro, 0, Frame alignment method for cross-lingual voice conversion, Proc Annu Conf Int Speech Commun Assoc, 1969
- 10.1016/S0167-6393(98)00085-5 · DOI · Paper page
- 10.1109/ICASSP.1998.675407 · doi:10.1109/icassp.1998.675407
- 10.1587/transinf.2015EDP7457 · DOI · Paper page
- wu, 0, Merlin: An open source neural network speech synthesis system, Proc ISCA Speech Synthesis Workshop, 1
- kingma, 0, Adam: A method for stochastic optimization, Proc Int Conf Learn Represent, 1
- 10.1109/ICASSP.2018.8462020 · DOI
- radford, 0, Unsupervised representation learning with deep convolutional generative adversarial networks, Proc Int Conf Learn Representations, 1
- chorowski, 0, Attention-based models for speech recognition, Proc Adv Neural Inf Process Syst, 577
- 10.1109/TASL.2010.2047683 · DOI · Paper page
- 10.1109/TASLP.2014.2353991 · DOI · Paper page
- 10.1109/ICASSP.2019.8683204 · DOI
- 10.1109/ICASSP.2015.7178896 · DOI · Paper page
- 10.1109/TASLP.2014.2379589 · doi:10.1109/taslp.2014.2379589
- sutskever, 0, Sequence to sequence learning with neural networks, Proc Adv Neural Inf Process Syst, 3104
- 10.3115/v1/D14-1179 · DOI · Paper page
- bahdanau, 0, Neural machine translation by jointly learning to align and translate, Proc Int Conf Learn Representations, 1
- 10.1109/TASLP.2019.2892235 · DOI · Paper page
- 10.1109/ICASSP.2019.8683282 · DOI · Paper page
- 10.1016/S0167-6393(99)00015-1 · doi:10.1016/s0167-6393(99)00015-1
- 10.1109/ICASSP.1998.674423 · DOI · Paper page
- 10.1016/j.specom.2017.01.008 · DOI · Paper page
- wu, 2006, Voice conversion using duration-embedded bi-HMMs for expressive speech synthesis, IEEE Trans Audio Speech Lang Process, 14, 1109, 10.1109/TASL.2006.876112 · DOI
- 10.1109/TASL.2007.907344 · DOI · Paper page
- 10.1007/978-3-540-74048-3_4 · DOI · Paper page
- 10.1109/ICASSP.2009.4960478 · DOI · Paper page
- 10.1109/ICASSP.2018.8461368 · DOI · Paper page
- 10.1109/CVPR.2005.202 · DOI · Paper page
- arik, 0, Neural voice cloning with a few samples, Proc Adv Neural Inf Process Syst, 10�040
- nachmani, 0, Fitting new speakers based on a short untranscribed sample, Proc Int Conf Mach Learn, 3683
Cited by
A review of deep learning techniques for speech processing
Mehrish, A., Majumder, N., Bharadwaj, R., Mihalcea, R., & Poria, S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101869 · Cited by 389
An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning
Sisman, B., Yamagishi, J., King, S., & Li, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2020.3038524 · Cited by 357
Khanjani, Z., Watson, G., & Janeja, V. P. (2023) · Frontiers in Big Data · doi:10.3389/fdata.2022.1001063 · Cited by 121
Review of audio deepfake detection techniques: Issues and prospects
Dixit, A., Kaur, N., & Kingra, S. (2023) · Expert Systems · doi:10.1111/exsy.13322 · Cited by 56
SpeechNet: A Universal Modularized Model for Speech Processing Tasks
Chen, Y.-C., Chi, P.-H., Yang, S.-W., Chang, K.-W., Lin, J.-H., Huang, S.-F., Liu, D.-R., Liu, C.-L., Lee, C.-K., & Lee, H.-Y. (2021) · arXiv · doi:10.48550/arxiv.2105.03070 · Cited by 6
“Some papers (Jain et al., 2018; Chen et al., 2020) use SC to improve ASR, and (Tang et al., 2016) studies MTL of ASR and SC. (Hsu et al., 2019; Jia et al., 2018) shows the effectiveness of SC to help TTS. (Zhang et al., 2019b) shows the performance of VC can be improved with text supervision.”
BackgroundMethod
Metadata
- Authors
- Zhang, J.-X., Ling, Z.-H., & Dai, L.-R.
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2020
- Volume
- 28
- Pages
- 540-552
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W2996414377
- Cited by
- 77
- References
- 10
- FWCI
- 4.60
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- National Key R&D Program of China (2019YFF0303001)
- National Natural Science Foundation of China (61871358)
- Key Science and Technology Project of Anhui Province (18030901016)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Non-Parallel Sequence-to-Sequence Voice Conversion With Disentangled Linguistic and Speaker Representations. https://app.cochat.ai/papers/10.1109/taslp.2019.2960721
- The paper
- Zhang, J.-X., Ling, Z.-H., & Dai, L.-R. (2020). Non-Parallel Sequence-to-Sequence Voice Conversion With Disentangled Linguistic and Speaker Representations. IEEE/ACM Transactions on Audio, Speech, and Language Processing. https://doi.org/10.1109/taslp.2019.2960721
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.