The DOI matches its Crossref record; 250 of the 250 references it cites were found in a registry or an index.
An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning
Open accessAbstract
Speaker identity is one of the important characteristics of human speech. In voice conversion, we change the speaker identity from one to another, while keeping the linguistic content unchanged. Voice conversion involves multiple speech processing techniques, such as speech analysis, spectral conversion, prosody conversion, speaker characterization, and vocoding. With the recent advances in theory and practice, we are now able to produce human-like voice quality with high speaker similarity. In this article, we provide a comprehensive overview of the state-of-the-art of voice conversion techniques and their performance evaluation methods from the statistical approaches to deep learning, and discuss their promise and limitations. We will also report the recent Voice Conversion Challenges (VCC), the performance of the current state of technology, and provide a summary of the available resources for voice conversion research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–250 of 250
Adaptive Wavenet Vocoder for Residual Compensation in GAN-Based Voice Conversion
Sisman, B., Zhang, M., Sakti, S., Li, H., & Nakamura, S. (2018) · 2018 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639507 · Cited by 36
Self-citationA Comparison of Recent Neural Vocoders for Speech Signal Reconstruction
Govalkar, P., Fischer, J., Zalkow, F., & Dittmar, C. (2019) · ISCA Speech Synthesis Workshop (SSW) · doi:10.21437/ssw.2019-2 · Cited by 36
On the Study of Generative Adversarial Networks for Cross-Lingual Voice Conversion
Şişman, B., Zhang, M., Dong, M., & Li, H. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003939 · Cited by 35
Self-citationEvent selection from phone posteriorgrams using matched filters
Kintzley, K., Jansen, A., & Heřmanský, H. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-354 · Cited by 34
Text-independent voice conversion based on state mapped codebook
Zhang, M., Tao, J., Tian, J., & Wang, X. (2008) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2008.4518682 · Cited by 33
Ak, K. E., Lim, J., Tham, J. Y., & Kassim, A. A. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00379 · Cited by 33
Wavelet Analysis of Speaker Dependent and Independent Prosody for Voice Conversion
Şişman, B., & Li, H. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1499 · Cited by 31
Self-citationWang, X., & Yamagishi, J. (2019) · ISCA Speech Synthesis Workshop (SSW) · doi:10.21437/ssw.2019-1 · Cited by 31
Self-citationLocally Linear Embedding for Exemplar-Based Spectral Conversion
Wu, Y.-C., Hwang, H.-T., Hsu, C.-C., Tsao, Y., & Wang, H. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-567 · Cited by 30
Williams, J. A., Rownicka, J., Gallegos, P. O., & King, S. (2020) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2020-32 · Cited by 30
Self-citationCute: A concatenative method for voice conversion using exemplar-based unit selection
Jin, Z., Finkelstein, A., DiVerdi, S., Lu, J., & Mysore, G. J. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472761 · Cited by 29
Yeh, C.-C., Hsu, P.-C., Chou, J.-C., Lee, H.-Y., & Lee, L.-S. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639647 · Cited by 29
Transformation of prosody in voice conversion
Şişman, B., Li, H., & Tan, K. C. (2017) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipa.2017.8282288 · Cited by 28
Self-citationTransferring Source Style in Non-Parallel Voice Conversion
Liu, S., Cao, Y., Kang, S., Hu, N., Liu, X., Su, D., Yu, D., & Meng, H. M. L. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2412 · Cited by 28
DeepConversion: Voice conversion with limited parallel training data
Zhang, M., Şişman, B., Zhao, L., & Li, H. (2020) · Speech Communication · doi:10.1016/j.specom.2020.05.004 · Cited by 27
Self-citationWavelet Speech Enhancement Based on Nonnegative Matrix Factorization
Wang, S., Chern, A., Tsao, Y., Hung, J.-W., Lu, X., Lai, Y.-H., & Su, B. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2571727 · Cited by 26
On the use of I-vectors and average voice model for voice conversion without parallel data
Wu, J., Wu, Z., & Xie, L. (2016) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipa.2016.7820901 · Cited by 26
Cross-domain speech recognition using nonparallel corpora with cycle-consistent adversarial networks
Mimura, M., Sakai, S., & Kawahara, T. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8268927 · Cited by 26
Okamoto, T., Toda, T., Shiga, Y., & Kawai, H. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1288 · Cited by 26
Modeling Prosodic Phrasing With Multi-Task Learning in Tacotron-Based TTS
Liu, R., Şişman, B., Bao, F., Gao, G., & Li, H. (2020) · IEEE Signal Processing Letters · doi:10.1109/lsp.2020.3016564 · Cited by 26
Self-citationArticulatory features for expressive speech synthesis
Black, A. W., Bunnell, H. T., Dou, Y., Muthukumar, P. K., Metze, F., Perry, D. J., Polzehl, T., Prahallad, K. S., Steidl, S., & Vaughn, C. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288796 · Cited by 25
High quality voice conversion using prosodic and high-resolution spectral features
Nguyen, H. Q., Lee, S. W., Tian, X., Dong, M., & Chng, E. S. (2015) · Multimedia Tools and Applications · doi:10.1007/s11042-015-3039-x · Cited by 25
Bootstrapping Non-Parallel Voice Conversion from Speaker-Adaptive Text-to-Speech
Luong, H.-T., & Yamagishi, J. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9004008 · Cited by 25
Self-citationMixture of Factor Analyzers Using Priors From Non-Parallel Speech for Voice Conversion
Wu, Z., Kinnunen, T., Chng, E. S., & Li, H. (2012) · IEEE Signal Processing Letters · doi:10.1109/lsp.2012.2225615 · Cited by 24
Self-citationHierarchical modeling of F0 contours for voice conversion
Sánchez, G., Silén, H., Nurminen, J., & Gabbouj, M. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-189 · Cited by 24
LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech
Zen, H., Dang, V., Clark, R., Zhang, Y., Weiss, R. J., Jia, Y., Chen, Z., & Wu, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.02882 · Cited by 23
Voice Conversion with Cyclic Recurrent Neural Network and Fine-tuned Wavenet Vocoder
Tobing, P. L., Wu, Y.-C., Hayashi, T., Kobayashi, K., & Toda, T. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682156 · Cited by 22
Parametric Resynthesis With Neural Vocoders
Maiti, S., & Mandel, M. (2019) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa.2019.8937165 · Cited by 21
The NU-NAIST Voice Conversion System for the Voice Conversion Challenge 2016
Kobayashi, K., Takamichi, S., Nakamura, S., & Toda, T. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-970 · Cited by 20
NU Voice Conversion System for the Voice Conversion Challenge 2018
Tobing, P. L., Wu, Y.-C., Hayashi, T., Kobayashi, K., & Toda, T. (2018) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2018-31 · Cited by 19
Voice conversion based on feature combination with limited training data
Ghorbandoost, M., Sayadiyan, A., Ahangar, M., Sheikhzadeh, H., Shahrebabaki, A. S., & Amini, J. (2014) · Speech Communication · doi:10.1016/j.specom.2014.12.004 · Cited by 18
Wu, Y.-C., Hayashi, T., Tobing, P. L., Kobayashi, K., & Toda, T. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1232 · Cited by 18
Semi-supervised noise dictionary adaptation for exemplar-based noise robust speech recognition
Luan, Y., Saito, D., Kashiwagi, Y., Minematsu, N., & Hirose, K. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6853897 · Cited by 15
Voice conversion for non-parallel datasets using dynamic kernel partial least squares regression
Silén, H., Nurminen, J., Helander, E. E., & Gabbouj, M. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-103 · Cited by 15
Error Reduction Network for DBLSTM-based Voice Conversion
Zhang, M., Şişman, B., Rallabandi, S. S., Li, H., & Li, Z. (2018) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipa.2018.8659543 · Cited by 15
Self-citationA Compact Framework for Voice Conversion Using Wavenet Conditioned on Phonetic Posteriorgrams
Lu, H., Wu, Z., Li, R., Kang, S., Jia, J., & Meng, H. M. L. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682938 · Cited by 15
Transforming Spectrum and Prosody for Emotional Voice Conversion with Non-Parallel Training DataPreprint
Zhou, K., Şişman, B., & Li, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.00198 · Cited by 15
Self-citationProbabilistic feature mapping based on trajectory HMMs
Zen, H., Nankaku, Y., & Tokuda, K. (2008) · Interspeech (USB) · doi:10.21437/interspeech.2008-330 · Cited by 13
Voice Conversion Using Conditional CycleGAN
Yook, D., Yoo, I.-C., & Yoo, S. (2018) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci46756.2018.00290 · Cited by 11
Multi-Target Emotional Voice Conversion With Neural VocodersPreprint
Liu, S., Cao, Y., & Meng, H. M. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03782 · Cited by 10
Speech enhancement using non negative matrix factorization and enhanced NMF
Akarsh, K. A., & Selvi, R. S. (2015) · International Conference on Circuits, Power and Computing Technologies (ICCPCT) · doi:10.1109/iccpct.2015.7159386 · Cited by 8
Parallel Dictionary Learning for Multimodal Voice Conversion Using Matrix Factorization
Aihara, R., Masaka, K., Takiguchi, T., & Ariki, Y. (2016) · Studies in computational intelligence · doi:10.1007/978-3-319-40171-3_3 · Cited by 8
On the analysis and evaluation of prosody conversion techniques
Şişman, B., Lee, G., Li, H., & Tan, K. C. (2017) · International Conference on Asian Language Processing · doi:10.1109/ialp.2017.8300542 · Cited by 8
Self-citationWang, X., & Yamagishi, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.10256 · Cited by 7
Self-citationMultidimensional scaling of systems in the Voice Conversion Challenge 2016
Wester, M., Wu, Z., & Yamagishi, J. (2016) · ISCA Speech Synthesis Workshop (SSW) · doi:10.21437/ssw.2016-7 · Cited by 6
Self-citationLi, Y., Lee, K. A., Yuan, Y., Li, H., & Yang, Z. (2018) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipa.2018.8659628 · Cited by 6
Self-citationWaveNet Factorization with Singular Value Decomposition for Voice Conversion
Du, H., Tian, X., Xie, L., & Li, H. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003801 · Cited by 5
Self-citationDictionary update for NMF-based voice conversion using an encoder-decoder network
Hsu, C.-C., Hwang, H.-T., Wu, Y.-C., Tsao, Y., & Wang, H. (2016) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp.2016.7918382 · Cited by 4
Statistical Voice Conversion with Quasi-periodic WaveNet Vocoder
Wu, Y.-C., Tobing, P. L., Hayashi, T., Kobayashi, K., & Toda, T. (2019) · ISCA Speech Synthesis Workshop (SSW) · doi:10.21437/ssw.2019-12 · Cited by 3
Research works printed but not linked: 66
- 10.1109/ICASSP.2018.8461960 · doi:10.1109/icassp.2018.8461960
- recommendation, 2001, 1534-1, Method for the subjective assessment of intermediate sound quality (MUSHRA), Int Telecommun Union
- benisty, 2011, Voice conversion using GMM with enhanced global variance, Proc 12th Annu Conf Int Speech Commun Assoc
- zielinski, 2007, Potential biases in MUSHRA listening tests, Proc 123rd Audio Eng Soc Convers
- 10.1016/j.asoc.2012.05.027 · doi:10.1016/j.asoc.2012.05.027
- song, 0, Text-independent voice conversion using speaker model alignment method from non-parallel speech, Proc Annu Conf Int Speech Commun Assoc INTERSPEECH, 2308
- tamura, 0, Speaker adaptation for HMM-based speech synthesis system using MLLR, Proc 3rd ESCA/COCOSDA Workshop (ETRW) Speech Synth
- remes, 2013, Objective evaluation measures for speaker-adaptive HMM-TTS systems, Proc 8th ISCA Workshop Speech Synth
- ping, 0, Deep voice 3: 2000-speaker neural text-to-speech, Proc ICLR, 214
- huang, 0, Prediction of perceived sound quality of synthetic speech, Proc Asia-Pacific Signal Inf Process Assoc Annu Summit Conf
- cernak, 0, An evaluation of synthetic speech using the PESQ measure, Proc Eur Congr Acoust, 2725
- erro, 0, Frame alignment method for cross-lingual voice conversion, Proc INTERSPEECH
- 10.1109/TASL.2010.2041688 · doi:10.1109/tasl.2010.2041688
- mashimo, 2002, Cross-language voice conversion evaluation using bilingual databases, Proc Int Conf Spoken Lang Process
- sundermann, 0, VTLN-based crosslanguage voice conversion, Proc IEEE Autom Speech Recognit Understanding Workshop
- 10.1109/ICASSP.2015.7178897 · doi:10.1109/icassp.2015.7178897
- 10.21437/Odyssey.2018-30 · DOI
- 10.21437/Odyssey.2018-34 · DOI
- 10.1109/TASLP.2019.2892235 · DOI
- fujii, 2007, High individuality voice conversion based on concatenative speech synthesis, Int J Elect Comput Energetic Electron Commun Eng, 1, 1617
- sagisaka, 0, Proc Int Conf Spoken Lang Process
- ney, 0, A first step towards text-independent voice conversion, Proc Int Conf Spoken Lang Process
- bi?kowski, 2018, Demystifying MMD GANs, arXiv 1801 01401
- 10.1109/ISCSLP.2014.6936599 · doi:10.1109/iscslp.2014.6936599
- 10.21437/Interspeech.2017-247 · doi:10.21437/interspeech.2017-247
- lorenzo-trueba, 2018, The voice conversion challenge: Promoting development of parallel and nonparallel methods, arXiv 1804 04262
- 10.1016/S0167-6393(99)00015-1 · doi:10.1016/s0167-6393(99)00015-1
- 10.23919/EUSIPCO.2019.8902651 · DOI
- kalchbrenner, 2018, Efficient neural audio synthesis, arXiv 1802 08435
- kim, 0, Flowavenet: A generative flow for raw audio, Proc Int Conf Mach Learn, 3370
- toda, 0, The voice conversion challenge, Proc INTERSPEECH, 1632
- 10.21437/Interspeech.2016-1331 · DOI
- 10.1109/ICCV.2017.244 · DOI · Paper page
- 10.21437/Interspeech.2018-1528 · DOI
- 10.1007/978-3-030-00794-2_48 · doi:10.1007/978-3-030-00794-2_48
- 10.1109/ASRU46091.2019.9003801 · DOI
- 10.21437/Interspeech.2019-1514 · doi:10.21437/interspeech.2019-1514
- 10.1109/TASLP.2018.2835720 · DOI
- huang, 0, TimbreTron: A wavenet (cycleGAN (CQT (audio))) pipeline for musical timbre transfer, Proc Int Conf Learn Representations
- 10.1088/1742-6596/1607/1/012046 · DOI
- 10.21437/Odyssey.2020-35 · DOI
- 10.1007/978-3-642-35292-8_4 · doi:10.1007/978-3-642-35292-8_4
- 10.1109/APSIPA.2013.6694179 · doi:10.1109/apsipa.2013.6694179
- 10.5923/j.ajsp.20120205.06 · doi:10.5923/j.ajsp.20120205.06
- ribeiro, 0, A perceptual investigation of wavelet-based decomposition of f0 for text-to-speech synthesis, Proc 16th Annu Conf Int Speech Commun Assoc
- 10.1109/ICASSP.2016.7472734 · doi:10.1109/icassp.2016.7472734
- 10.1109/TASL.2009.2038663 · DOI · Paper page
- 10.1109/CHINSL.2008.ECP.44 · doi:10.1109/chinsl.2008.ecp.44
- 10.1016/j.specom.2006.08.001 · doi:10.1016/j.specom.2006.08.001
- 10.1109/ICASSP.2004.1325911 · doi:10.1109/icassp.2004.1325911
- 10.1007/978-3-642-34584-5_24 · doi:10.1007/978-3-642-34584-5_24
- 10.1038/110311a0 · doi:10.1038/110311a0
- wu, 0, Examplar-based voice conversion using non-negative spectrogram deconvolution, Proc 8th ISCA Speech Synth Workshop
- liu, 2016, Coupled generative adversarial networks, Proc Adv Neural Inf Process Syst, 469
- zou, 2020, Edge-GAN: Edge conditioned multi-view face image generation, Proc IEEE Int Conf Image Process, 2401
- 10.1016/0167-6393(94)00053-D · DOI
- 10.1109/ICSP.2016.7877819 · doi:10.1109/icsp.2016.7877819
- 10.1109/TASL.2011.2165944 · DOI
- 10.1109/ICASSP.2005.1415037 · doi:10.1109/icassp.2005.1415037
- 10.1250/ast.14.353 · doi:10.1250/ast.14.353
- 10.1109/89.661472 · DOI · Paper page
- xuan, 0, Em algorithms of Gaussian mixture model and hidden Markov model, Proc Int Conf Image Process, 145
- 10.1109/APSIPA.2014.7041540 · doi:10.1109/apsipa.2014.7041540
- 10.1109/APSIPA.2017.8282110 · doi:10.1109/apsipa.2017.8282110
- watanabe, 0, Transformation of spectral envelope for voice conversion based on radial basis function networks, Proc Int Conf Spoken Lang Process
- ramani, 0, Cross-lingual voice conversion-based polyglot speech synthesizer for indian languages, Proc 15th Annu Conf Int Speech Commun Assoc
Cited by
Masood, M., Nawaz, M., Malik, K. M., Javed, A., Irtaza, A., & Malik, H. (2023) · Applied Intelligence · doi:10.1007/s10489-022-03766-z · Cited by 543
A survey on neural speech synthesis
Tan, X., Qin, T., Soong, F., & Liu, T.-Y. (2021) · arXiv · doi:10.48550/arxiv.2106.15561 · Cited by 216
A Survey on the Detection and Impacts of Deepfakes in Visual, Audio, and Textual Formats
Mubarak, R., Alsboui, T., Alshaikh, O., Inuwa-Dutse, I., Khan, S., & Parkinson, S. (2023) · IEEE Access · doi:10.1109/access.2023.3344653 · Cited by 132
Voice Conversion Challenge 2020 –- Intra-lingual semi-parallel and cross-lingual voice conversion –-
Yi, Z., Huang, W.-C., Tian, X., Yamagishi, J., Das, R. K., Kinnunen, T., Ling, Z.-H., & Toda, T. (2020) · Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge 2020 · doi:10.21437/vcc_bc.2020-14 · Cited by 126
Self-citationThe Vicomtech Audio Deepfake Detection System Based on Wav2vec2 for the 2022 ADD Challenge
Martin-Donas, J. M., & Alvarez, A. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9747768 · Cited by 100
Metadata
- Authors
- Sisman, B., Yamagishi, J., King, S., & Li, H.
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2021
- Volume
- 29
- Pages
- 132-157
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W3098557217
- Cited by
- 357
- References
- 250
- FWCI
- 15.63
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Funders
- SUTD Start-up Grant Artificial Intelligence for Human Voice Conversion (SRG ISTD 2020 158)
- SUTD AI
- The Understanding and Synthesis of Expressive Speech by AI (PIE-SGP-AI-2020-02)
- National Research Foundation Singapore (AISG-GC-2019-002, AISG-100E-2018-006)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.