Adaptive Wavenet Vocoder for Residual Compensation in GAN-Based Voice Conversion
The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
In this paper, we propose to use generative adversarial networks (GAN) together with a WaveNet vocoder to address the over-smoothing problem arising from the deep learning approaches to voice conversion, and to improve the vocoding quality over the traditional vocoders. As GAN aims to minimize the divergence between the natural and converted speech parameters, it effectively alleviates the over-smoothing problem in the converted speech. On the other hand, WaveNet vocoder allows us to leverage from the human speech of a large speaker population, thus improving the naturalness of the synthetic voice. Furthermore, for the first time, we study how to use WaveNet vocoder for residual compensation to improve the voice conversion performance. The experiments show that the proposed voice conversion framework consistently outperforms the baselines.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Benefiting from advancements in neural vocoders, modern VC systems have redefined their architecture. Since a neural vocoder itself is trainable, it enables joint training with conversion models [57] and, in some cases, speech analysis modules [58], achieving an E2E structure.”
From Audio Deepfake Detection: What Has Been Achieved and What Lies Ahead (2025) · cites this paper as [57]
Citations
Cites
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · cited 4×
“WaveNet [28] is a well-known deep neural network that can generates raw audio waveforms.”
BackgroundInfluentialStackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.629 · Cited by 3,021 · cited 1×
“GANs have recently been shown to be an effective training method and have been used for image generation [30], image synthesis [30], speech enhancement [23, 24], language identification [31], and text-to-speech synthesis [25, 26].”
MethodSelf-citationNatural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461368 · Cited by 2,791
SEGAN: Speech Enhancement Generative Adversarial Network
Pascual, S., Bonafonte, A., & Serrà, J. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1428 · Cited by 1,297 · cited 2×
“GAN has shown to be effective in many fields including speech enhancement [23, 24], speech synthesis [25,26].”
BackgroundMethodVoice Conversion Based on Maximum-Likelihood Estimation of Spectral Parameter Trajectory
Toda, T., Black, A. W., & Tokuda, K. (2007) · IEEE Transactions on Audio, Speech and Language Processing · doi:10.1109/tasl.2007.907344 · Cited by 937
Mel-cepstral distance measure for objective speech quality assessment
Kubichek, R. (2002) · Proceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing · doi:10.1109/pacrim.1993.407206 · Cited by 534 · cited 7×
“Such MCD doesn’t evaluate the effect of vocoders.”
MethodInfluentialVoice conversion through vector quantization
Abe, M., Nakamura, S., Shikano, K., & Kuwabara, H. (2003) · ICASSP-88., International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1988.196671 · Cited by 415 · cited 2×
“There are 30 hidden layers in our WaveNet Vocoder, and dilations are [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]× 3 respectively.”
BackgroundMethodSelf-citationDigital speech processing, synthesis, and recognition
(1989) · Speech Communication · doi:10.1016/0167-6393(89)90009-5 · Cited by 412
“StarGAN-VC: non-parallel many-to-many voice conversion using star generative adversarial networks,”
Kameoka, H., Kaneko, T., Tanaka, K., & Hojo, N. (2018) · 2018 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639535 · Cited by 371
Hsu, C.-C., Hwang, H.-T., Wu, Y.-C., Tsao, Y., & Wang, H.-M. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-63 · Cited by 320 · cited 2×
“To address the quality problem, GAN was proposed [17, 21, 22] as a solution.”
BackgroundPhonetic posteriorgrams for many-to-one voice conversion without parallel data training
Sun, L., Li, K., Wang, H., Kang, S., & Meng, H. (2016) · 2016 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2016.7552917 · Cited by 297 · cited 1×
“Recently, deep learning approaches such as restricted Boltzmann machine (RBM) [12, 13], long short-term memory (LSTM) [14, 15], and autoencoder [16, 17] are shown to be effective in modeling the relationship between source and target features more accurately than conventional techniques.”
BackgroundVoice conversion from non-parallel corpora using variational auto-encoder
Hsu, C.-C., Hwang, H.-T., Wu, Y.-C., Tsao, Y., & Wang, H.-M. (2016) · 2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA) · doi:10.1109/apsipa.2016.7820786 · Cited by 289 · cited 2×
“There are 30 hidden layers in our WaveNet Vocoder, and dilations are [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]× 3 respectively.”
BackgroundMethodSpeaker-Dependent WaveNet Vocoder
Tamamori, A., Hayashi, T., Kobayashi, K., Takeda, K., & Toda, T. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-314 · Cited by 269 · cited 2×
“independent WaveNet vocoders [27, 29] were studied and shown initial success.”
BackgroundVoice conversion using deep Bidirectional Long Short-Term Memory based Recurrent Neural Networks
Sun, L., Kang, S., Li, K., & Meng, H. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178896 · Cited by 257 · cited 1×
“Recently, deep learning approaches such as restricted Boltzmann machine (RBM) [12, 13], long short-term memory (LSTM) [14, 15], and autoencoder [16, 17] are shown to be effective in modeling the relationship between source and target features more accurately than conventional techniques.”
BackgroundMichelsanti, D., & Tan, Z.-H. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1620 · Cited by 233 · cited 2×
“GAN has shown to be effective in many fields including speech enhancement [23, 24], speech synthesis [25,26].”
BackgroundMethodStatistical Parametric Speech Synthesis Incorporating Generative Adversarial Networks
Saito, Y., Takamichi, S., & Saruwatari, H. (2018) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2017.2761547 · Cited by 223 · cited 2×
“GAN has shown to be effective in many fields including speech enhancement [23, 24], speech synthesis [25,26].”
BackgroundMethodVoice Conversion Using Deep Neural Networks With Layer-Wise Generative Training
Chen, L.-H., Ling, Z.-H., Liu, L.-J., & Dai, L.-R. (2014) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2014.2353991 · Cited by 210 · cited 1×
“Recently, deep learning approaches such as restricted Boltzmann machine (RBM) [12, 13], long short-term memory (LSTM) [14, 15], and autoencoder [16, 17] are shown to be effective in modeling the relationship between source and target features more accurately than conventional techniques.”
BackgroundVoice Conversion Using Partial Least Squares Regression
Helander, E., Virtanen, T., Nurminen, J., & Gabbouj, M. (2010) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2010.2041699 · Cited by 207 · cited 2×
“There are 30 hidden layers in our WaveNet Vocoder, and dilations are [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]× 3 respectively.”
MethodExemplar-Based Sparse Representation With Residual Compensation for Voice Conversion
Wu, Z., Virtanen, T., Chng, E. S., & Li, H. (2014) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2014.2333242 · Cited by 166 · cited 2×
“In VC literature, AB preference test and XAB test are the evaluation techniques which have been widely used [7].”
BackgroundMethodSelf-citationBest-worst scaling: theory and methods
Flynn, T. N., & Marley, A. A. J. (2014) · Edward Elgar Publishing eBooks · doi:10.4337/9781781003152.00014 · Cited by 161 · cited 2×
“To alleviate such a problem, in this paper we also use an evaluation technique called Best-Worst Scaling (BWS) [41, 42], which can handle a long list of options and always generates discriminating results as the respondents are asked to choose the BEST and WORST option in a choice set.”
MethodParallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks
Kaneko, T., & Kameoka, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.11293 · Cited by 158 · cited 4×
“To address the quality problem, GAN was proposed [17, 21, 22] as a solution.”
BackgroundMethodResultInfluentialHsu, C.-C., Hwang, H.-T., Wu, Y.-C., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.00849 · Cited by 151 · cited 2×
“To address the quality problem, GAN was proposed [17, 21, 22] as a solution.”
BackgroundUnsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
Hsu, W.-N., Zhang, Y., & Glass, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.07902 · Cited by 148
Learning Latent Representations for Speech Generation and Transformation
Hsu, W.-N., Zhang, Y., & Glass, J. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-349 · Cited by 141
Kaneko, T., Kameoka, H., Hiramatsu, K., & Kashino, K. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-970 · Cited by 127 · cited 2×
“[21] applied a GAN to sequenceto-sequence VC and demonstrated that the use of GANbased training criteria outperforms the use of traditional mean squared error (MSE)-based training criteria.”
BackgroundMethodVoice Conversion Using Dynamic Kernel Partial Least Squares Regression
Helander, E., Silen, H., Virtanen, T., & Gabbouj, M. (2012) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2011.2165944 · Cited by 126 · cited 2×
“Speech Samples: https://sites.google.com/view/berraksisman/ dynamic kernel partial least squares regression (DKPLS) [5] marked a success in spectrum conversion.”
BackgroundExemplar-based voice conversion in noisy environment
Takashima, R., Takiguchi, T., & Ariki, Y. (2012) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2012.6424242 · Cited by 122 · cited 1×
“As a solution to the limited training data problem, non-negative matrix factorization based voice conversion frameworks [6–11] have been proposed to address the over-smoothing problem in voice conversion.”
MethodGenerative adversarial network-based postfilter for statistical parametric speech synthesis
Kaneko, T., Kameoka, H., Hojo, N., Ijima, Y., Hiramatsu, K., & Kashino, K. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7953090 · Cited by 119 · cited 2×
“GAN has shown to be effective in many fields including speech enhancement [23, 24], speech synthesis [25,26].”
BackgroundMethodBest-worst scaling: theory and methods
Marley, A. A. J. (2024) · Edward Elgar Publishing eBooks · doi:10.4337/9781800375635.00014 · Cited by 102
An investigation of multi-speaker training for wavenet vocoder
Hayashi, T., Tamamori, A., Kobayashi, K., Takeda, K., & Toda, T. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8269007 · Cited by 97 · cited 7×
“independent WaveNet vocoders [27, 29] were studied and shown initial success.”
BackgroundMethodInfluentialStatistical Voice Conversion with WaveNet-Based Waveform Generation
Kobayashi, K., Hayashi, T., Tamamori, A., & Toda, T. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-986 · Cited by 80 · cited 2×
“WaveNet vocoder [27, 33] has been proposed and shown to achieve remarkable sound quality improvement over the traditional vocoders.”
MethodPostfilters to Modify the Modulation Spectrum for Statistical Parametric Speech Synthesis
Takamichi, S., Toda, T., Black, A. W., Neubig, G., Sakti, S., & Nakamura, S. (2016) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2016.2522655 · Cited by 71 · cited 1×
“Global variance (GV) [3] and modulation spectrum (MS) [18], among others [19, 20], are typical examples to address the problem.”
BackgroundSelf-citationNakashika, T., Takiguchi, T., & Ariki, Y. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-447 · Cited by 57 · cited 1×
“Recently, deep learning approaches such as restricted Boltzmann machine (RBM) [12, 13], long short-term memory (LSTM) [14, 15], and autoencoder [16, 17] are shown to be effective in modeling the relationship between source and target features more accurately than conventional techniques.”
BackgroundSpeaker adaptation and voice conversion by codebook mapping
Shikano, K., Nakamura, S., & Abe, M. (1991) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.1991.176405 · Cited by 54 · cited 2×
“There are 30 hidden layers in our WaveNet Vocoder, and dilations are [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]× 3 respectively.”
BackgroundMethodSelf-citationŞişman, B., Zhang, M., & Li, H. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1131 · Cited by 54 · cited 5×
“We note that the speaker adapted WaveNet vocoder is trained in a similar way as that in [39].”
MethodInfluentialSelf-citationVoice conversion based on Non-negative matrix factorization using phoneme-categorized dictionary
Aihara, R., Nakashika, T., Takiguchi, T., & Ariki, Y. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6855137 · Cited by 39 · cited 1×
“As a solution to the limited training data problem, non-negative matrix factorization based voice conversion frameworks [6–11] have been proposed to address the over-smoothing problem in voice conversion.”
MethodSparse representation of phonetic features for voice conversion with and without parallel data
Şişman, B., Li, H., & Tan, K. C. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8269002 · Cited by 39 · cited 2×
“The study [7, 11] has shown that residual compensation always enhances the speech quality.”
BackgroundMethodSelf-citationConditional Generative Adversarial Nets Classifier for Spoken Language Identification
Shen, P., Lu, X., Li, S., & Kawai, H. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-553 · Cited by 34 · cited 1×
“GANs have recently been shown to be an effective training method and have been used for image generation [30], image synthesis [30], speech enhancement [23, 24], language identification [31], and text-to-speech synthesis [25, 26].”
MethodCute: A concatenative method for voice conversion using exemplar-based unit selection
Jin, Z., Finkelstein, A., DiVerdi, S., Lu, J., & Mysore, G. J. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472761 · Cited by 29 · cited 1×
“As a solution to the limited training data problem, non-negative matrix factorization based voice conversion frameworks [6–11] have been proposed to address the over-smoothing problem in voice conversion.”
MethodTransformation of prosody in voice conversion
Şişman, B., Li, H., & Tan, K. C. (2017) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipa.2017.8282288 · Cited by 28 · cited 1×
“To alleviate such a problem, in this paper we also use an evaluation technique called Best-Worst Scaling (BWS) [41, 42], which can handle a long list of options and always generates discriminating results as the respondents are asked to choose the BEST and WORST option in a choice set.”
MethodSelf-citationModulation spectrum-constrained trajectory training algorithm for GMM-based Voice Conversion
Takamichi, S., Toda, T., Black, A. W., & Nakamura, S. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178894 · Cited by 28 · cited 1×
“Global variance (GV) [3] and modulation spectrum (MS) [18], among others [19, 20], are typical examples to address the problem.”
BackgroundSelf-citationProbabilistic feature mapping based on trajectory HMMs
Zen, H., Nankaku, Y., & Tokuda, K. (2008) · Interspeech (USB) · doi:10.21437/interspeech.2008-330 · Cited by 13 · cited 1×
“Many of the state-of-the-art voice conversion frameworks [1–3,17,22,34–36] use a statistical parametric vocoder, which is generally designed to simulate the complex mechanics of the human speech generation process under certain simple assumptions, for example, the interaction between F0 and formant structure is ignore…”
BackgroundParallel Dictionary Learning for Multimodal Voice Conversion Using Matrix Factorization
Aihara, R., Masaka, K., Takiguchi, T., & Ariki, Y. (2016) · Studies in computational intelligence · doi:10.1007/978-3-319-40171-3_3 · Cited by 8 · cited 1×
“There are 30 hidden layers in our WaveNet Vocoder, and dilations are [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]× 3 respectively.”
MethodAnalysis of spectral enhancement using global variance in HMM-based speech synthesis
Nose, T., & Ito, A. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-196 · Cited by 8 · cited 1×
“Global variance (GV) [3] and modulation spectrum (MS) [18], among others [19, 20], are typical examples to address the problem.”
Background
Cited by
An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning
Sisman, B., Yamagishi, J., King, S., & Li, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2020.3038524 · Cited by 357
“GANs offer a general framework for training a data generator in such a way that it can deceive a real/fake discriminator that attempts to distinguish real data and fake data produced by the generator [242]–[244].”
BackgroundMethodInfluentialSelf-citationEmotional voice conversion: Theory, databases and ESD
Zhou, K., Sisman, B., Liu, R., & Li, H. (2022) · Speech Communication · doi:10.1016/j.specom.2021.11.006 · Cited by 181
Self-citationExpressive TTS Training With Frame and Style Reconstruction Loss
Liu, R., Şişman, B., Gao, G., & Li, H. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3076369 · Cited by 97
“a neural vocoder [5], [16]–[21], neural TTS generates naturalsounding and human-like speech which achieves state-of-the-art”
BackgroundSelf-citationTransforming Spectrum and Prosody for Emotional Voice Conversion with Non-Parallel Training Data
Zhou, K., Şişman, B., & Li, H. (2020) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2020-33 · Cited by 93
Self-citationVQVAE Unsupervised Unit Discovery and Multi-Scale Code2Spec Inverter for Zerospeech Challenge 2019
Tjandra, A., Sisman, B., Zhang, M., Sakti, S., Li, H., & Nakamura, S. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-3232 · Cited by 65
Self-citation
Metadata
- Authors
- Sisman, B., Zhang, M., Sakti, S., Li, H., & Nakamura, S.
- Affiliations
- National University of Singapore (SG)
- Nara Institute of Science and Technology (JP)
- Venue
- 2018 IEEE Spoken Language Technology Workshop (SLT)
- Publisher
- IEEE
- Published
- 2018-12
- Pages
- 282-289
- OpenAlex
- W2911340057
- Cited by
- 36
- References
- 44
- Citations by year
- 2025: 1 · 2023: 1 · 2022: 2 · 2021: 7 · 2020: 14 · 2019: 11
- FWCI
- 3.25
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Spoken Language Technology Workshop (SLT) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- voice conversion, generative adversarial networks, residual compensation, over-smoothing, synthetic speech naturalness, speech synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.