Adaptive Wavenet Vocoder for Residual Compensation in GAN-Based Voice Conversion
The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
In this paper, we propose to use generative adversarial networks (GAN) together with a WaveNet vocoder to address the over-smoothing problem arising from the deep learning approaches to voice conversion, and to improve the vocoding quality over the traditional vocoders. As GAN aims to minimize the divergence between the natural and converted speech parameters, it effectively alleviates the over-smoothing problem in the converted speech. On the other hand, WaveNet vocoder allows us to leverage from the human speech of a large speaker population, thus improving the naturalness of the synthetic voice. Furthermore, for the first time, we study how to use WaveNet vocoder for residual compensation to improve the voice conversion performance. The experiments show that the proposed voice conversion framework consistently outperforms the baselines.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Benefiting from advancements in neural vocoders, modern VC systems have redefined their architecture. Since a neural vocoder itself is trainable, it enables joint training with conversion models [57] and, in some cases, speech analysis modules [58], achieving an E2E structure.”
From Audio Deepfake Detection: What Has Been Achieved and What Lies Ahead (2025) · cites this paper as [57]
Citations
Cites
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · cited 4×
“WaveNet [28] is a well-known deep neural network that can generates raw audio waveforms.”
BackgroundInfluentialStackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.629 · Cited by 3,021 · cited 1×
“GANs have recently been shown to be an effective training method and have been used for image generation [30], image synthesis [30], speech enhancement [23, 24], language identification [31], and text-to-speech synthesis [25, 26].”
MethodSelf-citationNatural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461368 · Cited by 2,791
SEGAN: Speech Enhancement Generative Adversarial Network
Pascual, S., Bonafonte, A., & Serrà, J. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1428 · Cited by 1,297 · cited 2×
“GAN has shown to be effective in many fields including speech enhancement [23, 24], speech synthesis [25,26].”
BackgroundMethodVoice Conversion Based on Maximum-Likelihood Estimation of Spectral Parameter Trajectory
Toda, T., Black, A. W., & Tokuda, K. (2007) · IEEE Transactions on Audio, Speech and Language Processing · doi:10.1109/tasl.2007.907344 · Cited by 937
Mel-cepstral distance measure for objective speech quality assessment
Kubichek, R. (2002) · Proceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing · doi:10.1109/pacrim.1993.407206 · Cited by 534 · cited 7×
“Such MCD doesn’t evaluate the effect of vocoders.”
MethodInfluentialVoice conversion through vector quantization
Abe, M., Nakamura, S., Shikano, K., & Kuwabara, H. (2003) · ICASSP-88., International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1988.196671 · Cited by 415 · cited 2×
“There are 30 hidden layers in our WaveNet Vocoder, and dilations are [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]× 3 respectively.”
BackgroundMethodSelf-citationDigital speech processing, synthesis, and recognition
(1989) · Speech Communication · doi:10.1016/0167-6393(89)90009-5 · Cited by 412
“StarGAN-VC: non-parallel many-to-many voice conversion using star generative adversarial networks,”
Kameoka, H., Kaneko, T., Tanaka, K., & Hojo, N. (2018) · 2018 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639535 · Cited by 371
Hsu, C.-C., Hwang, H.-T., Wu, Y.-C., Tsao, Y., & Wang, H.-M. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-63 · Cited by 320 · cited 2×
“To address the quality problem, GAN was proposed [17, 21, 22] as a solution.”
Background
Cited by
An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning
Sisman, B., Yamagishi, J., King, S., & Li, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2020.3038524 · Cited by 357
“GANs offer a general framework for training a data generator in such a way that it can deceive a real/fake discriminator that attempts to distinguish real data and fake data produced by the generator [242]–[244].”
BackgroundMethodInfluentialSelf-citationEmotional voice conversion: Theory, databases and ESD
Zhou, K., Sisman, B., Liu, R., & Li, H. (2022) · Speech Communication · doi:10.1016/j.specom.2021.11.006 · Cited by 181
Self-citationExpressive TTS Training With Frame and Style Reconstruction Loss
Liu, R., Şişman, B., Gao, G., & Li, H. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3076369 · Cited by 97
“a neural vocoder [5], [16]–[21], neural TTS generates naturalsounding and human-like speech which achieves state-of-the-art”
BackgroundSelf-citationTransforming Spectrum and Prosody for Emotional Voice Conversion with Non-Parallel Training Data
Zhou, K., Şişman, B., & Li, H. (2020) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2020-33 · Cited by 93
Self-citationVQVAE Unsupervised Unit Discovery and Multi-Scale Code2Spec Inverter for Zerospeech Challenge 2019
Tjandra, A., Sisman, B., Zhang, M., Sakti, S., Li, H., & Nakamura, S. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-3232 · Cited by 65
Self-citationAudio Deepfake Detection: What Has Been Achieved and What Lies Ahead
Zhang, B., Cui, H., Nguyen, V., & Whitty, M. (2025) · Sensors · doi:10.3390/s25071989 · Cited by 72
“Since a neural vocoder itself is trainable, it enables joint training with conversion models [57] and, in some cases, speech analysis modules [58], achieving an E2E structure.”
Cross-lingual Voice Conversion with Bilingual Phonetic Posteriorgram and Average Modeling
Zhou, Y., Tian, X., Xu, H., Das, R. K., & Li, H. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683746 · Cited by 86
“A number of models have been established to realize the spectral feature mapping from the source to target, such as Gaussian mixture models [1, 2], neural network based methods [3–8], frequency warping methods [9–12], exemplar based methods [13–16] and so on.”
BackgroundSelf-citationZhang, M., Wang, X., Fang, F., Li, H., & Yamagishi, J. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1357 · Cited by 80
Self-citationIs Natural Necessary? Human Voice versus Synthetic Voice for Intelligent Virtual Agents
Abdulrahman, A., & Richards, D. (2022) · Multimodal Technologies and Interaction · doi:10.3390/mti6070051 · Cited by 51
Teacher-Student Training For Robust Tacotron-Based TTS
Liu, R., Şişman, B., Li, J., Bao, F., Gao, G., & Li, H. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054681 · Cited by 51
“Furthermore, together with a neural vocoder [5, 11–15], natural-sounding human-like speech can be generated.”
BackgroundSelf-citationAk, K. E., Lim, J., Tham, J. Y., & Kassim, A. A. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.02.030 · Cited by 51
Huang, W.-C., Luo, H., Hwang, H.-T., Lo, C.-C., Peng, Y.-H., Tsao, Y., & Wang, H.-M. (2020) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2020.2977678 · Cited by 53
“It has been shown that employing neural vocoders as the waveform generation module can greatly improve the performance of VC systems [26]–[31].”
BackgroundImproving Zero-shot Voice Style Transfer via Disentangled Representation LearningPreprint
Yuan, S., Cheng, P., Zhang, R., Hao, W., Gan, Z., & Carin, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.09420 · Cited by 61
“Voice style transfer (VST) has received long-term research interest, due to its potential for applications in security (Sisman et al., 2018), medicine (Nakamura et al.”
BackgroundSINGAN: Singing Voice Conversion with Generative Adversarial Networks
Şişman, B., Vijayan, K., Dong, M., & Li, H. (2019) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc47483.2019.9023162 · Cited by 50
“In Figure 5, we report the SNR values under different training settings.”
BackgroundMethodInfluentialSelf-citationVaw-Gan For Disentanglement And Recomposition Of Emotional Elements In Speech
Zhou, K., Şişman, B., & Li, H. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383526 · Cited by 43
“We perform XAB test to assess the speech quality and emotion similarity, which has been widely used in speech synthesis, such as voice conversion [56], singing voice conversion [55].”
MethodSelf-citationLiu, R., Şişman, B., Bao, F., Yang, J., Gao, G., & Li, H. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3040523 · Cited by 34
Self-citationOn the Study of Generative Adversarial Networks for Cross-Lingual Voice Conversion
Şişman, B., Zhang, M., Dong, M., & Li, H. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003939 · Cited by 35
“[28] Chin-Cheng Hsu, Hsin-Te Hwang, Yi-Chiao Wu, Yu Tsao, and Hsin-Min Wang, “Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks,” arXiv:1704.00849 [cs.CL], 2017.”
BackgroundSelf-citationDeepConversion: Voice conversion with limited parallel training data
Zhang, M., Şişman, B., Zhao, L., & Li, H. (2020) · Speech Communication · doi:10.1016/j.specom.2020.05.004 · Cited by 31
Self-citationModeling Prosodic Phrasing With Multi-Task Learning in Tacotron-Based TTS
Liu, R., Şişman, B., Bao, F., Gao, G., & Li, H. (2020) · IEEE Signal Processing Letters · doi:10.1109/lsp.2020.3016564 · Cited by 26
“The recent progress in neural vocoder [4], [11]–[15] also contributes to the improvement of speech quality.”
Self-citationAnalysis of gender and identity issues in depression detection on de-identified speech
Lopez‐Otero, P., & Docío-Fernández, L. (2020) · Computer Speech & Language · doi:10.1016/j.csl.2020.101118 · Cited by 25
Expressive Voice Conversion: A Joint Framework for Speaker Identity and Emotional Style Transfer
Du, Z., Şişman, B., Zhou, K., & Li, H. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9687906 · Cited by 31
Self-citationWaveTTS: Tacotron-based TTS with Joint Time-Frequency Domain Loss
Liu, R., Şişman, B., Bao, F., Gao, G., & Li, H. (2020) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2020-35 · Cited by 17
Self-citationLearning Cross-Modal Representations for Language-Based Image Manipulation
Kenan, E. A., Sun, Y., & Lim, J. (2020) · Proceedings - International Conference on Image Processing · doi:10.1109/icip40778.2020.9191228 · Cited by 15
“With the introduction of Generative Adversarial Networks (GANs) [13, 22], there have been great advances in image [27, 38, 11, 10, 7] and speech [30, 34, 33, 31, 32, 35] synthesis.”
BackgroundContinuous vocoder applied in deep neural network based voice conversion
Al-Radhi, M. S., Csapó, T. G., & Németh, G. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-08198-5 · Cited by 12
“…various studies in voice conversion are still considering some of these vocoders [23], such as STRAIGHT [5, 53, 59], mixed excitation [34], Harmonic plus Noise Model [35], glottal source modeling [6], or even with more complex end-to-end acoustic models like adaptive WAVENET [54], or Tacotron [62].”
MethodSpeaker-independent Spectral Mapping for Speech-to-Singing Conversion
Gao, X., Tian, X., Das, R. K., Zhou, Y., & Li, H. (2019) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc47483.2019.9023056 · Cited by 12
“The Mean Opinion Score (MOS), AB preference tests were conducted for subjective evaluations [37]–[40] on the quality and naturalness of synthesized singing outputs.”
MethodSelf-citationA Modularized Neural Network with Language-Specific Output Layers for Cross-Lingual Voice Conversion
Zhou, Y., Tian, X., Yılmaz, E., Das, R. K., & Li, H. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003798 · Cited by 11
“Deep generative models like variational auto-encoder (VAE) [18, 19] and generative adversarial networks (GANs) [20,21] without requiring parallel data are found effective for VC, though in the same language.”
MethodSelf-citationMany-to-many Cross-lingual Voice Conversion with a Jointly Trained Speaker Embedding Network
Zhou, Y., Tian, X., Das, R. K., & Li, H. (2019) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc47483.2019.9023277 · Cited by 11
Self-citationIncorporating Reinforced Adversarial Learning in Autoregressive Image Generation
Ak, K. E., Xu, N., Lin, Z., & Wang, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58589-1_2 · Cited by 15
Generalization of Spectrum Differential based Direct Waveform Modification for Voice Conversion
Huang, W.-C., Wu, Y.-C., Kobayashi, K., Peng, Y., Hwang, H.-T., Tobing, P. L., Tsao, Y., Wang, H., & Toda, T. (2019) · ISCA Speech Synthesis Workshop (SSW) · doi:10.21437/ssw.2019-11 · Cited by 8
“State-of-the-art VC systems have shown remarkable results by combining such neural waveform generation process with conversion models such as GMMs [20], or other methods based on recent DNN methods such as bidirectional long short term memory [21], autoencoders [22], sequence-to-sequence learning [23] or generative ad…”
MethodSpectrum and Prosody Conversion for Cross-lingual Voice Conversion with CycleGANPreprint
Du, Z., Zhou, K., Şişman, B., & Li, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.04562 · Cited by 8
“in the fields of computer version [45]–[48], natural language processing [49], [50], speech synthesis [27], [51], [52] and voice conversion [9], [18], [26].”
BackgroundSelf-citationYang, J., Zhou, Y., & Huang, H. (2023) · Speech Communication · doi:10.1016/j.specom.2023.05.004 · Cited by 7
High-Quality Vocoding Design with Signal Processing for Speech Synthesis and Voice ConversionPreprint
Al-Radhi, M. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.10278 · Cited by 2
“I grouped the state-of-the-art vocoders based VC into three categories. a) Source-filter models: e.g., STRAIGHT [36]; b) Sinusoidal models: e.g., Harmonic plus Noise Model [37]; c) end-to-end complex models: e.g., adaptive WaveNet [38].”
MethodParallel Voice Conversion Based on a Continuous Sinusoidal Model
Al-Radhi, M. S., Csapó, T. G., & Németh, G. (2019) · International Conference on Speech Technology and Human-Computer Dialogue (SpeD) · doi:10.1109/sped.2019.8906565 · Cited by 1
“a) Source-filter models: STRAIGHT [15] and mixed excitation [16]; b) Sinusoidal models: Harmonic plus Noise Model [17] is the only model has been found in the literature based VC; c) end-to-end complex models: WaveNet-based waveform generation [18] and Tacotron [19].”
MethodWINVC: One-Shot Voice Conversion with Weight Adaptive Instance Normalization
Huang, S., Chen, M., Xu, Y., Ke, D., & Hain, T. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-89363-7_42 · Cited by 1
Metadata
- Authors
- Sisman, B., Zhang, M., Sakti, S., Li, H., & Nakamura, S.
- Affiliations
- National University of Singapore (SG)
- Nara Institute of Science and Technology (JP)
- Venue
- 2018 IEEE Spoken Language Technology Workshop (SLT)
- Publisher
- IEEE
- Published
- 2018-12
- Pages
- 282-289
- OpenAlex
- W2911340057
- Cited by
- 36
- References
- 44
- Citations by year
- 2025: 1 · 2023: 1 · 2022: 2 · 2021: 7 · 2020: 14 · 2019: 11
- FWCI
- 3.25
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Spoken Language Technology Workshop (SLT) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- voice conversion, generative adversarial networks, residual compensation, over-smoothing, synthetic speech naturalness, speech synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.