CoChat
Sign Up
PreprintCited by 20Speech Recognition and SynthesisSpeech and Audio ProcessingMusic and Audio Processing

The DOI matches its DataCite record; its reference list is not available.

NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis

Open access
Choi, Hyeong-Seok, Yang, Jinhyeok, Lee, Juheon, Kim, Hyeongju·2022·arXiv
DOI 10.48550/arxiv.2211.09407arXiv 2211.09407
Open this paper in CoChat — free accountOpen PDF

Abstract

Various applications of voice synthesis have been developed independently despite the fact that they generate "voice" as output in common. In addition, most of the voice synthesis models still require a large number of audio data paired with annotated labels (e.g., text transcription and music score) for training. To this end, we propose a unified framework of synthesizing and manipulating voice signals from analysis features, dubbed NANSY++. The backbone network of NANSY++ is trained in a self-supervised manner that does not require any annotations paired with audio. After training the backbone network, we efficiently tackle four voice applications - i.e. voice conversion, text-to-speech, singing voice synthesis, and voice designing - by partially modeling the analysis features required for each task. Extensive experiments show that the proposed framework offers competitive advantages such as controllability, data efficiency, and fast training convergence, while providing high quality synthesis. Audio samples: tinyurl.com/8tnsy3uc.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2211.09407

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 20

Research works printed but not linked: 52

  1. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems , 33:12449-12460, 2020.
  2. Evelina Bakhturina, Vitaly Lavrukhin, Boris Ginsburg, and Yang Zhang. Hi-fi multi-speaker english tts dataset. arXiv preprint arXiv:2104.01497 , 2021. · arXiv
  3. Piotr Bilinski, Thomas Merritt, Abdelhamid Ezzerg, Kamil Pokora, Sebastian Cygert, Kayoko Yanagisawa, Roberto Barra-Chicote, and Daniel Korzekwa. Creating New Voices using Normalizing Flows. In Proc. Interspeech 2022 , pp. 2958-2962, 2022. doi: 10.21437/Interspeech. 2022-10195. · DOI
  4. Mikołaj Bi´ nkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=r1gfQgSFDr .
  5. Paul Boersma et al. Accurate short-term analysis of the fundamental frequency and the harmonicsto-noise ratio of a sampled sound. In Proceedings of the institute of phonetic sciences , volume 17, pp. 97-110. Citeseer, 1993.
  6. Edresson Casanova, Julian Weber, Christopher D Shulby, Arnaldo Candido Junior, Eren G¨ olge, and Moacir A Ponti. Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning , pp. 2709-2720. PMLR, 2022.
  7. Jiawei Chen, Xu Tan, Jian Luan, Tao Qin, and Tie-Yan Liu. Hifisinger: Towards high-fidelity neural singing voice synthesis. arXiv preprint arXiv:2009.01776 , 2020a. · arXiv
  8. Mingjian Chen, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, Tie-Yan Liu, et al. Adaspeech: Adaptive text to speech for custom voice. In International Conference on Learning Representations , 2020b.
  9. Hyeong-Seok Choi, Juheon Lee, Wansoo Kim, Jie Lee, Hoon Heo, and Kyogu Lee. Neural analysis and synthesis: Reconstructing speech from self-supervised representations. Advances in Neural Information Processing Systems , 34:16251-16265, 2021a.
  10. Hyeong-Seok Choi, Sungjin Park, Jie Hwan Lee, Hoon Heo, Dongsuk Jeon, and Kyogu Lee. Realtime denoising and dereverberation wtih tiny recurrent u-net. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 5789-5793. IEEE, 2021b.
  11. Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning , pp. 933-941. PMLR, 2017.
  12. Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. ECAPA-TDNN: Emphasized Channel Attention, propagation and aggregation in TDNN based speaker verification. In Interspeech 2020 , pp. 3830-3834, 2020.
  13. Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, and Karen Simonyan. End-to-end adversarial text-to-speech. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=rsf1z-JSj87 .
  14. Jesse Engel, Lamtharn (Hanoi) Hantrakul, Chenjie Gu, and Adam Roberts. Ddsp: Differentiable digital signal processing. In International Conference on Learning Representations , 2020a. URL https://openreview.net/forum?id=B1x1ma4tDr .
  15. Jesse Engel, Rigel Swavely, Lamtharn Hanoi Hantrakul, Adam Roberts, and Curtis Hawthorne. Self-supervised pitch detection by inverse audio synthesis. In ICML 2020 Workshop on Selfsupervision in Audio and Speech , 2020b. URL https://openreview.net/forum?id= RlVTYWhsky7 .
  16. Beat Gfeller, Christian Frank, Dominik Roblek, Matt Sharifi, Marco Tagliasacchi, and Mihajlo Velimirovi´ c. Spice: Self-supervised pitch estimation. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 28:1118-1128, 2020.
  17. Jheng hao Lin, Yist Y. Lin, Chung-Ming Chien, and Hung yi Lee. S2VC: A Framework for Anyto-Any Voice Conversion with Self-Supervised Pretrained Representations. In Proc. Interspeech 2021 , pp. 836-840, 2021. doi: 10.21437/Interspeech.2021-1356. · DOI
  18. Yukiya Hono, Shinji Takaki, Kei Hashimoto, Keiichiro Oura, Yoshihiko Nankaku, and Keiichi Tokuda. Periodnet: A non-autoregressive waveform generation model with a structure separating periodic and aperiodic components. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 6049-6053. IEEE, 2021.
  19. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 29:3451-3460, 2021.
  20. Wenyong Huang, Zhenhe Zhang, Yu Ting Yeung, Xin Jiang, and Qun Liu. SPIRAL: Self-supervised perturbation-invariant representation learning for speech pre-training. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id= TBpg4PnXhYH .
  21. Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics , pp. 492518. Springer, 1992.
  22. Yannick Jadoul, Bill Thompson, and Bart De Boer. Introducing parselmouth: A python interface to praat. Journal of Phonetics , 71:1-15, 2018.
  23. Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pp. 4651-4664. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/jaegle21a.html .
  24. Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-tts: A denoising diffusion model for text-to-speech. arXiv preprint arXiv:2104.01409 , 2021. · arXiv
  25. Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems , 33:8067-8077, 2020.
  26. Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning , pp. 5530-5540. PMLR, 2021.
  27. Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello. Crepe: A convolutional representation for pitch estimation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 161-165. IEEE, 2018.
  28. Minchan Kim, Myeonghun Jeong, Byoung Jin Choi, Sunghwan Ahn, Joun Yeop Lee, and Nam Soo Kim. Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpus. In Proc. Interspeech 2022 , pp. 788-792, 2022. doi: 10.21437/Interspeech. 2022-225. · DOI
  29. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
  30. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems , 33:17022-17033, 2020.
  31. Juheon Lee, Hyeong-Seok Choi, Chang-Bin Jeon, Junghyun Koo, and Kyogu Lee. Adversarially trained end-to-end korean singing voice synthesis system. In Gernot Kubin and Zdravko Kacic (eds.), Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019 , pp. 2588-2592. ISCA, 2019. doi: 10.21437/Interspeech.2019-1722. URL https://doi.org/10.21437/Interspeech. 2019-1722 . · DOI
  32. Yist Y Lin, Chung-Ming Chien, Jheng-Hao Lin, Hung-yi Lee, and Lin-shan Lee. Fragmentvc: Anyto-any voice conversion by end-to-end extracting and fusing fine-grained voice fragments with attention. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 5939-5943. IEEE, 2021.
  33. Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pp. 11020-11028, 2022.
  34. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. · arXiv
  35. Matthias Mauch and Simon Dixon. pyin: A fundamental frequency estimator using probabilistic threshold distributions. In 2014 ieee international conference on acoustics, speech and signal processing (icassp) , pp. 659-663. IEEE, 2014.
  36. Dongchan Min, Dong Bok Lee, Eunho Yang, and Sung Ju Hwang. Meta-stylespeech: Multi-speaker adaptive text-to-speech generation. In International Conference on Machine Learning , pp. 77487759. PMLR, 2021.
  37. Max Morrison, Rithesh Kumar, Kundan Kumar, Prem Seetharaman, Aaron Courville, and Yoshua Bengio. Chunked autoregressive GAN for conditional waveform synthesis. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum? id=v3aeIsY\_vVX .
  38. Koji Okabe, Takafumi Koshinaka, and Koichi Shinoda. Attentive statistics pooling for deep speaker embedding. In Proc. Interspeech 2018 , pp. 2252-2256, 2018. doi: 10.21437/Interspeech. 2018-993. · DOI
  39. Olivier Perrotin, Hussein Amouri, G´ erard Bailly, and Thomas Hueber. Evaluating the extrapolation capabilities of neural vocoders to extreme pitch values. In Interspeech 2021 , pp. 11-15. ISCA, 2021.
  40. Kaizhi Qian, Yang Zhang, Heting Gao, Junrui Ni, Cheng-I Lai, David Cox, Mark HasegawaJohnson, and Shiyu Chang. Contentvec: An improved self-supervised speech representation by disentangling speakers. In International Conference on Machine Learning , pp. 18003-18017. PMLR, 2022.
  41. Hubert Siuzdak, Piotr Dura, Pol van Rijn, and Nori Jacoby. Wavthruvec: Latent speech representation as intermediate features for neural speech synthesis. arXiv preprint arXiv:2203.16930 , 2022. · arXiv
  42. Daisy Stanton, Matt Shannon, Soroosh Mariooryad, RJ Skerry-Ryan, Eric Battenberg, Tom Bagby, and David Kao. Speaker generation. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 7897-7901. IEEE, 2022.
  43. Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. Demand: a collection of multi-channel recordings of acoustic noise in diverse environments. In Proc. Meetings Acoust , pp. 1-6, 2013.
  44. Joseph Turian and Max Henry. I'm sorry for your loss: Spectrally-based audio distances are bad at pitch. arXiv preprint arXiv:2012.04572 , 2020. · arXiv
  45. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.
  46. Xin Wang, Shinji Takaki, and Junichi Yamagishi. Neural source-filter waveform models for statistical parametric speech synthesis. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 28:402-415, 2019.
  47. Yu Wang, Xinsheng Wang, Pengcheng Zhu, Jie Wu, Hanzhao Li, Heyang Xue, Yongmao Zhang, Lei Xie, and Mengxiao Bi. Opencpop: A high-quality open source chinese popular song corpus for singing voice synthesis. arXiv preprint arXiv:2201.07429 , 2022. · arXiv
  48. Ron J Weiss, RJ Skerry-Ryan, Eric Battenberg, Soroosh Mariooryad, and Diederik P Kingma. Wavetacotron: Spectrogram-free end-to-end text-to-speech synthesis. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 5679-5683. IEEE, 2021.
  49. Mirjam Wester, Zhizheng Wu, and Junichi Yamagishi. Analysis of the voice conversion challenge 2016 evaluation results. In Interspeech , pp. 1637-1641, 2016.
  50. Junichi Yamagishi, Xin Wang, Massimiliano Todisco, Md Sahidullah, Jose Patino, Andreas Nautsch, Xuechen Liu, Kong Aik Lee, Tomi Kinnunen, Nicholas Evans, and H´ ector Delgado. Asvspoof 2021: accelerating progress in spoofed and deepfake speech detection. In Proc. 2021 Edition of the Automatic Speaker Verification and Spoofing Countermeasures Challenge , pp. 47-54, 2021. doi: 10.21437/ASVSPOOF.2021-8. · DOI · Paper page
  51. Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 6199-6203. IEEE, 2020.
  52. Dacheng Yin, Xuanchi Ren, Chong Luo, Yuwang Wang, Zhiwei Xiong, and Wenjun Zeng. Retriever: Learning content-style representation as a token-level bipartite graph. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum? id=AXWygMvuT6Q .

Metadata

Authors
Choi, H.-S., Yang, J., Lee, J., & Kim, H.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2211.09407
arXiv
2211.09407
OpenAlex
W4309395027
Cited by
20
References
0
Citations by year
2025: 8 · 2024: 8 · 2023: 4
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-sa/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
Keywords
speech synthesis, voice conversion, text-to-speech, singing voice synthesis, voice design, self-supervised learning, data efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis. https://app.cochat.ai/papers/10.48550/arxiv.2211.09407
The paper
Choi, H.-S., Yang, J., Lee, J., & Kim, H. (2022). NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis. arXiv. https://doi.org/10.48550/arxiv.2211.09407