CoChat
Sign Up
PreprintCited by 580Speech Recognition and SynthesisSpeech and Audio ProcessingVoice and Speech Disorders

FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

Open access
Ren, Yi, Hu, Chenxu, Tan, Xu, Qin, Tao, Zhao, Sheng, Zhao, Zhou, Liu, Tie-Yan·2020·arXiv
DOI 10.48550/arxiv.2006.04558arXiv 2006.04558
Open this paper in CoChat — free accountOpen PDF

Abstract

Non-autoregressive text to speech (TTS) models such as FastSpeech can synthesize speech significantly faster than previous autoregressive models with comparable quality. The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated and time-consuming, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference. We further design FastSpeech 2s, which is the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of fully end-to-end inference. Experimental results show that 1) FastSpeech 2 achieves a 3x training speed-up over FastSpeech, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models. Audio samples are available at https://speechresearch.github.io/fastspeech2/.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.04558

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • End-to-end models such as DeepVoice 3, Tacotron 2, and FastSpeech 2 significantly reduced reliance on linguistic knowledge, lowering technical barriers to high-quality speech synthesis.

    “These models significantly reduced reliance on linguistic knowledge, thereby effectively lowering the technical barriers to high-quality speech synthesis.”

    From Audio Deepfake Detection: What Has Been Achieved and What Lies Ahead (2025) · cites this paper as [43]

    The Introduction (p. 2) and Conclusion (p. 9) describe FastSpeech 2 as achieving high voice quality with a simpler training pipeline. They do not say that FastSpeech 2, DeepVoice 3, and Tacotron 2 reduced reliance on linguistic knowledge or lowered technical barriers. The Conclusion also notes that FastSpeech 2 requires external alignment and pitch-extraction tools that may be complicated. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. Bistra Andreeva, Gra˙ zyna Demenko, Bernd M¨ obius, Frank Zimmerer, Jeanin J¨ ugler, and Magdalena Oleskowicz-Popiel. Differences of pitch profiles in germanic and slavic languages. In Fifteenth Annual Conference of the International Speech Communication Association , 2014.
  2. Sercan O Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman, et al. Deep voice: Real-time neural text-to-speech. arXiv preprint arXiv:1702.07825 , 2017. · arXiv
  3. Mingjian Chen, Xu Tan, Yi Ren, Jin Xu, Hao Sun, Sheng Zhao, and Tao Qin. Multispeech: Multispeaker text to speech with transformer. In INTERSPEECH , pp. 4024-4028, 2020.
  4. Mingjian Chen, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, sheng zhao, and Tie-Yan Liu. Adaspeech: Adaptive text to speech for custom voice. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=Drynvt7gg4L .
  5. Jeff Donahue, Sander Dieleman, Mikołaj Bi´ nkowski, Erich Elsen, and Karen Simonyan. End-to-end adversarial text-to-speech. arXiv preprint arXiv:2006.03575 , 2020. · arXiv
  6. Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, and Adam Roberts. Ddsp: Differentiable digital signal processing. arXiv preprint arXiv:2001.04643 , 2020. · arXiv
  7. Yuchen Fan, Yao Qian, Feng-Long Xie, and Frank K Soong. Tts synthesis with bidirectional lstm based recurrent neural networks. In Fifteenth Annual Conference of the International Speech Communication Association , 2014.
  8. Michael Gadermayr, Maximilian Tschuchnig, Dorit Merhof, Nils Kr¨ amer, Daniel Truhn, and Burkhard Gess. An asymetric cycle-consistency loss for dealing with many-to-one mappings in image translation: A study on thigh mr scans. arXiv preprint arXiv:2004.11001 , 2020. · arXiv
  9. Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. Deep voice 2: Multi-speaker neural text-to-speech. In Advances in neural information processing systems , pp. 2962-2970, 2017.
  10. Alexander Grossmann and Jean Morlet. Decomposition of hardy functions into square integrable wavelets of constant shape. SIAM journal on mathematical analysis , 15(4):723-736, 1984.
  11. Keikichi Hirose and Jianhua Tao. Speech Prosody in Speech Synthesis: Modeling and generation of prosody for high quality and flexible speech synthesis . Springer, 2015.
  12. Chrisina Jayne, Andreas Lanitis, and Chris Christodoulou. One-to-many neural network mapping techniques for face image synthesis. Expert Systems with Applications , 39(10):9778-9787, 2012.
  13. Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. arXiv preprint arXiv:2005.11129 , 2020. · arXiv
  14. Sungwon Kim, Sang-gil Lee, Jongyoon Song, Jaehyeon Kim, and Sungroh Yoon. Flowavenet: A generative flow for raw audio. arXiv preprint arXiv:1811.02155 , 2018. · arXiv
  15. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
  16. Adrian Ła´ ncucki. Fastpitch: Parallel text-to-speech with pitch prediction. arXiv preprint arXiv:2006.06873 , 2020. · arXiv
  17. Dan Lim, Won Jang, Hyeyeong Park, Bongwan Kim, Jesam Yoon, et al. Jdi-t: Jointly trained duration informed transformer for text-to-speech without explicit alignment. arXiv preprint arXiv:2005.07799 , 2020. · arXiv
  18. Philipos C Loizou. Speech quality assessment. In Multimedia analysis, processing and communications , pp. 623-654. Springer, 2011.
  19. Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Jinzhu Li, Sheng Zhao, Enhong Chen, and Tie-Yan Liu. Lightspeech: Lightweight and fast text to speech with neural architecture search. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021.
  20. Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech , pp. 498502, 2017.
  21. Chenfeng Miao, Shuang Liang, Minchuan Chen, Jun Ma, Shaojun Wang, and Jing Xiao. Flowtts: A non-autoregressive network for text to speech based on flow. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 7209-7213. IEEE, 2020.
  22. Meinard M¨ uller. Dynamic time warping. Information retrieval for music and motion , pp. 69-84, 2007.
  23. Oliver Niebuhr and Radek Skarnitzl. Measuring a speaker's acoustic correlates of pitch-but which? a contrastive analysis based on perceived speaker charisma. In Proceedings of 19th International Congress of Phonetic Sciences , 2019.
  24. Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C Cobo, Florian Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. arXiv preprint arXiv:1711.10433 , 2017. · arXiv
  25. Kainan Peng, Wei Ping, Zhao Song, and Kexin Zhao. Parallel neural text-to-speech. arXiv preprint arXiv:1905.08459 , 2019. · arXiv
  26. Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: 2000-speaker neural text-to-speech. In International Conference on Learning Representations , 2018.
  27. Wei Ping, Kainan Peng, and Jitong Chen. Clarinet: Parallel wave generation in end-to-end text-tospeech. In International Conference on Learning Representations , 2019.
  28. Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. In Advances in Neural Information Processing Systems , pp. 3165-3174, 2019.
  29. Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 4779-4783. IEEE, 2018.
  30. Hao Sun, Xu Tan, Jun-Wei Gan, Hongzhi Liu, Sheng Zhao, Tao Qin, and Tie-Yan Liu. Token-level ensemble distillation for grapheme-to-phoneme conversion. In INTERSPEECH , 2019.
  31. Antti Santeri Suni, Daniel Aalto, Tuomo Raitio, Paavo Alku, Martti Vainio, et al. Wavelets for intonation modeling in hmm speech synthesis. In 8th ISCA Workshop on Speech Synthesis, Proceedings, Barcelona, August 31-September 2, 2013 . ISCA, 2013.
  32. Franz B Tuteur. Wavelet transformations in signal detection. IFAC Proceedings Volumes , 21(9): 1061-1065, 1988.
  33. Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. In Advances in neural information processing systems , pp. 4790-4798, 2016.
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , pp. 5998-6008, 2017.
  35. Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135 , 2017. · arXiv
  36. Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 6199-6203. IEEE, 2020.
  37. Heiga Ze, Andrew Senior, and Mike Schuster. Statistical parametric speech synthesis using deep neural networks. In 2013 ieee international conference on acoustics, speech and signal processing , pp. 7962-7966. IEEE, 2013.
  38. Zhen Zeng, Jianzong Wang, Ning Cheng, Tian Xia, and Jing Xiao. Aligntts: Efficient feed-forward text-to-speech system without explicit alignment. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 6714-6718. IEEE, 2020.
  39. Chen Zhang, Yi Ren, Xu Tan, Jinglin Liu, Kejun Zhang, Tao Qin, Sheng Zhao, and Tie-Yan Liu. Denoispeech: Denoising text to speech with frame-level noise modeling. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021.
  40. Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in neural information processing systems , pp. 465-476, 2017.

Cited by

Sorted by citations

Authors
Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T.-Y.
Affiliations
  • Zhejiang University (CN)
  • Microsoft Research Asia (China) (CN)
  • Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2006.04558
arXiv
2006.04558
OpenAlex
W3033411150
Cited by
580
References
4
Citations by year
2026: 3 · 2025: 19 · 2024: 53 · 2023: 190 · 2022: 164 · 2021: 137 · 2020: 14
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • TIB Data Manager
Funders
  • National Natural Science Foundation of China (62072397, 61836002)
  • Natural Science Foundation of Zhejiang Province (LR19F020006)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Health Sciences › Medicine › Physiology › Voice and Speech Disorders
Keywords
text-to-speech, non-autoregressive models, end-to-end speech synthesis, one-to-many mapping, duration prediction, teacher-student distillation, training acceleration, voice quality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review