Tacotron: Towards End-to-End Speech Synthesis
The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module.Building these components often requires extensive domain expertise and may contain brittle design choices.In this paper, we present Tacotron, an end-to-end generative text-to-speech model that synthesizes speech directly from characters.Given pairs, the model can be trained completely from scratch with random initialization.We present several key techniques to make the sequence-tosequence framework perform well for this challenging task.Tacotron achieves a 3.82 subjective 5-scale mean opinion score on US English, outperforming a production parametric system in terms of naturalness.In addition, since Tacotron generates speech at the frame level, it's substantially faster than sample-level autoregressive methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The frameworks including their code as well as the datasets used that are available publicly (Sotelo et al., 2017; Wang et al., 2017; Ping et al., 2018; Shen et al., 2018; Vasquez and Lewis, 2019) are more likely to be used for nefarious purposes or research, so they will be more developed.”
From Audio deepfakes: A survey (2023) · cites this paper as Wang et al., 2017
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“We further pass the processed sequence to a few fixed-width 1-D convolutions, whose outputs are added with the original input sequence via residual connections [17].”
BackgroundAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 2×
“We use the Adam optimizer (Kingma & Ba, 2015) with learning rate decay, which starts from 0.001 and is reduced to 0.0005, 0.0003, and 0.0001 after 500K, 1M and 2M global steps, respectively.”
MethodInfluentialBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 2×
“Batch normalization [18] is used for all convolutional layers.”
MethodSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“In this paper, we propose Tacotron, an end-to-end generative TTS model based on the sequence-to-sequence (seq2seq) (Sutskever et al., 2014) with attention paradigm (Bahdanau et al., 2014).”
MethodSelf-citationNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“The backbone of Tacotron is a seq2seq model with attention [7, 14].”
MethodInfluentialEmpirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · cited 7×
“Second, we compare with a model with the CBHG encoder replaced by a 2-layer residual GRU encoder.”
BackgroundInfluentialTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · cited 2×
“We implemented Griffin-Lim in TensorFlow [22] hence it’s also part of the model.”
MethodSelf-citationGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 4×
“C L ] 6 A pr or machine translation (Wu et al., 2016), TTS outputs are continuous, and output sequences are usually much longer than those of the input.”
BackgroundMethodInfluentialSelf-citationWaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · cited 4×
“It could be used to predict alternative targets such as vocoder parameters, or as a WaveNetlike neural vocoder [9, 13, 10] that synthesizes waveform samples directly.”
BackgroundMethodInfluentialListen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226 · cited 2×
“Moreover, unlike end-to-end speech recognition [4] or machine translation [5], TTS outputs are continuous, and output sequences are usually much longer than those of the input.”
BackgroundSelf-citation
Cited by
Items 301–400 of 1,002
Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learning
Liu, X., Iqbal, T., Zhao, J., Huang, Q., Plumbley, M. D., & Wang, W. (2021) · Machine learning for signal processing ... · doi:10.1109/mlsp52302.2021.9596430 · Cited by 56
Harnessing the Power of Artificial Intelligence in Otolaryngology and the Communication Sciences
Wilson, B. S., Tucci, D. L., Moses, D. A., Chang, E. F., Young, N. M., Zeng, F., Lesica, N. A., Bur, A. M., Kavookjian, H. L., Mussatto, C. C., et al. (2022) · Journal of the Association for Research in Otolaryngology · doi:10.1007/s10162-022-00846-2 · Cited by 40
“…have developed “textto-speech” (TTS) approaches involving the synthesis of speech directly from text, with some approaches using AI methods to enable generation of naturalistic speech waveforms (Dutoit 1997; Ze et al. 2013 ; Oord et al. 2016; Wang et al. 2017 ; Shen et al. 2018; Ning et al. 2019).”
MethodFully Automated End-to-End Fake Audio Detection
Wang, C., Yi, J., Tao, J., Sun, H., Chen, X., Tian, Z., Ma, H., Fan, C., & Fu, R. (2022) · Proceedings of the 1st International Workshop on Deepfake Detection for Audio Multimedia · doi:10.1145/3552466.3556530 · Cited by 30
DiscreTalk: Text-to-Speech as a Machine Translation ProblemPreprint
Hayashi, T., & Watanabe, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.05525 · Cited by 38
“With the improvement of deep learning techniques, the presence of end-to-end text-to-speech (E2E-TTS) models has been growing not only in the research field but also as a production system [1]–[6].”
BackgroundLiu, R., Hu, Y., Zuo, H., Luo, Z., Wang, L., & Gao, G. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3348762 · Cited by 36
Improving Sequence-to-sequence Voice Conversion by Adding Text-supervision
Zhang, J.-X., Ling, Z.-H., Jiang, Y., Liu, L., Chen, L., & Dai, L.-R. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682380 · Cited by 44
“The SCENT followed the encoder-decoder with attention architecture [11–15].”
MethodGANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech Synthesis
Yang, J., Bae, J.-S., Bak, T., Kim, Y.-I., & Cho, H. Y. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-971 · Cited by 46
Unsupervised Text-to-Speech Synthesis by Unsupervised Automatic Speech Recognition
Ni, J., Wang, L., Gao, H., Qian, K., Zhang, Y., Chang, S., & Hasegawa‐Johnson, M. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-816 · Cited by 36
Daft-Exprt: Cross-Speaker Prosody Transfer on Any Text for Expressive Speech Synthesis
Zaïdi, J., Seuté, H., van Niekerk, B., & Carbonneau, M. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10761 · Cited by 49
“The first wave of neural text-to-speech systems showed the ability to synthesize natural-sounding speech [1–3].”
BackgroundGenerative adversarial network-based approach to signal reconstruction from magnitude spectrogram
Oyamada, K., Kameoka, H., Kaneko, T., Tanaka, K., Hojo, N., & Ando, H. (2018) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2018.8553396 · Cited by 36
ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformer
Liu, H., Huang, R., Lin, X., Xu, W., Zheng, M., Chen, H., He, J., & Zhao, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.990 · Cited by 42
Short-Term Precipitation Prediction with Skip-Connected PredNet
Sato, R., Kashima, H., & Yamamoto, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01424-7_37 · Cited by 38
Multi-speaker Emotional Acoustic Modeling for CNN-based Speech Synthesis
Choi, H., Park, S., Park, J., & Hahn, M. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683682 · Cited by 35
Sequence to Sequence Neural Speech Synthesis with Prosody Modification Capabilities
Shechtman, S., & Sorin, A. (2019) · ISCA Speech Synthesis Workshop (SSW) · doi:10.21437/ssw.2019-49 · Cited by 35
Latif, S., Rana, R. K., Khalifa, S., Jurdak, R., & Schuller, B. W. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-3190 · Cited by 44
Paul, D., Pantazis, Y., & Stylianou, Y. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2786 · Cited by 45
Zhang, G., Qin, Y. H., Zhang, W., Wu, J., Li, M., Gai, Y., Jiang, F., & Lee, T. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3268571 · Cited by 45
“In recent years, TTS systems have been predominantly based on deep neural network models [6]–[9].”
BackgroundNU-GAN: High resolution neural upsampling with GANPreprint
Kumar, R., Kumar, K., Anand, V., Bengio, Y., & Courville, A. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11362 · Cited by 35
“Progress over the past few years in deep learning based techniques for speech generative modeling [1, 2, 3, 4, 5] has allowed many practical applications of generative speech technology, such as text-to-speech (TTS) for powering voice assistants and characters in video games, audio denoising,…”
Lip to Speech Synthesis with Visual Context Attentional GANPreprint
Kim, M., Hong, J. H., & Ro, Y. M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.01726 · Cited by 54
“Since the Griffin-Lim algorithm transforms linear spectrogram to waveform, we use additional postnet which learns to map the mel-spectrogram to linear spectrogram similar to [47, 2].”
MethodAssistive systems for visually impaired people: A survey on current requirements and advancements
Kathiria, P., Mankad, S. H., Patel, J., Kapadia, M., & Lakdawala, N. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128284 · Cited by 46
Improving Performance of End-to-End ASR on Numeric Sequences
Peyser, C., Zhang, H., Sainath, T. N., & Wu, Z. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1345 · Cited by 34
One-Shot Voice Conversion with Global Speaker Embeddings
Lu, H., Wu, Z., Dai, D., Li, R., Kang, S., Jia, J., & Meng, H. M. L. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2365 · Cited by 47
“[11] Y. Wang, R. J. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. V. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous, “Tacotron: Towards end-to-end speech synthesis,” in INTERSPEECH, 2017.”
BackgroundOn the localness modeling for the self-attention based end-to-end speech synthesis
Yang, S., Lu, H., Kang, S., Xue, L., Xiao, J., Su, D., Xie, L., & Yu, D. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.01.034 · Cited by 34
“Since the above modules are correlative but trained independently, errors from each individual component may compound (Wang et al., 2017).”
BackgroundMethodInfluentialData Efficient Voice Cloning from Noisy Samples with Domain Adversarial Training
Cong, J., Yang, S., Xie, L., Yu, G., & Wan, G. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2530 · Cited by 41
Liu, R., Şişman, B., Bao, F., Yang, J., Gao, G., & Li, H. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3040523 · Cited by 40
“Nonetheless,somenewlydevelopedspeech synthesissystems,suchasTacotron[15]–[21],WaveNet-based approaches[22]–[28],andDeepVoice[29]havenotspecifi-callymodeledprosodiccuesfrominputtext.”
BackgroundDidispeech: A Large Scale Mandarin Speech Corpus
Guo, T., Cheng, W., Jiang, D., Luo, N., Zhang, R., Zhao, S., Li, W., Cheng, G., Zou, W., Han, K., et al. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414423 · Cited by 76
“The focus of speech synthesis is changing from synthesizing humanlike speech of a given target speaker to more complex tasks, such as voice conversion (VC) [13, 14] and multi-speaker TTS [15].”
BackgroundInfluentialTowards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language ModelsPreprint
Fang, W., Chung, Y.-A., & Glass, J. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.07307 · Cited by 34
“This pattern is also observed in the original Tacotron [1], as well as the baseline Tacotron-2 model.”
BackgroundMethodReview of end-to-end speech synthesis technology based on deep learningPreprint
Mu, Z., Yang, X., & Dong, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.09995 · Cited by 46
“Although there have been some reviews on TTS methods based on deep learning, only some of baseline models have been introduced, such as WaveNet [151], Tacotron [227] and SampleRNN [136].”
BackgroundInfluentialMake-A-Voice: Unified Voice Synthesis With Discrete RepresentationPreprint
Huang, R., Zhang, C., Wang, Y., Yang, D., Liu, L., Ye, Z., Jiang, Z. K., Weng, C., Zhao, Z., & Yu, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.19269 · Cited by 40
“2 Related Works 2.1 Speech/Singing Synthesis TTS or SVS models typically convert input text into mel-spectrogram (e.g., Tacotron [40], Fast-Speech [33]), which is then transformed to waveform using a separately trained vocoder [20, 12], or directly generate waveform from text (e.g., EATS [10] and…”
BackgroundMethodInfluentialSelf-citationSymbolicAI: A framework for logic-based approaches combining generative models and solversPreprint
Dinu, M.-C., Leoveanu-Condrei, C., Holzleitner, M., Zellinger, W., & Hochreiter, S. (2024) · arXiv (Cornell University) · doi:10.15203/99106-150-2-24 · Cited by 52
Emergence of deepfakes and video tampering detection approaches: A survey
Kingra, S., Aggarwal, N., & Kaur, N. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13100-x · Cited by 33
Deepfakes: A Grounded Threat Assessment
Center for Security and Emerging Technology, & Hwang, T. (2020) · doi:10.51593/20190030 · Cited by 33
Implementing Prosodic Phrasing in Chinese End-to-end Speech Synthesis
Lu, Y., Dong, M., & Chen, Y. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682368 · Cited by 33
“Later, Tacotron [11] directly predicted raw spectrogram from raw text.”
BackgroundZhu, X., Zhang, Y., Yang, S., Xue, L., & Xie, L. (2019) · IEEE Access · doi:10.1109/access.2019.2914149 · Cited by 33
“Third, different model structures have been investigated; for example, Wang et al. [3] have studied the impact of the CBHG (1-”
MethodInfluentialAtt-HACK: An Expressive Speech Database with Social Attitudes
Veillon, C. L. M., & Obin, N. (2020) · Speech prosody · doi:10.21437/speechprosody.2020-152 · Cited by 42
Multi-Reference Neural TTS Stylization with Adversarial Cycle Consistency
Whitehill, M., Ma, S., McDuff, D., & Song, Y. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2985 · Cited by 37
Ogawa, I., & Morise, M. (2021) · Nippon Onkyo Gakkaishi/Acoustical science and technology/Nihon Onkyo Gakkaishi · doi:10.1250/ast.42.140 · Cited by 54
Zheng, Z., Ali, M., Jamei, M., Xiang, Y., Karbasi, M., Yaseen, Z. M., & Farooque, A. A. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.105984 · Cited by 33
EfficientTTS: An Efficient and High-Quality Text-to-Speech ArchitecturePreprint
Miao, C., Liang, S., Liu, Z., Chen, M., Ma, J., Wang, S., & Xiao, J. Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.03500 · Cited by 34
“The most popular neural TTS models are autoregressive models based on an encoder-decoder framework (Wang et al., 2017; Shen et al.; Ping et al., 2018; 2019; Li et al., 2019; Valle et al., 2021).”
MethodExpressive Neural Voice CloningPreprint
Neekhara, P., Hussain, S., Dubnov, S., Koushanfar, F., & McAuley, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.00151 · Cited by 50
“Style conditioning in expressive TTS models is popularly done by learning a dictionary of latent style vectors called Global Style Tokens (GST) Wang et al. (2018). While GSTs can learn meaningful latent codes when trained on a dataset with high variation in expressions, we empirically find that it offers limited style…”
BackgroundMethodInfluentialMultimodal Forgery Detection Using Ensemble Learning
Hashmi, A., Shahzad, S. A., Ahmad, W., Lin, C. W., Tsao, Y., & Wang, H.-M. (2022) · 2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipaasc55919.2022.9980255 · Cited by 32
“However, it has been reported that current SV systems are easy to deceive by manipulating the audio signal [39] [40].”
BackgroundAn Emotion Speech Synthesis Method Based on VITS
Zhao, W., & Yang, Z. (2023) · Applied Sciences · doi:10.3390/app13042225 · Cited by 43
Self-citationEarmonitor: In-ear Motion-resilient Acoustic Sensing Using Commodity Earphones
Sun, X. Z., Xiong, J., Feng, C., Deng, W., Wei, X., Fang, D., & Chen, X. (2022) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3569472 · Cited by 36
“To further understand the effect of chirps on music, we conduct a subjective evaluation and adopt the mean opinion score (MOS) [41] as the metric to evaluate the users’ opinions on music quality change.”
MethodUnsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion
Liu, A. T., Hsu, P.-C., & Lee, H.-Y. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2048 · Cited by 36
“The ASR-Encoder is inspired by the CBHG module [1], where the linear output of ASR-Encoder is fed to the MBV encoding module.”
BackgroundMethodEnd-to-end Feedback Loss in Speech Chain Framework via Straight-through Estimator
Tjandra, A., Sakti, S., & Nakamura, S. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683480 · Cited by 45
Forward–Backward Decoding Sequence for Regularizing End-to-End TTS
Zheng, Y., Tao, J., Wen, Z., & Yi, J. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2935807 · Cited by 33
“[17]–[20] are proposed to simplify traditional TTS pipeline [3], [6], [11], [12] with a single neural network.”
BackgroundMethodGraphspeech: Syntax-Aware Graph Attention Network for Neural Speech Synthesis
Liu, R., Şişman, B., & Li, H. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413513 · Cited by 32
Cross-Speaker Style Transfer with Prosody Bottleneck in Neural Speech Synthesis
Pan, S., & He, L. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-979 · Cited by 32
Abootorabi, M. M., Ghahroodi, O., Zahraei, P. S., Behzadasl, H., Mirrokni, A., Salimipanah, M., Rasouli, A., Behzadipour, B., Azarnoush, S., Maleki, B., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.19056 · Cited by 47
Development and Evaluation of a Learning-Based Model for Real-Time Haptic Texture Rendering
Heravi, N., Culbertson, H., Okamura, A. M., & Bohg, J. (2024) · IEEE Transactions on Haptics · doi:10.1109/toh.2024.3382258 · Cited by 35
“Furthermore, we were inspired by the literature in the neural network-based audio generation community that predict the signal in the frequency domain followed by a wave reconstruction algorithm instead of predicting the wave form directly [36].”
Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generation
He, H., Shang, Z., Wang, C., Li, X., Gu, Y., Hua, H., Liu, L., Yang, C., Li, J., Shi, P., et al. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3612835 · Cited by 50
“Traditional speech generation models, such as Tacotron [25], FastSpeech [26], are limited by smaller datasets with only tens of hours of speech data.”
Self-citationWang, X., & Yamagishi, J. (2019) · ISCA Speech Synthesis Workshop (SSW) · doi:10.21437/ssw.2019-1 · Cited by 31
Tomashenko, N., Caubrière, A., & Estève, Y. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2158 · Cited by 40
“In the recent years, there has be a great interest of the research community in end-to-end systems for various speech and language technologies, such as ASR [1, 2, 3, 4], text-tospeech synthesis [5], machine translation [6], speaker verification [7] and many others.”
BackgroundTjandra, A., Sakti, S., & Nakamura, S. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.2977776 · Cited by 31
“to the earlier version, it consists of a sequence-to-sequence ASR [24], [46], a sequence-to-sequence TTS [25], and a loop connection from ASR to TTS and from TTS to ASR.”
BackgroundMethodInfluentialIncremental Text-to-Speech Synthesis with Prefix-to-Prefix Framework
Ma, M., Zheng, B., Liu, K., Zheng, R., Liu, H., Peng, K., Church, K., & Huang, L. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.346 · Cited by 31
Lee, K., Park, K., & Kim, D. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-838 · Cited by 39
S3PRL-VC: Open-Source Voice Conversion Framework with Self-Supervised Speech Representations
Huang, W.-C., Yang, S.-W., Hayashi, T., Lee, H.-Y., Watanabe, S., & Toda, T. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746430 · Cited by 45
Zheng, W.-Z., Han, J.-Y., Lee, C.-K., Lin, Y.-Y., Chang, S.-H., & Lai, Y.-H. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106602 · Cited by 31
Learning to Dub Movies via Hierarchical Prosody Models
Cong, G., Li, L., Qi, Y., Zha, Z.-J., Wu, Q., Wang, W., Jiang, B., Yang, M., & Huang, Q. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01411 · Cited by 62
Unsupervised Learning for Sequence-to-Sequence Text-to-Speech for Low-Resource Languages
Zhang, H., & Lin, Y. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1403 · Cited by 41
Learning pronunciation from a foreign language in speech synthesis networksPreprint
Lee, Y., Shon, S., & Kim, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.09364 · Cited by 40
“The proposed approach outperformed other pre-training frameworks as well as the baseline Tacotron.”
BackgroundMethodInfluentialI'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at PitchPreprint
Turian, J. P., & Henry, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.04572 · Cited by 41
“In [69], the authors argue that MFCCs are somewhat pitch invariant.”
BackgroundMethodHe, P., Wen, Z., Wang, Y., Wang, Y., Liu, X., Huang, J., Lei, Z., Gu, Z., Jin, X., Yang, J., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.07293 · Cited by 38
Self-citationStreaming Sequence-to-Sequence Learning with Delayed Streams ModelingPreprint
Zeghidour, N., Kharitonov, E., Orsini, M., Volhejn, V., de Marmiesse, G., Grave, É., Pérez, P., Mazaré, L., & Défossez, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.08753 · Cited by 38
“As an example, a Tacotron text-to-speech (TTS) model (Wang et al., 2017) would differ from an ASR model such as LAS (Chan et al., 2016).”
Combining Automatic Speaker Verification and Prosody Analysis for Synthetic Speech Detection
Attorresi, L., Salvi, D., Borrelli, C., Bestagini, P., & Tubaro, S. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-37742-6_21 · Cited by 30
DialogueNeRF: towards realistic avatar face-to-face conversation video generation
Yan, Y., Zhou, Z., Wang, Z., Gao, J., & Yang, X. (2024) · Visual Intelligence · doi:10.1007/s44267-024-00057-8 · Cited by 46
“In the area of natural language processing (NLP), there have been some classic methods that are able to generate realistic textual communication processes [22,26,45], and combine speech synthesis models [30, 46] to generate realistic vocal communication processes.”
Tyagi, S., Nicolis, M., Rohnke, J., Drugman, T., & Lorenzo-Trueba, J. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1411 · Cited by 30
High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency
Ellinas, N., Vamvoukakis, G., Μαρκόπουλος, Κ., Chalamandaris, A., Maniati, G., Kakoulidis, P., Raptis, S., Sung, J. S., Park, H.-M., & Tsiakoulis, P. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2464 · Cited by 30
“It is an attention based sequenceto-sequence model and a direct modification of Tacotron 1 and 2 [1][4].”
MethodInfluentialAccent and Speaker Disentanglement in Many-to-many Voice Conversion
Wang, Z., Ge, W., Wang, X., Yang, S., Gan, W., Chen, H., Li, H. H., Xie, L., & Li, X. (2021) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp49672.2021.9362120 · Cited by 30
“Different from Tacotron [22], we use the duration from the ASR model for frame expansion instead of attention-based soft alignment.”
MethodTTS-by-TTS: TTS-Driven Data Augmentation for Fast and High-Quality Speech Synthesis
Hwang, M.-J., Yamamoto, R., Song, E., & Kim, J.-M. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414408 · Cited by 47
Controllable Context-Aware Conversational Speech Synthesis
Cong, J., Yang, S., Hu, N., Li, G., Xie, L., & Su, D. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-412 · Cited by 30
Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Features
Lux, F., & Vu, T. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.472 · Cited by 45
Tell Me Where To Go: Voice-Controlled Hands-Free Locomotion for Virtual Reality Systems
Hombeck, J., Voigt, H., Heggemann, T., Datta, R. R., & Lawonn, K. (2023) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr55154.2023.00028 · Cited by 40
“Advances in natural language processing (NLP) technologies in recent years mostly based on deep learning [71] have increased the capabilities of speech interfaces to understand spoken text [6, 84,88], process natural language [25,80,105], generate text [26, 52, 78] and generate spoken words [45, 91, 101].”
BackgroundThe limits of the Mean Opinion Score for speech synthesis evaluation
Maguer, S. L., King, S., & Harte, N. (2023) · Computer Speech & Language · doi:10.1016/j.csl.2023.101577 · Cited by 67
Cycle consistent network for end-to-end style transfer TTS training
Xue, L., Pan, S., He, L., Xie, L., & Soong, F. K. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.03.005 · Cited by 31
“The KL annealing strategy used in Zhang et al. (2019) is also applied in our task to solve the collapse problem of KL loss.”
MethodLiu, Z., & Mak, B. K.-W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.11601 · Cited by 32
“In contrast, Char2wav [3] and Tacotron [4] and its improved version Tacotron2 [5] resort to a totally end-to-end neural model1 that use an attention mechanism to convert a sequence of text directly to its corresponding sequence of vocoder features, from which speech audios may be generated using a vocoder.”
InfluentialInvestigating Affective Use and Emotional Well-being on ChatGPT
Phang, J., Lampe, M., Ahmad, L., Agarwal, S., Fang, C. M., Liu, A. R., Danry, V., Lee, E., Chan, S. W. T., Pataranutaporn, P., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.03888 · Cited by 29
Exploiting Syntactic Features in a Parsed Tree to Improve End-to-End TTS
Guo, H., Soong, F. K., He, L., & Xie, L. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2167 · Cited by 29
On the Resilience of Biometric Authentication Systems against Random Inputs
Zhao, B. Z. H., Asghar, H. J., & Kâafar, M. A. (2020) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2020.24210 · Cited by 34
“For example, spoofing detection [52] is an active area in detecting speaker style transfer [53].”
BackgroundImproving End-to-End Speech Synthesis with Local Recurrent Neural Network Enhanced Transformer
Zheng, Y., Li, X., Xie, F., & Lü, L. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054148 · Cited by 29
“…traditional statistical parametric speech synthesis [4] which has a text frontend for various linguistic features extraction, end-to-end neural TTS [5, 6, 7, 8, 9, 10, 11] has grown active due to its simpler module and strong capabilities to produce highly natural speech without extensive domain…”
BackgroundHierarchical Multi-Grained Generative Model for Expressive Speech Synthesis
Hono, Y., Tsuboi, K., Sawada, K. B., Hashimoto, K., Oura, K., Nankaku, Y., & Tokuda, K. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2477 · Cited by 29
PPG-Based Singing Voice Conversion with Adversarial Representation Learning
Li, Z., Tang, B., Yin, X., Wan, Y., Xu, L., Shen, C., & Ma, Z. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414137 · Cited by 47
Prosospeech: Enhancing Prosody with Quantized Vector Pre-Training in Text-To-Speech
Ren, Y., Lei, M., Huang, Z., Zhang, S., Chen, Q., Yan, Z., & Zhao, Z. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746883 · Cited by 45
A ResNet attention model for classifying mosquitoes from wing-beating sounds
Wei, X., Hossain, M. Z., & Ahmed, K. A. (2022) · Scientific Reports · doi:10.1038/s41598-022-14372-x · Cited by 31
Rekimoto, J. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580706 · Cited by 43
Deep Fake Generation and Detection: Issues, Challenges, and Solutions
Salman, S., Shamsi, J. A., & Qureshi, R. (2023) · IT Professional · doi:10.1109/mitp.2022.3230353 · Cited by 34
“Applications, such as Google Tacotron2 14 and DeepVoice, 15 are used to create synthetic speech/voices.”
MethodLiu, R., Hu, Y., Ren, Y., Yin, X., & Li, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29833 · Cited by 49
Towards Robust Neural Vocoding for Speech Generation: A Survey
Hsu, P.-C., Wang, C.-H., Liu, A. T., & Lee, H.-Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.02461 · Cited by 21
“Most speech generation models, such as text-to-speech [1, 2, 3, 4] and voice conversion [5, 6, 7, 8], do not output waveform directly.”
BackgroundA comprehensive survey with critical analysis for deepfake speech detection
Pham, L. D., Lam, P., Tran, D., Tang, H., Nguyen, T., Schindler, A., Skopik, F., Polonsky, A., & Vu, H. C. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100757 · Cited by 43
Li, C.-Y., Yuan, P.-C., & Lee, H.-Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054675 · Cited by 37
“We avoid using attention-based models like Tacotron [9] where every output frame has access to the whole input signal.”
MethodInfluentialImproving the Prosody of RNN-Based English Text-To-Speech Synthesis by Incorporating a BERT Model
Kenter, T., Sharma, M. K., & Clark, R. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1430 · Cited by 45
“One of the reasons for this is that the input to speech synthesis systems typically consists of only text, read as characters or words [1, 2].”
BackgroundSelf-citationConverting Foreign Accent Speech Without a Reference
Zhao, G., Ding, S., & Gutiérrez‐Osuna, R. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3060813 · Cited by 46
“Therefore, the PostNet can use both past and future information to correct the prediction error for each individual frame [49].”
MethodAccentron: Foreign accent conversion to arbitrary non-native speakers using zero-shot learning
Ding, S., Zhao, G., & Gutiérrez‐Osuna, R. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101302 · Cited by 39
“[68] first proposed a seq2seq based TTS synthesizer (Tacotron), which significantly improved the acoustic quality of the syntheses over previous methods.”
MethodJain, R., Yiwere, M. Y., Bigioi, D., Corcoran, P., & Cucu, H. (2022) · IEEE Access · doi:10.1109/access.2022.3170836 · Cited by 31
Audio Splicing Detection and Localization Based on Acquisition Device Traces
Leonzio, D. U., Cuccovillo, L., Bestagini, P., Marcon, M., Aichroth, P., & Tubaro, S. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3293415 · Cited by 28
“This is especially true in the field of audio recordings, where the latest developed technologies can synthesize very realistic voices [1], clone voices with few training samples [2], or fill in small audio gaps controlling the synthesized words [3].”
BackgroundMethodRMSSinger: Realistic-Music-Score based Singing Voice Synthesis
He, J., Liu, J., Ye, Z., Huang, R., Cui, C., Liu, H., & Zhao, Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.16 · Cited by 37
SAPIEN: Affective Virtual Agents Powered by Large Language Models*
Hasan, M., Ozel, C., Potter, S., & Hoque, M. E. (2023) · International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) · doi:10.1109/aciiw59127.2023.10388188 · Cited by 42
“SAPIEN leverages state-of-the-art models in Speech-to-Text [5], [6], Text-to-Speech [7]–[9], and large language modeling [2], [4], [10]–[14].”
BackgroundAccented Text-to-Speech Synthesis With Limited Data
Zhou, X., Zhang, M., Zhou, Y., Wu, Z., & Li, H. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3363414 · Cited by 43
“STATE-of-the-art neural end-to-end (E2E) text-to-speech (TTS) systems [1]–[5] can generate human-like natural”
BackgroundGraph Neural Network for Music Score Data and Modeling Expressive Piano Performance
Jeong, D., Kwon, T., Kim, Y., & Nam, J. (2019) · International Conference on Machine Learning · Cited by 110
Unpaired Image-to-Speech Synthesis With Multimodal Information Bottleneck
Ma, S., McDuff, D., & Song, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00769 · Cited by 38
“Text → Speech: We compare with four text-to-speech (TTS) models: Tacotron [49], Tacotron 2 [38], DeepVoice3 [35], and GST [50].”
MethodInfluential
Metadata
- Authors
- Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., Le, Q., Agiomyrgiannakis, Y., Clark, R., & Saurous, R. A.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Wang, Yuxuan
- ORCID
- Venue
- Interspeech 2017
- Publisher
- ISCA
- Published
- 2017-08-20
- Pages
- 4006-4010
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2963609956
- Cited by
- 1,746
- References
- 25
- Citations by year
- 2026: 134 · 2025: 187 · 2024: 183 · 2023: 215 · 2022: 243 · 2021: 301 · 2020: 252 · 2019: 176 · 2018: 45 · 2017: 9
- FWCI
- 60.12
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
- Keywords
- Tacotron, end-to-end speech synthesis, text-to-speech, sequence-to-sequence models, naturalness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.