Tacotron: Towards End-to-End Speech Synthesis
The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module.Building these components often requires extensive domain expertise and may contain brittle design choices.In this paper, we present Tacotron, an end-to-end generative text-to-speech model that synthesizes speech directly from characters.Given pairs, the model can be trained completely from scratch with random initialization.We present several key techniques to make the sequence-tosequence framework perform well for this challenging task.Tacotron achieves a 3.82 subjective 5-scale mean opinion score on US English, outperforming a production parametric system in terms of naturalness.In addition, since Tacotron generates speech at the frame level, it's substantially faster than sample-level autoregressive methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The frameworks including their code as well as the datasets used that are available publicly (Sotelo et al., 2017; Wang et al., 2017; Ping et al., 2018; Shen et al., 2018; Vasquez and Lewis, 2019) are more likely to be used for nefarious purposes or research, so they will be more developed.”
From Audio deepfakes: A survey (2023) · cites this paper as Wang et al., 2017
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“We further pass the processed sequence to a few fixed-width 1-D convolutions, whose outputs are added with the original input sequence via residual connections [17].”
BackgroundAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 2×
“We use the Adam optimizer (Kingma & Ba, 2015) with learning rate decay, which starts from 0.001 and is reduced to 0.0005, 0.0003, and 0.0001 after 500K, 1M and 2M global steps, respectively.”
MethodInfluentialBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 2×
“Batch normalization [18] is used for all convolutional layers.”
MethodSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“In this paper, we propose Tacotron, an end-to-end generative TTS model based on the sequence-to-sequence (seq2seq) (Sutskever et al., 2014) with attention paradigm (Bahdanau et al., 2014).”
MethodSelf-citationNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“The backbone of Tacotron is a seq2seq model with attention [7, 14].”
MethodInfluentialEmpirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · cited 7×
“Second, we compare with a model with the CBHG encoder replaced by a 2-layer residual GRU encoder.”
BackgroundInfluentialTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · cited 2×
“We implemented Griffin-Lim in TensorFlow [22] hence it’s also part of the model.”
MethodSelf-citationGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 4×
“C L ] 6 A pr or machine translation (Wu et al., 2016), TTS outputs are continuous, and output sequences are usually much longer than those of the input.”
BackgroundMethodInfluentialSelf-citationWaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · cited 4×
“It could be used to predict alternative targets such as vocoder parameters, or as a WaveNetlike neural vocoder [9, 13, 10] that synthesizes waveform samples directly.”
BackgroundMethodInfluentialListen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226 · cited 2×
“Moreover, unlike end-to-end speech recognition [4] or machine translation [5], TTS outputs are continuous, and output sequences are usually much longer than those of the input.”
BackgroundSelf-citation
Cited by
Items 101–200 of 1,002
Conventional and contemporary approaches used in text to speech synthesis: a review
Kaur, N., & Singh, P. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10315-0 · Cited by 139
“Launched in 2017, Tacotron uses a generative model similar to WaveNet for end-to-end speech synthesis (Wang et al. 2017a, 2017b) Given a < text, audio > pair, Tacotron model used random initialization for training (Chung et al. 2018).”
MethodSemi-supervised Training for Improving Data Efficiency in End-to-end Speech Synthesis
Chung, Y.-A., Wang, Y., Hsu, W.-N., Zhang, Y., & Skerry-Ryan, R. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683862 · Cited by 130
Self-citationLocation-Relative Attention Mechanisms for Robust Long-Form Speech Synthesis
Battenberg, E., Skerry-Ryan, R., Mariooryad, S., Stanton, D., Kao, D., Shannon, M., & Bagby, T. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054106 · Cited by 138
“The original Tacotron system [3] used the content-based attention mechanism introduced in [2] to align the target text with the output spectrogram.”
BackgroundMethodInfluentialSelf-citationAdvances in anti-spoofing: from the perspective of ASVspoof challenges
Kamble, M. R., Sailor, H. B., Patil, H. A., & Li, H. (2020) · APSIPA Transactions on Signal and Information Processing · doi:10.1017/atsip.2019.21 · Cited by 102
Zhang, J.-X., Ling, Z.-H., & Dai, L.-R. (2020) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2019.2960721 · Cited by 77
“Here, the structure of the seq2seq decoder is similar to the Tacotron model [45], [46] for speech synthesis.”
MethodDurIAN: Duration Informed Attention Network For Multimodal SynthesisPreprint
Yu, C., Lu, H., Hu, N., Yu, M., Weng, C., Xu, K., Liu, P., Tuo, D., Kang, S., Lei, G., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01700 · Cited by 105
“End-to-end approaches [6, 7, 8, 9, 10, 11] have gained much attention recently due to the clearly better naturalness of their synthesized results and the simplified training pipelines.”
BackgroundMethodInfluentialShen, J., Ye, J., Chrzanowski, M., Zhang, Y., Elias, I., Zen, H., & Wu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.04301 · Cited by 117
“This led to works such as Char2Wav (Sotelo et al., 2017), Tacotron (Wang et al., 2017; Shen et al., 2018), Deep Voice 3 (Ping et al., 2018), and Transformer TTS (Li et al., 2019), each of them autoregressive networks that predict the output one frame at a time.”
MethodSelf-citationAn Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era
Triantafyllopoulos, A., Schuller, B. W., İymen, G., Sezgin, T. M., He, X., Yang, Z., Tzirakis, P., Liu, S., Mertes, S., André, E., et al. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3250266 · Cited by 151
“This was later followed by models trying to go directly from character/phoneme sequences to audio, such as Tacotron [68], [69].”
BackgroundMethodSelf-citationEmotional End-to-End Neural Speech SynthesizerPreprint
Lee, Y., Rabiee, A., & Lee, S.-Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05447 · Cited by 103
“Among the neural TTS models, Tacotron [11] has emerged as an end-to-end TTS model that can be trained from scratch on <text, audio> pairs.”
MethodNon-Autoregressive Neural Text-to-SpeechPreprint
Peng, K., Ping, W., Song, Z., & Zhao, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08459 · Cited by 123
“Taking into account that a reduction factor of 4 is used to simplify the learning of attention mechanism (Wang et al., 2017; Ping et al., 2018) , ωs is simply set as 6.”
BackgroundMethodInfluentialFully-Hierarchical Fine-Grained Prosody Modeling For Interpretable Speech Synthesis
Sun, G., Zhang, Y., Weiss, R. J., Cao, Y. J., Zen, H., & Wu, Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053520 · Cited by 143
Self-citationSynthesizing Audio with Generative Adversarial Networks
Donahue, C., McAuley, J., & Puckette, M. (2018) · Cited by 107
Flow-TTS: A Non-Autoregressive Network for Text to Speech Based on Flow
Miao, C., Liang, S., Chen, M., Ma, J., Wang, S., & Xiao, J. Z. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054484 · Cited by 139
“No distillation Tacotron [5] ParaNet [15] Flow-TTS Tacotron 2 [6] FastSpeech [14] Deep Voice 3 [7] TransformerTTS [16] WaveNet [9] ParallelWaveNet [17] WaveGlow [10] WaveRNN [18] ClariNet [8] FloWaveNet [19] LPCNet [20] Table 1 .”
BackgroundCai, Z., Stefanov, K., Dhall, A., & Hayat, M. (2022) · 2022 International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta56598.2022.10034605 · Cited by 99
Improved Speech Reconstruction from Silent Video
Ephrat, A., Halperin, T., & Peleg, S. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.61 · Cited by 108
Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers
Zhao, Y., Xu, R., Wang, X., Hou, P., Tang, H., & Song, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6174 · Cited by 114
Parallel Tacotron: Non-Autoregressive and Controllable TTS
Elias, I., Zen, H., Shen, J., Zhang, Y., Ye, J., Weiss, R. J., & Wu, Y. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414718 · Cited by 122
“When they were directly compared, Parallel Tacotron with LConv was more preferred.”
Self-citationMega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive BiasPreprint
Jiang, Z. K., Ren, Y., Ye, Z., Liu, J., Zhang, C., Yang, Q., Ji, S., Huang, R., Wang, C., Yin, X., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.03509 · Cited by 109
“4) Content has monotonic alignment with speech while the autoregressive language model cannot guarantee that, which can lead to repeating or missing word issues [59, 58, 67].”
BackgroundMethodInfluentialSound field reconstruction in rooms: Inpainting meets super-resolution
Lluís, F., Martínez-Nuevo, P., Møller, M. B., & Shepstone, S. E. (2020) · The Journal of the Acoustical Society of America · doi:10.1121/10.0001687 · Cited by 95
AI Literacy - Towards Measuring Human Competency in Artificial Intelligence
Pinski, M., & Benlian, A. (2023) · Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.24251/hicss.2023.021 · Cited by 94
MultiSpeech: Multi-Speaker Text to Speech with Transformer
Chen, M., Tan, X., Ren, Y., Xu, J., Sun, H., Zhao, S., & Qin, T. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-3139 · Cited by 92
Any-to-Many Voice Conversion With Location-Relative Sequence-to-Sequence Modeling
Liu, S., Cao, Y., Wang, D., Wu, X., Liu, X., & Meng, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3076867 · Cited by 83
“A CBHG encoder [34] is adopted to transform phoneme”
MethodSun, G., Zhang, Y., Weiss, R. J., Cao, Y. J., Zen, H., Rosenberg, A., Ramabhadran, B., & Wu, Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053436 · Cited by 119
“The fast-paced development of neural end-to-end TTS synthesis has enabled the generation of speech approaching human levels of naturalness [1–4].”
BackgroundSelf-citationWave-Tacotron: Spectrogram-Free End-to-End Text-to-Speech Synthesis
Weiss, R. J., Skerry-Ryan, R., Battenberg, E., Mariooryad, S., & Kingma, D. P. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413851 · Cited by 128
Self-citationAlmost Unsupervised Text to Speech and Automatic Speech Recognition
Ren, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z. L., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.06791 · Cited by 71
“In order to alleviate mispronunciation problem, we convert the text sequence into phoneme sequence before feeding into the model, as used in Arik et al. (2017); Wang et al. (2017); Shen et al. (2018).”
BackgroundMethodInfluentialSurvey on Publicly Available Sinhala Natural Language Processing Tools and ResearchPreprint
de Silva, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02358 · Cited by 102
“TacoSi introduced by Arachchige and Weerasinghe [315] is based on Tacotron [316] and has been evaluated with 10 human evaluators to determine its text-to-speech quality.”
MethodAttention-augmented End-to-end Multi-task Learning for Emotion Prediction from Speech
Zhang, Z., Wu, B., & Schuller, B. W. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682896 · Cited by 105
Shaaban, O. A., Yıldırım, R., & Alguttar, A. A. (2023) · IEEE Access · doi:10.1109/access.2023.3333866 · Cited by 107
“These include neural audio encoders [82], [83], auto encoder [84], autoregressive models [74], [85], [86], GAN [87], [88] as well as other emerging technologies.”
BackgroundMethodLRSpeech: Extremely Low-Resource Speech Synthesis and Recognition
Xu, J., Tan, X., Ren, Y., Qin, T., Li, J., Zhao, S., & Liu, T. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403331 · Cited by 127
“TTS and ASR are usually formulated as sequence to sequence problems [6, 41].”
BackgroundMethodInfluentialMsEmoTTS: Multi-Scale Emotion Transfer, Prediction, and Control for Emotional Speech Synthesis
Lei, Y., Yang, S., Wang, X., & Xie, L. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3145293 · Cited by 87
Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS
He, M., Deng, Y., & He, L. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1972 · Cited by 76
Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling
Elias, I., Zen, H., Shen, J., Zhang, Y., Ye, J., Skerry-Ryan, R., & Wu, Y. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1461 · Cited by 107
Self-citationAudio-deepfake detection: Adversarial attacks and countermeasures
Rabhi, M., Bakiras, S., & Pietro, R. D. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123941 · Cited by 106
Forward Attention in Sequence- To-Sequence Acoustic Modeling for Speech Synthesis
Zhang, J., Ling, Z.-H., & Dai, L.-R. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462020 · Cited by 101
A literature review and perspectives in deepfakes: generation, detection, and applications
Dagar, D., & Vishwakarma, D. K. (2022) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-022-00241-w · Cited by 113
Foundation Models for Music: A SurveyPreprint
Ma, Y., Øland, A., Ragni, A., Sette, B. M. D., Saitis, C., Donahue, C., Lin, C., Plachouras, C., Benetos, E., Shatri, E., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.14340 · Cited by 111
WaveFlow: A Compact Flow-based Model for Raw Audio
Ping, W., Peng, K., Zhao, K., & Song, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.01219 · Cited by 53
“Over the past few years, several neural text-to-speech (TTS) systems have been introduced, including WaveNet (van den Oord et al., 2016), Deep Voice (Arık et al., 2017a), Deep Voice 2 (Arık et al., 2017b), Deep Voice 3 (Ping et al., 2018), Tacotron (Wang et al., 2017), Tacotron 2 (Shen et al., 2018), Char2Wav (Sotelo…”
Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversionPreprint
Zhao, Y., Huang, W.-C., Tian, X., Yamagishi, J., Das, R. K., Kinnunen, T., Ling, Z.-H., & Toda, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.12527 · Cited by 102
“Two submitted systems (T01 and T28) were based on the Tacotron structure, where a speaker embedding is concatenated with encoded output to present the speaker identity.”
BackgroundMethodInfluentialGenerating Synthetic Audio Data for Attention-Based Speech Recognition Systems
Rossenbach, N., Zeyer, A., Schlüter, R., & Ney, H. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053008 · Cited by 123
“The usage of TTS in the context of ASR training was inspired by recent advances in end-to-end TTS systems with multispeaker capabilities such as Tacotron [12], Tacotron-2 [13] and Deep-Voice [14].”
MethodPnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS
Ye, J., Zen, H., Shen, J., Zhang, Y., & Wu, Y. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1757 · Cited by 108
Self-citationAI-Synthesized Voice Detection Using Neural Vocoder Artifacts
Sun, C., Jia, S., Hou, S., & Lyu, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00097 · Cited by 81
Yasuda, Y., Wang, X., Takaki, S., & Yamagishi, J. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682353 · Cited by 80
Fine-Grained Robust Prosody Transfer for Single-Speaker Neural Text-To-Speech
Klimkov, V., Ronanki, S., Rohnke, J., & Drugman, T. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2571 · Cited by 78
End-to-End Text-to-Speech for Low-Resource Languages by Cross-Lingual Transfer Learning
Chen, Y.-J., Tu, T., Yeh, C.-C., & Lee, H.-Y. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2730 · Cited by 72
WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis
Chen, N., Zhang, Y., Zen, H., Weiss, R. J., Norouzi, M., Dehak, N., & Chan, W. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1897 · Cited by 44
Self-citationNAUTILUS: A Versatile Voice Cloning System
Luong, H.-T., & Yamagishi, J. (2020) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2020.3034994 · Cited by 68
Joshi, S., Bhattacharya, S., Pathak, P., Natraj, N. A., Saini, J., & Goswami, S. (2024) · International Journal of Information Management Data Insights · doi:10.1016/j.jjimei.2024.100317 · Cited by 86
Transfer Learning From Speech Synthesis to Voice Conversion With Non-Parallel Training Data
Zhang, M., Zhou, Y., Li, Z., & Li, H. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3066047 · Cited by 90
“Tacotron is an end-to-end text-to-speech (TTS) system [46] that has a sequence-to-sequence encoder-decoder architecture with attention mechanism [47].”
BackgroundDong, W., Pan, D., & Kim, S. (2024) · Journal of Computational Science · doi:10.1016/j.jocs.2024.102397 · Cited by 95
Introduction to Voice Presentation Attack Detection and Recent Advances
Sahidullah, M., Delgado, H., Todisco, M., Kinnunen, T., Evans, N., Yamagishi, J., & Lee, K. A. (2019) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-319-92627-8_15 · Cited by 84
ConvS2S-VC: Fully Convolutional Sequence-to-Sequence Voice Conversion
Kameoka, H., Tanaka, K., Kwaśny, D., Kaneko, T., & Hojo, N. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3001456 · Cited by 75
“This approach has attracted a lot of attention in recent years after being introduced and applied with notable success in various tasks such as machine translation, automatic speech recognition (ASR) [22] and text-to-speech (TTS) [23]–[29].”
BackgroundMethodCopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech
Karlapati, S., Moinet, A., Joly, A., Klimkov, V., Sáez-Trigueros, D., & Drugman, T. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1251 · Cited by 81
Autoencoder With Emotion Embedding for Speech Emotion Recognition
Zhang, C., & Xue, L. (2021) · IEEE Access · doi:10.1109/access.2021.3069818 · Cited by 75
“In fact, magnitude spectrograms have been introduced into many audio processing fields including audio source separation and speech synthesis systems [32]–[36].”
BackgroundResultDiscop: Provably Secure Steganography in Practice Based on "Distribution Copies"
Ding, J., Chen, K., Wang, Y., Zhao, N., Zhang, W., & Yu, N. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179287 · Cited by 120
“For Tacotron [59] and WaveRNN [60], we employ the public pre-trained models from GitHub , .”
Self-citationFitting New Speakers Based on a Short Untranscribed SamplePreprint
Nachmani, E., Polyak, A., Taigman, Y., & Wolf, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.06984 · Cited by 96
“The recent neural TTS systems include the Deep Voice systems DV1 (Arik et al., 2017b), DV2 (Arik et al., 2017a) and DV3 (Ping et al., 2018), WaveNet (Oord et al., 2016) and Parallel WaveNet (van den Oord et al., 2017), Char2Wav (Sotelo et al., 2017), Tacotron (Wang et al., 2017) and Tacotron2 (Shen et al., 2017), and…”
InfluentialControllable Emotion Transfer For End-to-End Speech Synthesis
Li, T., Yang, S., Xue, L., & Xie, L. (2021) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp49672.2021.9362069 · Cited by 71
“With the fast development of the seq2seq modeling architecture, particularly the Tacotron family [1, 2], reference-based style transfer has emerged as another solution with great potential to solving the two problems simultaneously.”
SpecMix : A Mixed Sample Data Augmentation Method for Training with Time-Frequency Domain Features
Kim, G., Han, D. K., & Ko, H. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-103 · Cited by 91
Barakat, H. M., Türk, O., & Demiroğlu, C. (2024) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-024-00329-7 · Cited by 100
DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANsPreprint
Liu, S., Su, D., & Yu, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.11972 · Cited by 71
“Neural network-based autoregressive (AR) models have been adopted for TTS and have shown the capability of generating highly natural speech by producing acoustic features frame by frame (Wang et al., 2017; Gibiansky et al., 2017; Sotelo et al., 2017; Li et al., 2019).”
Adversarially Trained End-to-End Korean Singing Voice Synthesis System
Lee, J., Choi, H.-S., Jeon, C.-B., Koo, J., & Lee, K. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1722 · Cited by 70
Accuth+: Accelerometer-Based Anti-Spoofing Voice Authentication on Wrist-Worn Wearables
Han, F., Yang, P., Du, H., & Li, X.-Y. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3314837 · Cited by 73
“Since the microphone only captures the time-frequency information of the voice, the attackers can easily forge the victim’s voice with machine/deep learning methods [3], [4], and then they can launch replay attacks using loudspeakers.”
BackgroundTransformer Based Grapheme-to-Phoneme Conversion
Yolchuyeva, S., Németh, G., & Gyires-Tóth, B. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1954 · Cited by 69
“Recently, encoder-decoder architectures using convolutional neural networks have been studied and applied to various Natural Language Processing (NLP) tasks [9, 10].”
MethodZhang, M., Wang, X., Fang, F., Li, H., & Yamagishi, J. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1357 · Cited by 80
Human Perception of Audio Deepfakes
Müller, N. M., Pizzi, K., & Williams, J. (2022) · Proceedings of the 1st International Workshop on Deepfake Detection for Audio Multimedia · doi:10.1145/3552466.3556531 · Cited by 68
“The creation of audio deepfakes itself is a young discipline: the first text-to-speech (TTS) synthesis models using neural vocoders were published in 2017 [40].”
Lorenzo-Trueba, J., Fang, F., Wang, X., Echizen, I., Yamagishi, J., & Kinnunen, T. (2018) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2018-34 · Cited by 81
Cai, X., Dai, D., Wu, Z., Li, X., Li, J., & Meng, H. M. L. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413907 · Cited by 81
Lightspeech: Lightweight and Fast Text to Speech with Neural Architecture Search
Luo, R., Tan, X., Wang, R., Qin, T., Li, J., Zhao, S., Chen, E., & Liu, T.-Y. (2021) · ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp39728.2021.9414403 · Cited by 67
“Non-autoregressiveTTS models [5, 9, 7, 8, 6] generate speech in parallel and greatly speed up the inference process compared to autoregressive TTS models [1, 2, 3].”
BackgroundMethodMeta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech
Huang, S.-F., Lin, C.-J., Liu, D.-R., Chen, Y.-C., & Lee, H.-Y. (2022) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2022.3167258 · Cited by 59
“trained with a large amount of high-quality single-speaker speech data [1], [2], [3] or synthesize a set of voices with multi-speaker corpora [4], [5], [6], [7].”
BackgroundControllable Neural Text-to-Speech Synthesis Using Intuitive Prosodic Features
Raitio, T., Rasipuram, R., & Castellani, D. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2861 · Cited by 66
Cross-Speaker Emotion Disentangling and Transfer for End-to-End Speech Synthesis
Li, T., Wang, X., Xie, Q., Wang, Z., & Xie, L. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3164181 · Cited by 86
Speech Synthesis With Mixed Emotions
Zhou, K., Şişman, B., Rana, R. K., Schuller, B. W., & Li, H. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3233324 · Cited by 108
“The decoder has the same model architecture as that of Tacotron [12].”
BackgroundMethodSemi-Supervised Neural Architecture SearchPreprint
Luo, R., Tan, X., Wang, R., Qin, T., Chen, E., & Liu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10389 · Cited by 98
“Text to speech (TTS) [31, 25, 19, 13, 21] is an import task aiming to synthesize intelligible and natural speech from text.”
BackgroundMethodInfluentialTIMIT-TTS: A Text-to-Speech Dataset for Multimodal Synthetic Media Detection
Salvi, D., Hosler, B., Bestagini, P., Stamm, M. C., & Tubaro, S. (2023) · IEEE Access · doi:10.1109/access.2023.3276480 · Cited by 65
“This network has been overtaken over the years by other systems [27], [28], which made the synthesis of highly realistic artificial voices within everyone’s reach.”
Wang, X., Lorenzo-Trueba, J., Takaki, S., Juvela, L., & Yamagishi, J. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461452 · Cited by 70
Aligntts: Efficient Feed-Forward Text-to-Speech System Without Explicit Alignment
Zeng, Z., Wang, J., Cheng, N., Xia, T., & Xiao, J. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054119 · Cited by 63
“Recent advances in text-to-speech (TTS) are driven by the success of autoregressive neural network, such as Char2wav [1], VoiceLoop [2], Tacotron [3], Tacotron 2 [4], Deep Voice 1 [5],Deep Voice 2 [6], Deep Voice 3 [7], Clarinet [8] and Transformer TTS [9].”
InfluentialGeneralized end-to-end detection of spoofing attacks to automatic speaker recognizers
Monteiro, J., Alam, J., & Falk, T. H. (2020) · Computer Speech & Language · doi:10.1016/j.csl.2020.101096 · Cited by 62
End-to-End Emotional Speech Synthesis Using Style Tokens and Semi-Supervised Training
Wu, P., Ling, Z.-H., Liu, L., Jiang, Y., Wu, H.-C., & Dai, L.-R. (2019) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc47483.2019.9023186 · Cited by 65
“acoustic models which predicted acoustic features directly from phoneme or grapheme sequences [5], [6] have also been proposed and attracted the attention of many researchers.”
BackgroundText-Free Image-to-Speech Synthesis Using Learned Segmental Units
Hsu, W.-N., Harwath, D. F., Miller, T., Song, C., & Glass, J. R. (2021) · doi:10.18653/v1/2021.acl-long.411 · Cited by 89
Text-Free Prosody-Aware Generative Spoken Language Modeling
Kharitonov, E., Lee, A., Polyak, A., Adi, Y., Copet, J., Lakhotia, K., Nguyen, T. A., Rivière, M., Mohamed, A., Dupoux, E., et al. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.593 · Cited by 146
KERMIT: Generative Insertion-Based Modeling for Sequences
Chan, W., Kitaev, N., Guu, K., Stern, M., & Uszkoreit, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01604 · Cited by 57
Cai, Z., Ghosh, S., Dhall, A., Gedeon, T., Stefanov, K., & Hayat, M. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103818 · Cited by 63
Restoring speech intelligibility for hearing aid users with deep learning
Diehl, P., Singer, Y., Zilly, H., Schönfeld, U., Meyer-Rachner, P., Berry, M., Sprekeler, H., Sprengel, E., Pudszuhn, A., & Hofmann, V. (2022) · Scientific Reports · doi:10.1038/s41598-023-29871-8 · Cited by 63
“As obtaining MOS is time-consuming and expensive, they are typically only used to compare the performance of algorithms once they have been optimized, but not for the optimization procedure itself 29,30 .”
MethodGu, Y., Yin, X., Rao, Y., Wan, Y., Tang, B., Zhang, Y., Chen, J., Wang, Y., & Ma, Z. (2021) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp49672.2021.9362104 · Cited by 85
“[8] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio et al., “Tacotron: Towards end-to-end speech synthesis,” Proc.”
MethodInfluentialSelf-citationESPnet2-TTS: Extending the Edge of TTS ResearchPreprint
Hayashi, T., Yamamoto, R., Yoshimura, T., Wu, P., Shi, J., Saeki, T., Ju, Y., Yasuda, Y., Takamichi, S., & Watanabe, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07840 · Cited by 73
“E2E-TTS models can alleviate complex text pre-processing and learn alignments between input and output sequences in a data-driven manner, achieving high-fidelity speech comparable to professional recordings [3]–[5].”
BackgroundTransformer-Based Direct Speech-To-Speech Translation with Transcoder
Kano, T., Sakti, S., & Nakamura, S. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383496 · Cited by 62
“In the second Transcoder training, we train the second Transcoder using the pre-trained TTS encoder as a teacher model, with the same process as the first translation training process (Fig.”
BackgroundInfluentialNoise Adaptive Speech Enhancement Using Domain Adversarial Training
Liao, C.-F., Tsao, Y., Lee, H.-Y., & Wang, H. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1519 · Cited by 63
BOFFIN TTS: Few-Shot Speaker Adaptation by Bayesian Optimization
Moss, H. B., Aggarwal, V., Prateek, N., González, J., & Barra-Chicote, R. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054301 · Cited by 77
“Our base model (the model we adapt to target speakers) is a Tacotron2 [17] style multi-speaker system explained in detail by [1, 18], consisting of an acoustic context-generation model and neural vocoder.”
MethodDomain Generalization via Aggregation and Separation for Audio Deepfake Detection
Xie, Y., Cheng, H., Wang, Y., & Ye, L. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3324724 · Cited by 59
“Tacotron [24] is an innovative end-to-end TTS sys-tem that leverages an encoder–attention–decoder framework and uses the Griffin–Lim algorithm to generate waveforms.”
MethodSelf-citationChubb, J., Missaoui, S., Concannon, S. J., Maloney, L., & Walker, J. A. (2021) · International Journal of Child-Computer Interaction · doi:10.1016/j.ijcci.2021.100403 · Cited by 72
Lee, S.-H., Choi, H.-Y., Kim, S.-B., & Lee, S. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3584944 · Cited by 73
“Conventional sequence-to-sequence autoregressive TTS models, such as Tacotron [9], have successfully paved the way for speech synthesis technologies.”
SED-MDD: Towards Sentence Dependent End-To-End Mispronunciation Detection and Diagnosis
Feng, Y., Fu, G., Chen, Q., & Chen, K. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9052975 · Cited by 59
“CNN-RNN module is inspired by work in speech synthesis [16], where the main differences from [16] include the removal of conv1d bank and stacking, highway networks and residual connection.”
MethodVQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature
Du, C., Guo, Y., Chen, X., & Yu, K. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-489 · Cited by 75
“Compared with traditional statistical parametric speech synthesis [1], neural TTS model [2, 3] based on deep neural network shows a better performance.”
MethodWavJourney: Compositional Audio Creation With Large Language Models
Liu, X., Zhu, Z., Liu, H., Yuan, Y., Huang, Q., Cui, M., Liang, J., Cao, Y. P., Kong, Q., Plumbley, M. D., et al. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3574867 · Cited by 77
RF-Next: Efficient Receptive Field Search for Convolutional Neural Networks
Gao, S., Li, Z., Qi, H., Cheng, M., & Wang, L. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3183829 · Cited by 81
“2) transfer linguistic features or text to acoustic features [10], [77], [78], [79].”
BackgroundMethodEnd-to-End Video-to-Speech Synthesis Using Generative Adversarial Networks
Mira, R., Vougioukas, K., Ma, P., Petridis, S., Schuller, B. W., & Pantić, M. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3162495 · Cited by 72
“Another work [57] introduced a vastly different sequence-tosequence model that predicted linear-scale spectrograms from text, which were then converted into a waveform using the Griffin–Lim algorithm (GLA) [62].”
Review of audio deepfake detection techniques: Issues and prospects
Dixit, A., Kaur, N., & Kingra, S. (2023) · Expert Systems · doi:10.1111/exsy.13322 · Cited by 56
Li, H., Xing, W., Jiao, H., Yuen, K. F., Gao, R., Li, Y., Matthews, C., & Yang, Z. (2024) · Transportation Research Part E Logistics and Transportation Review · doi:10.1016/j.tre.2024.103770 · Cited by 62
Self-citationZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching
Zhu, H., Kang, W., Yao, Z., Guo, L., Kuang, F., Li, Z., Zhuang, W., Lin, L., & Povey, D. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11433841 · Cited by 68
“Early end-to-end TTS models [28], [29] suffered from slow inference due to AR speech generation.”
Machine Speech Chain with One-shot Speaker Adaptation
Tjandra, A., Sakti, S., & Nakamura, S. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1558 · Cited by 56
Fine-Grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis
Lei, Y., Yang, S., & Xie, L. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383524 · Cited by 56
Metadata
- Authors
- Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., Le, Q., Agiomyrgiannakis, Y., Clark, R., & Saurous, R. A.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Wang, Yuxuan
- ORCID
- Venue
- Interspeech 2017
- Publisher
- ISCA
- Published
- 2017-08-20
- Pages
- 4006-4010
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2963609956
- Cited by
- 1,746
- References
- 25
- Citations by year
- 2026: 134 · 2025: 187 · 2024: 183 · 2023: 215 · 2022: 243 · 2021: 301 · 2020: 252 · 2019: 176 · 2018: 45 · 2017: 9
- FWCI
- 60.12
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
- Keywords
- Tacotron, end-to-end speech synthesis, text-to-speech, sequence-to-sequence models, naturalness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.