Tacotron: Towards End-to-End Speech Synthesis
The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module.Building these components often requires extensive domain expertise and may contain brittle design choices.In this paper, we present Tacotron, an end-to-end generative text-to-speech model that synthesizes speech directly from characters.Given pairs, the model can be trained completely from scratch with random initialization.We present several key techniques to make the sequence-tosequence framework perform well for this challenging task.Tacotron achieves a 3.82 subjective 5-scale mean opinion score on US English, outperforming a production parametric system in terms of naturalness.In addition, since Tacotron generates speech at the frame level, it's substantially faster than sample-level autoregressive methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The frameworks including their code as well as the datasets used that are available publicly (Sotelo et al., 2017; Wang et al., 2017; Ping et al., 2018; Shen et al., 2018; Vasquez and Lewis, 2019) are more likely to be used for nefarious purposes or research, so they will be more developed.”
From Audio deepfakes: A survey (2023) · cites this paper as Wang et al., 2017
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“We further pass the processed sequence to a few fixed-width 1-D convolutions, whose outputs are added with the original input sequence via residual connections [17].”
BackgroundAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 2×
“We use the Adam optimizer (Kingma & Ba, 2015) with learning rate decay, which starts from 0.001 and is reduced to 0.0005, 0.0003, and 0.0001 after 500K, 1M and 2M global steps, respectively.”
MethodInfluentialBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 2×
“Batch normalization [18] is used for all convolutional layers.”
MethodSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“In this paper, we propose Tacotron, an end-to-end generative TTS model based on the sequence-to-sequence (seq2seq) (Sutskever et al., 2014) with attention paradigm (Bahdanau et al., 2014).”
MethodSelf-citationNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“The backbone of Tacotron is a seq2seq model with attention [7, 14].”
MethodInfluentialEmpirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · cited 7×
“Second, we compare with a model with the CBHG encoder replaced by a 2-layer residual GRU encoder.”
BackgroundInfluentialTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · cited 2×
“We implemented Griffin-Lim in TensorFlow [22] hence it’s also part of the model.”
MethodSelf-citationGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 4×
“C L ] 6 A pr or machine translation (Wu et al., 2016), TTS outputs are continuous, and output sequences are usually much longer than those of the input.”
BackgroundMethodInfluentialSelf-citationWaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · cited 4×
“It could be used to predict alternative targets such as vocoder parameters, or as a WaveNetlike neural vocoder [9, 13, 10] that synthesizes waveform samples directly.”
BackgroundMethodInfluentialListen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226 · cited 2×
“Moreover, unlike end-to-end speech recognition [4] or machine translation [5], TTS outputs are continuous, and output sequences are usually much longer than those of the input.”
BackgroundSelf-citation
Cited by
Items 201–300 of 1,002
Exploring Transfer Learning for Low Resource Emotional TTS
Tits, N., Haddad, K. E., & Dutoit, T. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-29516-5_5 · Cited by 76
Laptev, A., Korostik, R., Svischev, A., Andrusenko, A., Medennikov, I., & Rybin, S. (2020) · International Congress on Image and Signal Processing (CISP) · doi:10.1109/cisp-bmei51763.2020.9263564 · Cited by 87
“Authors used Tacotron 2 as synthesizer, WaveNet as vocoder, and Wave2Letter+ ASR model from OpenSeq2Seq toolkit2.”
InfluentialBehavioral Use Licensing for Responsible AI
Contractor, D., McDuff, D., Haines, J. K., Lee, J., Hines, C., Hecht, B., Vincent, N., & Li, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533143 · Cited by 54
Brain-Inspired Learning, Perception, and Cognition: A Comprehensive Review
Jiao, L., Ma, M., He, P., Geng, X., Liu, X., Liu, F., Ma, W., Yang, S., Hou, B., & Tang, X. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3401711 · Cited by 62
“Inspired by this, the current common tasks in the field of machine learning are speech recognition [191], [201], [202], speech synthesis [192], [203], [204], and speech enhancement [193], [205].”
Effect of Data Reduction on Sequence-to-sequence Neural TTS
Latorre, J., Lachowicz, J., Lorenzo-Trueba, J., Merritt, T., Drugman, T., Ronanki, S., & Klimkov, V. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682168 · Cited by 68
GlotNet—A Raw Waveform Model for the Glottal Excitation in Statistical Parametric Speech Synthesis
Juvela, L., Bollepalli, B., Tsiaras, V., & Alku, P. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2906484 · Cited by 53
“nets [3], [4], and most recently, sequence-to-sequence models with attention [5], [6] yield impressive results.”
BackgroundLeveraging Sequence-to-Sequence Speech Synthesis for Enhancing Acoustic-to-Word Speech Recognition
Mimura, M., Ueno, S., Inaguma, H., Sakai, S., & Kawahara, T. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639589 · Cited by 53
“Although a number of distinct architectures have been proposed [22, 18, 19, 20], these systems are commonly composed of an attention-based feature prediction network which maps character embedding to mel-scale spectrograms or vocoder parameters, followed by a vocoder which synthesize time-domain waveforms from these p…”
BackgroundControlling Emotion Strength with Relative Attribute for End-to-End Speech Synthesis
Zhu, X., Yang, S., Yang, G., & Xie, L. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003829 · Cited by 53
Autoregressive Diffusion Transformer for Text-to-Speech SynthesisPreprint
Liu, Z., Wang, S., Inoue, S., Bai, Q., & Li, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.05551 · Cited by 63
“(ii) An alternative is to represent p θ ( x n | x n ) using a Mixture Density Network (MDN) [42] that predicts parameters for continuous mixture distributions [43–49], such as a mixture of Gaussians (MoG).”
The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systems
Adigwe, A., Tits, N., Haddad, K. E., Ostadabbas, S., & Dutoit, T. (2018) · arXiv · doi:10.48550/arxiv.1806.09514 · Cited by 52
“The state-of-the-art speech synthesis systems such as wavenet and tacotron[19,23,21] are giving impressive results.”
BackgroundArias, P., Rachman, L., Liuni, M., & Aucouturier, J. (2020) · Emotion Review · doi:10.1177/1754073920934544 · Cited by 56
“The DNN architecture, which directly models waveforms using a neural network method trained with recordings of real speech, provides superior acoustic quality for several high-level vocal applications such as Text-to-Speech (Wang et al. 2017) and speech enhancement (Pascual et al.”
MethodPre-Trained Text Embeddings for Enhanced Text-to-Speech Synthesis
Hayashi, T., Watanabe, S., Toda, T., Takeda, K., Toshniwal, S., & Livescu, K. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3177 · Cited by 86
A Generative Model for Raw Audio Using Transformer Architectures
Verma, P., & Chafe, C. (2021) · International Conference on Digital Audio Effects (DAFx) · doi:10.23919/dafx51585.2021.9768298 · Cited by 59
Revisiting Over-Smoothness in Text to Speech
Ren, Y., Tan, X., Qin, T., Zhao, Z., & Liu, T. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.564 · Cited by 80
Yang, J., Luo, K. Z., Li, J., Deng, C., Guibas, L. J., Krishnan, D., Weinberger, K. Q., Tian, Y., & Wang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73013-9_26 · Cited by 51
Self-citationTeacher-Student Training For Robust Tacotron-Based TTS
Liu, R., Şişman, B., Li, J., Bao, F., Gao, G., & Li, H. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054681 · Cited by 51
Low-Resource Expressive Text-To-Speech Using Data Augmentation
Huybrechts, G., Merritt, T., Comini, G., Perz, B., Shah, R., & Lorenzo-Trueba, J. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413466 · Cited by 78
Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminability
Liu, R., Şişman, B., & Li, H. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1236 · Cited by 61
Fujita, Y., Watanabe, S., Horiguchi, S., Xue, Y., & Nagamatsu, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02966 · Cited by 57
“This neural-network-based end-to-end approach, in which only one neural network model directly computes the final outputs, has been successfully applied in a variety of tasks, including neural machine translation [34], [35], automatic speech recognition [36]–[38], and text-to-speech [39], [40].”
Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed DataPreprint
Kim, S., Kim, H., & Yoon, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.15370 · Cited by 63
RW-Resnet: A Novel Speech Anti-Spoofing Model Using Raw Waveform
Ma, Y., Ren, Z., & Xu, S. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-438 · Cited by 36
Semi-supervised End-to-end Speech Recognition Using Text-to-speech and Autoencoders
Karita, S., Watanabe, S., Iwata, T., Delcroix, M., Ogawa, A., & Nakatani, T. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682890 · Cited by 58
“(5) as b , x pre , Finally, we define the TTS loss Taco( · ) proposed in [25] that includes binary cross entropy for the flag b s and L1 [26] / L2 [4] norm for speech errors as follows:”
MethodImproving Speech Recognition Using Consistent Predictions on Synthesized Speech
Wang, G., Rosenberg, A., Chen, Z., Zhang, Y., Ramabhadran, B., Wu, Y., & Moreno, P. J. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053831 · Cited by 64
“We base our TTS model on Tacotron 2D [4, 31], which takes a text sequence as input, conditioned on a speaker d-vector, and outputs a sequence of mel spectrogram frames.”
MethodSelf-citationSynthesizing Dysarthric Speech Using Multi-Speaker Tts For Dysarthric Speech Recognition
Soleymanpour, M., Johnson, M. T., Soleymanpour, R., & Berry, J. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746585 · Cited by 67
Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis
Jiang, Z. K., Liu, J., Ren, Y., He, J., Ye, Z., Ji, S., Yang, Q., Zhang, C., Wei, P., Wang, C., et al. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2307.07218 · Cited by 60
“Skerry-Ryan et al. (2018) first integrate a prosody reference encoder into a TTS system based on Tacotron (Wang et al., 2017), which is capable of performing similar-text prosody transfer.”
MethodAudio-linguistic Embeddings for Spoken Sentences
Haque, A., Guo, M., Verma, P., & Fei-Fei, L. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682553 · Cited by 62
Bucher, M. J. J., Kraus, M. A., Rust, R., & Tang, S. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.105128 · Cited by 54
Recent Advances in Synthesis and Interaction of Speech, Text, and Vision
Orynbay, L., Shamshanovna, R. B., Peer, P., Meden, B., & Emeršič, Ž. (2024) · Electronics · doi:10.3390/electronics13091726 · Cited by 58
Ma, Y., Zheng, S., Yang, Z., Zheng, P., Leng, J., & Hong, J. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.07.019 · Cited by 56
Self-citationVoice In Ear: Spoofing-Resistant and Passphrase-Independent Body Sound Authentication
Gao, Y., Jin, Y., Chauhan, J., Choi, S., Li, J., & Jin, Z. (2021) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3448113 · Cited by 68
“There still are possibilities of incorporating the user’s non-passphrase voice and mouth movement to implement synthetic attacks [11] by utilizing deep learning-based Text-to-Speech (TTS) engines [36, 55].”
BackgroundWan, V., Chan, C.-A., Kenter, T., Vít, J., & Clark, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07195 · Cited by 49
“An alternative approach to modeling prosody explicitly is taken by end-to-end approaches to TTS, for example, (Ping et al., 2017; Wang et al., 2017) where there is no explicit prosody model at all, and any modeling of prosody is performed implicitly within the model.”
MethodSelf-citationTits, N., Wang, F., Haddad, K. E., Pagel, V., & Dutoit, T. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1426 · Cited by 52
A New GAN-Based End-to-End TTS Training Algorithm
Guo, H., Soong, F. K., He, L., & Xie, L. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2176 · Cited by 54
PromptTTS 2: Describing and Generating Voices with Text PromptPreprint
Leng, Y., Guo, Z., Shen, K., Tan, X., Ju, Z., Liu, Y., Liu, Y., Yang, D., Zhang, L., Song, K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.02285 · Cited by 131
Conversational End-to-End TTS for Voice Agents
Guo, H., Zhang, S., Soong, F. K., He, L., & Xie, L. (2021) · 2021 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383460 · Cited by 47
Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning
Liu, A. H., Tu, T., Lee, H.-Y., & Lee, L.-S. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053571 · Cited by 45
“We followed the previous work of TTS [19] to extract spectrogram with the window size of 50 ms and the hop size of 12.5 ms.”
BackgroundMethodInfluentialSpeech-to-Speech Translation Between Untranscribed Unknown Languages
Tjandra, A., Sakti, S., & Nakamura, S. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003853 · Cited by 72
“A traditional approach in S2ST systems requires effort to construct several components, including automatic speech recognition (ASR), machine translation (MT), and text-tospeech (TTS) synthesis, all of which are trained and tuned independently.”
BackgroundMethodInfluentialDesigners Characterize Naturalness in Voice User Interfaces: Their Goals, Practices, and Challenges
Kim, Y., Reza, M., McGrenere, J., & Yoon, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445579 · Cited by 42
“Hence, a demonstration-based prosody editing, leveraging our own voices [111], is a promising approach worth investigating.”
BackgroundEnd-to-End Speech Translation With Transcoding by Multi-Task Learning for Distant Language Pairs
Kano, T., Sakti, S., & Nakamura, S. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.2986886 · Cited by 55
“An attention-based encoder-decoder neural network is a powerful model for ASR, MT, and TTS [5]–[7].”
Background"Hello, It's Me": Deep Learning-based Speech Synthesis Attacks in the Real World
Wenger, E., Bronckers, M., Cianfarani, C., Cryan, J., Sha, A., Zheng, H.-T., & Zhao, B. Y. (2021) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3460120.3484742 · Cited by 73
“TTS systems transform arbitrary text into words spoken in the voice of a target speaker [21, 36, 37, 41, 64, 76, 83, 92].”
A Virtual Simulation-Pilot Agent for Training of Air Traffic Controllers
Zuluaga-Gómez, J., Prasad, A., Nigmatulina, I., Motlíček, P., & Kleinert, M. (2023) · Aerospace · doi:10.3390/aerospace10050490 · Cited by 55
OnePiece: Bringing Context Engineering and Reasoning to Industrial Cascade Ranking System
Dai, S., Tang, J., Wu, J., Wang, K., Zhu, Y., Chen, B., Hong, B., Zhao, Y., Fu, C., Wu, K., et al. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818456 · Cited by 49
Phase Reconstruction from Amplitude Spectrograms Based on Von-Mises-Distribution Deep Neural Network
Takamichi, S., Saito, Y., Takamune, N., Kitamura, D., & Saruwatari, H. (2018) · International Workshop on Acoustic Signal Enhancement · doi:10.1109/iwaenc.2018.8521313 · Cited by 46
Pretraining Techniques for Sequence-to-Sequence Voice Conversion
Huang, W.-C., Hayashi, T., Wu, Y.-C., Kameoka, H., & Toda, T. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3049336 · Cited by 57
“ing, most of which are inspired by the success of modern seq2seq TTS models [60], [61].”
Self-citationAdaspeech 2: Adaptive Text to Speech with Untranscribed Data
Yan, Y., Tan, X., Li, B., Qin, T., Zhao, S., Shen, Y., & Liu, T.-Y. (2021) · ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp39728.2021.9414872 · Cited by 45
“TTS adaptation pipeline [1, 2, 3, 4, 5, 6, 7, 8] usually finetunes a well-trained multi-speaker TTS model [9, 10] (source model) with few adaptation data, and achieves good similarity with ground-truth human voice and high voice quality.”
BackgroundTowards Multi-Scale Style Control for Expressive Speech Synthesis
Li, X., Song, C., Li, J., Wu, Z., Jia, J., & Meng, H. M. L. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-947 · Cited by 46
PortaSpeech: Portable and High-Quality Generative Text-to-SpeechPreprint
Ren, Y., Liu, J., & Zhao, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.15166 · Cited by 78
“• For inference speed, PortaSpeech (small) speeds up the end-to-end speech generation by 5.5x and 45.9x compared with Tacotron 2 and TransformerTTS and achieves similar RTF with other NAR-TTS models.”
BackgroundMethodInfluentialA review on subjective and objective evaluation of synthetic speech
Cooper, E., Huang, W.-C., Tsao, Y., Wang, H., Toda, T., & Yamagishi, J. (2024) · Nippon Onkyo Gakkaishi/Acoustical science and technology/Nihon Onkyo Gakkaishi · doi:10.1250/ast.e24.12 · Cited by 75
Video and Audio Deepfake Datasets and Open Issues in Deepfake Technology: Being Ahead of the Curve
Akhtar, Z., Pendyala, T. L., & Athmakuri, V. S. (2024) · Forensic Sciences · doi:10.3390/forensicsci4030021 · Cited by 60
Zhao, Y., Yang, M., Lin, Y., Zhang, X., Shi, F., Wang, Z., Ding, J., & Ning, H. (2025) · Electronics · doi:10.3390/electronics14061197 · Cited by 61
Yang, S., Xie, L., Chen, X., Lou, X., Zhu, X., Huang, D., & Li, H. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8269003 · Cited by 45
Audeo: Audio Generation for a Silent Performance VideoPreprint
Su, K., Liu, X., & Shlizerman, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.14348 · Cited by 45
“Tacotron [38, 39] proposed the encoder-decoder architecture to translate text to Mel-spectrogram and a Wavenet conditioned on generated Mel-spectrogram to generate final human speech waveform.”
MethodEMOQ-TTS: Emotion Intensity Quantization for Fine-Grained Controllable Emotional Text-to-Speech
Im, C.-B., Lee, S.-H., Kim, S.-B., & Lee, S. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747098 · Cited by 72
METTS: Multilingual Emotional Text-to-Speech by Cross-Speaker and Cross-Lingual Emotion Transfer
Zhu, X., Lei, Y., Li, T., Zhang, Y., Zhou, H., Lu, H., & Xie, L. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3363444 · Cited by 58
“R ECENT years have witnessed significant progress in the quality and naturalness of synthetic speech thanks to the advances of neural text-to-speech (TTS) systems [1], [2], [3], [4], [5], [6], [7], [8].”
BackgroundDeep Griffin–Lim Iteration: Trainable Iterative Phase Reconstruction Using Neural Network
Masuyama, Y., Yatabe, K., Koizumi, Y., Oikawa, Y., & Harada, N. (2021) · IEEE Journal on Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.3034486 · Cited by 32
“One of the most popular consistency-based phase reconstruction methods is the Griffin–Lim algorithm (GLA) [20] which has been applied to speech and audio synthesis [1]–[4].”
MethodUncovering Latent Style Factors for Expressive Speech SynthesisPreprint
Wang, Y., Skerry-Ryan, R., Xiao, Y., Stanton, D., Shor, J., Battenberg, E., Clark, R., & Saurous, R. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00520 · Cited by 44
“We use a similar single-speaker corpus as in [2]; while the speaker primarily speaks in a neutral prosody, a small subset of the corpus uses more expression (including performing as a game show host, reading jokes and poems, etc).”
InfluentialSelf-citationGuided-TTS: A Diffusion Model for Text-to-Speech via Classifier GuidancePreprint
Kim, H., Kim, S., & Yoon, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.11755 · Cited by 44
EAV: EEG-Audio-Video Dataset for Emotion Recognition in Conversational Contexts
Lee, M.-H., Shomanov, A., Begim, B., Kabidenova, Z., Nyssanbay, A., Yazıcı, A., & Lee, S. (2024) · Scientific Data · doi:10.1038/s41597-024-03838-4 · Cited by 63
“…underlying brain functions 29,30 ; 3) Examination of time-variant or contextual emotion analysis 31 ; 4) Synthesis of the facial or vocal outputs 32,33 ; and 5) Exploration of emerging machine learning topics, such as multimodal contrastive learning 34,35 , generative models 36 , or knowledge…”
BackgroundXiao, Y., He, L., Ming, H., & Soong, F. K. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054337 · Cited by 53
“The encoder part is replaced by a CBHG-like [11] module (1-”
MethodInfluentialSelf-citationEffective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis
Battenberg, E., Mariooryad, S., Stanton, D., Skerry-Ryan, R., Shannon, M., Kao, D., & Bagby, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.03402 · Cited by 41
“Model architecture The baseline model we start with is a Tacotron-based system (Wang et al., 2017) that incorporates modifications from Skerry-Ryan et al.”
BackgroundMethodResultInfluentialSelf-citationBoito, M. Z., Havard, W. N., Garnerin, M., Ferrand, É. L., & Besacier, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.12895 · Cited by 43
“Both speech encoders consist of a convolution bank [27] followed by two layers of bidirectional LSTM [28] and an attention mechanism [29] which computes a weighted average of the LTSM’s activations.”
BackgroundJDI-T: Jointly Trained Duration Informed Transformer for Text-To-Speech without Explicit Alignment
Lim, D., Jang, W., O, G., Park, H., Kim, B., & Yoon, J. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-2123 · Cited by 42
Neural Speech Phase Prediction Based on Parallel Estimation Architecture and Anti-Wrapping Losses
Ai, Y., & Ling, Z.-H. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096553 · Cited by 42
“Currently, several speech generation tasks, such as speech enhancement (SE) [1, 2, 3], bandwidth extension (BWE) [4, 5, 6] and speech synthesis (SS) [7, 8, 9, 10], mainly focus on the prediction of amplitude spectra or amplitude-derived features (e.”
BackgroundLatif, S., Shahid, A., & Qadir, J. (2023) · Applied Acoustics · doi:10.1016/j.apacoust.2023.109425 · Cited by 53
Wang, D., Yu, J., Wu, X., Liu, S., Sun, L., Liu, X., & Meng, H. M. L. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054596 · Cited by 57
“To tackle these two issues, this paper proposes a novel end-to-end VC method for DSR, which is inspired by the success of the seq2seq based TTS [14] and KD techniques [15, 16], where the latter is used in the teacher-student framework that transfers the distilled knowledge of the “ teacher ” to the…”
MethodAudio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis
Yang, K., Markovic, D., Krenn, S., Agrawal, V., & Richard, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00805 · Cited by 60
“Prominent examples of speech synthesis include text-to-speech synthesis and video-to-speech synthesis, where the first stage consists of generating the intermediate representation from text [33,43,51,61] or silent video [10,12,32,45] input.”
BackgroundMTTM: Metamorphic Testing for Textual Content Moderation Software
Wang, W., Huang, J.-T., Wu, W., Zhang, J., Huang, Y., Li, S., He, P., & Lyu, M. R.-T. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00200 · Cited by 60
“Typical scenarios include sentiment analysis [70], [71], machine translation [72]–[74] and text-to-speech synthesis [75], [76].”
BackgroundAn Effective Style Token Weight Control Technique for End-to-End Emotional Speech Synthesis
Kwon, O., Jang, I., Ahn, C., & Kang, H.-G. (2019) · IEEE Signal Processing Letters · doi:10.1109/lsp.2019.2931673 · Cited by 41
“The idea of the GST was first proposed in the original Tacotron framework, and the GST network was trained without using an explicit style label [12], [16].”
BackgroundMethodLearning Efficient Representations for Fake Speech Detection
Subramani, N., & Rao, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6044 · Cited by 38
“Neural text to speech models such as WaveNet (Oord et al., 2016), Tacotron (Wang et al., 2017), and DeepVoice (Gibiansky et al., 2017; Ping et al., 2017) have been extended to incorporate speaker identity Arik et al. (2018); Nachmani et al. (2018).”
MethodSemi-Supervised Generative Modeling for Controllable Speech SynthesisPreprint
Habib, R., Mariooryad, S., Shannon, M., Battenberg, E., Skerry-Ryan, R., Stanton, D., Kao, D., & Bagby, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.01709 · Cited by 49
“(2017) is used to convert the input phonemes into a sequence of text embeddings.”
BackgroundMethodInfluentialSelf-citationDetecting Voice Cloning Attacks via Timbre Watermarking
Liu, C., Zhang, J., Zhang, T., Yang, X., Zhang, W., & Yu, N. (2024) · Proceedings 2024 Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24200 · Cited by 40
The Impact of Silence on Speech Anti-Spoofing
Zhang, Y., Li, Z., Lu, J., Hua, H., Wang, W., & Zhang, P. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3306711 · Cited by 53
“W ITH the development of deep learning, text-to-speech (TTS) [1], [2] and voice conversion (VC) [3] technologies have made significant improvements.”
BackgroundGrapheme-to-Phoneme Conversion with Convolutional Neural Networks
Yolchuyeva, S., Németh, G., & Gyires-Tóth, B. (2019) · Applied Sciences · doi:10.3390/app9061143 · Cited by 40
“Sequence-to-sequence (seq2seq) learning, or encoder-decoder type neural networks have achieved remarkable success in various tasks, such as speech recognition, text-to-speech synthesis, machine translation [22–25].”
BackgroundMethodJSUT and JVS: Free Japanese voice corpora for accelerating speech synthesis research
Takamichi, S., Sonobe, R., Mitsui, K., Saito, Y., Koriyama, T., Tanji, N., & Saruwatari, H. (2020) · Nippon Onkyo Gakkaishi/Acoustical science and technology/Nihon Onkyo Gakkaishi · doi:10.1250/ast.41.761 · Cited by 40
“Utterance length is an important factor in speech synthesis using sequence-to-sequence mechanisms [43–45].”
BackgroundProtein Design and Variant Prediction Using Autoregressive Generative Models
Shin, J.-E., Riesselman, A. J., Kollasch, A. W., McMahon, C., Simon, E., Sander, C., Manglik, A., Kruse, A. C., & Marks, D. S. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/757252 · Cited by 32
“Many applications in text-to-speech (Graves et al., 2013; Wang et al., 2017) and translation (Bahdanau et al., 2014; Sutskever et al., 2014) tackle context dependent word and audio prediction and generation using generative models with an autoregressive likelihood (Sutskever et al., 2011; LeCun et…”
MethodChen, Q., Tan, M., Qi, Y., Zhou, J., Li, Y., & Wu, Q. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02056 · Cited by 67
Conditional End-to-End Audio Transforms
Haque, A., Guo, M., & Verma, P. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-38 · Cited by 42
Emotional Voice Conversion Using Multitask Learning with Text-To-Speech
Kim, T., Cho, S., Choi, S., Park, S., & Lee, S.-Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053255 · Cited by 53
“TTS is a task to transform textual or phonetic information to speech waveform, and abundant seq2seq-based studies have actively conducted [8, 9, 10].”
BackgroundMethodInfluentialDurIAN-SC: Duration Informed Attention Network Based Singing Voice Conversion System
Zhang, L., Yu, C., Lu, H., Weng, C., Zhang, C., Wu, Y., Xie, X., Li, Z., & Yu, D. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1789 · Cited by 45
Self-citationEMOVIE: A Mandarin Emotion Speech Dataset with a Simple Emotional Text-to-Speech Model
Cui, C., Ren, Y., Liu, J., Chen, F., Huang, R., Lei, M., & Zhao, Z. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1148 · Cited by 43
Zhou, K., Şişman, B., & Li, H. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-781 · Cited by 39
Audio representations for deep learning in sound synthesis: A review
Natsiou, A., & O’Leary, S. (2021) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa53542.2021.9686838 · Cited by 48
SpecRNet: Towards Faster and More Accessible Audio DeepFake Detection
Kawa, P., Plata, M., & Syga, P. (2022) · 2022 IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom56396.2022.00111 · Cited by 32
Avocodo: Generative Adversarial Network for Artifact-Free Vocoder
Bak, T., Lee, J., Bae, H., Yang, J., Bae, J.-S., & Joo, Y.-S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26479 · Cited by 63
CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model
Ye, Z., Xue, W., Tan, X., Chen, J., Liu, Q., & Guo, Y. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612061 · Cited by 68
CFAD: A Chinese dataset for fake audio detection
Ma, H., Yi, J., Wang, C., Yan, X., Tao, J., Wang, T., Wang, S., & Fu, R. (2024) · Speech Communication · doi:10.1016/j.specom.2024.103122 · Cited by 64
“But the acoustic model of F11 is Tacotron2 [48].”
BackgroundTextless Unit-to-Unit Training for Many-to-Many Multilingual Speech-to-Speech Translation
Kim, M., Choi, J., Kim, D., & Ro, Y. M. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3444470 · Cited by 46
Cheng, G., Wang, X., Cooper, E., Wells, D., Wang, L., Dang, J., Richmond, K., & Yamagishi, J. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3451951 · Cited by 57
“T EXT-TO-SPEECH (TTS) technology, which converts a text string into a speech waveform [1], [2], [3], [4], has received significant advancements driven by deep learning.”
BackgroundMethodCharacterization of Analytic Wavelet Transforms and a New Phaseless Reconstruction Algorithm
Holighaus, N., Koliander, G., Průša, Z., & Abreu, L. D. (2019) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2019.2920611 · Cited by 40
Singing Voice Synthesis Using Deep Autoregressive Neural Networks for Acoustic Modeling
Yi, Y., Ai, Y., Ling, Z.-H., & Dai, L.-R. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1563 · Cited by 38
Many-to-Many Voice Transformer Network
Kameoka, H., Huang, W.-C., Tanaka, K., Kaneko, T., Hojo, N., & Toda, T. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3047262 · Cited by 40
“Our VTN [49] is another example, which relies on TTS pretraining using text-speech pair data.”
MethodInfluentialKazakhTTS: An Open-Source Kazakh Text-to-Speech Synthesis Dataset
Mussakhojayeva, S., Janaliyeva, A., Mirzakhmetov, A., Khassanov, Y., & Varol, H. A. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-2124 · Cited by 48
“The recent surge of speech-enabled applications, such as virtual assistants for smart devices, has attracted substantial attention from both academia and industry to the TTS research [3, 4, 5].”
BackgroundThe Multi-Speaker Multi-Style Voice Cloning Challenge 2021
Xie, Q., Tian, X., Liu, G., Song, K., Xie, L., Wu, Z., Li, H. H., Shi, S. X., Li, H., Hong, F., et al. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414001 · Cited by 42
“In the submissions, Tacotron [1, 24] is the most popular one, where an encoder-attention-decoder based architecture is adopted for autoregressive generation.”
Improving Prosody Modelling with Cross-Utterance Bert Embeddings for End-to-End Speech Synthesis
Xu, G., Song, W. G., Zhang, Z., Zhang, C., He, X., & Zhou, B. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414102 · Cited by 38
“Regarding end-to-end speech synthesis based on the encoder-decoder models [12, 13], the Tacotron structure [2, 3] can be extended to include an additional encoder to learn prosody embeddings [14–16], which can be used to query a bank of randomly initialised global style tokens with an attention model to derive the sty…”
BackgroundMethodGeneralized Spoofing Detection Inspired from Audio Generation Artifacts
Gao, Y., Vuong, T., Elyasi, M., Bharaj, G., & Singh, R. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1705 · Cited by 25
A Deep Learning Framework of Autonomous Pilot Agent for Air Traffic Controller Training
Lin, Y., Wu, Y., Guo, D., Zhang, P., Yin, C., Yang, B., & Zhang, J. (2021) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2021.3102827 · Cited by 44
Self-citationA Review of Human Emotion Synthesis Based on Generative Technology
Ma, F., Xie, Y., Li, Y., He, Y., Zhang, Y., Ren, H., Liu, Z., Yao, W. J., Ren, F., Yu, F. R., et al. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3573878 · Cited by 52
“Speech emotion synthesis has progressed through GANs and Seq2Seq models like Tacotron [230], which add emotional intonation by modifying pitch, rhythm, and tone.”
End-to-end Code-switched TTS with Mix of Monolingual Recordings
Cao, Y., Wu, X., Liu, S., Yu, J., Xu, L., Wu, Z., Liu, X., & Meng, H. M. L. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682927 · Cited by 37
“In this paper, we explore the efficacy of using the Tacotron-based E2E framework [18], which is composed of an encoder and a de-coder with attention, for code-switched TTS using a combination of a Mandarin and an English monolingual speech corpus uttered by two female speakers.”
MethodAzizah, K., Adriani, M., & Jatmiko, W. (2020) · IEEE Access · doi:10.1109/access.2020.3027619 · Cited by 37
“Tacotron [15] and Tacotron-2 [13] do not model prosody explicitly.”
BackgroundFine-Grained Style Control In Transformer-Based Text-To-Speech Synthesis
Chen, L., & Rudnicky, A. I. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747747 · Cited by 37
Metadata
- Authors
- Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., Le, Q., Agiomyrgiannakis, Y., Clark, R., & Saurous, R. A.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Wang, Yuxuan
- ORCID
- Venue
- Interspeech 2017
- Publisher
- ISCA
- Published
- 2017-08-20
- Pages
- 4006-4010
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2963609956
- Cited by
- 1,746
- References
- 25
- Citations by year
- 2026: 134 · 2025: 187 · 2024: 183 · 2023: 215 · 2022: 243 · 2021: 301 · 2020: 252 · 2019: 176 · 2018: 45 · 2017: 9
- FWCI
- 60.12
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
- Keywords
- Tacotron, end-to-end speech synthesis, text-to-speech, sequence-to-sequence models, naturalness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.