Tacotron: Towards End-to-End Speech Synthesis
The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module.Building these components often requires extensive domain expertise and may contain brittle design choices.In this paper, we present Tacotron, an end-to-end generative text-to-speech model that synthesizes speech directly from characters.Given pairs, the model can be trained completely from scratch with random initialization.We present several key techniques to make the sequence-tosequence framework perform well for this challenging task.Tacotron achieves a 3.82 subjective 5-scale mean opinion score on US English, outperforming a production parametric system in terms of naturalness.In addition, since Tacotron generates speech at the frame level, it's substantially faster than sample-level autoregressive methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The frameworks including their code as well as the datasets used that are available publicly (Sotelo et al., 2017; Wang et al., 2017; Ping et al., 2018; Shen et al., 2018; Vasquez and Lewis, 2019) are more likely to be used for nefarious purposes or research, so they will be more developed.”
From Audio deepfakes: A survey (2023) · cites this paper as Wang et al., 2017
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“We further pass the processed sequence to a few fixed-width 1-D convolutions, whose outputs are added with the original input sequence via residual connections [17].”
BackgroundAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 2×
“We use the Adam optimizer (Kingma & Ba, 2015) with learning rate decay, which starts from 0.001 and is reduced to 0.0005, 0.0003, and 0.0001 after 500K, 1M and 2M global steps, respectively.”
MethodInfluentialBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 2×
“Batch normalization [18] is used for all convolutional layers.”
MethodSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“In this paper, we propose Tacotron, an end-to-end generative TTS model based on the sequence-to-sequence (seq2seq) (Sutskever et al., 2014) with attention paradigm (Bahdanau et al., 2014).”
MethodSelf-citationNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“The backbone of Tacotron is a seq2seq model with attention [7, 14].”
MethodInfluentialEmpirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · cited 7×
“Second, we compare with a model with the CBHG encoder replaced by a 2-layer residual GRU encoder.”
BackgroundInfluentialTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · cited 2×
“We implemented Griffin-Lim in TensorFlow [22] hence it’s also part of the model.”
MethodSelf-citationGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 4×
“C L ] 6 A pr or machine translation (Wu et al., 2016), TTS outputs are continuous, and output sequences are usually much longer than those of the input.”
BackgroundMethodInfluentialSelf-citationWaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · cited 4×
“It could be used to predict alternative targets such as vocoder parameters, or as a WaveNetlike neural vocoder [9, 13, 10] that synthesizes waveform samples directly.”
BackgroundMethodInfluentialListen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226 · cited 2×
“Moreover, unlike end-to-end speech recognition [4] or machine translation [5], TTS outputs are continuous, and output sequences are usually much longer than those of the input.”
BackgroundSelf-citation
Cited by
Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461368 · Cited by 2,791
Self-citationIEEE Symposium on Security and Privacy
(2023) · IT Professional · doi:10.1109/mitp.2023.3246667 · Cited by 1,167
Data Augmentation Generative Adversarial Networks
Antoniou, A., Storkey, A., & Edwards, H. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.04340 · Cited by 747
“Waveglow: a flow-based generative network for speech synthesis,”
Prenger, R., Valle, R., & Catanzaro, B. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683143 · Cited by 1,004
Deep Learning for Audio Signal Processing
Purwins, H., Li, B., Virtanen, T., Schluter, J., Chang, S.-Y., & Sainath, T. (2019) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2908700 · Cited by 855
“via the Griffin-Lim algorithm [65] in combination with the inverse Fourier transform [139] which does not reach high synthesis”
MethodFastSpeech: Fast, Robust and Controllable Text to Speech
Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z. L., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09263 · Cited by 559
A Comparative Study on Transformer vs RNN in Speech Applications
Karita, S., Chen, N., Hayashi, T., Hori, T., Inaguma, H., Jiang, Z., Someki, M., Soplin, N. E. Y., Yamamoto, R., Wang, X., et al. (2019) · 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru46091.2019.9003750 · Cited by 727
“• A reduction factor introduced in [51] was also effective for Transformer.”
BackgroundMelGAN: generative adversarial networks for conditional waveform synthesis
Kumar, K., Kumar, R., de Boissiere, T., Gestin, L., Teoh, W. Z., Sotelo, J., de Brebisson, A., Bengio, Y., & Courville, A. (2019) · arXiv · doi:10.48550/arxiv.1910.06711 · Cited by 722
“xample, the Griffin-Lim (Griffin & Lim, 1984) algorithm allows one to efficiently decode an STFT sequence back to the temporal signal at the cost of introducing strong, robotic artifacts as noted in Wang et al. (2017). More sophisticated representations and signal processing techniques have been investigated. For ins…”
MethodAdversarial Audio SynthesisPreprint
Donahue, C., McAuley, J., & Puckette, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04208 · Cited by 719
“Recently, several researchers have investigated parametric speech synthesis with end-to-end neural network approaches that learn to produce vocoder features directly from text or phonetic embeddings (Arik et al., 2017; Ping et al., 2018; Sotelo et al., 2017; Wang et al., 2017; Shen et al., 2018).”
BackgroundStyle Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
Wang, Y., Stanton, D., Zhang, Y., Skerry-Ryan, R., Battenberg, E., Shor, J., Xiao, Y., Ren, F., Ye, J., & Saurous, R. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.09017 · Cited by 497
“ermore, they often lack the ability to control the expression with which speech is synthesized. This work attempts to address the above issues by introducing “global style tokens” (GSTs) to Tacotron (Wang et al., 2017a;Shen et al.,2017), a state-of-the-art end-to-end TTS model. GSTs are trained without any prosodic la…”
BackgroundMethodResultInfluentialSelf-citationWaveGrad: Estimating Gradients for Waveform GenerationPreprint
Chen, N., Zhang, Y., Zen, H., Weiss, R. J., Norouzi, M., & Chan, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.00713 · Cited by 758
Self-citationFastSpeech 2: Fast and High-Quality End-to-End Text to Speech
Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T.-Y. (2020) · arXiv · doi:10.48550/arxiv.2006.04558 · Cited by 580
“Previous neural TTS models such as Tacotron [25], Tacotron 2 [21], Deep Voice 3 [17] and Transformer TTS [10] first generate mel-spectrograms autoregressively from text and then synthesize speech from the generated mel-spectrograms using a separately trained vocoder (e.”
MethodInfluentialDiffWave: A Versatile Diffusion Model for Audio Synthesis
Kong, Z., Ping, W., Huang, J., Zhao, K., & Catanzaro, B. (2020) · arXiv · doi:10.48550/arxiv.2009.09761 · Cited by 569
“, 2018), Tacotron 1 & 2 (Wang et al., 2017; Shen et al., 2018), Char2Wav (Sotelo et al.”
Masood, M., Nawaz, M., Malik, K. M., Javed, A., Irtaza, A., & Malik, H. (2023) · Applied Intelligence · doi:10.1007/s10489-022-03766-z · Cited by 543
Transfer learning from speaker verification to multispeaker text-to-speech synthesis
Jia, Y., Zhang, Y., Weiss, R. J., Wang, Q., Shen, J., Ren, F., Chen, Z., Nguyen, P., Pang, R., Moreno, I. L., et al. (2018) · arXiv · doi:10.48550/arxiv.1806.04558 · Cited by 519
Self-citationAn Unsupervised Autoregressive Model for Speech Representation Learning
Chung, Y.-A., Hsu, W.-N., Tang, H., & Glass, J. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1473 · Cited by 357
An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning
Sisman, B., Yamagishi, J., King, S., & Li, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2020.3038524 · Cited by 357
“Tacotron [89], [180], [218]–[220] represents one of the successful text-to-speech (TTS) implementations, that has been extended to voice conversion [3], [183], [221].”
BackgroundTowards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
Skerry-Ryan, R., Battenberg, E., Xiao, Y., Wang, Y., Stanton, D., Shor, J., Weiss, R. J., Clark, R., & Saurous, R. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.09047 · Cited by 207
Self-citationA review of deep learning techniques for speech processing
Mehrish, A., Majumder, N., Bharadwaj, R., Mihalcea, R., & Poria, S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101869 · Cited by 389
GANSynth: Adversarial Neural Audio Synthesis
Engel, J., Agrawal, K. K., Chen, S., Gulrajani, I., Donahue, C., & Roberts, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.08710 · Cited by 264
“Much of the work on deep generative models for audio tends to focus on speech synthesis (van den Oord et al., 2018; Sotelo et al., 2017; Wang et al., 2017).”
BackgroundDeep learning for determining a near-optimal topological design without any iteration
Yu, Y., Hur, T., Jung, J., & Jang, I. G. (2019) · Structural and Multidisciplinary Optimization · doi:10.1007/s00158-018-2101-5 · Cited by 353
DDSP: Differentiable Digital Signal ProcessingPreprint
Engel, J., Hantrakul, L., Gu, C., & Roberts, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.04643 · Cited by 575
“Fourier-based models–such as Tacotron (Wang et al., 2017) and GANSynth (Engel et al., 2019)– also suffer from the phase-alignment problem, as the Short-time Fourier Transform (STFT) is a representation over windowed wave packets.”
MethodMusic Source Separation in the Waveform Domain
Défossez, A., Usunier, N., Bottou, L., & Bach, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.13254 · Cited by 175
“Approaches that follow this line either generate spectrograms [e.g., Wang et al., 2017], or use a loss that compares power spectrograms of target/generated waveforms [Défossez et al., 2018].”
BackgroundFinding, visualizing, and quantifying latent structure across diverse animal vocal repertoires
Sainburg, T., Thielk, M., & Gentner, T. Q. (2020) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1008228 · Cited by 319
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation
Michelsanti, D., Tan, Z.-H., Zhang, S.-X., Xu, Y., Yu, M., Yu, D., & Jensen, J. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3066303 · Cited by 316
“system was designed to get a linear-scale spectrogram from a learned mel-scale one with the post-processing network in [268].”
MethodProtein design and variant prediction using autoregressive generative models
Shin, J.-E., Riesselman, A. J., Kollasch, A. W., McMahon, C., Simon, E., Sander, C., Manglik, A., Kruse, A. C., & Marks, D. S. (2021) · Nature Communications · doi:10.1038/s41467-021-22732-w · Cited by 320
“By contrast, the deep models that have transformed our ability to generate realistic speech such as text-to-speech 49,50 and translation 51,52 use generative models that do not require “ word alignment ” , e.g., between equisemantic sentences, but instead employ an autoregressive likelihood to…”
BackgroundClariNet: Parallel Wave Generation in End-to-End Text-to-SpeechPreprint
Ping, W., Peng, K., & Chen, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.07281 · Cited by 320
“In contrast, Tacotron, Deep Voice 3, and Char2Wav employ the attention based sequence-to-sequence models (Bahdanau et al., 2015), yielding more compact architectures.”
BackgroundMethodInfluentialDeep voice 3: scaling text-to-speech with convolutional sequence learning
Ping, W., Peng, K., Gibiansky, A., Arik, S. O., Kannan, A., Narang, S., Raiman, J., & Miller, J. (2017) · arXiv · doi:10.48550/arxiv.1710.07654 · Cited by 313
“For our system on single-speaker data, the average training iteration time (for batch size 4) is 0.06 seconds using one GPU as opposed to 0.59 seconds for Tacotron, indicating a ten-fold increase in training speed.”
BackgroundMethodResultInfluentialDeep Learning Enabled Semantic Communications With Speech Recognition and Synthesis
Weng, Z., Qin, Z., Tao, X., Pan, C., Liu, G., & Li, G. Y. (2023) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2023.3240969 · Cited by 286
“2, Tacotron 2 is adopted to reconstruct the speech sample sequences, M̂ , as close to the input sample sequences, M , as possible.”
MethodInfluentialNaturalSpeech: End-to-End Text-to-Speech Synthesis With Human-Level Quality
Tan, X., Chen, J., Liu, H., Cong, J., Zhang, C., Liu, Y., Wang, X., Leng, Y., Yi, Y., He, L., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3356232 · Cited by 195
Tachibana, H., Uenoyama, K., & Aihara, S. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461829 · Cited by 274
Diffsound: Discrete Diffusion Model for Text-to-Sound Generation
Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., & Yu, D. (2023) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2023.3268730 · Cited by 211
“AR models [22], [23] have shown powerful generation capability and have been applied for image generation [7], [24], [25], [26], [27], [28], [29], speech synthesis [30] and sound generation [3], [4].”
BackgroundChou, J.-C., & Lee, H.-Y. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2663 · Cited by 232
Does Audio Deepfake Detection Generalize?
Müller, N., Czempin, P., Diekmann, F., Froghyar, A., & Böttinger, K. (2022) · Interspeech 2022 · doi:10.21437/interspeech.2022-108 · Cited by 258
Deep Voice 2: Multi-Speaker Neural Text-to-Speech
Arık, S. Ö., Diamos, G., Gibiansky, A., Miller, J. J., Peng, K., Ping, W., Raiman, J., & Zhou, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08947 · Cited by 206
“Our contributions build upon recent work in entirely neural TTS systems, including Deep Voice 1 (Arik et al., 2017), Tacotron (Wang et al., 2017), and Char2Wav (Sotelo et al., 2017).”
MethodInfluentialDirect Speech-to-Speech Translation with a Sequence-to-Sequence Model
Jia, Y., Weiss, R. J., Biadsy, F., Macherey, W., Johnson, M., Chen, Z., & Wu, Y. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1951 · Cited by 184
Self-citationEnd-to-End Neural Speaker Diarization with Self-Attention
Fujita, Y., Kanda, N., Horiguchi, S., Xue, Y., Nagamatsu, K., & Watanabe, S. (2019) · 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru46091.2019.9003959 · Cited by 210
“This neural-network-based end-to-end approach, in which only one neural network model directly computes the final outputs, has been successfully applied in a variety of tasks, including neural machine translation [33,34], automatic speech recognition [35–37], and text-to-speech [38, 39].”
High Fidelity Speech Synthesis with Adversarial NetworksPreprint
Bińkowski, M., Donahue, J., Dieleman, S., Clark, A., Elsen, E., Casagrande, N., Cobo, L. C., & Simonyan, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11646 · Cited by 233
“Tacotron (Wang et al., 2017) and MelNet (Vasquez & Lewis, 2019) constitute notable exceptions, and they use the Griffin-Lim algorithm (Griffin & Lim, 1984) to reconstruct missing phase information, which the models themselves do not generate.”
ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech
Huang, R., Zhao, Z., Liu, H., Liu, J., Cui, C., & Ren, Y. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3547855 · Cited by 183
“, where we first convert the ground-truth audio into mel-spectrograms and then convert them back to audio using HiFi-GAN (V1) [19]; 3) Tacotron 2 [42]: the traditional autoregressive TTS model; 4) FastSpeech 2 [34]: one of the most popular non-autoregressive TTS models; 5) Glow-TTS [16]: the TTS model with the normali…”
BackgroundMethodInfluentialAISHELL-3: A Multi-speaker Mandarin TTS Corpus and the BaselinesPreprint
Shi, Y., Bu, H., Xu, X., Zhang, S., & Li, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11567 · Cited by 259
“This flourishing research progress is made largely due to the introduction of attention based sequence-to-sequence modeling architectures such as Tacotron [2, 3] or Transformer-TTS [4], and neural vocoders that map the lower dimensional acoustic representation to waveforms [5, 6, 7].”
InfluentialA Review on Methods and Applications in Multimodal Deep Learning
Jabeen, S., Li, X., Amin, M. S., Bourahla, O., Li, S., & Jabbar, A. (2023) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3545572 · Cited by 196
“[145] proposed "Tacotron," a sequence 2 sequence TTS framework that synthesizes speech from text and audio pairs.”
BackgroundMethodInfluentialJoint Audio-Visual Deepfake Detection
Zhou, Y., & Lim, S.-N. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01453 · Cited by 217
A survey on neural speech synthesis
Tan, X., Qin, T., Soong, F., & Liu, T.-Y. (2021) · arXiv · doi:10.48550/arxiv.2106.15561 · Cited by 216
“, Tacotron 1/2 [376, 297], Deep Voice 3 [264], and FastSpeech 1/2 [284, 286]) are proposed to simplify text analysis modules and directly take character/phoneme sequences as input, and simplify acoustic features with mel-spectrograms.”
BackgroundMethodInfluentialLearning the travelling salesperson problem requires rethinking generalization
Joshi, C. K., Cappart, Q., Rousseau, L.-M., & Laurent, T. (2022) · Constraints · doi:10.1007/s10601-022-09327-y · Cited by 285
Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
Jeong, M., Kim, H., Cheon, S. J., Choi, B. J., & Kim, N. S. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-469 · Cited by 207
Monotonic Chunkwise AttentionPreprint
Chiu, C., & Raffel, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.05382 · Cited by 207
“In future work, we are interested in applying MoChA to additional problems with (approximately) monotonic alignments, such as speech synthesis (Wang et al., 2017) and morphological inflection (Aharoni & Goldberg, 2016).”
BackgroundMethodInfluentialStep-Audio 2 Technical ReportPreprint
Wu, B., Yan, C., Chen, H., Yi, C., Feng, C., Tian, F., Shen, F., Yu, G., Zhang, H., Li, J., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.16632 · Cited by 233
“Text-to-Speech (TTS) technology has made remarkable strides in recent years, evolving from traditional concatenative and statistical parametric approaches [52, 55, 59, 68] to codec-based TTS systems.”
Self-citationMoGlow: probabilistic and controllable motion synthesis using normalising flows
Henter, G. E., Alexanderson, S., & Beskow, J. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417836 · Cited by 176
Generative AI-Driven Semantic Communication Networks: Architecture, Technologies, and Applications
Liang, C., Du, H., Sun, Y., Niyato, D. T., Kang, J., Zhao, D., & Imran, M. A. (2024) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2024.3435524 · Cited by 244
“A famous example is Tacotron [99], which is a Seq2Seq model that maps character embeddings to mel-scale spectrograms, which are then converted to audio using a vocoder like WaveNet or Griffin-Lim.”
Speech Technology for Healthcare: Opportunities, Challenges, and State of the Art
Latif, S., Qadir, J., Qayyum, A., Usama, M., & Younis, S. (2021) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2020.3006860 · Cited by 173
Espnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit
Hayashi, T., Yamamoto, R., Inoue, K., Yoshimura, T., Watanabe, S., Toda, T., Takeda, K., Zhang, Y., & Tan, X. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053512 · Cited by 172
“The conversion is performed by applying the inverse Mel basis or the convolutional bank highway network GRU (CBHG) network [5].”
BackgroundMethodA Review of Deep Learning Based Speech Synthesis
Ning, Y., He, S., Wu, Z., Xing, C., & Zhang, L.-J. (2019) · Applied Sciences · doi:10.3390/app9194050 · Cited by 163
“The overall framework of the Tacotron speech synthesis model can be seen in [63].”
BackgroundMERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound
Zellers, R., Lu, J., Lu, X., Yu, Y., Zhao, Y., Salehi, M., Kusupati, A., Hessel, J., Farhadi, A., & Choi, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01589 · Cited by 160
“We used a small number of mel spectrogram bins (64) because we found that at that threshold, we could reconstruct the original sequence at an acceptable level using the Griffin-Lim algorithm, [52] which itself might be a lower bound on quality as neural methods trained for this purpose have been shown to do better [11…”
BackgroundMethodAdaSpeech: Adaptive Text to Speech for Custom VoicePreprint
Chen, M., Tan, X., Li, B., Liu, Y., Qin, T., Zhao, S., & Liu, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.00993 · Cited by 252
“Text to speech (TTS) aims to synthesize natural and intelligible voice from text, and attracts a lot of interests in machine learning community (Arik et al., 2017; Wang et al., 2017; Gibiansky et al., 2017; Ping et al., 2018; Shen et al., 2018; Ren et al., 2019).”
BackgroundA neural attention model for speech command recognitionPreprint
de Andrade, D. C., Leo, S., Viana, M. L. D. S., & Bernkopf, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.08929 · Cited by 174
Deep learning for fabrication and maturation of 3D bioprinted tissues and organs
Ng, W. L., Chan, A., Ong, Y. S., & Chua, C. K. (2020) · Virtual and Physical Prototyping · doi:10.1080/17452759.2020.1771741 · Cited by 161
“…remarkable performance in inferring and manipulating data such as images (Li et al. 2019; Tan and Le 2019; Karras, Laine, and Aila 2019; Mao et al. 2018), language (Yang et al. 2019; Radford et al. 2019; Dong et al. 2019; Edunov et al. 2018) and speech (Wang et al. 2017; Amodei et al. 2016).”
BackgroundADD 2022: the first Audio Deep Synthesis Detection Challenge
Yi, J., Fu, R., Tao, J., Nie, S., Ma, H., Wang, C., Wang, T., Tian, Z., Bai, Y., Fan, C., et al. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9746939 · Cited by 165
JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesisPreprint
Sonobe, R., Takamichi, S., & Saruwatari, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00354 · Cited by 165
“Some studies on speech synthesis re-ported methods that do not use linguistic knowledge, e.g., no use of intermediate representations such as phonemes, in English, Spanish, and German [5, 6, 7].”
Zero-Shot Multi-Speaker Text-To-Speech with State-Of-The-Art Neural Speaker Embeddings
Cooper, E., Lai, C.-I., Yasuda, Y., Fang, F., Wang, X., Chen, N., & Yamagishi, J. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054535 · Cited by 164
“Our end-to-end multi-speaker text-to-speech model architecture is based on Tacotron [1], with the extension of self-attention described in [28] to better capture long-range dependencies illustrated in Figure 2.”
BackgroundMethodLeveraging Weakly Supervised Data to Improve End-to-end Speech-to-text Translation
Ye, J., Johnson, M., Macherey, W., Weiss, R. J., Cao, Y. J., Chiu, C., Ari, N., Laurenzo, S., & Wu, Y. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683343 · Cited by 210
Self-citationTextless Speech-to-Speech Translation on Real Data
Lee, A., Gong, H., Duquenne, P.-A., Schwenk, H., Chen, P., Wang, C., Popuri, S., Adi, Y., Pino, J., Gu, J., et al. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.63 · Cited by 227
A Comprehensive Review of Data‐Driven Co‐Speech Gesture Generation
Nyatsanga, S., Kucherenko, T., Ahuja, C., Henter, G. E., & Neff, M. (2023) · Computer Graphics Forum · doi:10.1111/cgf.14776 · Cited by 221
“This is similar to the focus on perceptual surface quality in other areas such as image generation [KLA ∗ 20, RPG ∗ 21, RDN ∗ 22, RBL ∗ 22] and speech synthesis [WSRS ∗ 17,vdODZ ∗ 16].”
End-to-End Adversarial Text-to-SpeechPreprint
Donahue, J., Dieleman, S., Bińkowski, M., Elsen, E., & Simonyan, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03575 · Cited by 175
“Tacotron 1 & 2 (Wang et al., 2017; Shen et al., 2018), Deep Voice 2 & 3 (Gibiansky et al.”
Wang, H., Peng, Z., Wang, D., Meng, Y., Wu, T., Sun, W., & Lu, Q.-C. (2020) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2020.102619 · Cited by 151
Listening while speaking: Speech chain by deep learning
Tjandra, A., Sakti, S., & Nakamura, S. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8268950 · Cited by 151
MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec TransformerPreprint
Wang, Y., Zhan, H., Liu, L., Zeng, R., Guo, H., Zheng, J., Zhang, Q. M., Zhang, X., Zhang, S., & Wu, Z. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.00750 · Cited by 189
“Large-scale TTS. Traditional TTS systems [21, 22, 23, 24, 25] are trained to generate speech from a single speaker or multiple speakers using hours of high-quality transcribed training data.”
BackgroundSelf-citationExpressive Speech Synthesis via Modeling Expressions with Variational Autoencoder
Akuzawa, K., Iwasawa, Y., & Matsuo, Y. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1113 · Cited by 147
SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks
Kimura, N., Kono, M., & Rekimoto, J. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300376 · Cited by 157
A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AIPreprint
Zhang, C., Zhang, C., Zheng, S., Zhang, M., Qamar, M., Bae, S., & Kweon, I. S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.13336 · Cited by 141
“, formant synthesis [40, 41, 95] and concatenative synthesis [29, 69, 75]), statistical parametric speech synthesis (SPSS)-based methods [115, 116, 133], and neural network-based stage [122, 135, 136].”
Emotion Intensity and its Control for Emotional Voice Conversion
Zhou, K., Şişman, B., Rana, R. K., Schuller, B. W., & Li, H. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3175578 · Cited by 181
“The Seq2Seq decoder has the same model architecture as that of Tacotron [37].”
BackgroundMethodInfluentialVioLA: Conditional Language Models for Speech Recognition, Synthesis, and Translation
Wang, T., Zhou, L., Zhang, Z., Wu, Y., Liu, S., Gaur, Y., Chen, Z., Li, J., & Wei, F. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3434425 · Cited by 148
“For speech synthesis tasks, AED methods such as Tacotron [22], Tacotron2 [23], andTransformerTTS[24],andduration-basedmethods,includingFast-speech[25],Fastspeech2[26],andRobuTrans[27]arewidelyused.”
MethodSelf-citationSample Efficient Adaptive Text-to-SpeechPreprint
Chen, Y., Assael, Y., Shillingford, B., Budden, D., Reed, S., Zen, H., Wang, Q., Cobo, L. C., Trask, A., Laurie, B., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10460 · Cited by 148
“, 2018) (building on (Wang et al., 2017)) which uses WaveNet as a vocoder to invert mel-spectrograms generated by an attentive sequence-to-sequence model.”
MethodRobust and Fine-grained Prosody Control of End-to-end Speech Synthesis
Lee, Y., & Kim, T. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683501 · Cited by 136
Lifelo, Z., Ding, J., Ning, H., Ain, Q. U., & Dhelim, S. (2024) · Electronics · doi:10.3390/electronics13244874 · Cited by 173
Chou, J.-C., Yeh, C.-C., Lee, H.-Y., & Lee, L.-S. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1830 · Cited by 135
Sequence-to-Sequence Acoustic Modeling for Voice Conversion
Zhang, J.-X., Ling, Z.-H., Liu, L.-J., Jiang, Y., & Dai, L.-R. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2892235 · Cited by 167
“Text-to-speech (TTS) methods based on seq2seq learning have emerged recently and attracted much attention [24]– [27].”
BackgroundMethodInfluentialA Survey on the Detection and Impacts of Deepfakes in Visual, Audio, and Textual Formats
Mubarak, R., Alsboui, T., Alshaikh, O., Inuwa-Dutse, I., Khan, S., & Parkinson, S. (2023) · IEEE Access · doi:10.1109/access.2023.3344653 · Cited by 132
“Recent advancements in deep learning-based approaches, such as WaveNet [73], Deepvoice [74], Tacotron [75], and NaturalSpeech [76], have significantly enhanced the TTS.”
MethodDeepSonar: Towards Effective and Robust Detection of AI-Synthesized Fake Voices
Wang, R., Juefei-Xu, F., Huang, Y., Guo, Q., Xie, X., Ma, L., & Liu, Y. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413716 · Cited by 131
Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech SynthesisPreprint
Valle, R., Shih, K. J., Prenger, R., & Catanzaro, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.05957 · Cited by 131
“We use the ADAM (Kingma & Ba, 2014) optimizer with default parameters, 1e-4 learning rate and 1e-6 weight decay for Flowtron and 1e-3 learning rate and 1e-5 weight decay for the other models, following guidelines in (Wang et al., 2017).”
BackgroundMethodFast Spectrogram Inversion Using Multi-Head Convolutional Neural Networks
Arık, S. Ö., Jun, H., & Diamos, G. (2018) · IEEE Signal Processing Letters · doi:10.1109/lsp.2018.2880284 · Cited by 127
InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt
Yang, D., Liu, S., Huang, R., Weng, C., & Meng, H. M. L. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3402088 · Cited by 218
“Nowadays, the state-of-the-art TTS systems [1], [2], [3] are able to produce natural and high-quality speech.”
BackgroundVoice Conversion Challenge 2020 –- Intra-lingual semi-parallel and cross-lingual voice conversion –-
Yi, Z., Huang, W.-C., Tian, X., Yamagishi, J., Das, R. K., Kinnunen, T., Ling, Z.-H., & Toda, T. (2020) · Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge 2020 · doi:10.21437/vcc_bc.2020-14 · Cited by 126
Direct Speech-to-Speech Translation With Discrete Units
Lee, A., Chen, P., Wang, C., Gu, J., Popuri, S., Ma, X., Polyak, A., Adi, Y., He, Q., Tang, Y., et al. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.235 · Cited by 225
Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis
Wang, Y., Wang, X., Zhu, P., Wu, J., Li, H., Xue, H., Zhang, Y., Xie, L., & Bi, M. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-48 · Cited by 126
Self-citationPredicting Expressive Speaking Style from Text in End-To-End Speech Synthesis
Stanton, D., Wang, Y., & Skerry-Ryan, R. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639682 · Cited by 155
Self-citationKhanjani, Z., Watson, G., & Janeja, V. P. (2023) · Frontiers in Big Data · doi:10.3389/fdata.2022.1001063 · Cited by 121
“Therefore, Tacotron 2 (Shen et al., 2018) system includes two components.”
BackgroundMethodInfluentialBiadsy, F., Weiss, R. J., Moreno, P. J., Kanvesky, D., & Ye, J. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1789 · Cited by 142
Self-citationAI-Generated Content (AIGC) for Various Data Modalities: A Survey
Foo, L. G., Rahmani, H., & Liu, J. (2025) · ACM Computing Surveys · doi:10.1145/3728633 · Cited by 137
A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond
Xiao, Y., Wu, L., Guo, J., Li, J., Zhang, M., Qin, T., & Liu, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3277122 · Cited by 174
“, simplify data distributions [132], [133], which provides more conditional input information, and enhance modeling methods [134], [135], which try to enhance the model capacity to fit the complex data distributions.”
BackgroundSelf-citationFakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset
Khalid, H., Tariq, S., Kim, M., & Woo, S. S. (2021) · arXiv · doi:10.48550/arxiv.2108.05080 · Cited by 116
“It generates a fixeddimensional embedding vector of input audio, a synthesis network based on Tacotron 2 that generates Mel spectrogram, and a WaveNet-based vocoder network that converts the Mel spectrogram into time-domain waveform samples.”
MethodTransformers in speech processing: Overcoming challenges and paving the future
Latif, S., Zaidi, S. A. M., Cuayáhuitl, H., Shamshad, F., Shoukat, M., Usama, M., & Qadir, J. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100768 · Cited by 124
VoiceLoop: Voice Fitting and Synthesis via a Phonological LoopPreprint
Taigman, Y., Wolf, L., Polyak, A., & Nachmani, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06588 · Cited by 143
Recent Advances and Trends in Multimodal Deep Learning: A ReviewPreprint
Jabeen, S., Li, X., Amin, M. S., Li, S., & Abdul, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.11087 · Cited by 139
“[156] proposed "Tacotron," a sequence 2 sequence TTS framework that synthesizes speech from text and audio pairs.”
BackgroundMethodInfluentialHuang, W.-C., Hayashi, T., Wu, Y.-C., Kameoka, H., & Toda, T. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1066 · Cited by 129
Self-citationATTS2S-VC: Sequence-to-sequence Voice Conversion with Attention and Context Preservation Mechanisms
Tanaka, K., Kameoka, H., Kaneko, T., & Hojo, N. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683282 · Cited by 112
Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors
Khalid, H., Kim, M., Tariq, S., & Woo, S. S. (2021) · Proceedings of the 1st Workshop on Synthetic Multimedia - Audiovisual Deepfake Generation and Detection · doi:10.1145/3476099.3484315 · Cited by 110
“Tacotron [85] is widely used for speech synthesis purposes.”
BackgroundMethodHalf-Truth: A Partially Fake Audio Detection Dataset
Yi, J., Bai, Y., Tao, J., Ma, H., Tian, Z., Wang, C., Wang, T., & Fu, R. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-930 · Cited by 156
Hsu, W.-N., Zhang, Y., Weiss, R. J., Chung, Y.-A., Wang, Y., Wu, Y., & Glass, J. R. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683561 · Cited by 140
Self-citationCSS10: A Collection of Single Speaker Speech Datasets for 10 Languages
Park, K., & Mulc, T. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1500 · Cited by 109
MelNet: a generative model for audio in the frequency domain
Vasquez, S., & Lewis, M. (2019) · arXiv · doi:10.48550/arxiv.1906.01083 · Cited by 107
“Many existing end-to-end TTS models are designed to generate a single high-quality sample for a given text [2, 43, 53, 37, 44].”
BackgroundResult
Metadata
- Authors
- Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., Le, Q., Agiomyrgiannakis, Y., Clark, R., & Saurous, R. A.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Wang, Yuxuan
- ORCID
- Venue
- Interspeech 2017
- Publisher
- ISCA
- Published
- 2017-08-20
- Pages
- 4006-4010
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2963609956
- Cited by
- 1,746
- References
- 25
- Citations by year
- 2026: 134 · 2025: 187 · 2024: 183 · 2023: 215 · 2022: 243 · 2021: 301 · 2020: 252 · 2019: 176 · 2018: 45 · 2017: 9
- FWCI
- 60.12
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
- Keywords
- Tacotron, end-to-end speech synthesis, text-to-speech, sequence-to-sequence models, naturalness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.