Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
On account of growing demands for personalization, the need for a so-called few-shot TTS system that clones speakers with only a few data is emerging.To address this issue, we propose Attentron, a few-shot TTS model that clones voices of speakers unseen during training.It introduces two special encoders, each serving different purposes.A fine-grained encoder extracts variable-length style information via an attention mechanism, and a coarse-grained encoder greatly stabilizes the speech synthesis, circumventing unintelligible gibberish even for synthesizing speech of unseen speakers.In addition, the model can scale out to an arbitrary number of reference audios to improve the quality of the synthesized speech.According to our experiments, including a human evaluation, the proposed model significantly outperforms state-of-the-art models when generating speech for unseen speakers in terms of speaker similarity and quality.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Due to rapid research progress, it is possible to produce a realistic-sounding clone using only a few audio samples [3].”
From Warning: Humans cannot reliably detect speech deepfakes (2023) · cites this paper as [3]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
librosa: Audio and Music Signal Analysis in Python
McFee, B., Raffel, C., Liang, D., Ellis, D., McVicar, M., Battenberg, E., & Nieto, O. (2015) · Proceedings of the Python in Science Conference · doi:10.25080/majora-7b98e3ed-003 · Cited by 3,033
Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461368 · Cited by 2,791
Efficient Neural Audio Synthesis
Kalchbrenner, N., Elsen, E., Simonyan, K., Noury, S., Casagrande, N., Lockhart, E., Stimberg, F., Oord, A. V. D., Dieleman, S., & Kavukcuoglu, K. (2018) · arXiv · doi:10.48550/arxiv.1802.08435 · Cited by 961
Mel-cepstral distance measure for objective speech quality assessment
Kubichek, R. (2002) · Proceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing · doi:10.1109/pacrim.1993.407206 · Cited by 534
Transfer learning from speaker verification to multispeaker text-to-speech synthesis
Jia, Y., Zhang, Y., Weiss, R. J., Wang, Q., Shen, J., Ren, F., Chen, Z., Nguyen, P., Pang, R., Moreno, I. L., et al. (2018) · arXiv · doi:10.48550/arxiv.1806.04558 · Cited by 519
Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
Wang, Y., Stanton, D., Zhang, Y., Skerry-Ryan, R., Battenberg, E., Shor, J., Xiao, Y., Ren, F., Ye, J., & Saurous, R. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.09017 · Cited by 497
SUPERSEDED - CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit
Veaux, C., Yamagishi, J., & MacDonald, K. (2016) · University of Edinburgh. The Centre for Speech Technology Research (CSTR) · doi:10.7488/ds/1495 · Cited by 434
ClariNet: Parallel Wave Generation in End-to-End Text-to-SpeechPreprint
Ping, W., Peng, K., & Chen, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.07281 · Cited by 351
Hierarchical Generative Modeling for Controllable Speech SynthesisPreprint
Hsu, W.-N., Zhang, Y., Weiss, R. J., Zen, H., Wu, Y., Wang, Y., Cao, Y. J., Ye, J., Chen, Z., Shen, J., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.07217 · Cited by 283
Learning Latent Representations for Style Control and Transfer in End-to-end Speech Synthesis
Zhang, Y.-J., Pan, S., He, L., & Ling, Z.-H. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683623 · Cited by 233
Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
Skerry-Ryan, R., Battenberg, E., Xiao, Y., Wang, Y., Stanton, D., Shor, J., Weiss, R. J., Clark, R., & Saurous, R. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.09047 · Cited by 207
Neural Voice Cloning with a Few Samples
Arık, S. Ö., Chen, J., Peng, K., Ping, W., & Zhou, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.06006 · Cited by 172
Zero-Shot Multi-Speaker Text-To-Speech with State-Of-The-Art Neural Speaker Embeddings
Cooper, E., Lai, C.-I., Yasuda, Y., Fang, F., Wang, X., Chen, N., & Yamagishi, J. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054535 · Cited by 164
Fully-Hierarchical Fine-Grained Prosody Modeling For Interpretable Speech Synthesis
Sun, G., Zhang, Y., Weiss, R. J., Cao, Y. J., Zen, H., & Wu, Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053520 · Cited by 143
Robust and Fine-grained Prosody Control of End-to-end Speech Synthesis
Lee, Y., & Kim, T. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683501 · Cited by 136
Predicting Expressive Speaking Style from Text in End-To-End Speech Synthesis
Stanton, D., Wang, Y., & Skerry-Ryan, R. (2018) · 2018 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639682 · Cited by 114
Hsu, W.-N., Zhang, Y., Weiss, R. J., Chung, Y.-A., Wang, Y., Wu, Y., & Glass, J. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683561 · Cited by 111
Fitting New Speakers Based on a Short Untranscribed SamplePreprint
Nachmani, E., Polyak, A., Taigman, Y., & Wolf, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.06984 · Cited by 96
Sample Efficient Adaptive Text-to-Speech
Chen, Y., Assael, Y., Shillingford, B., Budden, D., Reed, S., Zen, H., Wang, Q., Cobo, L. C., Trask, A., Laurie, B., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10460 · Cited by 94
Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS
He, M., Deng, Y., & He, L. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1972 · Cited by 76
Cross-Lingual, Multi-Speaker Text-To-Speech Synthesis Using Neural Speaker Embedding
Chen, M., Chen, M., Liang, S., Ma, J., Chen, L., Wang, S., & Xiao, J. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1632 · Cited by 59
High Quality, Lightweight and Adaptable TTS Using LPCNet
Kons, Z., Shechtman, S., Sorin, A., Rabinovitz, C., & Hoory, R. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1705 · Cited by 51
Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis
Battenberg, E., Mariooryad, S., Stanton, D., Skerry-Ryan, R., Shannon, M., Kao, D., & Bagby, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.03402 · Cited by 41
Neural Text-to-Speech Adaptation from Low Quality Public Recordings
Hu, Q., Marchi, E., Winarsky, D., Stylianou, Y., Naik, D., & Kajarekar, S. (2019) · 10th ISCA Workshop on Speech Synthesis (SSW 10) · doi:10.21437/ssw.2019-5 · Cited by 31
Deng, Y., He, L., & Soong, F. K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.05253 · Cited by 30
Lombard Speech Synthesis Using Transfer Learning in a Tacotron Text-to-Speech System
Bollepalli, B., Juvela, L., & Alku, P. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1333 · Cited by 25
Multi-Speaker End-to-End Speech SynthesisPreprint
Park, J., Zhao, K., Peng, K., & Ping, W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.04462 · Cited by 23
Cited by
A survey on neural speech synthesis
Tan, X., Qin, T., Soong, F., & Liu, T.-Y. (2021) · arXiv · doi:10.48550/arxiv.2106.15561 · Cited by 216
Warning: Humans cannot reliably detect speech deepfakes
Mai, K. T., Bray, S., Davies, T., & Griffin, L. D. (2023) · PLOS ONE · doi:10.1371/journal.pone.0285333 · Cited by 98
“Due to rapid research progress, it is possible to produce a realistic-sounding clone using only a few audio samples (Choi et al., 2020).”
BackgroundSC-GlowTTS: An Efficient Zero-Shot Multi-Speaker Text-To-Speech Model
Casanova, E., Shulby, C., Gölge, E., Müller, N. M., Oliveira, F. S. D., Candido Jr., A., Soares, A. D. S., Aluisio, S. M., & Ponti, M. A. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1774 · Cited by 83
Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech
Huang, S.-F., Lin, C.-J., Liu, D.-R., Chen, Y.-C., & Lee, H.-Y. (2022) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2022.3167258 · Cited by 59
AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset
Cai, Z., Ghosh, S., Adatia, A. P., Hayat, M., Dhall, A., Gedeon, T., & Stefanov, K. (2024) · Proceedings of the 32nd ACM International Conference on Multimedia · doi:10.1145/3664647.3680795 · Cited by 59
Metadata
- Authors
- Choi, S., Han, S., Kim, D., & Ha, S.
- Venue
- Interspeech 2020
- Publisher
- ISCA
- Published
- 2020-10-25
- Pages
- 2007-2011
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W3095199334
- Cited by
- 56
- References
- 29
- Citations by year
- 2026: 2 · 2025: 9 · 2024: 13 · 2023: 10 · 2022: 15 · 2021: 7
- FWCI
- 3.06
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
- Keywords
- attention mechanism, speaker similarity, synthetic speech quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.