Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
On account of growing demands for personalization, the need for a so-called few-shot TTS system that clones speakers with only a few data is emerging.To address this issue, we propose Attentron, a few-shot TTS model that clones voices of speakers unseen during training.It introduces two special encoders, each serving different purposes.A fine-grained encoder extracts variable-length style information via an attention mechanism, and a coarse-grained encoder greatly stabilizes the speech synthesis, circumventing unintelligible gibberish even for synthesizing speech of unseen speakers.In addition, the model can scale out to an arbitrary number of reference audios to improve the quality of the synthesized speech.According to our experiments, including a human evaluation, the proposed model significantly outperforms state-of-the-art models when generating speech for unseen speakers in terms of speaker similarity and quality.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Due to rapid research progress, it is possible to produce a realistic-sounding clone using only a few audio samples [3].”
From Warning: Humans cannot reliably detect speech deepfakes (2023) · cites this paper as [3]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
librosa: Audio and Music Signal Analysis in Python
McFee, B., Raffel, C., Liang, D., Ellis, D., McVicar, M., Battenberg, E., & Nieto, O. (2015) · Proceedings of the Python in Science Conference · doi:10.25080/majora-7b98e3ed-003 · Cited by 3,033
Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461368 · Cited by 2,791
Mel-cepstral distance measure for objective speech quality assessment
Kubichek, R. (2002) · Proceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing · doi:10.1109/pacrim.1993.407206 · Cited by 534
Transfer learning from speaker verification to multispeaker text-to-speech synthesis
Jia, Y., Zhang, Y., Weiss, R. J., Wang, Q., Shen, J., Ren, F., Chen, Z., Nguyen, P., Pang, R., Moreno, I. L., et al. (2018) · arXiv · doi:10.48550/arxiv.1806.04558 · Cited by 519
Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
Wang, Y., Stanton, D., Zhang, Y., Skerry-Ryan, R., Battenberg, E., Shor, J., Xiao, Y., Ren, F., Ye, J., & Saurous, R. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.09017 · Cited by 497
SUPERSEDED - CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit
Veaux, C., Yamagishi, J., & MacDonald, K. (2016) · University of Edinburgh. The Centre for Speech Technology Research (CSTR) · doi:10.7488/ds/1495 · Cited by 434
ClariNet: Parallel Wave Generation in End-to-End Text-to-SpeechPreprint
Ping, W., Peng, K., & Chen, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.07281 · Cited by 350
Cited by
A survey on neural speech synthesis
Tan, X., Qin, T., Soong, F., & Liu, T.-Y. (2021) · arXiv · doi:10.48550/arxiv.2106.15561 · Cited by 216
Warning: Humans cannot reliably detect speech deepfakes
Mai, K. T., Bray, S., Davies, T., & Griffin, L. D. (2023) · PLOS ONE · doi:10.1371/journal.pone.0285333 · Cited by 98
“Due to rapid research progress, it is possible to produce a realistic-sounding clone using only a few audio samples (Choi et al., 2020).”
BackgroundSC-GlowTTS: An Efficient Zero-Shot Multi-Speaker Text-To-Speech Model
Casanova, E., Shulby, C., Gölge, E., Müller, N. M., Oliveira, F. S. D., Candido Jr., A., Soares, A. D. S., Aluisio, S. M., & Ponti, M. A. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1774 · Cited by 83
Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech
Huang, S.-F., Lin, C.-J., Liu, D.-R., Chen, Y.-C., & Lee, H.-Y. (2022) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2022.3167258 · Cited by 59
AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset
Cai, Z., Ghosh, S., Adatia, A. P., Hayat, M., Dhall, A., Gedeon, T., & Stefanov, K. (2024) · Proceedings of the 32nd ACM International Conference on Multimedia · doi:10.1145/3664647.3680795 · Cited by 59
Detecting Voice Cloning Attacks via Timbre Watermarking
Liu, C., Zhang, J., Zhang, T., Yang, X., Zhang, W., & Yu, N. (2024) · Proceedings 2024 Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24200 · Cited by 40
The Multi-Speaker Multi-Style Voice Cloning Challenge 2021
Xie, Q., Tian, X., Liu, G., Song, K., Xie, L., Wu, Z., Li, H. H., Shi, S. X., Li, H., Hong, F., et al. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414001 · Cited by 42
StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis
Zhang, Y., Huang, R., Li, R., He, J., Xia, Y., Chen, F., Duan, X., Huai, B., & Zhao, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29932 · Cited by 66
Stanton, D., Shannon, M., Mariooryad, S., Skerry-Ryan, R., Battenberg, E., Bagby, T., & Kao, D. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747345 · Cited by 36
Gabryś, A., Huybrechts, G., Ribeiro, M. S., Chien, C.-M., Roth, J., Comini, G., Barra-Chicote, R., Perz, B., & Lorenzo-Trueba, J. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747239 · Cited by 29
Bhushan, S., Mishra, V. P., Rishiwal, V., Sharmila, A., & Agarwal, U. (2025) · IEEE Access · doi:10.1109/access.2025.3605236 · Cited by 22
Attacker Attribution of Audio Deepfakes
Müller, N., Diekmann, F., & Williams, J. R. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-129 · Cited by 30
Mandeel, A. R., Al-Radhi, M. S., & Csapó, T. G. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-14005-5 · Cited by 12
ZSE-VITS: A Zero-Shot Expressive Voice Cloning Method Based on VITS
Li, J., & Zhang, L. (2023) · Electronics · doi:10.3390/electronics12040820 · Cited by 21
Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speech
Lux, F., Koch, J., & Vu, N. T. (2023) · 2022 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022433 · Cited by 11
SC-CNN: Effective Speaker Conditioning Method for Zero-Shot Multi-Speaker Text-to-Speech Systems
Yoon, H., Kim, C., Um, S., Yoon, H.-W., & Kang, H.-G. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3277786 · Cited by 18
Unmasking Illusions: Understanding Human Perception of Audiovisual DeepfakesPreprint
Hashmi, A., Shahzad, S. A., Lin, C., Tsao, Y., & Wang, H. (2024) · doi:10.36227/techrxiv.171560542.29368554/v1 · Cited by 35
Prosodyspeech: Towards Advanced Prosody Model for Neural Text-to-Speech
Yi, Y., He, L., Pan, S., Wang, X., & Xiao, Y. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746744 · Cited by 12
High fidelity zero shot speaker adaptation in text to speech synthesis with denoising diffusion GAN
Liu, X., Ma, X., Song, W., Zhang, Y., & Zhang, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-90507-0 · Cited by 6
Zheng, J. P., Zhou, J., Zheng, W., Tao, L., & Kwan, H. K. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3412152 · Cited by 9
Semi-Supervised Learning Based on Reference Model for Low-resource TTS
Zhang, X., Wang, J., Cheng, N., & Xiao, J. Z. (2022) · International Conference on Mobility, Sensing and Networking (MSN) · doi:10.1109/msn57253.2022.00156 · Cited by 8
ADAPT-TTS: HIGH-QUALITY ZERO-SHOT MULTI-SPEAKER TEXT-TO-SPEECH ADAPTIVE-BASED FOR VIETNAMESE
Ngoc, P. P., Quang, C. T., & Luong, M. C. (2023) · Journal of Computer Science and Cybernetics · doi:10.15625/1813-9663/18136 · Cited by 9
The Thinkit System for Icassp2021 M2voc Challenge
Shang, Z., Zhang, H., Chen, Z., Zhou, B., & Zhang, P. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413669 · Cited by 5
Personalized Lightweight Text-to-Speech: Voice Cloning with Adaptive Structured Pruning
Huang, S.-F., Chen, C.-P., Chen, Z., Tsai, Y.-P., & Lee, H.-Y. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097178 · Cited by 10
Investigating a neural all pass warp in modern TTS applications
Schnell, B., & Garner, P. N. (2022) · Speech Communication · doi:10.1016/j.specom.2021.12.002 · Cited by 7
Fine-grained Noise Control for Multispeaker Speech Synthesis
Nikitaras, K., Vamvoukakis, G., Ellinas, N., Klapsas, K., Μαρκόπουλος, Κ., Raptis, S., Sung, J. S., Jho, G., Chalamandaris, A., & Tsiakoulis, P. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10765 · Cited by 6
Advancing Accessibility: Voice Cloning and Speech Synthesis for Individuals with Speech DisordersPreprint
Vinotha, R., Hepsiba, D. E., Anand, L. D. V., & Reji, D. J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4481522 · Cited by 5
Speaker Characteristics Guided Speech Synthesis
Yang, Z., Wu, Z., & Jia, J. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892169 · Cited by 5
Vinotha, R., Hepsiba, D. E., & D, V. A. L. (2025) · Journal of Trends in Computer Science and Smart Technology · doi:10.36548/jtcsst.2025.1.003 · Cited by 9
Eren, E., & Demiroğlu, C. (2023) · Computer Speech & Language · doi:10.1016/j.csl.2023.101520 · Cited by 6
Wu, Q., Shen, Q., Luan, J., & Wang, Y. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747032 · Cited by 4
Variable-Length Speaker Conditioning in Flow-Based Text-to-Speech
Choi, B. J., Jeong, M., Kim, M., & Kim, N. S. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3377588 · Cited by 4
The Multi-speaker Multi-style Voice Cloning Challenge 2021Preprint
Xie, Q., Tian, X., Liu, G., Song, K., Xie, L., Wu, Z., Li, H. H., Shi, S. X., Li, H., Hong, F., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01818 · Cited by 3
Improving Language Model-Based Zero-Shot Text-to-Speech Synthesis with Multi-Scale Acoustic Prompts
Lei, S., Zhou, Y., Chen, L., Luo, D., Wu, Z., Wu, X., Kang, S., Jiang, T., Zhou, Y., Han, Y., et al. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447815 · Cited by 5
Text-to-Voice Conversion for Indic Languages
Hebbar, A., Rasheed, S., Kl, S., Mustafa, S., & Santhosh, S. (2025) · International Conference on Artificial Intelligence and Data Engineering (AIDE) · doi:10.1109/aide64228.2025.10986870 · Cited by 2
Enriching Style Transfer in multi-scale control based personalized end-to-end speech synthesis
Lyu, Z., & Zhu, J. B. (2022) · International Conference on Information Science and Technology · doi:10.1109/icist55546.2022.9926908 · Cited by 3
VStyclone: Real-time Chinese voice style clone
Wu, Y., Zhao, H., Liang, X., & Sun, Y. (2022) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2022.108534 · Cited by 4
Watanabe, A., Takamichi, S., Saito, Y., Xin, D., & Saruwatari, H. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097113 · Cited by 5
MRMI-TTS: Multi-Reference Audios and Mutual Information Driven Zero-Shot Voice Cloning
Chen, Y.-T., Li, W., & Tang, B. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3649501 · Cited by 3
STFT-GradTTS: a robust, diffusion-based speech synthesis system with iSTFT decoder for Bangla
Intesum, M., Masud, A. I., Karim, R., & Islam, M. A. (2026) · International Journal of Speech Technology · doi:10.1007/s10772-025-10241-w · Cited by 1
A Study on Neural-Network-Based Text-to-Speech Adaptation Techniques for Vietnamese
Phuong, P. N., Quang, C. T., Truong, Q. C., & Luong, M. C. (2021) · International Conference on Speech Database and Assessments · doi:10.1109/o-cocosda202152914.2021.9660445 · Cited by 3
Multi-Scale Speaker Vectors for Zero-Shot Speech Synthesis
Cory, T., & Iqbal, R. (2022) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac54236.2022.00093 · Cited by 2
Speaker GenerationPreprint
Stanton, D., Shannon, M., Mariooryad, S., Skerry-Ryan, R., Battenberg, E., Bagby, T., & Kao, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05095 · Cited by 5
Síntese de fala aplicada à geração de conjunto de dados para reconhecimento automático de fala
Casanova, E. (2022) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2022.tde-02092022-142539
Advancing Accessibility: Voice Cloning and Speech Synthesis for Individuals with Speech DisordersPreprint
Vinotha, R., Hepsiba, D. E., Anand, L. D. V., & Reji, D. J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4622757 · Cited by 1
A Voice Cloning Model for Rhythm and Content De-Entanglement
王, 萌. (2024) · Artificial Intelligence and Robotics Research · doi:10.12677/airr.2024.131018
One-shot multi-speaker text-to-speech using RawNet3 speaker representation*
Han, S., Um, J. S., & Kim, H. (2024) · Phonetics and Speech Sciences · doi:10.13064/ksss.2024.16.1.067 · Cited by 1
Self-citationGenerative Spoken Language Modeling with Quantized Feature Enhancement
Duan, F., Li, C., Wang, K., Wu, S., Yin, C., & Rong, W. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651390
NRAdapt: Noise-Robust Adaptive Text to Speech Using Untranscribed Data
Cheng, M., Lei, S., Dai, D., Wu, Z., & Chong, D. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651038
As-Speech: Adaptive Style For Speech Synthesis
Ли, Ж., Xing, X., Wang, J., Chen, S., Yu, G., Wan, G., & Xu, X. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832337
Personalized Lao language synthesis via disentangled neural codec language model
Mao, C., Tian, T., Wang, L., Yu, Z., Gao, S., & Dong, L. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02535-x
Zero-Shot Voice Cloning Based on Target Adaptation and Phoneme-Level Local Feature Embedding
Cui, X., Tuerhong, G., & Wushouer, M. (2025) · Lecture notes in electrical engineering · doi:10.1007/978-981-96-2468-3_18
USIAL-VC: A One-Shot Voice Conversion by U-Net-Based Encoder and Speaker Identity Adaptive Learning
Peng, Y., & Wang, Y. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4279-3_10
Personalized Speech Synthesis Based on Gated Network
Qiu, X., Luo, Y., Xin, Z., & Sun, Z. (2024) · Asian Conference on Artificial Intelligence Technology (ACAIT) · doi:10.1109/acait63902.2024.11022245
Adaptive Singing Voice Enhancement for Live Stages
Hsu, J.-L., & Chien, P.-W. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-07959-6_14
Conan: A Chunkwise Online Network for Zero-Shot Adaptive Voice Conversion
Zhang, Y., Tian, B., & Duan, Z. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434701 · Cited by 9
Exploring Dysphonic Artificial Intelligence Voice Cloning for Speech Intelligibility in Noise
Bottalico, P., Nudelman, C. J., Fogerty, D., Tardini, V., & Ishikawa, K. (2026) · Journal of Voice · doi:10.1016/j.jvoice.2026.04.034
Metadata
- Authors
- Choi, S., Han, S., Kim, D., & Ha, S.
- Venue
- Interspeech 2020
- Publisher
- ISCA
- Published
- 2020-10-25
- Pages
- 2007-2011
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W3095199334
- Cited by
- 56
- References
- 29
- Citations by year
- 2026: 2 · 2025: 9 · 2024: 13 · 2023: 10 · 2022: 15 · 2021: 7
- FWCI
- 3.06
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
- Keywords
- attention mechanism, speaker similarity, synthetic speech quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.