Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Personalizing a speech synthesis system is a highly desired application, where the system can generate speech with the user’s voice with rare enrolled recordings. There are two main approaches to build such a system in recent works: speaker adaptation and speaker encoding. On the one hand, speaker adaptation methods fine-tune a trained multi-speaker text-to-speech (TTS) model with few enrolled samples. However, they require at least thousands of fine-tuning steps for high-quality adaptation, making it hard to apply on devices. On the other hand, speaker encoding methods encode enrollment utterances into a speaker embedding. The trained TTS model can synthesize the user’s speech conditioned on the corresponding speaker embedding. Nevertheless, the speaker encoder suffers from the generalization gap between the seen and unseen speakers. In this paper, we propose applying a meta-learning algorithm to the speaker adaptation method. More specifically, we use Model Agnostic Meta-Learning (MAML) as the training algorithm of a multi-speaker TTS model, which aims to find a great meta-initialization to adapt the model to any few-shot speaker adaptation tasks quickly. Therefore, we can also adapt the meta-trained TTS model to unseen speakers efficiently. Our experiments compare the proposed method (Meta-TTS) with two baselines: a speaker adaptation method baseline and a speaker encoding method baseline. The evaluation results show that Meta-TTS can synthesize high speaker-similarity speech from few enrollment samples with fewer adaptation steps than the speaker adaptation baseline and outperforms the speaker encoding baseline under the same training scheme. When the speaker encoder of the baseline is pre-trained with extra 8371 speakers of data, Meta-TTS can still outperform the baseline on LibriTTS dataset and achieve comparable results on VCTK dataset.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256
VoxCeleb: A Large-Scale Speaker Identification Dataset
Nagrani, A., Chung, J. S., & Zisserman, A. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-950 · Cited by 2,210 · cited 1×
“The model was trained on a combination of subsets of the three datasets: Librispeech train-other-500 [24], VoxCeleb1 Dev A-D [25], and VoxCeleb2 Dev A-H [26], which is 3201 hours of utterances in total from 8371 speakers3.”
MethodTacotron: towards end-to-end speech synthesis
Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., et al. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1452 · Cited by 1,746 · cited 1×
“trained with a large amount of high-quality single-speaker speech data [1], [2], [3] or synthesize a set of voices with multi-speaker corpora [4], [5], [6], [7].”
BackgroundMelGAN: generative adversarial networks for conditional waveform synthesis
Kumar, K., Kumar, R., de Boissiere, T., Gestin, L., Teoh, W. Z., Sotelo, J., de Brebisson, A., Bengio, Y., & Courville, A. (2019) · arXiv · doi:10.48550/arxiv.1910.06711 · Cited by 722 · cited 1×
“We use MelGAN [35] as our neural vocoder during inference, which is pre-trained on an internal 6-speaker dataset with 3 male and 3 female speakers by the MelGAN’s authors.”
MethodTransfer learning from speaker verification to multispeaker text-to-speech synthesis
Jia, Y., Zhang, Y., Weiss, R. J., Wang, Q., Shen, J., Ren, F., Chen, Z., Nguyen, P., Pang, R., Moreno, I. L., et al. (2018) · arXiv · doi:10.48550/arxiv.1806.04558 · Cited by 519 · cited 6×
“SV2TTS [6] uses the speaker encoding method, where the speaker encoder is transferred from a pre-trained speaker verification model, and the TTS architecture is a Tacotron 2 [36].”
BackgroundMethodInfluentialCSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92)
Yamagishi, J., Veaux, C., & MacDonald, K. (2019) · University of Edinburgh. The Centre for Speech Technology Research (CSTR) · doi:10.7488/ds/2645 · Cited by 356
Zero-Shot Multi-Speaker Text-To-Speech with State-Of-The-Art Neural Speaker Embeddings
Cooper, E., Lai, C.-I., Yasuda, Y., Fang, F., Wang, X., Chen, N., & Yamagishi, J. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054535 · Cited by 164 · cited 2×
“There are two general approaches to deal with such task [8], speaker adaptation [8], [9], [10], [11], [12] and speaker encoding [6], [13], [7], [14], [15], [16], [17].”
Background
Research works printed but not linked: 25
- Ren, 2021, FastSpeech 2: Fast and high-quality end-to-end text-to-speech
- Gibiansky, 2017, Deep voice 2: Multi-speaker neural text-to-speech, 30, 2966
- Ping, 2018, Deep voice 3: 2000-speaker neural text-to-speech, 214
- Arik, 2018, Neural voice cloning with a few samples, 10040
- Chen, 2019, Sample efficient adaptive text-to-speech
- 10.21437/Interspeech.2020-1745 · DOI · Paper page
- Chen, 2021, AdaSpeech: Adaptive text to speech for custom voice
- 10.1109/ICASSP39728.2021.9414727 · DOI
- Taigman, 2018, VoiceLoop: Voice fitting and synthesis via a phonological loop
- 10.21437/Interspeech.2020-1737 · DOI
- 10.21437/Interspeech.2020-1032 · DOI
- 10.1109/ICASSP39728.2021.9413880 · DOI
- Finn, 2017, Model-agnostic meta-learning for fast adaptation of deep networks, 1126
- Raghu, 2020, Rapid learning or feature reuse? Towards understanding the effectiveness of MAML
- 10.1109/ICASSP.2014.6854363 · DOI · Paper page
- 10.1109/ICASSP.2016.7472652 · DOI · Paper page
- 10.1109/ICASSP.2018.8462665 · DOI · Paper page
- 10.21437/Interspeech.2019-2003 · DOI · Paper page
- 10.1109/ICASSP39728.2021.9413877 · DOI
- 10.21437/Interspeech.2021-2013 · DOI
- Baevski, 2020, Wav2Vec 2.0: A framework for self-supervised learning of speech representations, 33, 12449
- 10.1109/TASLP.2021.3095662 · DOI · Paper page
- 10.21437/Interspeech.2019-2441 · DOI · Paper page
- Yamagishi, 2019, CSTR VCTK Corpus: English multi-speaker corpus for CSTR voice cloning toolkit
- 10.1109/ICASSP.2018.8461368 · DOI · Paper page
Cited by
A review of deep learning techniques for speech processing
Mehrish, A., Majumder, N., Bharadwaj, R., Mihalcea, R., & Poria, S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101869 · Cited by 389
ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild
Liu, X., Wang, X., Sahidullah, M., Patino, J., Delgado, H., Kinnunen, T., Todisco, M., Yamagishi, J., Evans, N., Nautsch, A., et al. (2023) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2023.3285283 · Cited by 312
“TTS and VC technology has advanced rapidly in recent years, especially in terms of requiring far less target speaker training data [53], [54],”
BackgroundNeural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.02111 · Cited by 250
Singh, S., & Dhumane, A. (2025) · MethodsX · doi:10.1016/j.mex.2025.103632 · Cited by 24
Metadata
- Authors
- Huang, S.-F., Lin, C.-J., Liu, D.-R., Chen, Y.-C., & Lee, H.-Y.
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2022
- Volume
- 30
- Pages
- 1558-1571
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W3213544594
- Cited by
- 59
- References
- 12
- FWCI
- 4.52
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- Ministry of Science and Technology, Taiwan (111-2628-E-002-013, 110-2223-E-002 -007-MY3)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech. https://app.cochat.ai/papers/10.1109/taslp.2022.3167258
- The paper
- Huang, S.-F., Lin, C.-J., Liu, D.-R., Chen, Y.-C., & Lee, H.-Y. (2022). Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech. IEEE/ACM Transactions on Audio, Speech, and Language Processing. https://doi.org/10.1109/taslp.2022.3167258
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.