CoChat
Sign Up
PreprintCited by 256Speech Recognition and SynthesisSpeech and Audio ProcessingVoice and Speech Disorders

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

Open access
Wang, Chengyi, Chen, Sanyuan, Wu, Yu, Zhang, Ziqiang, Zhou, Long, Liu, Shujie, Chen, Zhuo, Liu, Yanqing +5 more·2023·arXiv
DOI 10.48550/arxiv.2301.02111arXiv 2301.02111
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce a language modeling approach for text to speech synthesis (TTS). Specifically, we train a neural codec language model (called Vall-E) using discrete codes derived from an off-the-shelf neural audio codec model, and regard TTS as a conditional language modeling task rather than continuous signal regression as in previous work. During the pre-training stage, we scale up the TTS training data to 60K hours of English speech which is hundreds of times larger than existing systems. Vall-E emerges in-context learning capabilities and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt. Experiment results show that Vall-E significantly outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity. In addition, we find Vall-E could preserve the speaker's emotion and acoustic environment of the acoustic prompt in synthesis. See https://aka.ms/valle for demos of our work.

AI snapshot

Written from the paper’s full text.

TL;DR

VALL-E treats text-to-speech as conditional codec language modeling over discrete EnCodec acoustic tokens, pretrained on 60K hours of English speech, and can synthesize personalized speech from a 3-second enrolled recording of an unseen speaker. It outperforms the state-of-the-art zero-shot TTS baseline on LibriSpeech and VCTK in naturalness and speaker similarity, while preserving prompt acoustic environment/emotion and producing diverse outputs.

Key findings

  • On LibriSpeech test-clean, VALL-E reaches WER 5.9 and speaker similarity 0.580, versus YourTTS at WER 7.7 and 0.337; VALL-E-continual reaches WER 3.8 and 0.508, with ground truth at WER 2.2 and 0.754.
  • Human evaluation on LibriSpeech with 40 speakers and 3-second enrollment gives VALL-E SMOS 4.38 ±0.10 versus YourTTS 3.45 ±0.09, and +0.12 CMOS over YourTTS; on VCTK with 60 speakers, VALL-E gives SMOS 3.81 ±0.09 versus YourTTS 3.70 ±0.09, +0.23 CMOS over YourTTS, and +0.04 CMOS over ground truth.
  • On VCTK automatic speaker similarity, VALL-E beats YourTTS even though YourTTS saw 97 VCTK speakers in training: with 3s/5s/10s prompts, VALL-E scores 0.382/0.423/0.484 versus YourTTS 0.357/0.377/0.394 on 108 full speakers, and 0.389/0.380/0.414 versus 0.331/0.337/0.344 on 11 unseen speakers.
  • Ablations show prompts matter: NAR-no prompt gives WER 19.6 and SPK 0.518, NAR-phn prompt gives WER 3.0 and SPK 0.541, and NAR-2 prompts gives WER 2.8 and SPK 0.732; removing the AR acoustic prompt drops speaker similarity to 0.236 versus 0.585 for VALL-E.

Methods

  • VALL-E is trained on LibriLight: 60K hours of unlabelled English audiobook speech with around 7000 speakers; a hybrid DNN-HMM ASR model trained on 960 hours of LibriSpeech generates phoneme alignments, and EnCodec produces discrete acoustic codes at 24 kHz, 75 Hz embeddings, 8 RVQ codebooks with 1024 entries each at 6K bitrate.
  • The model uses an autoregressive decoder-only transformer for the first codebook conditioned on phoneme and acoustic prompts, plus a non-autoregressive transformer for codebooks 2–8 conditioned on phoneme, acoustic prompt, and previous codebooks; both have 12 layers, 16 attention heads, embedding dimension 1024, feed-forward dimension 4096, and dropout 0.1, trained on 16 NVIDIA TESLA V100 32GB GPUs with batch size 6k acoustic tokens per GPU for 800k steps using AdamW.
  • Evaluation is zero-shot on LibriSpeech test-clean (4–10s samples, 2.2 hours) and VCTK (108 speakers, none seen in training; also 11 unseen speakers for fair comparison), using WavLM-TDNN speaker similarity, HuBERT-Large ASR WER, and human CMOS/SMOS with 12 and 6 native speakers; the baseline is YourTTS.

Limitations

  • Stated: synthesis robustness remains an issue, with some words unclear, missed, or duplicated because the phoneme-to-acoustic autoregressive model has disordered attention alignments and no constraints to solve the issue.
  • Stated: data coverage is insufficient even with 60K hours; it cannot cover everyone’s voice, especially accent speakers, and LibriLight’s audiobook reading style limits speaking-style diversity, with worse VCTK results implying insufficient accent coverage.
  • Stated: the model currently uses two models for different quantizers, and broader impacts include potential misuse such as spoofing voice identification or impersonating a specific speaker; evident from the text, evaluations are English-only on LibriSpeech/VCTK with human evaluations limited to 40 and 60 speakers, so generalization to other languages, spontaneous speech, and broader accents is not demonstrated.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2301.02111

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

2 of 7 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    VALL-E is able to keep the same emotion of the prompt in speech synthesis, even if the model is not fine-tuned on an emotional TTS dataset.

    “VALL-E is able to keep the same emotion of the prompt in speech synthesis, even if the model is not fine-tuned on an emotional TTS dataset.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Adigwe et al., 2018 source

      The introduction (p. 1) discusses emotional control in speech synthesis, and §5.1 (p. 5) describes the paper’s own neutral-to-angry voice transformation system. Searches for VALL-E, preservation of a prompt’s emotion, and synthesis without fine-tuning found no passage addressing the claim. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
  • Findingown data

    The NAR model without any prompts performs poorly on both ASR and speaker similarity evaluations, even though the acoustic input token is ground truth.

    “Results show that the model without any prompts performs poorly on both ASR and speaker similarity evaluations, even though the acoustic input token is ground truth.”
    Quote in the paperSubstantial claim
  • Findingown data

    VALL-E achieves new state-of-the-art zero-shot TTS results on LibriSpeech and VCTK.

    “We achieve new state-of-the-art zero-shot TTS results on LibriSpeech and VCTK.”
    Quote in the paperSubstantial claim
  • Findingown data

    VALL-E is very close to ground truth in SMOS on LibriSpeech human evaluation.

    “VALL-E is very closed to ground truth in terms of SMOS, indicating the synthesized speech is similar to the given unseen speaker in testing.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: There is no speaker overlap between LibriLight training data and LibriSpeech test-clean data.
  • Findingown data

    VALL-E outperforms the YourTTS baseline on VCTK speaker similarity even though YourTTS has seen 97 speakers in training.

    “From Table 6, we can see that VALL-E outperforms the baseline even if the baseline has seen 97 speakers in training, indicating our model is able to synthesize speech with higher speaker similarity.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: YourTTS has seen 97 speakers in VCTK as training.
    • Rests on an assumption: VCTK consists of 108 speakers, and none of the speakers are observed during VALL-E training.
  • May combine two claims. 23
  • Classified as probably not a substantial claim. 16

  • May combine two claims.

    Premisecitation

    Audio codec contains abundant speaker information and acoustic information, which could maintain speaker identity in reconstruction compared to HuBERT codes.

    “It contains abundant speaker information and acoustic information, which could maintain speaker identity in reconstruction compared to HuBERT codes [Hsu et al., 2021].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Hsu et al., 2021 source

      The HuBERT paper’s Introduction discusses speaker identity as an aspect of high-fidelity speech representation, and its Implementation section describes HuBERT codewords. Neither section compares HuBERT codes with audio codecs or assesses speaker-identity preservation in reconstruction; searches for those terms found no such comparison in our copy. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    VALL-E preserves the speaker's emotion and acoustic environment of the acoustic prompt in synthesis.

    “In addition, we find VALL-E could preserve the speaker's emotion and acoustic environment of the acoustic prompt in synthesis.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The quantized tokens contain sufficient information about the speaker and recording conditions.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In Figure 4(a), the two samples have different lengths and phrase durations, with the first having a faster speech rate.

    “In Figure 4(a), we observe the two samples have different lengths and phrase durations, where the first has a faster speech rate.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    VALL-E emerges in-context learning capabilities and can synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt.

    “VALL-E emerges in-context learning capabilities and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The quantized tokens contain sufficient information about the speaker and recording conditions.
    • The authors state this as an assumption: The neural language model learns to extract the content and speaker information from the phoneme sequence and the acoustic prompt, respectively.

  • May combine two claims.

    Findingown data

    VALL-E achieves +0.04 CMOS over ground-truth on VCTK, demonstrating no statistically significant difference from human recordings on this dataset.

    “Furthermore, VALL-E achieves +0.04 CMOS over ground-truth, demonstrating no statistically significant difference from human recordings on this dataset.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    Beam search may lead the LM into an infinity loop.

    “For the AR model, we use samplingbased decoding conditioned on the prompts since we observe that beam search may lead the LM into an infinity loop.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    VALL-E significantly outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity.

    “Experiment results show that VALL-E significantly outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    When the acoustic prompt has reverberation, VALL-E synthesizes speech with reverberation as well, whereas the baseline outputs clean speech.

    “When the acoustic prompt has reverberation, VALL-E could synthesize speech with reverberation as well, whereas the baseline outputs clean speech.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    VALL-E is significantly better than the YourTTS baseline in both robustness (WER) and speaker similarity (SPK) on LibriSpeech zero-shot TTS evaluation.

    “model is significantly better in both robustness and speaker similarity, showing that our generated speech is highly faithful to the given text and the given enrolled speech.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: There is no speaker overlap between LibriLight training data and LibriSpeech test-clean data.

  • May combine two claims.

    Findingown data

    Even if the NAR model could see the prompt, the prompt for the AR model also contributes a lot to speaker similarity.

    “Even if the NAR model could see the prompt, the prompt for the AR model also contributes a lot to speaker similarity.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In Figure 4(b), the accents of the two samples are different; the second output emphasizes the word must with a larger amplitude whereas the first output does not.

    “In Figure 4(b), we observe that the accents of the two samples are different.The second output emphasizes the word “must” with a larger amplitude whereas the first output does not.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    VALL-E is +0.23 CMOS over YourTTS on VCTK, indicating significantly better performance on speaking naturalness.

    “The side-by-side CMOS evaluation shows that VALL-E is +0.23 over YourTTS, indicating a significantly better performance on speaking of naturalness.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    VALL-E is the first language model based TTS framework leveraging large, diverse, and multi-speaker speech data.

    “Transferring this success to the field of speech synthesis, we introduce VALL-E, the first language model based TTS framework leveraging the large, diverse, and multi-speaker speech data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Because RVQ is employed, the first quantizer plays the most important role in reconstruction, and the impact from others gradually decreases.

    “Because RVQ is employed, the first quantizer plays the most important role in reconstruction, and the impact from others gradually decreases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    VALL-E generated speech is highly faithful to the given text and the given enrolled speech.

    “model is significantly better in both robustness and speaker similarity, showing that our generated speech is highly faithful to the given text and the given enrolled speech.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Sampling-based decoding significantly increases the diversity of the output.

    “Furthermore, the sampling-based method could significantly increase the diversity of the output.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Prompting is necessary to enable in-context learning in the zero-shot scenario.

    “For language models, prompting is necessary to enable in-context learning in the zero-shot scenario.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: In-context learning is an ability of text-based language models to predict labels for unseen inputs without additional parameter updates.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    VALL-E is trained on a large-scale dataset consisting of more acoustic conditions than the data used by the baseline, so VALL-E could learn acoustic consistency instead of a clean environment only during training.

    “Our explanation is that VALL-E is trained on a large-scale dataset consisting of more acoustic conditions than the data used by the baseline, so VALL-E could learn the acoustic consistency instead of a clean environment only during training.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: VALL-E is trained on a large-scale dataset consisting of more acoustic conditions than the data used by the baseline.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    VALL-E is the first to emerge in-context learning capability in zero-shot TTS.

    “Furthermore, VALL-E is the first to use audio codec codes as intermediate representations, and emerge in-context learning capability in zero-shot TTS.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    VCTK is more challenging for speaker similarity than LibriSpeech because it contains speakers with various accents while the training data and LibriSpeech test data do not contain various accent speakers.

    “In terms of speaker similarity, VCTK is more challenging as it contains speakers with various accents while the training data and LibriSpeech test data do not contain various accent speakers.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The training data and LibriSpeech test data do not contain various accent speakers.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    The unclear, missed, or duplicated words are mainly because the phoneme-to-acoustic language part is an autoregressive model, in which disordered attention alignments exist and no constraints to solving the issue.

    “It is mainly because the phoneme-to-acoustic language part is an autoregressive model, in which disordered attention alignments exist and no constraints to solving the issue.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    VALL-E is the first to use audio codec codes as intermediate representations.

    “Furthermore, VALL-E is the first to use audio codec codes as intermediate representations, and emerge in-context learning capability in zero-shot TTS.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    One major reason VALL-E is better than speech-to-speech LM-based systems is that VALL-E is trained with pseudo-phoneme instead of HuBERT/w2v-BERT codes, which enjoys better alignment quality with the input text.

    “One major reason is VALL-E trained with pseudo-phoneme instead of HuBERT/w2v-BERT codes, which enjoys better alignment quality with the input text.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    The in-context learning capability of existing TTS systems is not strong, because they either require additional fine-tuning or degrade dramatically for unseen speakers.

    “However, the in-context learning capability of existing TTS systems is not strong, because they either require additional fine-tuning or degrade dramatically for unseen speakers.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The diversity of speaking styles in VALL-E is not enough, as LibriLight is an audiobook dataset in which most utterances are in reading style.

    “Moreover, the diversity of speaking styles is not enough, as LibriLight is an audiobook dataset, in which most utterances are in reading style.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: LibriLight is an audiobook dataset in which most utterances are in reading style.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisereasoning

    Previous TTS systems have a strong one-one mapping between input text and output waveform, because mel spectrum generation is based on reconstruction for each step without randomness.

    “Previous TTS systems have a strong one-one mapping between input text and output waveform, because mel spectrum generation is based on reconstruction for each step without randomness.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Previous TTS pre-training work leverages less than 1K hours of data, whereas VALL-E is pre-trained with 60K hours of data.

    “Previous TTS pre-training work leverages less than 1K hours of data, whereas VALL-E is pre-trained with 60K hours of data.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    The EnCodec model can encode waveform into discrete acoustic codes and reconstruct high-quality waveform even if the speaker is unseen in training.

    “To compress audio for network transmission, codec models are able to encode waveform into discrete acoustic codes and reconstruct high-quality waveform even if the speaker is unseen in training.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    Current TTS systems suffer from poor generalization; speaker similarity and speech naturalness decline dramatically for unseen speakers in the zero-shot scenario.

    “Because the training data is relatively small, current TTS systems still suffer from poor generalization.Speaker similarity and speech naturalness decline dramatically for unseen speakers in the zero-shot scenario.”
    Quote in the paperSubstantial claim
29 of 29

  • The authors state this as an assumption.

    The quantized tokens contain sufficient information about the speaker and recording conditions.

    “Compared to traditional audio codec approaches, the neural-based codec is significantly better at low bitrates, and we believe the quantized tokens contain sufficient information about the speaker and recording conditions.”

  • The authors state this as an assumption.

    The proposed approach is robust to the noise and generalizes well by leveraging large data.

    “We believe the proposed approach is robust to the noise and generalize well by leveraging large data.”

  • The authors state this as an assumption.

    The neural language model learns to extract the content and speaker information from the phoneme sequence and the acoustic prompt, respectively.

    “We expect the neural language model learns to extract the content and speaker information from the phoneme sequence and the acoustic prompt, respectively.”

  • The paper relies on this without support.

    In-context learning is an ability of text-based language models to predict labels for unseen inputs without additional parameter updates.

    “In-context learning is a surprising ability of the text-based language model, which is able to predict labels for unseen inputs without additional parameter updates.”

  • The paper relies on this without support.

    There is no speaker overlap between LibriLight training data and LibriSpeech test-clean data.

    “since there is no speaker overlap between LibriLight training data and LibriSpeech test-clean data.”

  • The paper relies on this without support.

    HuBERT codes discard speaker identity.

    “Since the HuBERT codes discard the speaker identity”

  • The paper relies on this without support.

    The training data and LibriSpeech test data do not contain various accent speakers.

    “while the training data and LibriSpeech test data do not contain various accent speakers.”

  • The paper relies on this without support.

    VALL-E is trained on a large-scale dataset consisting of more acoustic conditions than the data used by the baseline.

    “VALL-E is trained on a large-scale dataset consisting of more acoustic conditions than the data used by the baseline”

  • The paper relies on this without support.

    LibriLight is an audiobook dataset in which most utterances are in reading style.

    “LibriLight is an audiobook dataset, in which most utterances are in reading style.”

  • The paper relies on this without support.

    YourTTS has seen 97 speakers in VCTK as training.

    “Since YourTTS has seen 97 speakers in VCTK as training”

  • The paper relies on this without support.

    VCTK consists of 108 speakers, and none of the speakers are observed during VALL-E training.

    “We evaluate our model on VCTK consisting of 108 speakers, where none of the speakers are observed during training.”
11 of 11

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 11

  1. Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, et al. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5723-5738, 2022.
  2. Sercan Ömer Arik, Jitong Chen, Kainan Peng, Wei Ping, and Yanqi Zhou. Neural voice cloning with a few samples. In NeurIPS, pages 10040-10050, 2018.
  3. He Bai, Renjie Zheng, Junkun Chen, Mingbo Ma, Xintong Li, and Liang Huang. A 3 t: Alignment- aware acoustic and text pretraining for speech synthesis and editing. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 1399-1411. PMLR, 2022.
  4. Ye Jia, Yu Zhang, Ron J. Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez-Moreno, and Yonghui Wu. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. In NeurIPS, pages 4485-4495, 2018.
  5. Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In AAAI, pages 6706-6713. AAAI, 2019.
  6. Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail A. Kudinov. Grad-tts: A diffusion probabilistic model for text-to-speech. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8599-8608.
  7. PMLR, 2021. URL http://proceedings.mlr.press/v139/popov21a.html.
  8. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding, number CONF. IEEE Signal Processing Society, 2011.
  9. Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. In NeurIPS, pages 3165-3174, 2019.
  10. Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Infor- mation Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6306-6315, 2017.
  11. Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 4383-4393, 2019.

Cited by

Sorted by citations

Related

Authors
Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., He, L., Zhao, S., & Wei, F.
Affiliations
Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2301.02111
arXiv
2301.02111
OpenAlex
W4313679638
Semantic Scholar
c2f91f35df893714418cc29096083dce0b441229
Cited by
250
References
0
Citations by year
2026: 4 · 2025: 86 · 2024: 115 · 2023: 45
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, public-domain)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Health Sciences › Medicine › Physiology › Voice and Speech Disorders
Keywords
text-to-speech, zero-shot speech synthesis, neural audio codecs, personalized speech synthesis, in-context learning, speaker similarity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-28.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-28.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review