The DOI matches its DataCite record; 42 of the 42 references it cites were found in a registry or an index; 2 of the 7 sources its claims cite were checked for support.
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Open accessAbstract
We introduce a language modeling approach for text to speech synthesis (TTS). Specifically, we train a neural codec language model (called Vall-E) using discrete codes derived from an off-the-shelf neural audio codec model, and regard TTS as a conditional language modeling task rather than continuous signal regression as in previous work. During the pre-training stage, we scale up the TTS training data to 60K hours of English speech which is hundreds of times larger than existing systems. Vall-E emerges in-context learning capabilities and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt. Experiment results show that Vall-E significantly outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity. In addition, we find Vall-E could preserve the speaker's emotion and acoustic environment of the acoustic prompt in synthesis. See https://aka.ms/valle for demos of our work.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
VALL-E treats text-to-speech as conditional codec language modeling over discrete EnCodec acoustic tokens, pretrained on 60K hours of English speech, and can synthesize personalized speech from a 3-second enrolled recording of an unseen speaker. It outperforms the state-of-the-art zero-shot TTS baseline on LibriSpeech and VCTK in naturalness and speaker similarity, while preserving prompt acoustic environment/emotion and producing diverse outputs.
Key findings
- On LibriSpeech test-clean, VALL-E reaches WER 5.9 and speaker similarity 0.580, versus YourTTS at WER 7.7 and 0.337; VALL-E-continual reaches WER 3.8 and 0.508, with ground truth at WER 2.2 and 0.754.
- Human evaluation on LibriSpeech with 40 speakers and 3-second enrollment gives VALL-E SMOS 4.38 ±0.10 versus YourTTS 3.45 ±0.09, and +0.12 CMOS over YourTTS; on VCTK with 60 speakers, VALL-E gives SMOS 3.81 ±0.09 versus YourTTS 3.70 ±0.09, +0.23 CMOS over YourTTS, and +0.04 CMOS over ground truth.
- On VCTK automatic speaker similarity, VALL-E beats YourTTS even though YourTTS saw 97 VCTK speakers in training: with 3s/5s/10s prompts, VALL-E scores 0.382/0.423/0.484 versus YourTTS 0.357/0.377/0.394 on 108 full speakers, and 0.389/0.380/0.414 versus 0.331/0.337/0.344 on 11 unseen speakers.
- Ablations show prompts matter: NAR-no prompt gives WER 19.6 and SPK 0.518, NAR-phn prompt gives WER 3.0 and SPK 0.541, and NAR-2 prompts gives WER 2.8 and SPK 0.732; removing the AR acoustic prompt drops speaker similarity to 0.236 versus 0.585 for VALL-E.
Methods
- VALL-E is trained on LibriLight: 60K hours of unlabelled English audiobook speech with around 7000 speakers; a hybrid DNN-HMM ASR model trained on 960 hours of LibriSpeech generates phoneme alignments, and EnCodec produces discrete acoustic codes at 24 kHz, 75 Hz embeddings, 8 RVQ codebooks with 1024 entries each at 6K bitrate.
- The model uses an autoregressive decoder-only transformer for the first codebook conditioned on phoneme and acoustic prompts, plus a non-autoregressive transformer for codebooks 2–8 conditioned on phoneme, acoustic prompt, and previous codebooks; both have 12 layers, 16 attention heads, embedding dimension 1024, feed-forward dimension 4096, and dropout 0.1, trained on 16 NVIDIA TESLA V100 32GB GPUs with batch size 6k acoustic tokens per GPU for 800k steps using AdamW.
- Evaluation is zero-shot on LibriSpeech test-clean (4–10s samples, 2.2 hours) and VCTK (108 speakers, none seen in training; also 11 unseen speakers for fair comparison), using WavLM-TDNN speaker similarity, HuBERT-Large ASR WER, and human CMOS/SMOS with 12 and 6 native speakers; the baseline is YourTTS.
Limitations
- Stated: synthesis robustness remains an issue, with some words unclear, missed, or duplicated because the phoneme-to-acoustic autoregressive model has disordered attention alignments and no constraints to solve the issue.
- Stated: data coverage is insufficient even with 60K hours; it cannot cover everyone’s voice, especially accent speakers, and LibriLight’s audiobook reading style limits speaking-style diversity, with worse VCTK results implying insufficient accent coverage.
- Stated: the model currently uses two models for different quantizers, and broader impacts include potential misuse such as spoofing voice identification or impersonating a specific speaker; evident from the text, evaluations are English-only on LibriSpeech/VCTK with human evaluations limited to 40 and 60 speakers, so generalization to other languages, spontaneous speech, and broader accents is not demonstrated.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
28 Own interpretation · 26 Own result · 15 No basis given · 2 Couldn’t confirm · 2 Sources not found · 1 Not checked yet · accuracy not yet measured
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“VALL-E is able to keep the same emotion of the prompt in speech synthesis, even if the model is not fine-tuned on an emotional TTS dataset.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 5.4 Qualitative Analysis
What the sources say
- Could not verifycites Adigwe et al., 2018 sourceExperimental — accuracy not yet measured
The introduction (p. 1) discusses emotional control in speech synthesis, and §5.1 (p. 5) describes the paper’s own neutral-to-angry voice transformation system. Searches for VALL-E, preservation of a prompt’s emotion, and synthesis without fine-tuning found no passage addressing the claim. (LLM, conf=0.78)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Results show that the model without any prompts performs poorly on both ASR and speaker similarity evaluations, even though the acoustic input token is ground truth.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > +0.17
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We achieve new state-of-the-art zero-shot TTS results on LibriSpeech and VCTK.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 6 Conclusion, Limitations, and Future Work
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“VALL-E is very closed to ground truth in terms of SMOS, indicating the synthesized speech is similar to the given unseen speaker in testing.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 5.2 LibriSpeech Evaluation
hedgedn = 40- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: There is no speaker overlap between LibriLight training data and LibriSpeech test-clean data.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From Table 6, we can see that VALL-E outperforms the baseline even if the baseline has seen 97 speakers in training, indicating our model is able to synthesize speech with higher speaker similarity.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 5.3 VCTK Evaluation
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: YourTTS has seen 97 speakers in VCTK as training.
- Rests on an assumption: VCTK consists of 108 speakers, and none of the speakers are observed during VALL-E training.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regarding naturalness, VALL-E beats the baseline with +0.12 CMOS, indicating the proposed method could synthesize more natural and realistic speech against baselines.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 5.2 LibriSpeech Evaluation
hedgeddirectn = 40- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: There is no speaker overlap between LibriLight training data and LibriSpeech test-clean data.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“VALL-E also beats the baseline on VCTK with +0.11 SMOS and +0.23 CMOS improvements.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When we compare with the baseline in a fair setting (11 speakers), the performance gap becomes larger, especially when only 3s prompts are available.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 5.3 VCTK Evaluation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: YourTTS has seen 97 speakers in VCTK as training.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluation results show that VALL-E significantly outperforms the state-of-the-art zero-shot TTS system on LibriSpeech and VCTK.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 1 Introduction
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, the qualitative analysis shows that VALL-E is able to synthesize diverse outputs with the same text and target speaker, which could benefit pseudo-data creation for the speech recognition task.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 23
- Probably not a substantial claim. 16
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“It contains abundant speaker information and acoustic information, which could maintain speaker identity in reconstruction compared to HuBERT codes [Hsu et al., 2021].”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 3 Background: Speech Quantization
What the sources say
- Could not verifycites Hsu et al., 2021 sourceExperimental — accuracy not yet measured
The HuBERT paper’s Introduction discusses speaker identity as an aspect of high-fidelity speech representation, and its Implementation section describes HuBERT codewords. Neither section compares HuBERT codes with audio codecs or assesses speaker-identity preservation in reconstruction; searches for those terms found no such comparison in our copy. (LLM, conf=0.76)
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we find VALL-E could preserve the speaker's emotion and acoustic environment of the acoustic prompt in synthesis.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > Abstract
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The quantized tokens contain sufficient information about the speaker and recording conditions.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Figure 4(a), we observe the two samples have different lengths and phrase durations, where the first has a faster speech rate.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 5.4 Qualitative Analysis
plaincorrelationaln = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“VALL-E emerges in-context learning capabilities and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The quantized tokens contain sufficient information about the speaker and recording conditions.
- The authors state this as an assumption: The neural language model learns to extract the content and speaker information from the phoneme sequence and the acoustic prompt, respectively.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, VALL-E achieves +0.04 CMOS over ground-truth, demonstrating no statistically significant difference from human recordings on this dataset.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 5.3 VCTK Evaluation
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For the AR model, we use samplingbased decoding conditioned on the prompts since we observe that beam search may lead the LM into an infinity loop.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 4.3 Inference: In-Context Learning via Prompting
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experiment results show that VALL-E significantly outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“When the acoustic prompt has reverberation, VALL-E could synthesize speech with reverberation as well, whereas the baseline outputs clean speech.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 5.4 Qualitative Analysis
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“model is significantly better in both robustness and speaker similarity, showing that our generated speech is highly faithful to the given text and the given enrolled speech.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 5.2 LibriSpeech Evaluation
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: There is no speaker overlap between LibriLight training data and LibriSpeech test-clean data.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even if the NAR model could see the prompt, the prompt for the AR model also contributes a lot to speaker similarity.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > +0.17
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Figure 4(b), we observe that the accents of the two samples are different.The second output emphasizes the word “must” with a larger amplitude whereas the first output does not.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 5.4 Qualitative Analysis
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The side-by-side CMOS evaluation shows that VALL-E is +0.23 over YourTTS, indicating a significantly better performance on speaking of naturalness.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 5.3 VCTK Evaluation
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Transferring this success to the field of speech synthesis, we introduce VALL-E, the first language model based TTS framework leveraging the large, diverse, and multi-speaker speech data.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Because RVQ is employed, the first quantizer plays the most important role in reconstruction, and the impact from others gradually decreases.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > Data coverage:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“model is significantly better in both robustness and speaker similarity, showing that our generated speech is highly faithful to the given text and the given enrolled speech.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 5.2 LibriSpeech Evaluation
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, the sampling-based method could significantly increase the diversity of the output.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 4.3 Inference: In-Context Learning via Prompting
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For language models, prompting is necessary to enable in-context learning in the zero-shot scenario.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 4.3 Inference: In-Context Learning via Prompting
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
- Rests on an assumption: In-context learning is an ability of text-based language models to predict labels for unseen inputs without additional parameter updates.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our explanation is that VALL-E is trained on a large-scale dataset consisting of more acoustic conditions than the data used by the baseline, so VALL-E could learn the acoustic consistency instead of a clean environment only during training.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 5.4 Qualitative Analysis
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: VALL-E is trained on a large-scale dataset consisting of more acoustic conditions than the data used by the baseline.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, VALL-E is the first to use audio codec codes as intermediate representations, and emerge in-context learning capability in zero-shot TTS.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In terms of speaker similarity, VCTK is more challenging as it contains speakers with various accents while the training data and LibriSpeech test data do not contain various accent speakers.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > Data coverage:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The training data and LibriSpeech test data do not contain various accent speakers.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is mainly because the phoneme-to-acoustic language part is an autoregressive model, in which disordered attention alignments exist and no constraints to solving the issue.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > Synthesis robustness:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, VALL-E is the first to use audio codec codes as intermediate representations, and emerge in-context learning capability in zero-shot TTS.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One major reason is VALL-E trained with pseudo-phoneme instead of HuBERT/w2v-BERT codes, which enjoys better alignment quality with the input text.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 5.2 LibriSpeech Evaluation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, the in-context learning capability of existing TTS systems is not strong, because they either require additional fine-tuning or degrade dramatically for unseen speakers.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 4.3 Inference: In-Context Learning via Prompting
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, the diversity of speaking styles is not enough, as LibriLight is an audiobook dataset, in which most utterances are in reading style.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > Data coverage:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: LibriLight is an audiobook dataset in which most utterances are in reading style.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Previous TTS systems have a strong one-one mapping between input text and output waveform, because mel spectrum generation is based on reconstruction for each step without randomness.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 5.4 Qualitative Analysis
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Previous TTS pre-training work leverages less than 1K hours of data, whereas VALL-E is pre-trained with 60K hours of data.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To compress audio for network transmission, codec models are able to encode waveform into discrete acoustic codes and reconstruct high-quality waveform even if the speaker is unseen in training.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 3 Background: Speech Quantization
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Because the training data is relatively small, current TTS systems still suffer from poor generalization.Speaker similarity and speech naturalness decline dramatically for unseen speakers in the zero-shot scenario.”
§ Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The authors state this as an assumption.
The quantized tokens contain sufficient information about the speaker and recording conditions.
“Compared to traditional audio codec approaches, the neural-based codec is significantly better at low bitrates, and we believe the quantized tokens contain sufficient information about the speaker and recording conditions.”
The authors state this as an assumption.
The proposed approach is robust to the noise and generalizes well by leveraging large data.
“We believe the proposed approach is robust to the noise and generalize well by leveraging large data.”
The authors state this as an assumption.
The neural language model learns to extract the content and speaker information from the phoneme sequence and the acoustic prompt, respectively.
“We expect the neural language model learns to extract the content and speaker information from the phoneme sequence and the acoustic prompt, respectively.”
The paper relies on this without support.
In-context learning is an ability of text-based language models to predict labels for unseen inputs without additional parameter updates.
“In-context learning is a surprising ability of the text-based language model, which is able to predict labels for unseen inputs without additional parameter updates.”
The paper relies on this without support.
There is no speaker overlap between LibriLight training data and LibriSpeech test-clean data.
“since there is no speaker overlap between LibriLight training data and LibriSpeech test-clean data.”
The paper relies on this without support.
HuBERT codes discard speaker identity.
“Since the HuBERT codes discard the speaker identity”
The paper relies on this without support.
The training data and LibriSpeech test data do not contain various accent speakers.
“while the training data and LibriSpeech test data do not contain various accent speakers.”
The paper relies on this without support.
VALL-E is trained on a large-scale dataset consisting of more acoustic conditions than the data used by the baseline.
“VALL-E is trained on a large-scale dataset consisting of more acoustic conditions than the data used by the baseline”
The paper relies on this without support.
LibriLight is an audiobook dataset in which most utterances are in reading style.
“LibriLight is an audiobook dataset, in which most utterances are in reading style.”
The paper relies on this without support.
YourTTS has seen 97 speakers in VCTK as training.
“Since YourTTS has seen 97 speakers in VCTK as training”
The paper relies on this without support.
VCTK consists of 108 speakers, and none of the speakers are observed during VALL-E training.
“We evaluate our model on VCTK consisting of 108 speakers, where none of the speakers are observed during training.”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 8
- Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, et al. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5723-5738, 2022.
- Sercan Ömer Arik, Jitong Chen, Kainan Peng, Wei Ping, and Yanqi Zhou. Neural voice cloning with a few samples. In NeurIPS, pages 10040-10050, 2018.
- He Bai, Renjie Zheng, Junkun Chen, Mingbo Ma, Xintong Li, and Liang Huang. A 3 t: Alignment- aware acoustic and text pretraining for speech synthesis and editing. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 1399-1411. PMLR, 2022.
- Ye Jia, Yu Zhang, Ron J. Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez-Moreno, and Yonghui Wu. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. In NeurIPS, pages 4485-4495, 2018.
- Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In AAAI, pages 6706-6713. AAAI, 2019.
- PMLR, 2021. URL http://proceedings.mlr.press/v139/popov21a.html.
- Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding, number CONF. IEEE Signal Processing Society, 2011.
- Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 4383-4393, 2019.
Cited by
Sorted by citations
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
A Survey on In-context Learning
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.64 · Cited by 649
Generative artificial intelligence
Banh, L., & Strobel, G. (2023) · Electronic Markets · doi:10.1007/s12525-023-00680-1 · Cited by 879
A review of deep learning techniques for speech processing
Mehrish, A., Majumder, N., Bharadwaj, R., Mihalcea, R., & Poria, S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101869 · Cited by 389
Artificial Intelligence & Creativity: A Manifesto for Collaboration
Vinchon, F., Lubart, T., Bartolotta, S., Gironnay, V., Botella, M., Bourgeois-Bougrine, S., Burkhardt, J., Bonnardel, N., Corazza, G. E., Glăveanu, V., et al. (2023) · The Journal of Creative Behavior · doi:10.1002/jocb.597 · Cited by 273
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., & Qiu, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1055 · Cited by 549
MoMask: Generative Masked Modeling of 3D Human Motions
Guo, C., Mu, Y., Javed, M. G., Wang, S., & Cheng, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00186 · Cited by 280
Language Is Not All You Need: Aligning Perception with Language Models
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.14045 · Cited by 235
Self-citationCharacter-LLM: A Trainable Agent for Role-Playing
Shao, Y., Li, L., Dai, J., & Qiu, X. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.814 · Cited by 140
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Chen, S., Wang, C., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, T., Wang, H., Li, J., et al. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3530270 · Cited by 162
Self-citationSpeak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision
Kharitonov, E., Vincent, D., Borsos, Z., Marinier, R., Girgin, S., Pietquin, O., Sharifi, M., Tagliasacchi, M., & Zeghidour, N. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00618 · Cited by 173
On Decoder-Only Architecture For Speech-to-Text and Large Language Model Integration
Wu, J., Gaur, Y., Chen, Z., Zhou, L., Zhu, Y., Wang, T., Li, J., Liu, S., Ren, B., Liu, L., et al. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389705 · Cited by 118
Self-citationCombined scaling for zero-shot transfer learning
Pham, H., Dai, Z., Ghiasi, G., Kawaguchi, K., Liu, H., Yu, A. W., Yu, J., Chen, Y.-T., Luong, M.-T., Wu, Y., et al. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126658 · Cited by 85
Self-citationInstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt
Yang, D., Liu, S., Huang, R., Weng, C., & Meng, H. M. L. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3402088 · Cited by 125
Self-citationFoundation models and intelligent decision-making: Progress, challenges, and perspectives
Huang, J., Xu, Y., Wang, Q., Wang, Q., Liang, X., Wang, F., Zhang, Z., Wei, W., Zhang, B., Huang, L., et al. (2025) · The Innovation · doi:10.1016/j.xinn.2025.100948 · Cited by 87
Self-citationStyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis
Li, Y. A., Han, C., & Mesgarani, N. (2025) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2025.3530171 · Cited by 89
Deepfakes as a threat to a speaker and facial recognition: An overview of tools and attack vectors
Firc, A., Malinka, K., & Hanáček, P. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e15090 · Cited by 58
Seamless: Multilingual Expressive and Streaming Speech Translation
Communication, S., Barrault, L., Chung, Y.-A., Meglioli, M. C., Dale, D., Dong, N., Duppenthaler, M., Duquenne, P.-A., Ellis, B., Elsahar, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.05187 · Cited by 45
Self-citationFrom Speaker to Dubber: Movie Dubbing with Prosody and Duration Consistency Learning
Zhang, Z., Li, L., Cong, G., Yin, H., Gao, Y., Yan, C. C., van den Hengel, A., & Qi, Y. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680777 · Cited by 46
Self-citationThe Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audio
Xie, Y., Lu, Y., Fu, R., Wen, Z., Wang, Z., Tao, J., Qi, X., Wang, X., Liu, Y., Cheng, H., et al. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3525966 · Cited by 67
Self-citationE2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS
Eskimez, Ş. E., Wang, X., Thakker, M., Li, C., Tsai, C.-H., Xiao, Z. Y., Yang, H., Zhu, Z., Tang, M., Tan, X., et al. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832320 · Cited by 105
Self-citationAPCodec: A Neural Audio Codec With Parallel Amplitude and Phase Spectrum Encoding and Decoding
Ai, Y., Jiang, X.-H., Lu, Y.-X., Du, H.-P., & Ling, Z.-H. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3417347 · Cited by 42
SpeechX: Neural Codec Language Model as a Versatile Speech Transformer
Wang, X., Thakker, M., Chen, Z., Kanda, N., Eskimez, Ş. E., Chen, S., Tang, M., Liu, S., Li, J., & Yoshioka, T. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3419418 · Cited by 83
Self-citationFunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec
Du, Z., Zhang, S., Hu, K., & Zheng, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447523 · Cited by 62
Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation
Yariv, G., Gat, I., Benaim, S., Wolf, L., Schwartz, I., & Adi, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i7.28486 · Cited by 59
SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound
Liu, H., Xu, X., Yuan, Y., Wu, M., Wang, W., & Plumbley, M. D. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3506286 · Cited by 75
Expresso: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
Nguyen, T. A., Hsu, W.-N., D’Avirro, A., Shi, B., Gat, I., Fazel-Zarani, M., Remez, T., Copet, J., Synnaeve, G., Hassid, M., et al. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-1905 · Cited by 48
SafeEar: Content Privacy-Preserving Audio Deepfake Detection
Li, X., Li, K., Zheng, Y., Yan, C., Ji, X., & Xu, W. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670285 · Cited by 39
Chen, Z., Huang, H., Andrusenko, A., Hrinchuk, O., Puvvada, K. C., Li, J., Ghosh, S., Balam, J., & Ginsburg, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447553 · Cited by 44
Self-citationChang, X., Yan, B., Choi, K., Jung, J.-W., Lu, Y., Maiti, S., Sharma, R., Shi, J., Tian, J., Watanabe, S., et al. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447929 · Cited by 37
Self-citationXiao, L., Wang, D., Zhong, H., Wang, Q., Li, R., Jia, R., & Wan, B. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103805 · Cited by 33
Speaker Anonymization Using Neural Audio Codec Language Models
Panariello, M., Nespoli, F., Todisco, M., & Evans, N. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447871 · Cited by 34
Maiti, S., Peng, Y., Choi, S., Jung, J.-W., Chang, X., & Watanabe, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447112 · Cited by 53
Cheng, G., Wang, X., Cooper, E., Wells, D., Wang, L., Dang, J., Richmond, K., & Yamagishi, J. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3451951 · Cited by 34
Wang, S., Chen, Z., Lee, K. A., Qian, Y., & Li, H. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3492793 · Cited by 36
Self-citationSELM: Speech Enhancement using Discrete Tokens and Language Models
Wang, Z., Zhu, X., Zhang, Z., Lv, Y., Jiang, N., Zhao, G., & Xie, L. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447464 · Cited by 42
Self-citationIntegrating Generative Artificial Intelligence in Intelligent Vehicle Systems
Stappen, L., Dillmann, J., Striegel, S., Vögel, H.-J., Flores-Herr, N., & Schuller, B. W. (2023) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc57777.2023.10422003 · Cited by 35
UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding
Du, C., Guo, Y., Shen, F., Liu, Z., Zheng, L., Chen, X., Wang, S., Zhang, H., & Yu, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29747 · Cited by 27
Amphion: an Open-Source Audio, Music, and Speech Generation Toolkit
Zhang, X., Xue, L., Gu, Y., Wang, Y., Li, J., He, H., Wang, C., Liu, S., Chen, X., Zhang, J. S., et al. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832255 · Cited by 26
Self-citationLi, Y. A., Han, C., Raghavan, V., Mischler, G., & Mesgarani, N. (2023) · PubMed · doi:10.48550/arxiv.2306.07691 · Cited by 58
VioLA: Conditional Language Models for Speech Recognition, Synthesis, and Translation
Wang, T., Zhou, L., Zhang, Z., Wu, Y., Liu, S., Gaur, Y., Chen, Z., Li, J., & Wei, F. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3434425 · Cited by 135
Self-citationLM-VC: Zero-Shot Voice Conversion via Speech Generation Based on Language Models
Wang, Z., Chen, Y., Xie, L., Tian, Q., & Wang, Y. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3308474 · Cited by 24
TextrolSpeech: A Text Style Control Speech Corpus with Codec Language Text-to-Speech Models
Ji, S., Zuo, J., Fang, M., Jiang, Z., Chen, F., Duan, X., Huai, B., & Zhao, Z. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10445879 · Cited by 24
Xie, Y., Cheng, H., Wang, Y., & Ye, L. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448196 · Cited by 27
Textless Unit-to-Unit Training for Many-to-Many Multilingual Speech-to-Speech Translation
Kim, M., Choi, J., Kim, D., & Ro, Y. M. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3444470 · Cited by 34
TellMeTalk: Multimodal-driven talking face video generation
Li, P., Zhao, H., Liu, Q., Tang, P., & Zhang, L. (2024) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2023.109049 · Cited by 22
SpoofCeleb: Speech Deepfake Detection and SASV in the Wild
Jung, J.-W., Wu, Y., Wang, X., Kim, J.-H., Maiti, S., Matsunaga, Y., Shim, H.-J., Tian, J., Evans, N., Chung, J. S., et al. (2025) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2025.3529377 · Cited by 26
Self-citationJiang, X., Li, Y. A., Florea, A. N., Han, C., & Mesgarani, N. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889391 · Cited by 41
Wang, B., Tang, Y., Wei, F., Ba, Z., & Ren, K. (2024) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2024.3492796 · Cited by 20
Self-citationUbiq-Genie: Leveraging External Frameworks for Enhanced Social VR Experiences
Numan, N., Giunchi, D., Congdon, B., & Steed, A. J. (2023) · IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) · doi:10.1109/vrw58643.2023.00108 · Cited by 20
Fewer-Token Neural Speech Codec with Time-Invariant Codes
Ren, Y. S., Wang, T., Yi, J., Xu, L., Tao, J., Zhang, C. Y., & Zhou, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448454 · Cited by 19
VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech
Du, C., Guo, Y., Wang, H., Yang, Y., Niu, Z., Wang, S., Zhang, H., Chen, X., & Yu, K. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890943 · Cited by 31
Self-citationGuo, H., Liu, C., Ishi, C. T., & Ishiguro, H. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389651 · Cited by 16
MM-TTS: Multi-Modal Prompt Based Style Transfer for Expressive Text-to-Speech Synthesis
Guan, W., Li, Y., Li, T., Huang, H., Wang, F., Lin, J., Huang, L., Li, L., & Hong, Q. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29769 · Cited by 21
Generative Expressive Conversational Speech Synthesis
Liu, R., Hu, Y., Ren, Y., Yin, X. G., & Li, H. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681697 · Cited by 16
Yet Another Generative Model for Room Impulse Response Estimation
Lee, S., Choi, H.-S., & Lee, K. (2023) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa58266.2023.10248189 · Cited by 15
Jiang, X.-H., Ai, Y., Zheng, R.-C., Du, H.-P., Lu, Y.-X., & Ling, Z.-H. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832214 · Cited by 16
Puvvada, K. C., Koluguri, N. R., Dhawan, K., Balam, J., & Ginsburg, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446998 · Cited by 18
USAT: A Universal Speaker-Adaptive Text-to-Speech Approach
Wang, W., Song, Y., & Jha, S. K. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3393714 · Cited by 15
Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimization
Gao, X., Zhang, C., Chen, Y., Zhang, H., & Chen, N. F. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888737 · Cited by 38
Vision + Language Applications: A Survey
Zhou, Y., & Shimada, N. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00090 · Cited by 14
Qiang, C., Li, H., Ni, H., Qu, H., Fu, R., Wang, T., Wang, L., & Dang, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446203 · Cited by 16
Leveraging VQ-VAE tokenization for autoregressive modeling of medical time series
Lee, Y., Chae, Y., & Jung, K. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102925 · Cited by 13
ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs For Audio, Music, and Speech
Shi, J., Tian, J., Wu, Y., Jung, J.-W., Yip, J. Q., Masuyama, Y., Chen, W., Wu, Y., Tang, Y., Baali, M., et al. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832289 · Cited by 18
Self-citationEnhancing Low-Resource ASR through Versatile TTS: Bridging the Data Gap
Yang, G., Yu, F., Ma, Z., Du, Z., Gao, Z., Zhang, S. S., & Chen, X. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889894 · Cited by 13
Text-Dominant Speech-Enhanced for Multimodal Aspect-Based Sentiment Analysis network
Huang, X., Sun, H., Xue-chun, L., Wu, J., & He, L. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103543 · Cited by 12
Self-citationSpeaker Adaptive Text-to-Speech With Timbre-Normalized Vector-Quantized Feature
Du, C., Guo, Y., Chen, X., & Yu, K. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3308374 · Cited by 13
E3 TTS: Easy End-to-End Diffusion-Based Text To Speech
Gao, Y., Morioka, N., Zhang, Y., & Chen, N. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389766 · Cited by 18
tinyDigiClones: A Multi-Modal LLM-Based Framework for Edge-optimized Personalized Avatars
Basit, A., & Shafique, M. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10649909 · Cited by 12
Boosting Large Language Model for Speech Synthesis: An Empirical Study
Hao, H., Zhou, L., Liu, S., Li, J., Hu, S., Wang, R., & Wei, F. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890588 · Cited by 29
Self-citationComparison of the Ability of Neural Network Model and Humans to Detect a Cloned Voice
Milewski, K., Zaporowski, S., & Czyżewski, A. (2023) · Electronics · doi:10.3390/electronics12214458 · Cited by 16
PS3DT: Synthetic Speech Detection Using Patched Spectrogram Transformer
Yadav, A. K. S., Xiang, Z., Bhagtani, K., Bestagini, P., Tubaro, S., & Delp, E. J. (2023) · International Conference on Machine Learning and Applications · doi:10.1109/icmla58977.2023.00075 · Cited by 12
Sparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17
Toward Joint Language Modeling for Speech Units and Text
Chou, J.-C., Chien, C.-M., Hsu, W.-N., Livescu, K., Babu, A., Conneau, A., Baevski, A., & Auli, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.438 · Cited by 22
Transduce and Speak: Neural Transducer for Text-To-Speech with Semantic Token Prediction
Kim, M., Jeong, M., Choi, B. J., Lee, D., & Kim, N. S. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389791 · Cited by 11
Extending Multilingual Speech Synthesis to 100+ Languages without Transcribed Data
Saeki, T., Wang, G., Morioka, N., Elias, I., Kastner, K., Rosenberg, A., Ramabhadran, B., Zen, H., Beaufays, F., & Shemtov, H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448074 · Cited by 12
Generative De-Quantization for Neural Speech Codec Via Latent Diffusion
Yang, H., Jang, I., & Kim, M. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446556 · Cited by 12
High-Fidelity Neural Phonetic Posteriorgrams
Churchwell, C., Morrison, M., & Pardo, B. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10669905 · Cited by 11
Vec-Tok Speech: Speech Vectorization and Tokenization for Neural Speech Generation
Zhu, X., Lv, Y., Lei, Y., Li, T., He, W., Zhou, H., Lu, H., & Xie, L. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3546559 · Cited by 17
Fujita, K., Satō, H., Ashihara, T., Kanagawa, H., Delcroix, M., Moriya, T., & Ijima, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447809 · Cited by 11
Shirasaka, M., Matsushima, T., Tsunashima, S., Ikeda, Y., Horo, A., Ikoma, S., Tsuji, C., Wada, H., Omija, T., Komukai, D., et al. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611640 · Cited by 15
Speak While You Think: Streaming Speech Synthesis During Text Generation
Dekel, A., Shechtman, S., Fernandez, R., Haws, D., Kons, Z., & Hoory, R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446214 · Cited by 13
AccentBox: Towards High-Fidelity Zero-Shot Accent Generation
Zhong, J., Richmond, K., Su, Z., & Sun, S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888332 · Cited by 12
Triantafyllopoulos, A., Spiesberger, A. A., Tsangko, I., Jing, X., Distler, V., Dietz, F., Alt, F., & Schuller, B. W. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2025.101802 · Cited by 20
Wu, H., Wang, X., Eskimez, Ş. E., Thakker, M., Tompkins, D., Tsai, C.-H., Li, C., Xiao, Z. Y., Zhao, S., Li, J., et al. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832181 · Cited by 9
Self-citationLiu, R., Liang, K., Hu, D., Li, T., Yang, D., & Li, H. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3533361 · Cited by 9
Speech Enhancement Using Continuous Embeddings of Neural Audio Codec
Li, H., Yip, J. Q., Fan, T., & Chng, E. S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890379 · Cited by 9
RefXVC: Cross-Lingual Voice Conversion With Enhanced Reference Leveraging
Zhang, M., Zhou, Y., Ren, Y., Zhang, C., Yin, X., & Li, H. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3439996 · Cited by 9
BeCAPTCHA-Type: Biometric Keystroke Data Generation for Improved Bot Detection
DeAlcala, D., Morales, A., Tolosana, R., et al. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00112 · Cited by 8
Филиппович, Б. Н., & Lebedeva, A. K. (2023) · Юридические исследования · doi:10.25136/2409-7136.2023.11.69014 · Cited by 8
T5G2P: Text-to-Text Transfer Transformer Based Grapheme-to-Phoneme Conversion
Řezáčková, M., Tihelka, D., & Matoušek, J. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3426332 · Cited by 9
Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgetting
Chen, H., & Garner, P. N. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3463395 · Cited by 13
From One Stolen Utterance: Assessing the Risks of Voice Cloning in the AIGC Era
Wang, K., Chen, M., Lü, L., Feng, J., Chen, Q., Ba, Z., Ren, K., & Chen, C. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00238 · Cited by 8
Low-Latency Speech Enhancement via Speech Token Generation
Xue, H., Peng, X., & Lü, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447774 · Cited by 8
AutoPrep: An Automatic Preprocessing Framework for In-The-Wild Speech Data
Yu, J., Chen, H., Bian, Y., Li, X., Luo, Y., Tian, J., Liu, M., Jiang, J., & Wang, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447759 · Cited by 8
Everson, K., Gu, Y., Yang, H., Shivakumar, P. G., Lin, G.-T., Kolehmainen, J., Bulyko, I., Gandhe, A., Ghosh, S., Hamza, W., et al. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447938 · Cited by 12
U-Style: Cascading U-Nets With Multi-Level Speaker and Style Modeling for Zero-Shot Voice Cloning
Li, T., Wang, Z., Zhu, X., Cong, J., Tian, Q., Wang, Y., & Xie, L. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3453606 · Cited by 8
Codec-Superb @ SLT 2024: A Lightweight Benchmark For Neural Audio Codec Models
Wu, H., Chen, X., Lin, Y., Chang, K., Du, J., Lu, K.-H., Liu, A. H., Chung, H.-L., Wu, Y.-K., Yang, D., et al. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832364 · Cited by 16
Self-citationA comprehensive study of audio profiling: Methods, applications, challenges, and future directions
Pudasaini, A., Al-Hawawreh, M. S., Bouadjenek, M. R., Hacid, H., & Aryal, S. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130334 · Cited by 9
Metadata
- Authors
- Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., He, L., Zhao, S., & Wei, F.
- Affiliations
- Microsoft (United States) (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2301.02111
- OpenAlex
- W4313679638
- Semantic Scholar
- c2f91f35df893714418cc29096083dce0b441229
- Cited by
- 250
- References
- 0
- Citations by year
- 2026: 4 · 2025: 86 · 2024: 115 · 2023: 45
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- public-domain
- arxiv.org/licenses/nonexclusive-distrib/1.0/
- Locations
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
- Keywords
- text-to-speech, zero-shot speech synthesis, neural audio codecs, personalized speech synthesis, in-context learning, speaker similarity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
