Cited by 92Speech Recognition and SynthesisSpeech and Audio ProcessingVoice and Speech Disorders

MultiSpeech: Multi-Speaker Text to Speech with Transformer

Open access
Chen, Mingjian, Tan, Xu, Ren, Yi, Xu, Jin, Sun, Hao, Zhao, Sheng, Qin, Tao·2020·Interspeech 2020

Abstract

Transformer-based text to speech (TTS) model (e.g., Transformer TTS [1], FastSpeech [2]) has shown the advantages of training and inference efficiency over Tacotron [3]) due to its parallel computation in training and/or inference.However, the parallel computation increases the difficulty while learning the alignment between text and speech in Transformer, which is further magnified in the multi-speaker scenario with noisy data and diverse speakers, and hinders the applicability of Transformer for multi-speaker TTS.In this paper, we develop a robust and high-quality multispeaker Transformer TTS system called MultiSpeech, with several specially designed components/techniques to improve textto-speech alignment: 1) a diagonal constraint on the weight matrix of encoder-decoder attention in both training and inference; 2) layer normalization on phoneme embedding in encoder to better preserve position information; 3) a bottleneck in decoder pre-net to prevent copy between consecutive speech frames.Experiments on VCTK and LibriTTS multi-speaker datasets demonstrate the effectiveness of MultiSpeech: 1) it synthesizes more robust and better quality multi-speaker voice than naive Transformer based TTS; 2) with a MutiSpeech model as the teacher, we obtain a strong multi-speaker FastSpeech model with almost zero quality degradation while enjoying extremely fast inference speed.

Integrity signals

Registry

Crossref · 10.21437/interspeech.2020-3139

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Several ideas introduced recently include random sampling, learning concept dictionary, uniform masking, patch projection, joint learning of a region detector, and representation aligning before mask prediction.

    “Several ideas introduced recently include random sampling [113], learning concept dictionary [203], uniform masking [270], patch projection [192], joint learning of a region detector [271], and representation aligning before mask prediction [263].”

    From Multimodal learning with transformers: A survey (2023) · cites this paper as [113]

  • Recent remedies for fine-grained semantic alignment include random sampling, learning a concept dictionary, jointly learning a region detector, and representation aligning before mask prediction.

    “Several remedies introduced recently include random sampling [113], learning concept dictionary [203], jointly learning a region detector [271], and representation aligning before mask prediction [263].”

    From Multimodal learning with transformers: A survey (2023) · cites this paper as [113]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Chen, M., Tan, X., Ren, Y., Xu, J., Sun, H., Zhao, S., & Qin, T.
Affiliations
  • Peking University (CN)
  • Microsoft Research Asia (China) (CN)
  • Zhejiang University (CN)
  • Tsinghua University (CN)
  • Microsoft (United States) (US)
Venue
Interspeech 2020
Publisher
ISCA
Published
2020-10-25
Pages
4024-4028
ISSN
2308-457X (ISSN-L)
OpenAlex
W3095545636
Cited by
92
References
23
Citations by year
2026: 3 · 2025: 16 · 2024: 10 · 2023: 20 · 2022: 14 · 2021: 27 · 2020: 2
FWCI
5.93
Type
conference paper
Language
en
Open access
Licence
public-domain
Locations
  • Interspeech (USB) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, public-domain)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Health Sciences › Medicine › Physiology › Voice and Speech Disorders
Keywords
multi-speaker text-to-speech, inference speed, VCTK, LibriTTS

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review