Speech Synthesis Evaluation — State-of-the-Art Assessment and Suggestion for a Novel Research Program
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Wagner P, Beskow J, Betz S, et al. Speech Synthesis Evaluation—State-of-the-Art Assessment and Suggestion for a Novel Research Program. In: Proceedings of the 10th Speech Synthesis Workshop (SSW10). 2019.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It is preferred to correlation-based metrics, such as computing a Pearson correlation between ground truth and decoded acoustic waveforms, as MCD weighs the features in the produced sound that are most perceptually relevant for conveying information151.”
From The speech neuroprosthesis (2024) · cites this paper as [151]
Citations
Cites
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · cited 3×
“How can novel machine learning and high quality synthesis such as WaveNet be put to use in TTS evaluation?”
MethodBartneck, C., Kulić, D., Croft, E., & Zoghbi, S. (2009) · International Journal of Social Robotics · doi:10.1007/s12369-008-0001-3 · Cited by 3,193 · cited 1×
“To overcome this problem, typical surveys employed in HCI or HRI tend to be very extensive [35], thereby trying to address all potential quality dimensions a user may have employed in her or his assessment.”
BackgroundEncyclopedia of Survey Research Methods
Lavrakas, P. (2008) · doi:10.4135/9781412963947 · Cited by 2,689
Explaining Phonetic Variation: A Sketch of the H&H Theory
Lindblom, B. (1990) · Speech Production and Speech Modelling · doi:10.1007/978-94-009-2037-8_16 · Cited by 1,824
The Meaning of Intonational Contours in the Interpretation of Discourse
Pierrehumbert, J., & Hirschberg, J. (1990) · Intentions in Communication · doi:10.7551/mitpress/3839.003.0016 · Cited by 1,433
The Meaning of Intonational Contours in the Interpretation of Discourse
Pierrehumbert, J. B., & Hirschberg, J. (1990) · doi:10.7916/d8kd24fp · Cited by 830 · cited 1×
“…(tutoring, chatting, arguing, counseling...), individual (hearing problems, attitude, level of distraction, motivation, familiarity), linguistic (frequency, predictability, suprisal, importance) or environmental settings (external noise, mutual visibility, ...) [11, 12, 13, 14, 15, 16, 17, 18].”
Aylett, M., & Turk, A. (2004) · Language and Speech · doi:10.1177/00238309040470010201 · Cited by 781 · cited 1×
“…(tutoring, chatting, arguing, counseling...), individual (hearing problems, attitude, level of distraction, motivation, familiarity), linguistic (frequency, predictability, suprisal, importance) or environmental settings (external noise, mutual visibility, ...) [11, 12, 13, 14, 15, 16, 17, 18].”
Probabilistic relations between words: Evidence from reduction in lexical production
Jurafsky, D., Bell, A., Gregory, M., & Raymond, W. D. (2001) · Typological Studies in Language · doi:10.1075/tsl.45.13jur · Cited by 762
A Bayesian explanation of the ‘Uncanny Valley’ effect and related psychological phenomena
Moore, R. K. (2012) · Scientific Reports · doi:10.1038/srep00864 · Cited by 212 · cited 1×
“This is in line with the analysis by [20, 21], claiming that long-known problems of human machine interaction such as the uncanny valley can be modeled as a mismatch between a user’s expectations and a machine’s actual expression.”
ResultQuality-Net: An End-to-End Non-intrusive Speech Quality Assessment Model Based on BLSTM
Fu, S.-W., Tsao, Y., Hwang, H.-T., & Wang, H.-M. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1802 · Cited by 178
Cited by
Saka, A. B., Oyedele, L. O., Akanbi, L. A., Ganiyu, S. A., Chan, D. W. M., & Bello, S. A. (2023) · Advanced Engineering Informatics · doi:10.1016/j.aei.2022.101869 · Cited by 161
Conversational AI: Dialogue Systems, Conversational Agents, and Chatbots
McTear, M. (2021) · Synthesis Lectures on Human Language Technologies · doi:10.1007/978-3-031-02176-3 · Cited by 155
Kühne, K., Fischer, M. H., & Zhou, Y. (2020) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2020.593732 · Cited by 163
“While most synthesized voices have reached a high level of intelligibility, it is the perceived quality that still requires clarification (Polkosky and Lewis, 2003; Wagner et al., 2019).”
BackgroundSilva, A. B., Littlejohn, K. T., Liu, J. R., Moses, D. A., & Chang, E. F. (2024) · Nature Reviews Neuroscience · doi:10.1038/s41583-024-00819-9 · Cited by 103
“The performance of TTS systems is often described using mel-cepstral distortion (MCD) and open-ended human-transcribed WER 149,150 .”
Choice of Voices: A Large-Scale Evaluation of Text-to-Speech Voice Quality for Long-Form Content
Cambre, J., Colnago, J., Maddock, J., Tsai, J., & Kaye, J. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376789 · Cited by 102
“However, speech technology is advancing so rapidly that the state-of-the-art is likely outpacing our own publishing cycles [41,49].”
BackgroundInfluential
Metadata
- Authors
- Wagner, P., Beskow, J., Betz, S., Edlund, J., Gustafson, J., Eje Henter, G., Le Maguer, S., Malisz, Z., Székely, É., Tånnander, C., & Voße, J.
- Affiliations
- Bielefeld University (DE)
- KTH Royal Institute of Technology (SE)
- Corresponding author
- Wagner, Petra
- ORCID
- Venue
- 10th ISCA Workshop on Speech Synthesis (SSW 10)
- Publisher
- ISCA
- Published
- 2019-09-20
- Pages
- 105-110
- OpenAlex
- W3011535310
- Cited by
- 74
- References
- 40
- Citations by year
- 2026: 1 · 2025: 9 · 2024: 12 · 2023: 17 · 2022: 12 · 2021: 11 · 2020: 12
- FWCI
- 3.35
- Type
- conference paper
- Language
- en
- Open access
- green
- nbn-resolving.org
- Licence
- cc-by-nc-nd
- Locations
- ISCA Speech Synthesis Workshop (SSW) (publishedVersion)
- PUB – Publications at Bielefeld University (Bielefeld University) (submittedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
- Keywords
- speech synthesis evaluation, speech quality assessment, state-of-the-art assessment, research program proposal, speech synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Speech Synthesis Evaluation — State-of-the-Art Assessment and Suggestion for a Novel Research Program. https://app.cochat.ai/papers/10.21437/ssw.2019-19
- The paper
- Wagner, P., Beskow, J., Betz, S., Edlund, J., Gustafson, J., Eje Henter, G., Le Maguer, S., Malisz, Z., Székely, É., Tånnander, C., & Voße, J. (2019). Speech Synthesis Evaluation — State-of-the-Art Assessment and Suggestion for a Novel Research Program. 10th ISCA Workshop on Speech Synthesis (SSW 10). https://doi.org/10.21437/ssw.2019-19
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.