Listening like a speech-training app: Expert and non-expert listeners’ goodness ratings of children’s speech
Open accessAbstract
Speech training apps are being developed that provide automatic feedback concerning children's production of known target words, as a score on a 1-5 scale. However, this 'goodness' scale is still poorly understood. We investigated listeners' ratings of 'how many stars the app should provide as feedback' on children's utterances, and whether listener agreement is affected by clinical experience and/or access to anchor stimuli. In addition, we explored the association between goodness ratings and clinical measures of speech accuracy; the Percentage of Consonants Correct (PCC) and the Percentage of Phonemes Correct (PPC). Twenty speech-language pathologists and 20 non-expert listeners participated; half of the listeners in each group had access to anchor stimuli. The listeners rated 120 words, collected from children with and without speech sound disorder. Concerning reliability, intra-rater agreement was generally high, whereas inter-rater agreement was moderate. Access to anchor stimuli was associated with higher agreement, but only for non-expert listeners. Concerning the association between goodness ratings and the PCC/PPC, correlations were moderate for both listener groups, under both conditions. The results indicate that the task of rating goodness is difficult, regardless of clinical experience, and that access to anchor stimuli is insufficient for achieving reliable ratings. This raises concerns regarding the 1-5 rating scale as the means of feedback in speech training apps. More specific listener instructions, particularly regarding the intended context for the app, are suggested in collection of human ratings underlying the development of speech training apps. Until then, alternative means of feedback should be preferred.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 29,639 · see citing works
The REDCap consortium: building an international community of software platform partners
Harris, P. A., Taylor, R., Minor, B. L., Elliott, V., Fernandez, M., O'Neal, L., McLeod, L., Delacqua, G., Delacqua, F., Kirby, J., et al. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103208 · Cited by 24,013 · see citing works
Practical Statistics for Medical Research
Altman, D. G. (1990) · doi:10.1201/9780429258589 · Cited by 10,668 · see citing works
Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M., & de Vet, H. C. W. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.006 · Cited by 4,465 · see citing works
Statistics corner: A guide to appropriate use of correlation coefficient in medical research.
(2012) · Europe PMC (PubMed Central) · Cited by 4,239 · see citing works
Perceptual Evaluation of Voice Quality
Kreiman, J. E., Gerratt, B. R., Kempster, G. B., Erman, A. A., & Berke, G. S. (1993) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3601.21 · Cited by 737 · see citing works
The Percentage of Consonants Correct (PCC) Metric: Extensions and Reliability Data
Shriberg, L. D., Austin, D., Lewis, B. A., McSweeny, J. L., & Wilson, D. L. (1997) · Journal of Speech, Language, and Hearing Research · doi:10.1044/jslhr.4004.708 · Cited by 647 · see citing works
Shriberg, L. D., & Kwiatkowski, J. (1982) · Journal of Speech and Hearing Disorders · doi:10.1044/jshd.4703.256 · Cited by 599 · see citing works
Universal Parameters for Reporting Speech Outcomes in Individuals with Cleft Palate
Henningsson, G., Kuehn, D. P., Sell, D. A., Sweeney, T., Trost-Cardamone, J. E., & Whitehill, T. L. (2008) · The Cleft Palate-Craniofacial Journal · doi:10.1597/06-086.1 · Cited by 553 · see citing works
Comparing Internal and External Standards in Voice Quality Judgments
Gerratt, B. R., Kreiman, J. E., Antoñanzas-Barroso, N., & Berke, G. S. (1993) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3601.14 · Cited by 280 · see citing works
The Effect of Anchors and Training on the Reliability of Perceptual Voice Evaluation
Chan, K. M. K., & Yiu, E. M. ‐. (2002) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2002/009 · Cited by 204 · see citing works
Skahan, S., Watson, M. M., & Lof, G. L. (2007) · American Journal of Speech-Language Pathology · doi:10.1044/1058-0360(2007/029 · Cited by 199 · see citing works
Mcleod, S., & Baker, E. (2014) · Clinical Linguistics & Phonetics · doi:10.3109/02699206.2014.926994 · Cited by 193 · see citing works
The Intelligibility of Children’s Speech
Kent, R. D., Miolo, G., & Bloedel, S. L. (1994) · American Journal of Speech-Language Pathology · doi:10.1044/1058-0360.0302.81 · Cited by 153 · see citing works
Random versus blocked practice in treatment for childhood apraxia of speech.
Maas, E., & Farinella, K. (2012) · Journal of Speech, Language and Hearing Research · doi:10.1044/1092-4388(2011/11-0120 · Cited by 146 · see citing works
Munson, B., Edwards, J., Schellinger, S. K., Beckman, M. E., & Meyer, M. K. (2010) · Clinical Linguistics & Phonetics · doi:10.3109/02699200903532524 · Cited by 122 · see citing works
The influence of listener experience and academic training on ratings of nasality
Lewis, K. E., Watterson, T., & Houghton, S. M. (2002) · Journal of Communication Disorders · doi:10.1016/s0021-9924(02)00134-x · Cited by 109 · see citing works
The Effect of Listener Experience and Anchors on Judgments of Dysphonia
Eadie, T. L., & Kapsner-Smith, M. R. (2010) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2010/09-0205 · Cited by 98 · see citing works
Shriberg, L. D., Strand, E. A., Jakielski, K. J., & Mabie, H. L. (2019) · Clinical Linguistics & Phonetics · doi:10.1080/02699206.2019.1595732 · Cited by 84 · see citing works
Brunnegård, K., Lohmander, A., & van Doorn, J. (2008) · International Journal of Language & Communication Disorders · doi:10.1080/13682820802295203 · Cited by 84 · see citing works
Helou, L. B., Solomon, N. P., Henry, L. R., Coppit, G. L., Howard, R. S., & Stojadinovic, A. (2010) · American Journal of Speech-Language Pathology · doi:10.1044/1058-0360(2010/09-0012 · Cited by 75 · see citing works
Munson, B., Johnson, J. M., & Edwards, J. (2012) · American Journal of Speech-Language Pathology · doi:10.1044/1058-0360(2011/11-0009 · Cited by 69 · see citing works
Intelligibility in Context Scale: cross-linguistic use, validity, and reliability
McLeod, S. L. (2020) · Speech Language and Hearing · doi:10.1080/2050571x.2020.1718837 · Cited by 68 · see citing works
The visual sort and rate method for perceptual evaluation in listening tests
Granqvist, S. (2003) · Logopedics, Phoniatrics, Vocology · doi:10.1080/14015430310015255 · Cited by 66 · see citing works
Speech-driven mobile games for speech therapy: User experiences and feasibility
Ahmed, B., Monroe, P., Hair, A., Tan, C. T., Gutierrez-Osuna, R., & Ballard, K. J. (2018) · International Journal of Speech-Language Pathology · doi:10.1080/17549507.2018.1513562 · Cited by 65 · see citing works
Computer-based speech therapy for childhood speech sound disorders
Furlong, L., Erickson, S., & Morris, M. E. (2017) · Journal of Communication Disorders · doi:10.1016/j.jcomdis.2017.06.007 · Cited by 62 · see citing works
Brinca, L., Batista, A. P., Tavares, A. I., Pinto, P. S. L. G. D. S., & Araújo, L. M. C. (2015) · Journal of Voice · doi:10.1016/j.jvoice.2015.01.007 · Cited by 59 · see citing works
Hustad, K. C., Mahr, T., Natzke, P., & Rathouz, P. J. (2021) · Journal of Speech Language and Hearing Research · doi:10.1044/2021_jslhr-21-00142 · Cited by 54 · see citing works
The impact of selecting different contrasts in phonological therapy
Dodd, B. J., Crosbie, S., McIntosh, B., Holm, A., Harvey, C., Liddy, M., Fontyne, K., Pinchin, B., & Rigby, H. (2008) · International Journal of Speech-Language Pathology · doi:10.1080/14417040701732590 · Cited by 54 · see citing works
Automated speech analysis tools for children’s speech production: A systematic literature review
McKechnie, J., Ahmed, B., Gutierrez-Osuna, R., Monroe, P., McCabe, P., & Ballard, K. J. (2018) · International Journal of Speech-Language Pathology · doi:10.1080/17549507.2018.1477991 · Cited by 54 · see citing works
Tabby Talks: An automated tool for the assessment of childhood apraxia of speech
Shahin, M. A., Ahmed, B., Parnandi, A., Karappa, V., McKechnie, J., Ballard, K. J., & Gutiérrez‐Osuna, R. (2015) · Speech Communication · doi:10.1016/j.specom.2015.04.002 · Cited by 49 · see citing works
Talking to Teo: Video game supported speech therapy
Newball, A. A. N., Loaiza, D. F., Oviedo, C., Castillo, A., Portilla, A., Linares, D., & Alvarez, G. (2014) · Entertainment Computing · doi:10.1016/j.entcom.2014.10.005 · Cited by 48 · see citing works
Gradient perception of children’s productions of /s/ and /θ/: A comparative study of rating methods
Schellinger, S. K., Munson, B., & Edwards, J. (2016) · Clinical Linguistics & Phonetics · doi:10.1080/02699206.2016.1205665 · Cited by 42 · see citing works
Developing a Weighted Measure of Speech Sound Accuracy
Preston, J. L., Ramsdell, H. L., Oller, D. K., Edwards, M. L., & Tobin, S. J. (2010) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2010/10-0030 · Cited by 40 · see citing works
Jesus, L. M. T., Martinez, J., Santos, J. P., Hall, A. O., & Joffe, V. L. (2019) · Journal of Speech Language and Hearing Research · doi:10.1044/2019_jslhr-s-18-0301 · Cited by 39 · see citing works
Processes and challenges in clinical decision‐making for children with speech‐sound disorders
Furlong, L., Serry, T. A., Erickson, S., & Morris, M. E. (2018) · International Journal of Language & Communication Disorders · doi:10.1111/1460-6984.12426 · Cited by 35 · see citing works
Klein, H. B., Grigos, M. I., Byun, T. M., & Davidson, L. (2012) · Clinical Linguistics & Phonetics · doi:10.3109/02699206.2012.682695 · Cited by 29 · see citing works
Enhancing Intervention for Residual Rhotic Errors Via App-Delivered Biofeedback: A Case Study
Byun, T. M., Kabakoff, H. M., Carey, H., Liang, W., Park, T. H., & Svirsky, M. A. (2017) · Journal of Speech Language and Hearing Research · doi:10.1044/2017_jslhr-s-16-0248 · Cited by 28 · see citing works
Developing an AI-Assisted Low-Resource Spoken Language Learning App for Children
Getman, Y., Phan, N., Al-Ghezi, R., Voskoboinik, E., Singh, M., Grósz, T., Kurimo, M., Salvi, G., Svendsen, T., Strömbergsson, S., et al. (2023) · IEEE Access · doi:10.1109/access.2023.3304274 · Cited by 27 · see citing works
Self-citationBarrow, C. W., Körner, K., & Strömbergsson, S. (2021) · Logopedics Phoniatrics Vocology · doi:10.1080/14015439.2021.1977383 · Cited by 24 · see citing works
Self-citationCleland, J., & Scobbie, J. M. (2021) · Journal of Speech Language and Hearing Research · doi:10.1044/2020_jslhr-20-00373 · Cited by 19 · see citing works
Gaming enhances learning-induced plastic changes in the brain
Junttila, K., Smolander, A.-R. M., Karhila, R., Giannakopoulou, A., Uther, M., Kurimo, M., & Ylinen, S. (2022) · Brain and Language · doi:10.1016/j.bandl.2022.105124 · Cited by 18 · see citing works
Self-citationThe Measurement of Intervention Effects in Developmental Phonological Disorders
Hall, R., Adams, C., Hesketh, A., & Nightingale, K. (1998) · International Journal of Language & Communication Disorders · doi:10.3109/13682829809179466 · Cited by 17 · see citing works
Clinical experience and categorical perception of children's speech
Meyer, M. K., & Munson, B. (2021) · International Journal of Language & Communication Disorders · doi:10.1111/1460-6984.12610 · Cited by 16 · see citing works
Equitable access to speech practice for rural Australian children using the SayBananas! mobile game
McLeod, S., Kelly, G., Ahmed, B., & Ballard, K. J. (2023) · International Journal of Speech-Language Pathology · doi:10.1080/17549507.2023.2205057 · Cited by 13 · see citing works
Transparent Pronunciation Scoring Using Articulatorily Weighted Phoneme Edit Distance
Karhila, R., Smolander, A.-R. M., Ylinen, S., & Kurimo, M. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1785 · Cited by 11 · see citing works
Self-citationSpeech-Language Pathologists' Ratings of Speech Accuracy in Children With Speech Sound Disorders
Jing, L., & Grigos, M. I. (2021) · American Journal of Speech-Language Pathology · doi:10.1044/2021_ajslp-20-00381 · Cited by 10 · see citing works
Dealing with the unknown – addressing challenges in evaluating unintelligible speech
Alves, M. Ö. C., Ode, C., & Strömbergsson, S. (2019) · Clinical Linguistics & Phonetics · doi:10.1080/02699206.2019.1622787 · Cited by 9 · see citing works
Self-citationUnderstanding acceptability of disordered speech through Audience Response Systems-based evaluation
Strömbergsson, S., Edlund, J., McAllister, A. M., & Lagerberg, T. B. (2021) · Speech Communication · doi:10.1016/j.specom.2021.05.005 · Cited by 9 · see citing works
Self-citationSIAK – A Game for Foreign Language Pronunciation Learning
Karhila, R., Ylinen, S., Enarvi, S., Palomäki, K. J., Nikulin, A., Rantula, O., Viitanen, V., Dhinakaran, K., Smolander, A.-R. M., Kallio, H., et al. (2017) · Aaltodoc (Aalto University) · Cited by 8 · see citing works
Self-citationLagerberg, T. B., Lam, J., Olsson, R., Abelin, Å., & Strömbergsson, S. (2019) · Journal of Speech Language and Hearing Research · doi:10.1044/2019_jslhr-s-18-0492 · Cited by 6 · see citing works
Self-citationMulti-task wav2vec2 Serving as a Pronunciation Training System for Children
Getman, Y., Al-Ghezi, R., Grósz, T., & Kurimo, M. (2023) · ISCA Workshop on Speech and Language Technology in Education (SLaTE) · doi:10.21437/slate.2023-8 · Cited by 5 · see citing works
Self-citationPerception of Children’s Productions of /l/: Acoustic Correlates and Effects of Listener Experience
Coniglio, E., Chung, H., & Schellinger, S. K. (2022) · Folia Phoniatrica et Logopaedica · doi:10.1159/000524395 · Cited by 4 · see citing works
Measuring intelligibility in spontaneous speech using syllables perceived as understood
Lagerberg, T. B., Holm, K., McAllister, A. M., & Strömbergsson, S. (2021) · Journal of Communication Disorders · doi:10.1016/j.jcomdis.2021.106108 · Cited by 4 · see citing works
Self-citationBrown, T., Murray, E. L., & McCabe, P. (2018) · Clinical Linguistics & Phonetics · doi:10.1080/02699206.2018.1463395 · Cited by 3 · see citing works
Alighieri, C., Bettens, K., Bruneel, L., D’haeseleer, E., Gaever, E. V., & Lierde, K. M. V. (2021) · Journal of Speech Language and Hearing Research · doi:10.1044/2021_jslhr-20-00628 · Cited by 3 · see citing works
Aakko, I., Kauramäki, J., Cleland, J., Lee, A., Vainio, M., & Saalasti, S. (2022) · Clinical Linguistics & Phonetics · doi:10.1080/02699206.2022.2118079 · Cited by 2 · see citing works
Sakash, A., Mahr, T., & Hustad, K. C. (2023) · Clinical Linguistics & Phonetics · doi:10.1080/02699206.2022.2150893 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Strömbergsson, S., Edlund, E., Pettersson, M., Phan, N., & Kurimo, M. (2026) · International Journal of Language & Communication Disorders · doi:10.1111/1460-6984.70163
“As elaborated in (Strömbergsson et al. 2024), the use of the 1–5 rating scale in the context of SSD can be questioned; both SLP clinicians and non-expert adults have difficulties using the scale in their ratings of children’s disordered speech.”
Self-citation
- Authors
- Strömbergsson, S., Fröjdh, M., Pettersson, M., Grósz, T., Getman, Y., & Kurimo, M.
- Affiliations
- Karolinska Institutet (SE)
- Aalto University (FI)
- Corresponding author
- Strömbergsson, Sofia
- Venue
- Clinical Linguistics & Phonetics
- Publisher
- Informa UK Limited
- Published
- 2024-06-09
- Volume
- 39
- Issue
- 2
- Pages
- 144-165
- ISSN
- 0269-9206 (ISSN-L)
- 1464-5076
- OpenAlex
- W4399506378
- Cited by
- 1
- References
- 58
- Citations by year
- 2025: 1
- FWCI
- 0.79
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Clinical Linguistics & Phonetics (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Aaltodoc (Aalto University) (submittedVersion, cc-by)
- Funders
- NordForsk (103893)
- Topics
- Social Sciences › Psychology › Developmental and Educational Psychology › Language Development and Disorders
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Health Sciences › Medicine › Physiology › Voice and Speech Disorders
- Keywords
- listener ratings, speech sound disorders, children's speech, automated feedback, interobserver agreement, intraobserver agreement, speech-language pathologists, speech accuracy, rating scale reliability
- MeSH terms
- Adult, Child, Child, Preschool, Female, Humans, Male, Phonetics, Speech, Speech-Language Pathology, Speech Perception, Speech Therapy, Reproducibility of Results, Mobile Applications, Speech Sound Disorder
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
