Progress toward estimating the minimal clinically important difference of intelligibility: a crowdsourced perceptual experiment
Open accessAbstract
Purpose: The purpose of the current study was to estimate the minimal clinically important difference (MCID) of sentence intelligibility in control speakers and in speakers with dysarthria due to multiple sclerosis (MS) and Parkinson's disease (PD). Method: Sixteen control speakers, 16 speakers with MS, and 16 speakers with PD were audio-recorded reading aloud sentences in habitual, clear, fast, loud, and slow speaking conditions. Two hundred forty nonexpert crowdsourced listeners heard paired conditions of the same sentence content from a speaker and indicated if one condition was more understandable than another. Listeners then used the Global Ratings of Change (GROC) Scale to indicate how much more understandable that condition was than the other. Listener ratings were compared with objective intelligibility scores obtained previously via orthographic transcriptions from nonexpert listeners. Receiver operating characteristic (ROC) curves and average magnitude of intelligibility difference per level of the GROC Scale were evaluated to determine the sensitivity, specificity, and accuracy of potential cutoff scores in intelligibility for establishing thresholds of important change. Results: MCIDs derived from the ROC curves were invalid. However, the average magnitude of intelligibility difference derived valid and useful thresholds. The MCID of intelligibility was determined to be about 7% for a small amount of difference and about 15% for a large amount of difference. Conclusions: This work demonstrates the feasibility of the novel experimental paradigm for collecting crowdsourced perceptual data to estimate MCIDs. Results provide empirical evidence that clinical tools for the perception of intelligibility by nonexpert listeners could consist of three categories, which emerged from the data (“no difference,” “a little bit of difference,” “a lot of difference”). The current work is a critical step toward development of a universal language with which to evaluate changes in intelligibility as a result of neurological injury, disease progression, and speech-language therapy.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The mean within-group improvement in the control group was 7.17 points, corresponding to the lower threshold of the minimal clinically important difference range.
“The mean within-group improvement in the control group was 7.17 points, corresponding to the lower threshold of the minimal clinically important difference range [41].”
From Efficacy of Digital Speech Therapy for Poststroke Dysarthria: Randomized Noninferiority Trial (2026) · cites this paper as [41]
The mean improvement in the workbook-based control arm aligns with the lower bound of the cited minimal clinically important difference range.
“which aligns with the lower bound of the cited minimal clinically important difference range [41]”
From Efficacy of Digital Speech Therapy for Poststroke Dysarthria: Randomized Noninferiority Trial (2026) · cites this paper as [41]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 29,639 · see citing works
<b>lmerTest</b> Package: Tests in Linear Mixed Effects Models
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017) · Journal of Statistical Software · doi:10.18637/jss.v082.i13 · Cited by 24,346 · see citing works
Hosmer, D. W., & Lemeshow, S. (2000) · doi:10.1002/0471722146 · Cited by 20,302 · see citing works
Jaeschke, R., Singer, J., & Guyatt, G. H. (1989) · Controlled Clinical Trials · doi:10.1016/0197-2456(89)90005-6 · Cited by 4,982 · see citing works
Prolific.ac—A subject pool for online experiments
Palan, S., & Schitter, C. (2018) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2017.12.004 · Cited by 3,450 · see citing works
Cedarbaum, J. M., Stambler, N., Malta, E., Fuller, C., Hilt, D., Thurmond, B., & Nakanishi, A. (1999) · Journal of the Neurological Sciences · doi:10.1016/s0022-510x(99)00210-5 · Cited by 3,307 · see citing works
jsPsych: A JavaScript library for creating behavioral experiments in a Web browser
de Leeuw, J. R. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0458-y · Cited by 1,783 · see citing works
Understanding the minimum clinically important difference: A review of concepts and methods
Copay, A. G., Subach, B. R., Glassman, S. D., Polly, D. W., & Schuler, T. C. (2007) · The Spine Journal · doi:10.1016/j.spinee.2007.01.008 · Cited by 1,662 · see citing works
Development and Validation of the Voice Handicap Index‐10
Rosen, C. A., Lee, A. S., Osborne, J. C., Zullo, T. G., & Murry, T. (2004) · The Laryngoscope · doi:10.1097/00005537-200409000-00009 · Cited by 1,414 · see citing works
IEEE Recommended Practice for Speech Quality Measurements
(1969) · IEEE Transactions on Audio and Electroacoustics · doi:10.1109/tau.1969.1162058 · Cited by 1,233 · see citing works
Chen, A., Frankowski, R. F., Bishop‐Leone, J. K., Hébert, T. M., Leyk, S., Lewin, J. S., & Goepfert, H. (2001) · PubMed · Cited by 940 · see citing works
Jacobson, N. S., Roberts, L. J., Berns, S. B., & McGlinchey, J. B. (1999) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.67.3.300 · Cited by 853 · see citing works
Assessment and prognosis of coma after head injury
Teasdale, G. M., & Jennett, B. (1976) · Acta Neurochirurgica · doi:10.1007/bf01405862 · Cited by 824 · see citing works
ROC-ing along: Evaluation and interpretation of receiver operating characteristic curves
Carter, J. V., Pan, J., Rai, S. N., & Galandiuk, S. (2016) · Surgery · doi:10.1016/j.surg.2015.12.029 · Cited by 755 · see citing works
Minimal Clinically Important Difference
McGlothlin, A. E., & Lewis, R. J. (2014) · JAMA · doi:10.1001/jama.2014.13128 · Cited by 709 · see citing works
Beaton, D., Boers, M., & Wells, G. (2002) · Current Opinion in Rheumatology · doi:10.1097/00002281-200203000-00006 · Cited by 547 · see citing works
Tilson, J. K., Sullivan, K. J., Cen, S., Rose, D. K., Koradia, C. H., Azen, S. P., Duncan, P. W., & for the Locomotor Experience Applied Post Stroke (LEAPS) Investigative Team (2009) · Physical Therapy · doi:10.2522/ptj.20090079 · Cited by 501 · see citing works
The Concept of Clinically Meaningful Difference in Health-Related Quality of Life Research
Hays, R. D., & Woolley, J. M. (2000) · PharmacoEconomics · doi:10.2165/00019053-200018050-00001 · Cited by 491 · see citing works
The minimal detectable change cannot reliably replace the minimal important difference
Turner, D., Schünemann, H. J., Griffith, L. E., Beaton, D., Griffiths, A. M., Critch, J., & Guyatt, G. H. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.01.024 · Cited by 352 · see citing works
Minimal clinically important difference for grip strength: a systematic review
Bohannon, R. W. (2019) · Journal of Physical Therapy Science · doi:10.1589/jpts.31.75 · Cited by 312 · see citing works
Responsiveness of the Neck Disability Index in patients with mechanical neck disorders
Young, B. A., Walker, M. J., Strunce, J. B., Boyles, R. E., Whitman, J. M., & Childs, J. D. (2009) · The Spine Journal · doi:10.1016/j.spinee.2009.06.002 · Cited by 284 · see citing works
Building Experiments with PsychoPy
Wang, Z. (2021) · Springer eBooks · doi:10.1007/978-3-030-82635-2_2 · Cited by 227 · see citing works
Yorkston, K. M., & Beukelman, D. R. (1981) · Journal of Speech and Hearing Disorders · doi:10.1044/jshd.4603.296 · Cited by 202 · see citing works
Measuring up to speech intelligibility
Miller, N. (2013) · International Journal of Language & Communication Disorders · doi:10.1111/1460-6984.12061 · Cited by 197 · see citing works
Tjaden, K., Sussman, J. E., & Wilding, G. E. (2014) · Journal of Speech Language and Hearing Research · doi:10.1044/2014_jslhr-s-12-0372 · Cited by 182 · see citing works
Self-citationMinimal Clinically Important Difference of Shoulder Outcome Measures and Diagnoses
Dabija, D. I., & Jain, N. B. (2019) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0000000000001169 · Cited by 170 · see citing works
Hutcheson, K. A., Barrow, M. P., Lisec, A., Barringer, D. A., Gries, K., & Lewin, J. S. (2015) · The Laryngoscope · doi:10.1002/lary.25778 · Cited by 167 · see citing works
Furlan, L., & Sterr, A. (2018) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2018.00095 · Cited by 162 · see citing works
Copay, A. G., Eyberg, B. A., Chung, A. S., Zurcher, K. S., Chutkan, N. B., & Spangehl, M. J. (2018) · JBJS Reviews · doi:10.2106/jbjs.rvw.17.00160 · Cited by 148 · see citing works
1. Scaling procedures for the measurement of speech intelligibility
Schiavetti, N. (1992) · Studies in speech pathology and clinical linguistics · doi:10.1075/sspcl.1.02sch · Cited by 115 · see citing works
Sussman, J. E., & Tjaden, K. (2012) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2011/11-0048 · Cited by 106 · see citing works
Self-citationMinimal Clinically Important Difference
Engel, L., Beaton, D., & Touma, Z. (2018) · Rheumatic Disease Clinics of North America · doi:10.1016/j.rdc.2018.01.011 · Cited by 99 · see citing works
Stipancic, K. L., Tjaden, K., & Wilding, G. E. (2015) · Journal of Speech Language and Hearing Research · doi:10.1044/2015_jslhr-s-15-0271 · Cited by 96 · see citing works
Self-citationOnline crowdsourcing for efficient rating of speech: A validation study
Byun, T. M., Halpin, P. F., & Szeredi, D. (2014) · Journal of Communication Disorders · doi:10.1016/j.jcomdis.2014.11.003 · Cited by 95 · see citing works
Construct Validity of Direct Magnitude Estimation and Interval Scaling of Speech Intelligibility
Schiavetti, N., Metz, D. E., & Sitler, R. W. (1981) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.2403.441 · Cited by 88 · see citing works
Effect of Rate Control on Speech Production and Intelligibility in Dysarthria
Nuffelen, G. V., Bodt, M. S. D., Vanderwegen, J., et al. (2010) · Folia Phoniatrica et Logopaedica · doi:10.1159/000287209 · Cited by 87 · see citing works
Minimal Clinically Important Difference of the Functional Gait Assessment in Older Adults
Beninato, M., Fernandes, A., & Plummer, L. S. (2014) · Physical Therapy · doi:10.2522/ptj.20130596 · Cited by 86 · see citing works
Minimal clinically important difference of voice handicap index‐10 in vocal fold paralysis
Young, V. N., Jeong, K., Rothenberger, S. D., Gillespie, A. I., Smith, L. J., Gartner‐Schmidt, J. L., & Rosen, C. A. (2017) · The Laryngoscope · doi:10.1002/lary.27001 · Cited by 83 · see citing works
Stipancic, K. L., Yunusova, Y., Berry, J. D., & Green, J. R. (2018) · Journal of Speech Language and Hearing Research · doi:10.1044/2018_jslhr-s-17-0366 · Cited by 83 · see citing works
Self-citationStratford, P. W., & Riddle, D. L. (2012) · Physical Therapy · doi:10.2522/ptj.20120002 · Cited by 78 · see citing works
Pattern of spread and prognosis in lower limb-onset ALS
Turner, M. R., Brockington, A., Scaber, J., Hollinger, H. C., Marsden, R., Shaw, P. J., & Talbot, K. (2009) · Amyotrophic Lateral Sclerosis · doi:10.3109/17482960903420140 · Cited by 70 · see citing works
Hustad, K. C. (2006) · American Journal of Speech-Language Pathology · doi:10.1044/1058-0360(2006/025 · Cited by 66 · see citing works
Influence of Cognitive Function on Speech and Articulation Rate in Multiple Sclerosis
Rodgers, J. D., Tjaden, K., Feenaughty, L., Weinstock‐Guttman, B., & Benedict, R. H. B. (2012) · Journal of the International Neuropsychological Society · doi:10.1017/s1355617712001166 · Cited by 65 · see citing works
Self-citationReliability and validity of speech & pause measures during passage reading in ALS
Barnett, C., Green, J. R., Marzouqah, R., Stipancic, K. L., Berry, J. D., Korngut, L. W., Genge, A. L., Shoesmith, C. L., Briemberg, H., Abrahão, A., et al. (2019) · Amyotrophic Lateral Sclerosis and Frontotemporal Degeneration · doi:10.1080/21678421.2019.1697888 · Cited by 60 · see citing works
Self-citationMinimal Clinically Important Difference
Gatchel, R. J., Lurie, J. D., & Mayer, T. G. (2010) · Spine · doi:10.1097/brs.0b013e3181d3cfc9 · Cited by 58 · see citing works
Abur, D., Enos, N. M., & Stepp, C. E. (2019) · American Journal of Speech-Language Pathology · doi:10.1044/2019_ajslp-18-0275 · Cited by 39 · see citing works
Gutz, S. E., Stipancic, K. L., Yunusova, Y., Berry, J. D., & Green, J. R. (2022) · Journal of Speech Language and Hearing Research · doi:10.1044/2022_jslhr-21-00589 · Cited by 33 · see citing works
Self-citationStipancic, K. L., & Tjaden, K. (2022) · Journal of Speech Language and Hearing Research · doi:10.1044/2022_jslhr-21-00648 · Cited by 32 · see citing works
Self-citationIs This Change Real? Interpreting Patient Outcomes in Physical Therapy.
Kozlowski, A. J. (2014) · PubMed Central · doi:10.3138/ptc.66.4.rev1 · Cited by 31 · see citing works
Marks, K. L., Verdi, A., Toles, L. E., Stipancic, K. L., Ortiz, A. J., Hillman, R. E., & Mehta, D. D. (2021) · American Journal of Speech-Language Pathology · doi:10.1044/2021_ajslp-21-00111 · Cited by 28 · see citing works
Self-citationIs This Change Real?: Interpreting Patient Outcomes in Physical Therapy
Riddle, D. L., & Stratford, P. W. (2013) · Medical Entomology and Zoology · Cited by 27 · see citing works
Okano, I., Miller, C. O., Salzmann, S. N., Hoshino, Y., Shue, J., Sama, A. A., Cammisa, F. P., Girardi, F. P., & Hughes, A. P. (2020) · Clinical Orthopaedics and Related Research · doi:10.1097/corr.0000000000001236 · Cited by 21 · see citing works
Mallinson, T., Pape, T., & Guernon, A. (2015) · Journal of Head Trauma Rehabilitation · doi:10.1097/htr.0000000000000184 · Cited by 20 · see citing works
Intelligibility Across a Reading Passage: The Effect of Dysarthria and Cued Speaking Styles
van Brenk, F., Stipancic, K. L., Kain, A., & Tjaden, K. (2022) · American Journal of Speech-Language Pathology · doi:10.1044/2021_ajslp-21-00151 · Cited by 20 · see citing works
Self-citationInternally Versus Externally Cued Speech in Parkinson's Disease and Cerebellar Disease
Weir-Mayta, P., Spencer, K. A., Eadie, T. L., Yorkston, K. M., Savaglio, S., & Woollcott, C. (2017) · American Journal of Speech-Language Pathology · doi:10.1044/2017_ajslp-16-0109 · Cited by 18 · see citing works
Stipancic, K. L., Golzy, M., Zhao, Y., et al. (2023) · Journal of Speech Language and Hearing Research · doi:10.1044/2023_jslhr-23-00322 · Cited by 14 · see citing works
Self-citationShin, H.-B., Shivabasappa, P., & Koul, R. (2021) · International Journal of Speech-Language Pathology · doi:10.1080/17549507.2021.1943522 · Cited by 10 · see citing works
Stipancic, K. L., Wilding, G. E., & Tjaden, K. (2023) · Journal of Speech Language and Hearing Research · doi:10.1044/2023_jslhr-22-00279 · Cited by 10 · see citing works
Self-citationHustad, K. C., & Borrie, S. A. (2021) · doi:10.1002/9781119606987.ch4 · Cited by 6 · see citing works
Darling-White, M., & Polkowitz, R. (2023) · American Journal of Speech-Language Pathology · doi:10.1044/2023_ajslp-23-00093 · Cited by 5 · see citing works
Tipping the Scales: Indiscriminate Use of Interval Scales to Rate Diverse Dysarthric Features
Stipancic, K. L., Whelan, B., Laur, L., et al. (2024) · Journal of Speech Language and Hearing Research · doi:10.1044/2024_jslhr-23-00785 · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 4
- Beukelman, D. R., Fager, S., Ullman, C., Hanson, E., & Logemann, J. (2002). The impact of speech supplementation and clear speech on the intelligibility and speaking rate of people with traumatic brain injury. Journal of Medical Speech-Language Pathology, 10(4), 237-242.
- Chen, A., Frankowski, R., Bishop-Leone, J., Hebert, T., Leyk, S., Lewin, J., & Goepfert, H. (2001). The development and vali- dation of a dysphagia-specific quality-of-life questionnaire for patients with head and neck cancer. Archives of Otolaryngology- Head & Neck Surgery, 127(7), 870-876.
- 0b013e3181d3cfc9 · DOI
- Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., Markus, M., Siegert, S., Doering, M., & Billings, Z. (2023). Package "pROC."
Cited by
Sorted by citations
Efficacy of Digital Speech Therapy for Poststroke Dysarthria: Randomized Noninferiority Trial
Kim, Y., Kim, M., Kim, S., Kim, J., Shin, J.-H., Chang, Y., Na, J. Y., Kim, J., & Song, T.-J. (2026) · Journal of Medical Internet Research · doi:10.2196/81938
“In the present sample, the mean improvement in the workbook-based control arm was 7.17 points, which aligns with the lower bound of the cited minimal clinically important difference range [41]; this should be considered when interpreting the strength of the noninferiority inference [44].”
Influential
- Authors
- Stipancic, K. L., van Brenk, F., Qiu, M., & Tjaden, K.
- Affiliations
- University at Buffalo, State University of New York (US)
- Department of Communicative Disorders and Sciences, University at Buffalo, New York
- Trent University (CA)
- Venue
- Journal of Speech, Language, and Hearing Research
- Published
- 2025-07-29
- Volume
- 68
- Issue
- 7S
- Pages
- 3480-3494
- ISSN
- 1092-4388 (ISSN-L)
- 1558-9102
- PMC
- PMC12337110
- OpenAlex
- W4403761734
- Cited by
- 11
- References
- 63
- Citations by year
- 2026: 8 · 2025: 3
- FWCI
- 3.11
- Type
- journal article
- Language
- en
- Open access
- green
- pmc.ncbi.nlm.nih.gov
- Locations
- Journal of Speech Language and Hearing Research (publishedVersion)
- PubMed Central (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institute on Deafness and Other Communication Disorders (R01DC004689)
- Topics
- Health Sciences › Medicine › Physiology › Voice and Speech Disorders
- Life Sciences › Neuroscience › Cognitive Neuroscience › Hearing Loss and Rehabilitation
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Keywords
- minimal clinically important difference, sentence intelligibility, dysarthria, multiple sclerosis, Parkinson's disease, speech rate, loudness
- MeSH terms
- Minimal Clinically Important Difference, Adult, Aged, Dysarthria, Female, Humans, Male, Middle Aged, Multiple Sclerosis, Parkinson Disease, ROC Curve, Speech Intelligibility, Speech Perception, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
