The DOI matches its Crossref record; 77 of the 78 references it cites were found in a registry or an index; 25 of the 87 sources its claims cite were checked for support.
Equipping Speech-Language Clinicians for the Critical Appraisal of an Artificial Intelligence–Driven, Evidence-Based Future
Open accessAbstract
Purpose: Artificial intelligence (AI) is more capable and accessible than ever before. But what does this mean for clinical practice? How can speech-language clinicians evaluate the efficacy, validity, and reliability of AI and machine learning tools for automating assessment and treatment? How can speech-language clinicians ethically use these clinical AI technologies? We contend that clinical AI will best serve clinicians and clients when aligned with an evidence-based framework. Therefore, this tutorial presents guidelines for the critical appraisal of clinical AI through the lens of validity, reliability, ethical use, and equitable use, facilitated by the Critical Appraisal Rubric for Ethical and Equitable Clinical Artificial Intelligence. Similarly, in order for developers of clinical AI to meet the needs of the profession, these principles should guide the development and assessment of new clinical technologies. Conclusions: The questions of efficacy, validity, reliability, ethical use, and equitable use of clinical AI can be answered through the examination of a specific clinical AI for a given user, as emphasized by culturally responsive professional practice. A framework is provided to assist clinicians in the critical appraisal of clinical AI tools.
Overview
CoChat Summary
TL;DR
This invited tutorial argues that speech-language clinicians already possess the critical-appraisal skills needed to evaluate clinical artificial intelligence (AI), because the same reasoning used to judge standardized psychometric tests can be applied to AI tools. It introduces the CARE-AI framework (Appendix A) to help clinicians document evidence of validity, reliability, and ethical/equitable use of clinical AI within an evidence-based practice (EBP) framework.
Key findings
- The paper contends that clinical AI should be appraised through two questions: "Is there evidence that a given tool is valid and reliable for clinical use?" and "Is there evidence that a given tool is designed to work without bias in real clinical populations with speakers it has never encountered before?"
- Data scarcity is described as a primary barrier to clinical AI development; the authors' Figure 1 illustrates how training on insufficient data produces overfit classifiers with low external validity that misclassify never-before-seen speakers.
- The authors cite that 86% of speech sound therapy systems available in app stores do not show potential for therapeutic benefit (Furlong et al., 2018), and that the overall methodological quality of research on computerized speech and language interventions is moderate to low (Y.-P. P. Chen et al., 2016; Furlong et al., 2017; Simpson et al., 2015).
- For interpreting classifier performance, the paper cites Plante and Vance (1994): discriminant accuracy above .8 at a given cut-score shows "fair" evidence of clinical utility and above .9 shows "good" evidence; McKechnie et al. (2018) argued clinical utility is demonstrated at ≥80% agreement with human raters on incorrect sounds (specificity above .8).
- The authors state that typical data splits are 70% training, 15% validation, and 15% test, and that test sets should be speaker-disjoint from training/validation sets.
Methods
- Design: an invited tutorial/conceptual paper (based on material presented during a seminar receiving the Changemaker Award at the 2023 ASHA Convention), not an empirical study.
- Data: the Data availability statement says "Data sharing is not applicable to this article as no data sets were generated or analyzed in this tutorial."
- Content: introduces AI/machine learning terminology (supervised learning, forward pass/backpropagation, classification, regression, automatic speech recognition), reviews EBP and clinical AI, and presents the CARE-AI rubric (Appendix A), which is also published with a CC BY-NC-ND 4.0 license at https://osf.io/nqzd9/.
Limitations
- Stated: the authors note that developing a well-vetted, standardized quality appraisal instrument that evaluates clinical AI research and EBP is a "long-term goal of this line of research," and that the CARE-AI framework is "a first effort toward this goal."
- Stated: the authors note that clinical AI is undergoing rapid development that will likely impact professional best practices, and that ongoing development and research is required to understand likely future scenarios (e.g., whether effective clinical AI reduces or increases clinician burden).
- Evident from the text: the CARE-AI framework is presented as preliminary guidance rather than a validated instrument, and the paper reports no primary data or empirical validation of the framework itself; the accompanying glossary was generated with NotebookLM and reviewed by the authors for accuracy.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
20 Valid · 4 Not valid · 37 Not checked · 77 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, and perhaps most critically, there has not been enough publicly available audio or video examples of speech and language disorders to develop AI systems that work well for children or adults with these types of disorders (e.g., Benway et al., 2023;Berisha et al., 2021;Shahin et al., 2020).”
§ Equipping Speech-Language Clinicians for the Critical Appraisal of an Artificial Intelligence-Driven, Evidence-Based Future > Abstract
What the sources say
- Partly supportedcites Benway et al., 2023 sourceExperimental — accuracy not yet measured
The Speech Technology and Ultrasuite section says large, labeled speech datasets for children with speech sound disorders are rare and that existing child mispronunciation-detection systems fall short of clinically acceptable accuracy. It does not establish the same shortage or performance limitation for adults or for language disorders, nor does it specifically address a lack of video examples. (LLM, conf=0.88)
- Partly supportedcites Berisha et al., 2021 sourceExperimental — accuracy not yet measured
The Introduction says clinical speech databases are often small, with tens or hundreds of patients and only a few minutes of speech per person, making speech-based clinical AI challenging. The cited paper does not establish that publicly available audio or video examples of speech and language disorders are insufficient, or specifically assess AI performance for children and adults with those disorders. (LLM, conf=0.86)
- Not checked — no full text of the sourcecites Shahin et al., 2020 source
states a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The more complex the association between the input and the output, the more data are needed to avoid overfitting and ensure a tool will work well in the future (Berisha et al., 2021), which highlights the importance of machine learning experimental design for data-scarce clinical AI (Ghasemzadeh et al., 2024).”
§ Equipping Speech-Language Clinicians for the Critical Appraisal of an Artificial Intelligence-Driven, Evidence-Based Future > Examining the External Validity and Generalization Potential of Machine Learning
What the sources say
- Supportedcites Berisha et al., 2021 source
In “The curse of dimensionality and dataset blind spots,” the paper says the sample size needed to train a model and estimate its performance on new data scales with model complexity and the difficulty of the problem. Its “Model validation” section also links small samples to overfitting and overoptimistic performance estimates. Page numbers were not provided in our copy. (LLM, conf=0.91)
- Supportedcites Ghasemzadeh et al., 2024 source
The Ghasemzadeh citation supports the sentence’s point about experimental design for data-scarce clinical AI; the preceding claim about input–output complexity is cited to Berisha et al., not Ghasemzadeh. In “The Problem of Overfitting in ML,” the paper says speech, language, and hearing studies often have small samples that are especially susceptible to overfitting and require careful cross-validation. Its Discussion also describes the impact of cross-validation choices on models trained with clinical data. (LLM, conf=0.92)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“even expert clinicians cannot reliably do so (Murray et al., 2024).”
§ Equipping Speech-Language Clinicians for the Critical Appraisal of an Artificial Intelligence-Driven, Evidence-Based Future > AI Clinicians and AI-Assisted Clinical Practice
What the sources say
- Could not verifycites Murray et al., 2024 sourceExperimental — accuracy not yet measured
The Method (Participant selection) describes children without a documented speech sound disorder and an expert’s ratings of voice severity. The Discussion (Limitations and future directions) addresses dysphonia assessment, not childhood apraxia of speech. Searches for apraxia diagnosis and clinician diagnostic reliability found no passage addressing the claim. (LLM, conf=0.77)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“Also, the overwhelming majority of (speech sound therapy) systems available in app stores do not show potential for therapeutic benefit (86%; Furlong et al., 2018).”
§ Equipping Speech-Language Clinicians for the Critical Appraisal of an Artificial Intelligence-Driven, Evidence-Based Future > Abstract
What the sources say
- Supportedcites Furlong et al., 2018 source
The Results (Descriptive data for the reviewed apps; page unavailable) report that 19 of the 132 reviewed apps—14%—were considered to have potential therapeutic benefit. The remaining 86% were not. This percentage applies to the reviewed apps, not every app available in the stores. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Another point is that AI clinicians could not be expected to reliably diagnose conditions like, for example, childhood apraxia of speech, because even expert clinicians cannot reliably do so (Murray et al., 2024).”
§ Equipping Speech-Language Clinicians for the Critical Appraisal of an Artificial Intelligence-Driven, Evidence-Based Future > AI Clinicians and AI-Assisted Clinical Practice
What the sources say
- Could not verifycites Murray et al., 2024 sourceExperimental — accuracy not yet measured
The Method (Participant selection; page unavailable) describes a sample of children without documented speech sound disorders, and the Discussion (Limitations and future directions; page unavailable) addresses limitations of voice measurements. Searches for childhood apraxia of speech, clinician diagnostic reliability, and AI diagnosis found no passage addressing the cited claim. (LLM, conf=0.76)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
51 more claims · 7 with their cited sources checked
Create a free account to see all 138 claims and every source verdict.
- It may state two separate claims as one. 26
- Probably not a substantial claim. 71
Citations
Cites
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
American Speech‐Language‐Hearing Association
Weiner, F. F., & Ketzenberger, K. E. (2008) · Encyclopedia of Special Education · doi:10.1002/9780470373699.speced0119 · Cited by 2,557
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369
Transparency and the black box problem: why we do not trust AI
von Eschenbach, W. J. (2021) · Philosophy & Technology · doi:10.1007/s13347-021-00477-0 · Cited by 786
Algorithmic fairness in artificial intelligence for medicine and healthcare
Chen, R. J., Wang, J. J., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Lu, M. Y., Sahai, S., & Mahmood, F. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01056-8 · Cited by 750
Research works printed but not linked: 7
- Casilli, A. A. (2024). ‘End-to-end’ ethical AI. Taking into account the social and natural environments of automation. In A. Ponce del Castillo (Ed.), Artificial intelligence, labour and society (pp. 83–92). ETUI.
- Holliday N. (2023b). A sociolinguistic perspective on speech technology [Survey talk]. Annual Conference of the International Speech Communication Association (INTERSPEECH) Dublin Ireland.
- International Organization for Standardization. (2022). Information technology—Artificial intelligence. In Artificial intelligence concepts and terminology (Vol. 22989 p. 60). https://www.iso.org/standard/74296.html
- Lundberg S. & Lee S.-I. (2017). A unified approach to interpreting model predictions. arXiv. https://doi.org/10.48550/arXiv.1705.07874Focus to learn more · arXiv · doi:10.48550/arxiv.1705.07874focus
- Nelson, N. W., Plante, E., Helm-Estabrooks, N., & Hotz, G. (2016). Test of Integrated Language and Literacy Skills (TILLS). Brookes
- Preston, J. L., Benway, N. R., Prestopnik, N., & Preston, N. (2024). The speech motor chaining web app for speech motor learning. Proceedings of Interspeech, 5208–5209.
- Richardson, R., Schultz, J. M., & Crawford, K. (2019). Dirty data, bad predictions: How civil rights violations impact police data, predictive policing systems, and justice. NYU Law Review Online, 94(15), 15–55. https://nyulawreview.org/online-features/dirty-data-bad-predictions-how-civil-rights-violations-impact-police-data-predictive-policing-systems-and-justice/
Cited by
Benway, N. R., Tabatabaee, S., Wang, D., Munson, B., Preston, J. L., & Espy-Wilson, C. (2026) · Journal of Speech Language and Hearing Research · doi:10.1044/2025_jslhr-25-00515 · Cited by 5
Self-citationBailey, D. J., Anjum, J., Park, H., Carpenter, E., Mamlekar, C. R., & Kurland, J. (2026) · American Journal of Speech-Language Pathology · doi:10.1044/2025_ajslp-25-00262 · Cited by 7
Introduction to the Forum: Changemakers Igniting Innovation
DeVeney, S. L., & Pelatti, C. Y. (2025) · Language Speech and Hearing Services in Schools · doi:10.1044/2025_lshss-25-00077
Saligram, C., Shrivastava, V., & Speights, M. L. (2026) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2026.1802703
Kashif, M., Fatima, A., Shah, M. S., Ahmad, D. S., & Maqbool, A. (2026) · Qlantic journal of social sciences. · doi:10.55737/qjss.vii-iv.26580
Metadata
- Authors
- Benway, N. R., & Preston, J. L.
- Affiliations
- University of Maryland, College Park (US)
- Syracuse University (US)
- Venue
- Language, Speech, and Hearing Services in Schools
- Published
- 2025-07-16
- Volume
- 56
- Issue
- 3
- Pages
- 442-468
- ISSN
- 0161-1461 (ISSN-L)
- 1558-9129
- OpenAlex
- W4410388726
- Semantic Scholar
- 2feb585e7592cc0777fcf726b721173b4bd727a0
- Cited by
- 12
- References
- 78
- Citations by year
- 2026: 11 · 2025: 1
- FWCI
- 2.46
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- Language Speech and Hearing Services in Schools (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health
- National Institute on Deafness and Other Communication Disorders (R01DC020959, T32DC000046)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- clinical AI, critical appraisal, validity, reliability statistics, ethical use of AI, machine learning tools, culturally responsive practice, evidence-based practice
- MeSH terms
- Artificial Intelligence, Humans, Speech-Language Pathology, Reproducibility of Results, Evidence-Based Practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Equipping Speech-Language Clinicians for the Critical Appraisal of an Artificial Intelligence–Driven, Evidence-Based Future. https://app.cochat.ai/papers/10.1044/2025_lshss-24-00085
- The paper
- Benway, N. R., & Preston, J. L. (2025). Equipping Speech-Language Clinicians for the Critical Appraisal of an Artificial Intelligence–Driven, Evidence-Based Future. Language, Speech, and Hearing Services in Schools. https://doi.org/10.1044/2025_lshss-24-00085
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.