Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 77 of the 78 references it cites were found in a registry or an index; 25 of the 87 sources its claims cite were checked for support.

Equipping Speech-Language Clinicians for the Critical Appraisal of an Artificial Intelligence–Driven, Evidence-Based Future

Open access
Benway, Nina R., Preston, Jonathan L. · 2025 · Language, Speech, and Hearing Services in Schools

Abstract

Purpose: Artificial intelligence (AI) is more capable and accessible than ever before. But what does this mean for clinical practice? How can speech-language clinicians evaluate the efficacy, validity, and reliability of AI and machine learning tools for automating assessment and treatment? How can speech-language clinicians ethically use these clinical AI technologies? We contend that clinical AI will best serve clinicians and clients when aligned with an evidence-based framework. Therefore, this tutorial presents guidelines for the critical appraisal of clinical AI through the lens of validity, reliability, ethical use, and equitable use, facilitated by the Critical Appraisal Rubric for Ethical and Equitable Clinical Artificial Intelligence. Similarly, in order for developers of clinical AI to meet the needs of the profession, these principles should guide the development and assessment of new clinical technologies. Conclusions: The questions of efficacy, validity, reliability, ethical use, and equitable use of clinical AI can be answered through the examination of a specific clinical AI for a given user, as emphasized by culturally responsive professional practice. A framework is provided to assist clinicians in the critical appraisal of clinical AI tools.

Overview

CoChat Summary

TL;DR

This invited tutorial argues that speech-language clinicians already possess the critical-appraisal skills needed to evaluate clinical artificial intelligence (AI), because the same reasoning used to judge standardized psychometric tests can be applied to AI tools. It introduces the CARE-AI framework (Appendix A) to help clinicians document evidence of validity, reliability, and ethical/equitable use of clinical AI within an evidence-based practice (EBP) framework.

Key findings

  • The paper contends that clinical AI should be appraised through two questions: "Is there evidence that a given tool is valid and reliable for clinical use?" and "Is there evidence that a given tool is designed to work without bias in real clinical populations with speakers it has never encountered before?"
  • Data scarcity is described as a primary barrier to clinical AI development; the authors' Figure 1 illustrates how training on insufficient data produces overfit classifiers with low external validity that misclassify never-before-seen speakers.
  • The authors cite that 86% of speech sound therapy systems available in app stores do not show potential for therapeutic benefit (Furlong et al., 2018), and that the overall methodological quality of research on computerized speech and language interventions is moderate to low (Y.-P. P. Chen et al., 2016; Furlong et al., 2017; Simpson et al., 2015).
  • For interpreting classifier performance, the paper cites Plante and Vance (1994): discriminant accuracy above .8 at a given cut-score shows "fair" evidence of clinical utility and above .9 shows "good" evidence; McKechnie et al. (2018) argued clinical utility is demonstrated at ≥80% agreement with human raters on incorrect sounds (specificity above .8).
  • The authors state that typical data splits are 70% training, 15% validation, and 15% test, and that test sets should be speaker-disjoint from training/validation sets.

Methods

  • Design: an invited tutorial/conceptual paper (based on material presented during a seminar receiving the Changemaker Award at the 2023 ASHA Convention), not an empirical study.
  • Data: the Data availability statement says "Data sharing is not applicable to this article as no data sets were generated or analyzed in this tutorial."
  • Content: introduces AI/machine learning terminology (supervised learning, forward pass/backpropagation, classification, regression, automatic speech recognition), reviews EBP and clinical AI, and presents the CARE-AI rubric (Appendix A), which is also published with a CC BY-NC-ND 4.0 license at https://osf.io/nqzd9/.

Limitations

  • Stated: the authors note that developing a well-vetted, standardized quality appraisal instrument that evaluates clinical AI research and EBP is a "long-term goal of this line of research," and that the CARE-AI framework is "a first effort toward this goal."
  • Stated: the authors note that clinical AI is undergoing rapid development that will likely impact professional best practices, and that ongoing development and research is required to understand likely future scenarios (e.g., whether effective clinical AI reduces or increases clinician burden).
  • Evident from the text: the CARE-AI framework is presented as preliminary guidance rather than a validated instrument, and the paper reports no primary data or empirical validation of the framework itself; the accompanying glossary was generated with NotebookLM and reviewed by the authors for accuracy.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1044/2025_lshss-24-00085

Cites

77 of 78 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 An illustrative example showing how training data scarcity (shown in light orange) might lead to overfit machine learning with low external validity. Panel A shows the decision boundary that would be placed if all data were available at the time of ing, correctly classifying the orange star as a member of the orange group. Panel B shows that, even with data scarcity i orange group, the classifier will still appear to perform well at the time of training relative to the data available. However, Panel when the starred sample is considered relative to this new decision boundary, the starred point is assigned to the incorrect gro the machine learning algorithm was not trained on enough orange data to accurately predict its label.
Figure 1 An illustrative example showing how training data scarcity (shown in light orange) might lead to overfit machine learning with low external validity. Panel A shows the decision boundary that would be placed if all data were available at the time of ing, correctly classifying the orange star as a member of the orange group. Panel B shows that, even with data scarcity i orange group, the classifier will still appear to perform well at the time of training relative to the data available. However, Panel when the starred sample is considered relative to this new decision boundary, the starred point is assigned to the incorrect gro the machine learning algorithm was not trained on enough orange data to accurately predict its label.CC-BY-NC-ND · From the PDF
Figure 3 An illustrative example of the relationship between validity and reliability. In this example, the bullseye locations represent an underlying speech or language construct to be measured, and the locations of the stars represent the measurement attempts.
Figure 3 An illustrative example of the relationship between validity and reliability. In this example, the bullseye locations represent an underlying speech or language construct to be measured, and the locations of the stars represent the measurement attempts.CC-BY-NC-ND · From the PDF
Figure 4 Classification performance metrics such as balanced accuracy and F1 score are different ways to represent a confusio which summarizes true positives, true negatives, false positives, and false negatives.
Figure 4 Classification performance metrics such as balanced accuracy and F1 score are different ways to represent a confusio which summarizes true positives, true negatives, false positives, and false negatives.CC-BY-NC-ND · From the PDF
Figure 5 An illustrative example showing how ethical and equitable clinical AI performs without bias across different linguistic a graphic groups. Panel A shows that an unbiased tool will have measurements (blue stars) that are equally valid (centered on t and reliable (tightly clustered) across demographic and linguistic groups; in other words, the measurements attempts are aligned w community standard for the speech or language construct to be measured (location of the bullseye). Panel B shows that this tool is only valid and reliable to measure a speech or language construct from one particular community and does not capture nity standard for the speech and language construct relevant to other communities. AI = artificial intelligence.
Figure 5 An illustrative example showing how ethical and equitable clinical AI performs without bias across different linguistic a graphic groups. Panel A shows that an unbiased tool will have measurements (blue stars) that are equally valid (centered on t and reliable (tightly clustered) across demographic and linguistic groups; in other words, the measurements attempts are aligned w community standard for the speech or language construct to be measured (location of the bullseye). Panel B shows that this tool is only valid and reliable to measure a speech or language construct from one particular community and does not capture nity standard for the speech and language construct relevant to other communities. AI = artificial intelligence.CC-BY-NC-ND · From the PDF

Verification

Claims the paper makes

20 Valid · 4 Not valid · 37 Not checked · 77 Own results and interpretations

51 more claims · 7 with their cited sources checked

Create a free account to see all 138 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 26
  • Probably not a substantial claim. 71

Create a free account to see these claims.

Citations

Cites

Show all 78 references

Research works printed but not linked: 7

  1. Casilli, A. A. (2024). ‘End-to-end’ ethical AI. Taking into account the social and natural environments of automation. In A. Ponce del Castillo (Ed.), Artificial intelligence, labour and society (pp. 83–92). ETUI.
  2. Holliday N. (2023b). A sociolinguistic perspective on speech technology [Survey talk]. Annual Conference of the International Speech Communication Association (INTERSPEECH) Dublin Ireland.
  3. International Organization for Standardization. (2022). Information technology—Artificial intelligence. In Artificial intelligence concepts and terminology (Vol. 22989 p. 60). https://www.iso.org/standard/74296.html
  4. Lundberg S. & Lee S.-I. (2017). A unified approach to interpreting model predictions. arXiv. https://doi.org/10.48550/arXiv.1705.07874Focus to learn more · arXiv · doi:10.48550/arxiv.1705.07874focus
  5. Nelson, N. W., Plante, E., Helm-Estabrooks, N., & Hotz, G. (2016). Test of Integrated Language and Literacy Skills (TILLS). Brookes
  6. Preston, J. L., Benway, N. R., Prestopnik, N., & Preston, N. (2024). The speech motor chaining web app for speech motor learning. Proceedings of Interspeech, 5208–5209.
  7. Richardson, R., Schultz, J. M., & Crawford, K. (2019). Dirty data, bad predictions: How civil rights violations impact police data, predictive policing systems, and justice. NYU Law Review Online, 94(15), 15–55. https://nyulawreview.org/online-features/dirty-data-bad-predictions-how-civil-rights-violations-impact-police-data-predictive-policing-systems-and-justice/

Cited by

See all 12

Metadata

Authors
Benway, N. R., & Preston, J. L.
Affiliations
  • University of Maryland, College Park (US)
  • Syracuse University (US)
Venue
Language, Speech, and Hearing Services in Schools
Published
2025-07-16
Volume
56
Issue
3
Pages
442-468
ISSN
  • 0161-1461 (ISSN-L)
  • 1558-9129
OpenAlex
W4410388726
Cited by
12
References
78
Citations by year
2026: 11 · 2025: 1
FWCI
2.46
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • Language Speech and Hearing Services in Schools (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
Funders
  • National Institutes of Health
  • National Institute on Deafness and Other Communication Disorders (R01DC020959, T32DC000046)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
clinical AI, critical appraisal, validity, reliability statistics, ethical use of AI, machine learning tools, culturally responsive practice, evidence-based practice
MeSH terms
Artificial Intelligence, Humans, Speech-Language Pathology, Reproducibility of Results, Evidence-Based Practice

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Equipping Speech-Language Clinicians for the Critical Appraisal of an Artificial Intelligence–Driven, Evidence-Based Future. https://app.cochat.ai/papers/10.1044/2025_lshss-24-00085
The paper
Benway, N. R., & Preston, J. L. (2025). Equipping Speech-Language Clinicians for the Critical Appraisal of an Artificial Intelligence–Driven, Evidence-Based Future. Language, Speech, and Hearing Services in Schools. https://doi.org/10.1044/2025_lshss-24-00085

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.