Evaluating Fluency in Aphasia: Fluency Scales, Trichotomous Judgements, or Machine Learning
Open accessThe DOI matches its Crossref record; the references it cites are not checked yet.
Abstract
Background: Speech-language pathologists (SLPs) and other clinicians often use aphasia batteries, such as the Western Aphasia Battery-Revised (WAB-R), to evaluate both severity and classification of aphasia. However, the fluency scale on the WAB-R is not entirely objective and has been found to have less than ideal inter-rater reliability, due to variability in weighing the importance of one dimension (e.g. articulatory effort or grammaticality) over another. This limitation has implications for aphasia classification. The subjectivity might be mitigated through the implementation of machine learning to identify fluent and non-fluent speech. Aims: We hypothesized that two models consisting of convolutional and recurrent neural networks can be used to identify fluent and non-fluent aphasia as judged by SLPs, with greater reliability than use of the WAB-R fluency scale. Methods & Procedures: The training and testing dataset for the networks was collected from the public domain, and the validation dataset was collected from participants in post-stroke aphasia studies. We used Kappa scores to evaluate inter-rater reliability among SLPs, and between the networks and SLPs. Outcome and Results: Using public domain samples, the model for detecting non-fluent aphasia achieved high accuracy on the training dataset after 10 epochs (i.e., when algorithm scans the entire dataset) and 81% testing accuracy using public domain samples. The model for detecting fluent speech had high training accuracy and 83% testing. Across samples, using the WAB-R fluency scale, there was poor to perfect agreement among SLPs on the precise WAB-R fluency score, but substantial agreement on non-fluent (score 0-4) versus fluent (score of 5-9). The agreement between the model and the SLPs was moderate for identifying non-fluent speech and substantial fpr identifying fluent speech. When SLPs were asked to identify each sample as fluent, non-fluent, or mixed (without using the fluency scale), the agreement between SLPs was almost perfect (Kappa 0.94). The agreement between the SLPs' trichotomous judgement and the models was fair for detecting non-fluent speech and substantial for detecting fluent speech. Conclusions: Results indicate that neither the WAB-R fluency scale nor the machine learning algorithms were as useful (reliable and valid) as a simple trichotomous judgement of fluent, non-fluent, or mixed by SLPs. These results, together with data from the literature, indicate that it is time to re-consider use of the WAB-R fluency scale for classification of aphasia. It is also premature, at present, to rely on machine learning to rate spoken language fluency.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The analysis results showed that the machine learning algorithms used in post-stroke aphasia models were mainly supervised methods, including random forests, neural networks, and support vector machines.”
From Machine learning models in post-stroke aphasia: a scoping review (2026) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some studies (22, 27, 31) also included speech or behavioral data of healthy subjects as a control or training basis.”
From Machine learning models in post-stroke aphasia: a scoping review (2026) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Four studies (19, 22, 30, 37) constructed diagnostic and classification models of aphasia by evaluating general demographic data, neuroimaging data, and clinical indicators of patients, but there were differences in model function, fitting, and application effect.”
From Machine learning models in post-stroke aphasia: a scoping review (2026) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In addition, only one study (22) preliminarily analyzed the consistency between the model judgment and the clinical evaluation of speech therapists”
From Machine learning models in post-stroke aphasia: a scoping review (2026) · cites this paper as [22]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“One model (22) uses speech acoustic features to automatically identify the fluency type of aphasia.”
From Machine learning models in post-stroke aphasia: a scoping review (2026) · cites this paper as [22]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
The citation lists are queued
They are fetched later and appear here then.
References 22 · Cited by 10
Metadata
- Authors
- Metu, J., Kotha, V., & Hillis, A. E.
- Affiliations
- Rock Ridge High School
- Thomas Jefferson High School for Science and Technology
- Johns Hopkins University (US)
- Corresponding author
- Hillis, Argye E.
- ORCID
- Hillis, Argye E.
- Venue
- Aphasiology
- Publisher
- Informa UK Limited
- Published
- 2023-02-06
- Volume
- 38
- Issue
- 1
- Pages
- 168-180
- ISSN
- 0268-7038 (ISSN-L)
- 1464-5041
- PMC
- PMC10901507
- OpenAlex
- W4319442412
- Cited by
- 10
- References
- 22
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 3 · 2023: 1
- FWCI
- 1.49
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Aphasiology (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion, public-domain)
- Funders
- the National Institute on Deafness and Other Communication Disorders) (P50 DC014664)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
- Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
- Social Sciences › Psychology › Clinical Psychology › Stuttering Research and Treatment
- Keywords
- aphasia, speech fluency, Western Aphasia Battery-Revised, inter-rater reliability, convolutional neural networks, recurrent neural networks, machine learning, post-stroke aphasia, aphasia classification, speech and language assessment, automated speech analysis, kappa agreement, bilingual aphasia
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating Fluency in Aphasia: Fluency Scales, Trichotomous Judgements, or Machine Learning. https://app.cochat.ai/papers/10.1080/02687038.2023.2171261
- The paper
- Metu, J., Kotha, V., & Hillis, A. E. (2024). Evaluating Fluency in Aphasia: Fluency Scales, Trichotomous Judgements, or Machine Learning. Aphasiology. https://doi.org/10.1080/02687038.2023.2171261
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.