CoChat

Performance Evaluation of Machine Learning Frameworks for Aphasia Assessment

Open access

The DOI matches its Crossref record; 36 of the 40 references it cites were found in a registry or an index; 6 of the 32 sources its claims cite were checked for support.

Abstract

Speech assessment is an essential part of the rehabilitation procedure for patients with aphasia (PWA). It is a comprehensive and time-consuming process that aims to discriminate between healthy individuals and aphasic patients, determine the type of aphasia syndrome, and determine the patients’ impairment severity levels (these are referred to here as aphasia assessment tasks). Hence, the automation of aphasia assessment tasks is essential. In this study, the performance of three automatic speech assessment models based on the speech dataset-type was investigated. Three types of datasets were used: healthy subjects’ dataset, aphasic patients’ dataset, and a combination of healthy and aphasic datasets. Two machine learning (ML)-based frameworks, classical machine learning (CML) and deep neural network (DNN), were considered in the design of the proposed speech assessment models. In this paper, the DNN-based framework was based on a convolutional neural network (CNN). Direct or indirect transformation of these models to achieve the aphasia assessment tasks was investigated. Comparative performance results for each of the speech assessment models showed that quadrature-based high-resolution time-frequency images with a CNN framework outperformed all the CML frameworks over the three dataset-types. The CNN-based framework reported an accuracy of 99.23 ± 0.003% with the healthy individuals’ dataset and 67.78 ± 0.047% with the aphasic patients’ dataset. Moreover, direct or transformed relationships between the proposed speech assessment models and the aphasia assessment tasks are attainable, given a suitable dataset-type, a reasonably sized dataset, and appropriate decision logic in the ML framework.

Overview

CoChat Summary

TL;DR

This paper compares classical machine learning (CML) and a CNN-based deep neural network framework for three aphasia-related speech assessment models built from different dataset types: healthy, aphasic, and combined healthy-aphasic speech. Across all three models, the ResNet-34 CNN with high-resolution time-frequency images outperformed the CML classifiers, with the best reported accuracy of 99.23 ± 0.003% on healthy speech and 67.78 ± 0.047% on aphasic speech.

Key findings

  • On the healthy dataset (Model-A), the ResNet-34 CNN with HTD time-frequency images outperformed all CML classifiers; it achieved 99.23 ± 0.003% accuracy on the only-words dataset, versus 95.28 ± 0.79% for the best CML method (LDA). For the larger 26-class dataset, CNN reached 97.39 ± 0.004%, and all classifiers except kNN exceeded 90% accuracy.
  • On the aphasic dataset (Model-B), CNN again outperformed the CML methods, but overall performance was much lower; CNN achieved 67.78 ± 0.047% accuracy on the only-words dataset, compared with 49.06 ± 1.91% for the best CML method (random forest).
  • On the combined healthy-aphasic dataset (Model-C), CNN remained the top-performing framework, with accuracies of 59.17% on the only-words dataset and 57.29% on the vowels + words dataset; all ML algorithms performed poorly overall on this dataset.
  • The paper reports that transforming Model-A into a binary normal-vs-aphasic classifier achieved 100% accuracy when the decision threshold κ was set to 0.9.

Methods

  • The study reused Mandarin speech datasets from a prior collection: 34 healthy subjects and 12 aphasic patients, with 6 Mandarin vowels, 10 nouns, and 10 verbs; data were organized into two task-specific subsets per dataset type: 26 classes (vowels + words) and 20 classes (only-words).
  • Two framework families were evaluated: CML with five classifiers (QSVM, RBF SVM, LDA, random forest, kNN) using 51 ASR features (MFCCs, energy, formants), and a CNN framework using ResNet-34 fine-tuned on HTD-based time-frequency images resized to 224 × 224 × 3.
  • Model-A and Model-B were evaluated with 5-fold cross-validation; Model-C used a train-test split. Performance was measured by accuracy, precision, and recall, and the paper reports mean and standard deviation across folds where applicable.

Limitations

  • The paper states that the aphasic dataset is limited and highly variable: the recruited patients had different impairment severity levels, and aphasic syndrome datasets are scarce with small samples per severity group, which contributed to poor performance, especially in Models B and C.
  • The authors note that Model-B is not suitable for aphasia syndrome discrimination in its current form because it was trained only on aphasic speech and would need properly labeled data covering multiple aphasia syndromes.
  • Evident from the text: the dataset is small overall (34 healthy, 12 aphasic), and the paper itself notes that 5 of the 12 patients recorded vowels only, which likely further restricts some analyses.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/s21082582

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 A typical classical machine learning framework for the three speech assessment models.
Figure 1 A typical classical machine learning framework for the three speech assessment models.CC BY · PubMed Central
Figure 2 A typical convolutional neural network (CNN)-based classification framework for the three speech assessment models.
Figure 2 A typical convolutional neural network (CNN)-based classification framework for the three speech assessment models.CC BY · PubMed Central
Figure 3 A general framework for aphasia assessment tasks.
Figure 3 A general framework for aphasia assessment tasks.CC BY · PubMed Central
Figure 4 Performance evaluation for the CNN and classical machine learning (CML) classification frameworks, Model-A, on (a) the Only words (20 classes) of healthy dataset-type and (b) the Vowels + Words (26 classes) of healthy dataset-type.
Figure 4 Performance evaluation for the CNN and classical machine learning (CML) classification frameworks, Model-A, on (a) the Only words (20 classes) of healthy dataset-type and (b) the Vowels + Words (26 classes) of healthy dataset-type.CC BY · PubMed Central
Figure 5 Performance evaluation for the CNN and CML classification frameworks, Model-B, on (a) the Only words of aphasic patients’ dataset-type and (b) the Vowels + Words of aphasic patients’ dataset-type.
Figure 5 Performance evaluation for the CNN and CML classification frameworks, Model-B, on (a) the Only words of aphasic patients’ dataset-type and (b) the Vowels + Words of aphasic patients’ dataset-type.CC BY · PubMed Central
Figure 6 Performance evaluation for the CNN and CML classification frameworks, Model-C, on (a) the Only words of aphasic patients and healthy subjects’ dataset-types and (b) the Vowels + Words of aphasic patients and healthy subjects’ dataset-types.
Figure 6 Performance evaluation for the CNN and CML classification frameworks, Model-C, on (a) the Only words of aphasic patients and healthy subjects’ dataset-types and (b) the Vowels + Words of aphasic patients and healthy subjects’ dataset-types.CC BY · PubMed Central

Verification

Claims the paper makes

4 Valid · 1 Not valid · 12 Not checked · 34 Own results and interpretations

26 more claims

Create a free account to see all 51 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 15
  • Probably not a substantial claim. 10

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 4

  1. Mahmoud, S.S., Fang, Q., Tang, Y., Alsulami, M., and Alotaibi, M. (2019, January 28–29). Automatic Mandarin Vowels Recognition Framework for Aphasic Patients Rehabilitation. Proceedings of the 13th IEEE-EMBS International Summer School and Symposium on Medical Devices and Biosensors (MDBS2019), Chengdu, China.
  2. Li, S. (2014). Speech Therapy, Huaxia Press.
  3. Rabiner, L.R., and Schafer, R.W. (2010). Theory and Application of Digital Speech Processing, Pearson.
  4. Suresh, 2018, Prasad Kannojia, Effects of Varying Resolution on Performance of CNN based Image Classification: An Experi-mental Study, Int. J. Comput. Sci. Eng., 6, 451

Cited by

See all 27

Metadata

Authors
Mahmoud, S. S., Kumar, A., Li, Y., Tang, Y., & Fang, Q.
Affiliations
Shantou University (CN)
Corresponding author
  • Mahmoud, Seedahmed S.
  • Fang, Qiang
Venue
Sensors
Publisher
MDPI AG
Published
2021-04-07
Volume
21
Issue
8
Pages
2582
ISSN
1424-8220 (ISSN-L)
OpenAlex
W3149523164
Cited by
27
References
40
Citations by year
2026: 2 · 2025: 7 · 2024: 5 · 2023: 5 · 2022: 7 · 2021: 1
FWCI
2.82
Type
journal article
Language
en
Open access
Locations
  • Sensors (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
2020 Li Ka Shing Foundation Cross-disciplinary Research Grant (2020LKSFG04C)
Topics
  • Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
Keywords
aphasia assessment, convolutional neural networks, classical machine learning, time-frequency images, aphasia severity, speech rehabilitation, model performance comparison
MeSH terms
Machine Learning, Aphasia, Humans, Speech, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.