An End-to-End Approach to Automatic Speech Assessment for Cantonese-speaking People with Aphasia
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Conventional automatic assessment of pathological speech usually follows two main steps: (1) extraction of pathology-specific features; (2) classification or regression on extracted features. Given the great variety of speech and language disorders, feature design is never a straightforward task, and yet it is most crucial to the performance of assessment. This paper presents an end-to-end approach to automatic speech assessment for Cantonese-speaking People With Aphasia (PWA). The assessment is formulated as a binary classification task to discriminate PWA with high scores of subjective assessment from those with low scores. The 2-layer Gated Recurrent Unit (GRU) and Convolutional Neural Network (CNN) models are applied to realize the end-to-end mapping from basic speech features to the classification outcome. The pathology-specific features used for assessment are learned implicitly by the neural network model. The Class Activation Mapping (CAM) method is utilized to visualize how the learned features contribute to the assessment result. Experimental results show that the end-to-end approach can achieve comparable performance to the conventional two-step approach in the classification task, and the CNN model is able to learn impairment-related features that are similar to the hand-crafted features. The experimental results also indicate that CNN model performs better than 2-layer GRU model in this specific task.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Most research on automatic aphasia assessment [5,9,10,12,13,14,18] has focused on a single aphasia assessment task using automatic speech recognition techniques and a machine learning (ML) framework with a fixed dataset-type.”
From Performance Evaluation of Machine Learning Frameworks for Aphasia Assessment (2021) · cites this paper as [9]
In the citing paper
“The naming, repetition, and auditory verbal comprehension subtests can be automated using customized machine learning [8,11,12].”
From A Comparative Investigation of Automatic Speech Recognition Platforms for Aphasia Assessment Batteries (2023) · cites this paper as [8]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“me acoustic features are extracted and concatenated sequentially as a 2-dimensional time-frequency representation. The CNN model structure described in Table 2 is inspired by the AlexNet [31] and VGG [32] models. It is constructed with a Global Average Pooling (GAP) layer instead of a fully-connected layer after the l…”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“The CNN model structure described in Table 2 is inspired by the AlexNet [31] and VGG [32] models.”
MethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“al learning rate is set to 10 4 for the sequence-to-one GRU-RNN model and it is set to 10 3 for the CNN model. Model training aims at minimizing the binary cross-entropy loss using the Adam optimizer [34]. Weight decay coecient is set to 5 10 4 for both models to tackle the overtting problem. All models are implemente…”
MethodInfluentialGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
(2017) · Cited by 11,625
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · cited 2×
“. Batch normalization (BN) layer and ReLU activation function are added after each convolutional layer. 4.2 Class Activation Mapping Given a CNN model, the technique of Class Activation Mapping (CAM) [24] is capable of visualizing the class-discriminative regions in the input images. Therefore, the CAM can improve und…”
BackgroundMethodInfluentialOn the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014) · Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation · doi:10.3115/v1/w14-4012 · Cited by 7,099 · cited 1×
“ng numerous neural network structures, Recurrent Neural Network (RNN) has the ability to model sequential signals and to deal with variable-length utterances. LSTM [18] and Gated Recurrent Unit (GRU) [19] are well-established \gating" models that tackle the problem of gradient vanishing in vanilla RNN. The GRU is…”
BackgroundGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 1×
“eature extraction and assessment model are optimized separately instead of jointly. Recently, the so-called \end-to-end" approach has demonstrated good successes in ASR [16], machine translation [17], and other applications. It requires relatively little human eort on designing tailored features and shows superio…”
BackgroundSelf-citationNote on the Sampling Error of the Difference Between Correlated Proportions or Percentages
McNemar, Q. (1947) · Psychometrika · doi:10.1007/bf02295996 · Cited by 4,754
Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
Dietterich, T. G. (1998) · Neural Computation · doi:10.1162/089976698300017197 · Cited by 3,840
Towards End-To-End Speech Recognition with Recurrent Neural Networks
Graves, A., & Jaitly, N. (2014) · Cited by 2,557
On the Properties of Neural Machine Translation: Encoder-Decoder ApproachesPreprint
Cho, K., van Merriënboer, B., Bahdanau, D., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.1259 · Cited by 1,112 · cited 1×
“ng numerous neural network structures, Recurrent Neural Network (RNN) has the ability to model sequential signals and to deal with variable-length utterances. LSTM [18] and Gated Recurrent Unit (GRU) [19] are well-established \gating" models that tackle the problem of gradient vanishing in vanilla RNN. The GRU is…”
BackgroundAphasiaBank: Methods for studying discourse
MacWhinney, B., Fromm, D., Forbes, M., & Holland, A. (2011) · Aphasiology · doi:10.1080/02687038.2011.589893 · Cited by 432
Journal of Language and Linguistic Studies
(2021) · Journal of Language and Linguistic Studies · doi:10.52462/jlls · Cited by 387
Aphasia : a clinical perspective
Benson, D. F., & Ardila, A. (1996) · Oxford University Press eBooks · Cited by 301
Automated classification of primary progressive aphasia subtypes from narrative speech transcripts
Fraser, K. C., Meltzer, J. A., Graham, N. L., Leonard, C., Hirst, G., Black, S. E., & Rochon, E. (2014) · Cortex · doi:10.1016/j.cortex.2012.12.006 · Cited by 229
Aphasia: A Clinical Perspective
Benson, D. F., & Ardila, A. (1996) · doi:10.1093/oso/9780195089349.001.0001 · Cited by 214
ROC curve, lift chart and calibration plot
Vuk, M., & Curk, T. (2006) · Advances in Methodology and Statistics · doi:10.51936/noqf3710 · Cited by 176 · cited 1×
“For the binary classification on test utterances from High-AQ and Low-AQ subjects, the Area Under receiver operating characteristic Curve (AUC) [36] is adopted as the performance metric.”
MethodGated Recurrent Unit (GRU) for Emotion Classification from Noisy SpeechPreprint
Rana, R. K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.07778 · Cited by 176 · cited 1×
“, question detection [20] and emotion classification [21].”
BackgroundLinguistic assessment of Chinese-speaking aphasics: Development of a Cantonese aphasia battery
Yiu, E. M. ‐. (1992) · Journal of Neurolinguistics · doi:10.1016/0911-6044(92)90025-r · Cited by 137
Automatic quantitative analysis of spontaneous aphasic speech
Le, D., Licata, K., & Mower Provost, E. (2018) · Speech Communication · doi:10.1016/j.specom.2018.04.001 · Cited by 97 · cited 1×
“[10, 11] improved the PWA speech recognition using discriminative pre-training with out-of-domain dataset and multi-task acoustic model.”
BackgroundConvolutional Neural Network to Model Articulation Impairments in Patients with Parkinson’s Disease
Vásquez-Correa, J. C., Orozco‐Arroyave, J. R., & Nöth, E. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1078 · Cited by 89 · cited 1×
“hes have been investigated in the speech assessment area. It was successfully applied to spoken uency scoring [22] and dierentiate speech of people with Parkinson disease from that of healthy people [23]. In this paper, we focus on investigating the eectiveness of GRU-RNN model and CNN model in PWA speech assessment.…”
MethodKong, A. P., Law, S., Kwan, C., Lai, C., & Lam, V. (2014) · Journal of Nonverbal Behavior · doi:10.1007/s10919-014-0200-6 · Cited by 77
Self-citationQuestion detection from acoustic features using recurrent neural network with gated recurrent unit
Tang, Y., Huang, Y., Wu, Z., Meng, H. M. L., Xu, M., & Cai, L. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472854 · Cited by 71 · cited 1×
“ing in vanilla RNN. The GRU is able to achieve comparable performance to LSTM, with a more simplied architecture. It has been applied to utterance-based classication tasks, e.g., question detection [20] and emotion classication [21]. On the other hand, Convolutional Neural Network (CNN) based approaches have been inve…”
MethodAutomatic speech recognition in the diagnosis of primary progressive aphasia
Fraser, K., Rudzicz, F., Graham, N. L., & Rochon, E. A. (2013) · Cited by 67
Using text and acoustic features to diagnose progressive aphasia and its subtypes
Fraser, K. C., Rudzicz, F., & Rochon, E. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-514 · Cited by 61 · cited 1×
“nonuent aphasia. They proposed a number of phone duration features, part-of-speech features as well as linguistic inquiry and word count features, which were extracted from ASR outputs. Fraser et al. [7,8] performed an automatic classication of sub-types of primary progressive aphasia using acoustic features and text…”
MethodImproving Automatic Recognition of Aphasic Speech with AphasiaBank
Le, D., & Provost, E. M. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-213 · Cited by 58 · cited 1×
“was adopted to automatically generate transcriptions for text feature extraction. However, the eectiveness of assessment was limited by the low recognition accuracy on impaired speech. Duc Le et al. [10,11] improved the PWA speech recognition Title Suppressed Due to Excessive Length 3 using discriminative pre-training…”
MethodEnhancing Sound Texture in CNN-based Acoustic Scene Classification
Wu, Y., & Lee, T. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683490 · Cited by 53 · cited 2×
“GAP has been proved to be a good regularizer for CNNs in image classification [35] and audio scene classification [27] tasks.”
BackgroundMethodSelf-citationLee, T., Liu, Y., Huang, P.-W., Chien, J.-T., Lam, W. K., Yeung, Y. T., Law, T. K. T., Lee, K. Y. S., Kong, A. P.-H., & Law, S.-P. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472924 · Cited by 44 · cited 1×
“A general-purpose ASR system was applied on PWA speech in [14] and a higher syllable error rate was observed for aphasia subjects compared with unimpaired subjects.”
MethodSelf-citationLearning diagnostic models using speech and language measures
Peintner, B., Jarrold, W., Vergyri, D., Richey, C., Tempini, M. L. G., & Ogar, J. (2008) · 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/iembs.2008.4650249 · Cited by 41 · cited 1×
“[6] performed an automatic classification on three sub-types of frontotemporal lobar degeneration in a relatively small dataset which includes the progressive non-fluent aphasia.”
MethodAutomatic Speech Assessment for People with Aphasia Using TDNN-BLSTM with Multi-Task Learning
Qin, Y., Lee, T., Feng, S., & Kong, A. P. H. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1630 · Cited by 30 · cited 2×
“ASR system. The timedelay layers stacked with bidirectional long short term memory layers (TDNNBLSTM) are used as acoustic model of the ASR system and it is trained using multi-task learning strategy [15]. These ASR-generated features were shown to be eective to classify High-AQ speakers from Low-AQ ones in the aspect…”
MethodSelf-citationAnalysis of Neurogenic Disordered Discourse Production
Kong, A. P. (2016) · Routledge eBooks · doi:10.4324/9781315639376 · Cited by 28
Self-citationKong, A. P., & Law, S. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1043-6 · Cited by 24
Self-citationAnalysis of Neurogenic Disordered Discourse Production: From Theory to Practice
Kong, A. P. (2016) · Cited by 18
Self-citationQin, Y., Lee, T., & Hin Kong, A. P. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461289 · Cited by 16 · cited 3×
“otal Predicted: High-AQ 32 9 41 Predicted: Low-AQ 7 43 50 Total 39 52 91 The baseline assessment system in this study follows a conventional twostep assessment approach proposed in our previous study [14]. A 5-dimensional feature vector of supra-segmental duration features is evaluated on the same task of binary class…”
MethodInfluentialSelf-citationLee, T., Kong, A., Chan, V., & Wang, H. (2013) · Procedia - Social and Behavioral Sciences · doi:10.1016/j.sbspro.2013.09.032 · Cited by 12
Self-citationDysprosody in aphasia: An acoustic analysis evidence from Palestinian Arabic
Adam, H. (2014) · Journal of Language and Linguistic Studies · Cited by 9
Spoken English fluency scoring using convolutional neural networks
Chung, H., Lee, Y. K., Lee, S. J., & Park, J. G. (2017) · International Conference on Speech Database and Assessments · doi:10.1109/icsda.2017.8384444 · Cited by 7 · cited 1×
“It was successfully applied to spoken fluency scoring [22] and differentiate speech of people with Parkinson disease from that of healthy people [23].”
BackgroundKohlschein, C., Klischies, D., Schuller, B. W., Meisen, T., & Werner, C. J. (2018) · RWTH Publications (RWTH Aachen) · Cited by 6
Enhancing Sound Texture in CNN-Based Acoustic Scene ClassificationPreprint
Wu, Y., & Lee, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.01502 · Cited by 2 · cited 2×
“GAP has been proved to be a good regularizer for CNNs in image classification [35] and audio scene classification [27] tasks.”
BackgroundMethodSelf-citation
Research works printed but not linked: 1
- Lin, M., Chen, Q., Yan, S. (2013). Network in network. arXiv:cs.NE/1312.4400.
Cited by
Adikari, A., Hernandez, N., Alahakoon, D., Rose, M. L., & Pierce, J. E. (2024) · Disability and Rehabilitation · doi:10.1080/09638288.2023.2199463 · Cited by 54
“The remaining four aimed at providing automated feedback within therapy software, including automated judgements of utterance quality [25] or within immersive VR [30].”
BackgroundMethodAutomatic Assessment of Speech Impairment in Cantonese-Speaking People with Aphasia
Qin, Y., Lee, T., & Kong, A. P. H. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2956371 · Cited by 32
“In [17], an end-to-end “utteranceto-score” approach was attempted in aphasic speech assessment.”
BackgroundSelf-citationPerformance Evaluation of Machine Learning Frameworks for Aphasia Assessment
Mahmoud, S. S., Kumar, A., Li, Y., Tang, Y., & Fang, Q. (2021) · Sensors · doi:10.3390/s21082582 · Cited by 27
“The CNN models are widely used across various applications including aphasia assessment tasks [9].”
BackgroundMethodInfluentialAI and Aphasia in the Digital Age: A Critical Review
Privitera, A. J., Ng, S. H. S., Kong, A. P.-H., & Weekes, B. S. (2024) · Brain Sciences · doi:10.3390/brainsci14040383 · Cited by 26
“However, two studies by Qin and colleagues utilized automatic speech recognition in their assessments to differentiate between aphasic and non-aphasic speech [72,73].”
BackgroundSelf-citationMahmoud, S. S., Pallaud, R. F., Kumar, A., Faisal, S., Wang, Y., & Fang, Q. (2023) · Sensors · doi:10.3390/s23020857 · Cited by 25
“The naming, repetition, and auditory verbal comprehension subtests can be automated using customized machine learning [8,11,12].”
Influential
Metadata
- Authors
- Qin, Y., Wu, Y., Lee, T., & Kong, A. P. H.
- Affiliations
- Chinese University of Hong Kong (HK)
- University of Central Florida (US)
- Corresponding author
- Qin, Ying
- Venue
- Journal of Signal Processing Systems
- Published
- 2020-02-18
- Volume
- 92
- Issue
- 8
- Pages
- 819-830
- ISSN
- 1939-8018 (ISSN-L)
- 1939-8115
- OpenAlex
- W3006736300
- Semantic Scholar
- 9068f3383f06f4f130f5e8fae99f5861ffd14ca1
- Cited by
- 25
- References
- 40
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 5 · 2023: 9 · 2022: 3 · 2021: 3
- FWCI
- 1.47
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Journal of Signal Processing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Research Committee of the Chinese University of Hong Kong
- GRF project grant from the Hong Kong Research Grants Council (CUHK14227216)
- National Institutes of Health (NIH-R01-DC010398)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- aphasia, Cantonese, end-to-end learning, pathological speech, class activation mapping, gated recurrent unit, convolutional neural networks, representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.