Automatic Speech Assessment for People with Aphasia Using TDNN-BLSTM with Multi-Task Learning
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
This paper describes an investigation on automatic speech assessment for people with aphasia (PWA) using a DNN based automatic speech recognition (ASR) system. The main problems being addressed are the lack of training speech in the intended application domain and the relevant degradation of ASR performance for impaired speech of PWA. We adopt the TDNN-BLSTM structure for acoustic modeling and apply the technique of multi-task learning with large amount of domain-mismatched data. This leads to a significant improvement on the recognition accuracy, as compared with a conventional single-task learning DNN system. To facilitate the extraction of robust text features for quantifying language impairment in PWA speech, we propose to incorporate N-best hypotheses and confusion network representation of the ASR output. The severity of impairment is predicted from text features and supra-segmental duration features using different regression models. Experimental results show a high correlation of 0.842 between the predicted severity level and the subjective Aphasia Quotient score.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482 · cited 1×
“In order to select the most effective features and reduce the feature di-mension for the subsequent regression process, we jointly consider the ranking order of 13 candidate features suggested by the LASSO regression [34, 35] and the correlation between each feature and the AQ value.”
MethodEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“The syllable vectors were obtained from a continuous bag-of-words model [31] trained with syllable-level transcriptions of the training set of CanAB.”
MethodBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973 · cited 1×
“Time delay neural network (TDNN) [11] and long-short-term memory recurrent neural network (LSTM-RNN) [12] are well known of the capability of capturing long-term temporal dependency of acoustic events.”
BackgroundHybrid speech recognition with Deep Bidirectional LSTM
Graves, A., Jaitly, N., & Mohamed, A.-R. (2013) · 2013 IEEE Workshop on Automatic Speech Recognition and Understanding · doi:10.1109/asru.2013.6707742 · Cited by 1,736 · cited 1×
“Bidirectional LSTM (BLSTM) [13] is an extension to LSTM with both preceding and succeeding contexts considered.”
BackgroundA time delay neural network architecture for efficient modeling of long temporal contexts
Peddinti, V., Povey, D., & Khudanpur, S. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-647 · Cited by 986 · cited 1×
“Time delay neural network (TDNN) [11] and long-short-term memory recurrent neural network (LSTM-RNN) [12] are well known of the capability of capturing long-term temporal dependency of acoustic events.”
BackgroundA pitch extraction algorithm tuned for automatic speech recognition
Ghahremani, P., BabaAli, B., Povey, D., Riedhammer, K., Trmal, J., & Khudanpur, S. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6854049 · Cited by 288 · cited 2×
“Pitch features have been shown useful to boost ASR performance on tonal language like Cantonese [29].”
BackgroundAutomated classification of primary progressive aphasia subtypes from narrative speech transcripts
Fraser, K. C., Meltzer, J. A., Graham, N. L., Leonard, C., Hirst, G., Black, S. E., & Rochon, E. (2014) · Cortex · doi:10.1016/j.cortex.2012.12.006 · Cited by 229 · cited 1×
“In [2, 3], Fraser et al. used acoustic and text features for automatic classification of sub-types of primary progressive aphasia.”
BackgroundFinding consensus among words: lattice-based word error minimization
Mangu, L., Brill, E., & Stolcke, A. (1999) · 6th European Conference on Speech Communication and Technology · doi:10.21437/eurospeech.1999-127 · Cited by 214 · cited 1×
“On the other hand, rich representation of ASR output, such as N-best hypotheses [15] and confusion networks [16], has been widely used in spoken language understanding and speech translation [17, 18].”
BackgroundExplicit word error minimization in n-best list rescoring
Stolcke, A., Konig, Y., & Weintraub, M. (1997) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1997-68 · Cited by 153 · cited 1×
“On the other hand, rich representation of ASR output, such as N-best hypotheses [15] and confusion networks [16], has been widely used in spoken language understanding and speech translation [17, 18].”
Background
Cited by
Careless Whisper: Speech-to-Text Hallucination Harms
Koenecke, A., Choi, A. S. G., Mei, K. X., Schellmann, H., & Sloane, M. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658996 · Cited by 89
“Research shows that patients exhibiting aphasia are usually older than stroke patients who do not exhibit aphasia, while those stroke patients who are younger are more likely to exhibit less fluent (e.g., non-fluent or Broca’s type) aphasia [7].”
BackgroundAdikari, A., Hernandez, N., Alahakoon, D., Rose, M. L., & Pierce, J. E. (2024) · Disability and Rehabilitation · doi:10.1080/09638288.2023.2199463 · Cited by 54
An Efficient Deep Learning Based Method for Speech Assessment of Mandarin-Speaking Aphasic Patients
Mahmoud, S. S., Kumar, A., Tang, Y., Li, Y., Gu, X., Fu, J., & Fang, Q. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3011104 · Cited by 36
“In[22],theautomaticspeechassessment forPWAreliesonthemanualtranscriptionofaudiospeechinto Chinesecharacters.”
Automatic Assessment of Speech Impairment in Cantonese-Speaking People with Aphasia
Qin, Y., Lee, T., & Kong, A. P. H. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2956371 · Cited by 32
“In the previous study [25], we incorporated rich representation of ASR output (i.”
MethodSelf-citationPerformance Evaluation of Machine Learning Frameworks for Aphasia Assessment
Mahmoud, S. S., Kumar, A., Li, Y., Tang, Y., & Fang, Q. (2021) · Sensors · doi:10.3390/s21082582 · Cited by 27
Mahmoud, S. S., Pallaud, R. F., Kumar, A., Faisal, S., Wang, Y., & Fang, Q. (2023) · Sensors · doi:10.3390/s23020857 · Cited by 25
“Qin et al., 2018 [15] DNN: Time Delay Neural Network Bidirectional Long-Short Term Memory Recurrent Neural Network (TDNN-BLSTM-RNN) • The aim was to predict aphasia severity using speech utterances under the constraint of a lack of training speech data in the intended application domain and the…”
An End-to-End Approach to Automatic Speech Assessment for Cantonese-speaking People with Aphasia
Qin, Y., Wu, Y., Lee, T., & Kong, A. P. H. (2020) · Journal of Signal Processing Systems · doi:10.1007/s11265-019-01511-3 · Cited by 25
“ASR system. The timedelay layers stacked with bidirectional long short term memory layers (TDNNBLSTM) are used as acoustic model of the ASR system and it is trained using multi-task learning strategy [15]. These ASR-generated features were shown to be eective to classify High-AQ speakers from Low-AQ ones in the aspect…”
MethodSelf-citationHerath, H. M. D. P. M., Weraniyagoda, W. A. S. A., Rajapaksha, R. T. M., Wijesekara, P. A. D. S. N., Sudheera, K. L. K., & Chong, P. H. J. (2022) · Sensors · doi:10.3390/s22186966 · Cited by 23
“A lower syllable error rate for aphasic speech has been obtained by using a Time Delay Neural Network (TDNN) along with a Bidirectional Long Short Term Memory (BLSTM) for acoustic modeling in ASR [56].”
MethodResultTang, J., Chen, W., Chang, X., Watanabe, S., & MacWhinney, B. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-2191 · Cited by 22
Corrales-Astorgano, M., Martínez‐Castilla, P., Escudero-Mancebo, D., Aguilar, L., González-Ferreras, C., & Cardeñoso-Payo, V. (2019) · Applied Sciences · doi:10.3390/app9071440 · Cited by 20
“Some works focus on the speech intelligibility of people with aphasia [23,24] or speech intelligibility in pathological voices [25,26].”
BackgroundRanjith, R., & Chandrasekar, A. (2023) · Computer Speech & Language · doi:10.1016/j.csl.2023.101568 · Cited by 17
Low-resource automatic speech recognition and error analyses of oral cancer speech
Halpern, B. M., Feng, S., van Son, R. J. J. H., van den Brekel, M. W. M., & Scharenborg, O. (2022) · Speech Communication · doi:10.1016/j.specom.2022.04.006 · Cited by 21
“This approach has shown to be effective in improving acoustic modelling for pathological speech (Christensen et al., 2013; Liu et al., 2017), including dysarthria (Yilmaz et al., 2017; Hermann and Doss, 2020), and aphasia (Qin et al., 2018) for hybrid models.”
Self-citationWNARS: WFST based Non-autoregressive Streaming End-to-End Speech RecognitionPreprint
Wang, Z., Yang, W., Zhou, P., & Chen, W. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.03587 · Cited by 12
Automatic recognition and detection of aphasic natural speech
Barberis, M., Clercq, P. D., Tamm, B., hamme, H. V., & Vandermosten, M. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-517 · Cited by 10
“First, there is a relevant decrease in the accuracy of ASR for aphasic speech [11], [12], [13].”
BackgroundConv-transformer-based Jaya Gazelle optimization for speech intelligibility with aphasia
Ranjith, R., & Chandrasekar, A. (2023) · Signal Image and Video Processing · doi:10.1007/s11760-023-02844-0 · Cited by 9
Nivedha, E., Chandrasekar, A., & Jothi, S. (2023) · Signal Image and Video Processing · doi:10.1007/s11760-023-02620-0 · Cited by 9
Qin, Y. H., Lee, T., & Kong, A. P. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683835 · Cited by 9
“Refer to [11] for the details of unimpaired corpora and CD-GMM-HMM AM training.”
MethodInfluentialSelf-citationThe CUHK-TUDELFT System for The SLT 2021 Children Speech Recognition ChallengePreprint
Ng, S. I., Liu, W., Peng, Z., Feng, S., Huang, H.-P., Scharenborg, O., & Lee, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.06239 · Cited by 8
“Previous studies showed that the recognition accuracy could drop drastically for a mismatch caused by speech pathology [3, 4], background noise [5] or speaker’s attributes (e.g. age group [6,7], gender [8]).”
BackgroundSelf-citationAn End-to-End Approach to Automatic Speech Assessment for People with Aphasia
Qin, Y. H., Lee, T., Wu, Y., & Kong, A. P. (2018) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp.2018.8706690 · Cited by 7
“We also made efforts to improve the ASR accuracy on PWA speech using multi-task learning strategy, and used N-best lists and confusion network to mitigate the effect of ASR errors on the text features [8].”
MethodSelf-citationEscudero-Mancebo, D., Corrales-Astorgano, M., Cardeñoso-Payo, V., & González-Ferreras, C. (2021) · IEEE Access · doi:10.1109/access.2021.3090516 · Cited by 8
Classifying Speech Intelligibility Levels of Children in Two Continuous Speech Styles
Lin, Y.-S., & Tseng, S.-C. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414986 · Cited by 5
“These two approaches have been adopted for building automatic speech assessment systems [10, 16-19], e.”
MethodAccurate Anomia Severity Detection in Post-Stroke Aphasia Patients using MobileNetV2
Saran, M. D. S. R., Akshayaa, B. K., Raghavendra, R. S., Poornima, N., & Lal, G. J. (2024) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt61001.2024.10724563 · Cited by 5
“Qin et al. [14] discussed the use of TDNN-BLSTM with MTL stratergy for automatic speech assessment for PWA.”
MethodCrossed-Time Delay Neural Network for Speaker Recognition
Chen, L., Liang, Y., Shi, X., Zhou, Y., & Wu, C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67832-6_1 · Cited by 4
Introducing ISAP and MATSS: Mental stress induced speech utterance procedure and obtained dataset
Singh, B., & Phartiyal, G. S. (2022) · Speech Communication · doi:10.1016/j.specom.2022.09.005 · Cited by 3
Learning Co-Speech Gesture for Multimodal Aphasia Type Detection
Lee, D., Son, S., Jeon, H., Kim, S., & Han, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.577 · Cited by 6
“Note that we exclude the segments of the interviewer in the video recordings since their subjective reactions can potentially affect the patient’s performance (Parveen and Santhanam, 2021; Qin et al., 2018b).”
BackgroundMethodInfluentialAnalyzing Machine Learning Algorithms for Speech Impairment Related Issues
Singh, V. P., Agarwal, J., & Kumar, M. (2021) · International Conference on Signal Processing and Integrated Networks (SPIN) · doi:10.1109/spin52536.2021.9566072 · Cited by 3
Automatic Assessment of Language Impairment Based on Raw ASR Output
Qin, Y. H., Lee, T., & Kong, A. P. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1688 · Cited by 2
“In our previous work, syllable-level embedding features, phone posteriorgrams, and supra-segmental duration features, were shown to be effective in assessment of language impairment with Cantonese-speaking PWA [8, 9, 10].”
BackgroundMethodSelf-citationWang, Y., Pang, H., Qin, Y. H., Wei, S., & Zhao, Y. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107464 · Cited by 1
Self-citationRykova, E., & Walther, M. (2025) · Loquens · doi:10.3989/loquens.2025.e116 · Cited by 3
amor, I. B. (2024) · theses.fr (ABES) · doi:10.70675/7f49c144zf750z4f04za5efzb9ee13802913 · Cited by 3
Corrales-Astorgano, M. (2019) · UVaDOC UVaDOC University of Valladolid Documentary Repository (University of Valladolid) · doi:10.35376/10324/38470
Al-Marridi, A. Z., Said, A. B., & Khwaileh, T. (2025) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc58623.2025.11252647
“The studies in [11], [12], [13] developed a fully automated assessment system for Cantonese-speaking PWA based on MFCC features extracted from narrative speech.”
Söderbäck, A. (2025) · Lund University Publications Student Papers (Lund University)
Metadata
- Authors
- Qin, Y., Lee, T., Feng, S., & Kong, A. P. H.
- Venue
- Interspeech 2018
- Publisher
- ISCA
- Published
- 2018-09-02
- Pages
- 3418-3422
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2889294685
- Semantic Scholar
- 27b92a0da0ee4014c16770499764f9b06386050f
- Cited by
- 30
- References
- 26
- Citations by year
- 2025: 3 · 2024: 2 · 2023: 7 · 2022: 3 · 2021: 6 · 2020: 3 · 2019: 5 · 2018: 1
- FWCI
- 1.52
- Type
- conference paper
- Language
- en
- Open access
- Locations
- Interspeech (USB) (publishedVersion)
- Journal of International Crisis and Risk Communication Research (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- aphasia, multi-task learning, acoustic modeling, confusion networks, aphasia quotient
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.