Automatic Speech Assessment for People with Aphasia Using TDNN-BLSTM with Multi-Task Learning
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
This paper describes an investigation on automatic speech assessment for people with aphasia (PWA) using a DNN based automatic speech recognition (ASR) system. The main problems being addressed are the lack of training speech in the intended application domain and the relevant degradation of ASR performance for impaired speech of PWA. We adopt the TDNN-BLSTM structure for acoustic modeling and apply the technique of multi-task learning with large amount of domain-mismatched data. This leads to a significant improvement on the recognition accuracy, as compared with a conventional single-task learning DNN system. To facilitate the extraction of robust text features for quantifying language impairment in PWA speech, we propose to incorporate N-best hypotheses and confusion network representation of the ASR output. The severity of impairment is predicted from text features and supra-segmental duration features using different regression models. Experimental results show a high correlation of 0.842 between the predicted severity level and the subjective Aphasia Quotient score.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482 · cited 1×
“In order to select the most effective features and reduce the feature di-mension for the subsequent regression process, we jointly consider the ranking order of 13 candidate features suggested by the LASSO regression [34, 35] and the correlation between each feature and the AQ value.”
MethodEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“The syllable vectors were obtained from a continuous bag-of-words model [31] trained with syllable-level transcriptions of the training set of CanAB.”
MethodBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973 · cited 1×
“Time delay neural network (TDNN) [11] and long-short-term memory recurrent neural network (LSTM-RNN) [12] are well known of the capability of capturing long-term temporal dependency of acoustic events.”
BackgroundHybrid speech recognition with Deep Bidirectional LSTM
Graves, A., Jaitly, N., & Mohamed, A.-R. (2013) · 2013 IEEE Workshop on Automatic Speech Recognition and Understanding · doi:10.1109/asru.2013.6707742 · Cited by 1,736 · cited 1×
“Bidirectional LSTM (BLSTM) [13] is an extension to LSTM with both preceding and succeeding contexts considered.”
BackgroundA time delay neural network architecture for efficient modeling of long temporal contexts
Peddinti, V., Povey, D., & Khudanpur, S. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-647 · Cited by 986 · cited 1×
“Time delay neural network (TDNN) [11] and long-short-term memory recurrent neural network (LSTM-RNN) [12] are well known of the capability of capturing long-term temporal dependency of acoustic events.”
BackgroundA pitch extraction algorithm tuned for automatic speech recognition
Ghahremani, P., BabaAli, B., Povey, D., Riedhammer, K., Trmal, J., & Khudanpur, S. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6854049 · Cited by 288 · cited 2×
“Pitch features have been shown useful to boost ASR performance on tonal language like Cantonese [29].”
BackgroundAutomated classification of primary progressive aphasia subtypes from narrative speech transcripts
Fraser, K. C., Meltzer, J. A., Graham, N. L., Leonard, C., Hirst, G., Black, S. E., & Rochon, E. (2014) · Cortex · doi:10.1016/j.cortex.2012.12.006 · Cited by 229 · cited 1×
“In [2, 3], Fraser et al. used acoustic and text features for automatic classification of sub-types of primary progressive aphasia.”
BackgroundFinding consensus among words: lattice-based word error minimization
Mangu, L., Brill, E., & Stolcke, A. (1999) · 6th European Conference on Speech Communication and Technology · doi:10.21437/eurospeech.1999-127 · Cited by 214 · cited 1×
“On the other hand, rich representation of ASR output, such as N-best hypotheses [15] and confusion networks [16], has been widely used in spoken language understanding and speech translation [17, 18].”
BackgroundExplicit word error minimization in n-best list rescoring
Stolcke, A., Konig, Y., & Weintraub, M. (1997) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1997-68 · Cited by 153 · cited 1×
“On the other hand, rich representation of ASR output, such as N-best hypotheses [15] and confusion networks [16], has been widely used in spoken language understanding and speech translation [17, 18].”
BackgroundBeyond ASR 1-best: Using word confusion networks in spoken language understanding
Hakkani-Tur, D. Z., Béchet, F., Riccardi, G., & Tür, G. (2005) · Computer Speech & Language · doi:10.1016/j.csl.2005.07.005 · Cited by 142 · cited 1×
“On the other hand, rich representation of ASR output, such as N-best hypotheses [15] and confusion networks [16], has been widely used in spoken language understanding and speech translation [17, 18].”
BackgroundLinguistic assessment of Chinese-speaking aphasics: Development of a Cantonese aphasia battery
Yiu, E. M. ‐. (1992) · Journal of Neurolinguistics · doi:10.1016/0911-6044(92)90025-r · Cited by 137 · cited 2×
“All aphasic subjects went through a standardized assessment system using the Cantonese Aphasia Battery [21].”
BackgroundMethodSpoken language resources for Cantonese speech processing
Lee, T., Lo, W. K., Ching, P. C., & Meng, H. M. L. (2002) · Speech Communication · doi:10.1016/s0167-6393(00)00101-1 · Cited by 133 · cited 1×
“It was developed by The Chinese University of Hong Kong [22].”
MethodSelf-citationMinimum Bayes Risk decoding and system combination based on a recursion for edit distance
Xu, H., Povey, D., Mangu, L., & Zhu, J. (2011) · Computer Speech & Language · doi:10.1016/j.csl.2011.03.001 · Cited by 130 · cited 1×
“In present study, we adopt the advanced method implemented in the Kaldi (by function lattice-mbr-decode ) [32] to generate CNs.”
MethodParallel training of Deep Neural Networks with Natural Gradient and Parameter AveragingPreprint
Povey, D., Zhang, X., & Khudanpur, S. P. (2014) · arXiv (Cornell University) · Cited by 126
Consensus Network Decoding for Statistical Machine Translation System Combination
Sim, K. C., Byrne, B., Gales, M., Sahbi, H., & Woodland, P. C. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.367174 · Cited by 105 · cited 1×
“On the other hand, rich representation of ASR output, such as N-best hypotheses [15] and confusion networks [16], has been widely used in spoken language understanding and speech translation [17, 18].”
BackgroundAn Exploration of Dropout with LSTMs
Cheng, G., Peddinti, V., Povey, D., Manohar, V., Khudanpur, S. P., & Yan, Y. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-129 · Cited by 105 · cited 1×
“1 is applied to improve generalization of neural networks [30].”
BackgroundAutomatic speech recognition in the diagnosis of primary progressive aphasia
Fraser, K., Rudzicz, F., Graham, N. L., & Rochon, E. A. (2013) · Cited by 67
Combining in-domain and out-of-domain speech data for automatic recognition of disordered speech
Christensen, H., Aniol, M. B., Bell, P., Green, P. D., Hain, T., King, S., & Swietojanski, P. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-324 · Cited by 63 · cited 1×
“In [7] and [8], tandem feature and discriminative pretraining based out-of-domain adaptation methods were applied to improve the ASR performance on a smaller impaired speech corpus.”
MethodUsing text and acoustic features to diagnose progressive aphasia and its subtypes
Fraser, K. C., Rudzicz, F., & Rochon, E. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-514 · Cited by 61 · cited 1×
“In [2, 3], Fraser et al. used acoustic and text features for automatic classification of sub-types of primary progressive aphasia.”
BackgroundImproving Automatic Recognition of Aphasic Speech with AphasiaBank
Le, D., & Provost, E. M. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-213 · Cited by 58 · cited 2×
“In addition, i-vector based neural network adaptation has been proved to benefit the ASR on PWA speech [8].”
BackgroundMethodFeature selection for automated speech scoring
Loukina, A., Zechner, K., Chen, L., & Heilman, M. (2015) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.3115/v1/w15-0602 · Cited by 52
Multi-Task Learning for Speech Recognition: An Overview
Pironkov, G., Dupont, S., & Dutoit, T. (2016) · ORBi UMONS · Cited by 27
Aalto system for the 2017 Arabic multi-genre broadcast challenge
Smit, P., Gangireddy, S. R., Enarvi, S., Virpioja, S., & Kurimo, M. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8268955 · Cited by 24 · cited 1×
“TDNN-BLSTM has demonstrated its effectiveness in DNN-HMM hybrid acoustic modeling for ASR [10, 14].”
MethodDysprosody in aphasia: An acoustic analysis evidence from Palestinian Arabic
Adam, H. (2014) · Journal of Language and Linguistic Studies · Cited by 9
Towards automatic assessment of aphasia speech using automatic speech recognition techniques
Qin, Y., Lee, T., Kong, A. P. H., & Law, S. P. (2016) · 2016 10th International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp.2016.7918445 · Cited by 7 · cited 1×
“In [5, 6], although a limited amount of domain-matched unimpaired speech were available, they were not sufficient to support the use of most advanced deep learning techniques.”
BackgroundSelf-citation
Cited by
Careless Whisper: Speech-to-Text Hallucination Harms
Koenecke, A., Choi, A. S. G., Mei, K. X., Schellmann, H., & Sloane, M. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658996 · Cited by 89
“Research shows that patients exhibiting aphasia are usually older than stroke patients who do not exhibit aphasia, while those stroke patients who are younger are more likely to exhibit less fluent (e.g., non-fluent or Broca’s type) aphasia [7].”
BackgroundAdikari, A., Hernandez, N., Alahakoon, D., Rose, M. L., & Pierce, J. E. (2024) · Disability and Rehabilitation · doi:10.1080/09638288.2023.2199463 · Cited by 54
An Efficient Deep Learning Based Method for Speech Assessment of Mandarin-Speaking Aphasic Patients
Mahmoud, S. S., Kumar, A., Tang, Y., Li, Y., Gu, X., Fu, J., & Fang, Q. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3011104 · Cited by 36
“In[22],theautomaticspeechassessment forPWAreliesonthemanualtranscriptionofaudiospeechinto Chinesecharacters.”
Automatic Assessment of Speech Impairment in Cantonese-Speaking People with Aphasia
Qin, Y., Lee, T., & Kong, A. P. H. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2956371 · Cited by 32
“In the previous study [25], we incorporated rich representation of ASR output (i.”
MethodSelf-citationPerformance Evaluation of Machine Learning Frameworks for Aphasia Assessment
Mahmoud, S. S., Kumar, A., Li, Y., Tang, Y., & Fang, Q. (2021) · Sensors · doi:10.3390/s21082582 · Cited by 27
Metadata
- Authors
- Qin, Y., Lee, T., Feng, S., & Kong, A. P. H.
- Venue
- Interspeech 2018
- Publisher
- ISCA
- Published
- 2018-09-02
- Pages
- 3418-3422
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2889294685
- Semantic Scholar
- 27b92a0da0ee4014c16770499764f9b06386050f
- Cited by
- 30
- References
- 26
- Citations by year
- 2025: 3 · 2024: 2 · 2023: 7 · 2022: 3 · 2021: 6 · 2020: 3 · 2019: 5 · 2018: 1
- FWCI
- 1.52
- Type
- conference paper
- Language
- en
- Open access
- Locations
- Interspeech (USB) (publishedVersion)
- Journal of International Crisis and Risk Communication Research (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- aphasia, multi-task learning, acoustic modeling, confusion networks, aphasia quotient
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.