Predicting MCI Status From Multimodal Language Data Using Cascaded Classifiers
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Recent work has indicated the potential utility of automated language analysis for the detection of mild cognitive impairment (MCI). Most studies combining language processing and machine learning for the prediction of MCI focus on a single language task; here, we consider a cascaded approach to combine data from multiple language tasks. A cohort of 26 MCI participants and 29 healthy controls completed three language tasks: picture description, reading silently, and reading aloud. Information from each task is captured through different modes (audio, text, eye-tracking, and comprehension questions). Features are extracted from each mode, and used to train a series of cascaded classifiers which output predictions at the level of features, modes, tasks, and finally at the overall session level. The best classification result is achieved through combining the data at the task level (AUC = 0.88, accuracy = 0.83). This outperforms a classifier trained on neuropsychological test scores (AUC = 0.75, accuracy = 0.65) as well as the "early fusion" approach to multimodal classification (AUC = 0.79, accuracy = 0.70). By combining the predictions from the multimodal language classifier and the neuropsychological classifier, this result can be further improved to AUC = 0.90 and accuracy = 0.84. In a correlation analysis, language classifier predictions are found to be moderately correlated (ρ = 0.42) with participant scores on the Rey Auditory Verbal Learning Test (RAVLT). The cascaded approach for multimodal classification improves both system performance and interpretability. This modular architecture can be easily generalized to incorporate different types of classifiers as well as other heterogeneous sources of data (imaging, metabolic, etc.).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kok, J. S., Berg, I. J., & Scherder, E. J. A. (2013) · Dementia and Geriatric Cognitive Disorders Extra · doi:10.1159/000353441 · Cited by 20,589
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Morris, J. C. (1997) · International Psychogeriatrics · doi:10.1017/s1041610297004870 · Cited by 1,641
Multimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279
Interpreting the magnitudes of correlation coefficients.
Hemphill, J. F. (2003) · American Psychologist · doi:10.1037/0003-066x.58.1.78 · Cited by 1,237
Hierarchical Feature Representation and Multimodal Fusion with Deep Learning for AD/MCI Diagnosis
Suk, H.-I., Lee, S.-W., & Shen, D. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.06.077 · Cited by 968 · cited 1×
“With respect to dementia detection, multimodal approaches have been most effective in the medical imaging domain, where such methodologies have been used to combine information from various brain imaging technologies (Suk et al., 2014; Thung et al., 2017).”
MethodLinguistic Features Identify Alzheimer’s Disease in Narrative Speech
Fraser, K. C., Meltzer, J. A., & Rudzicz, F. (2015) · Journal of Alzheimer's Disease · doi:10.3233/jad-150520 · Cited by 797 · cited 2×
“Previous classification studies using the picture description task have combined acoustic and linguistic features (Roark et al., 2011; Fraser et al., 2016; Gosztolya et al., 2019).”
BackgroundMethodSelf-citationLanguage performance in Alzheimer's disease and mild cognitive impairment: A comparative review
Taler, V., & Phillips, N. A. (2008) · Journal of Clinical and Experimental Neuropsychology · doi:10.1080/13803390701550128 · Cited by 529 · cited 1×
“Language tasks have been shown to be sensitive to subtle cognitive deficits (Taler and Phillips, 2008).”
BackgroundAutomatic speech analysis for the assessment of patients with predementia and Alzheimer's disease
König, A., Satt, A., Sorin, A., Hoory, R., Toledo‐Ronen, O., Derreumaux, A., Manera, V., Verhey, F., Aalten, P., Robert, P. H., et al. (2015) · Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring · doi:10.1016/j.dadm.2014.11.012 · Cited by 380 · cited 1×
“One potential alternative to conventional cognitive testing is the analysis of naturalistic language use, which can be less stressful (König et al., 2015), more easily repeatable (Forbes-McKay et al., 2013), and a better predictor of actual functional ability (Sajjadi et al., 2012).”
BackgroundConnected speech as a marker of disease progression in autopsy-proven Alzheimer’s disease
Ahmed, S., Haigh, A.-M. F., de Jager, C. A., & Garrard, P. (2013) · Brain · doi:10.1093/brain/awt269 · Cited by 362
Connected Speech in Neurodegenerative Language Disorders: A Review
Boschi, V., Catricalà, E., Consonni, M., Chesi, C., Moro, A., & Cappa, S. F. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00269 · Cited by 362
Coupled neural systems underlie the production and comprehension of naturalistic narrative speech
Silbert, L. J., Honey, C. J., Simony, E., Poeppel, D., & Hasson, U. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1323812111 · Cited by 352
Reisberg, B., & Gauthier, S. (2007) · International Psychogeriatrics · doi:10.1017/s1041610207006412 · Cited by 338 · cited 1×
“…(MCI) describes a level of cognitive decline characterized by a clinically observable deficit in at least one cognitive domain, but which does not interfere significantly with a person’s ability to complete the activities of daily living (Gauthier et al., 2006; Reisberg and Gauthier, 2008).”
BackgroundMueller, K. D., Hermann, B., Mecollari, J., & Turkstra, L. S. (2018) · Journal of Clinical and Experimental Neuropsychology · doi:10.1080/13803395.2018.1446513 · Cited by 336
Toth, L., Hoffmann, I., Gosztolya, G., Vincze, V., Szatloczki, G., Banreti, Z., Pakaski, M., & Kalman, J. (2018) · Current Alzheimer Research · doi:10.2174/1567205014666171121114930 · Cited by 290 · cited 2×
“…learning studies on detecting MCI, which reported AUC = 0.732 (when excluding neuropsychological features) (Roark et al., 2011), AUC = 0.724 (Toth et al., 2018), AUC = 0.758 (König et al., 2018a), AUC = 0.796 (Asgari et al., 2017), and AUC = 0.82 (Lehr et al., 2013; Prud’hommeaux and Roark,…”
InfluentialForbes-McKay, K. E., & Venneri, A. (2005) · Neurological Sciences · doi:10.1007/s10072-005-0467-9 · Cited by 261
Gosztolya, G., Vincze, V., Tóth, L., Pákáski, M., Kálmán, J., & Hoffmann, I. (2019) · Computer Speech & Language · doi:10.1016/j.csl.2018.07.007 · Cited by 196 · cited 7×
“Previous classification studies using the picture description task have combined acoustic and linguistic features (Roark et al., 2011; Fraser et al., 2016; Gosztolya et al., 2019).”
BackgroundMethodInfluentialMultimodal integration-a statistical view
Wu, L., Oviatt, S. L., & Cohen, P. R. (1999) · IEEE Transactions on Multimedia · doi:10.1109/6046.807953 · Cited by 190
Predicting mild cognitive impairment from spontaneous spoken utterances
Asgari, M., Kaye, J., & Dodge, H. (2017) · Alzheimer's & Dementia: Translational Research & Clinical Interventions · doi:10.1016/j.trci.2017.01.006 · Cited by 185
Abnormalities of connected speech in semantic dementia vs Alzheimer's disease
Sajjadi, S. A., Patterson, K., Tomek, M., & Nestor, P. J. (2012) · Aphasiology · doi:10.1080/02687038.2012.654933 · Cited by 142 · cited 1×
“One potential alternative to conventional cognitive testing is the analysis of naturalistic language use, which can be less stressful (König et al., 2015), more easily repeatable (Forbes-McKay et al., 2013), and a better predictor of actual functional ability (Sajjadi et al., 2012).”
BackgroundKonig, A., Satt, A., Sorin, A., Hoory, R., Derreumaux, A., David, R., & Robert, P. H. (2018) · Current Alzheimer Research · doi:10.2174/1567205014666170829111942 · Cited by 119
Thomas, C., Keselj, V., Cercone, N., Rockwood, K., & Asp, E. (2006) · IEEE International Conference Mechatronics and Automation, 2005 · doi:10.1109/icma.2005.1626789 · Cited by 110 · cited 1×
“, 2013; Prud’hommeaux and Roark, 2015), conversation (Thomas et al., 2005; Asgari et al., 2017), or tests of semantic verbal fluency (Pakhomov and Hemmy, 2014; Linz et al.”
BackgroundLinguistic changes in verbal expression: A preclinical marker of Alzheimer's disease
CUETOS, F., ARANGO-LASPRILLA, J. C., URIBE, C., VALENCIA, C., & LOPERA, F. (2007) · Journal of the International Neuropsychological Society · doi:10.1017/s1355617707070609 · Cited by 102 · cited 1×
“Cuetos et al. (2007) used the Cookie Theft picture description task from the Boston Diagnostic Aphasia Examination (BDAE) (Goodglass et al., 1983) to elicit speech samples from asymptomatic, middle-aged participants with and without the E280A mutation (which inevitably leads to AD).”
MethodFully Automatic Speech-Based Analysis of the Semantic Verbal Fluency Task
König, A., Linz, N., Tröger, J., Wolters, M., Alexandersson, J., & Robert, P. (2018) · Dementia and Geriatric Cognitive Disorders · doi:10.1159/000487852 · Cited by 77 · cited 5×
“In a similar vein, König et al. (2018b) combine features from eight language tasks into a single classifier, and distinguish between MCI and subjective cognitive impairment with an accuracy of 0.86, but include only a qualitative discussion of the relative contributions of each of the tasks to the…”
BackgroundMethodInfluentialSeverity of Alzheimer’s disease and language features in picture descriptions
Kavé, G., & Dassa, A. (2018) · Aphasiology · doi:10.1080/02687038.2017.1303441 · Cited by 72 · cited 1×
“Some correlations between characteristics of narrative speech and neuropsychological test scores have been reported for AD (Ash et al., 2007; Kavé and Goral, 2016; Kavé and Dassa, 2018), however, fewer studies have examined possible correlations in the MCI stage.”
BackgroundThung, K.-H., Yap, P.-T., & Shen, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-67558-9_19 · Cited by 63
Vector-space topic models for detecting Alzheimer's disease
Yancheva, M., & Rudzicz, F. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1221 · Cited by 57
Toward the Automation of Diagnostic Conversation Analysis in Patients with Memory Complaints
Mirheidari, B., Blackburn, D., Harkness, K., Walker, T., Venneri, A., Reuber, M., & Christensen, H. (2017) · Journal of Alzheimer’s Disease · doi:10.3233/jad-160507 · Cited by 54 · cited 1×
“Mirheidari et al. (2017) used automated conversation analysis in the differential diagnosis of participants with mild cognitive difficulties due to neurodegenerative disorders and those with functional memory disorders, finding both non-verbal and semantic cues to be highly predictive.”
Method
Cited by
Multimodal machine learning in precision health: A scoping review
Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., & Luo, Y. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00712-8 · Cited by 528
Speech- and language-based classification of Alzheimer's disease: a systematic review
Vigo, I., Coelho, L., & Reis, S. (2022) · Bioengineering · doi:10.3390/bioengineering9010027 · Cited by 83
An automatic Alzheimer’s disease classifier based on spontaneous spoken English
Bertini, F., Allevi, D., Lutero, G., Calzà, L., & Montesi, D. (2022) · Computer Speech & Language · doi:10.1016/j.csl.2021.101298 · Cited by 66
Clarke, N., Barrick, T. R., & Garrard, P. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.634360 · Cited by 65
A remote speech-based AI system to screen for early Alzheimer's disease via smartphones
Fristed, E., Skirrow, C., Meszaros, M., Lenain, R., Meepegama, U., Cappa, S., Aarsland, D., & Weston, J. (2022) · Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring · doi:10.1002/dad2.12366 · Cited by 47
Metadata
- Authors
- Fraser, K. C., Lundholm Fors, K., Eckerström, M., Öhman, F., & Kokkinakis, D.
- Venue
- Frontiers in Aging Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2019-08-02
- Volume
- 11
- Pages
- 205-205
- ISSN
- 1663-4365 (ISSN-L)
- OpenAlex
- W2964250703
- Cited by
- 97
- References
- 30
- FWCI
- 6.85
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.