CoChat
Sign Up
Cited by 732Dementia and Cognitive Impairment ResearchAlzheimer's disease research and treatmentsArtificial Intelligence in Healthcare

The DOI matches its Crossref record; 54 of the 57 references it cites were found in a registry or an index.

Multimodal deep learning models for early detection of Alzheimer’s disease stage

Open access
Venugopalan, Janani, Tong, Li, Hassanzadeh, Hamid Reza, Wang, May D.·2021·Scientific Reports
DOI 10.1038/s41598-020-74399-wPMC PMC7864942
Open this paper in CoChat — free accountOpen PDF

Abstract

Most current Alzheimer's disease (AD) and mild cognitive disorders (MCI) studies use single data modality to make predictions such as AD stages. The fusion of multiple data modalities can provide a holistic view of AD staging analysis. Thus, we use deep learning (DL) to integrally analyze imaging (magnetic resonance imaging (MRI)), genetic (single nucleotide polymorphisms (SNPs)), and clinical test data to classify patients into AD, MCI, and controls (CN). We use stacked denoising auto-encoders to extract features from clinical and genetic data, and use 3D-convolutional neural networks (CNNs) for imaging data. We also develop a novel data interpretation method to identify top-performing features learned by the deep-models with clustering and perturbation analysis. Using Alzheimer's disease neuroimaging initiative (ADNI) dataset, we demonstrate that deep models outperform shallow models, including support vector machines, decision trees, random forests, and k-nearest neighbors. In addition, we demonstrate that integrating multi-modality data outperforms single modality models in terms of accuracy, precision, recall, and meanF1 scores. Our models have identified hippocampus, amygdala brain areas, and the Rey Auditory Verbal Learning Test (RAVLT) as top distinguished features, which are consistent with the known AD literature.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-020-74399-w

Cites

54 of 54 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 (a) Description of ADNI data. Clinical data consists of demographics, neurological exams and assessments, medications, imaging volumes, and biomarkers. (b) Number of patients by modality and disease stage. (CN controls, MCI mild cognitive disorder, and AD Alzheimer’s disease). 220 patients have all the three data modalities, 588 patients have SNP and EHR, 283 patients have imaging and EHR, the remaining patients have only EHR data.
Figure 1 (a) Description of ADNI data. Clinical data consists of demographics, neurological exams and assessments, medications, imaging volumes, and biomarkers. (b) Number of patients by modality and disease stage. (CN controls, MCI mild cognitive disorder, and AD Alzheimer’s disease). 220 patients have all the three data modalities, 588 patients have SNP and EHR, 283 patients have imaging and EHR, the remaining patients have only EHR data.CC BY · PubMed Central
Figure 2 Deep model for data integration compared with shallow models of data integration. (a) Feature level integration on shallow models, where the features are concatenated before passing into shallow models. (b) Deep intermediate feature level integration where the original features are transformed separatelyusing deep models prior to integration and prediction. (c) Decision level integration where voting is performed using decisions of individual classifiers. In this study, we comparee the performance of deep intermediate level integration against shallow feature and decision levels integrations for the prediction of Alzheimer’s stages.
Figure 2 Deep model for data integration compared with shallow models of data integration. (a) Feature level integration on shallow models, where the features are concatenated before passing into shallow models. (b) Deep intermediate feature level integration where the original features are transformed separatelyusing deep models prior to integration and prediction. (c) Decision level integration where voting is performed using decisions of individual classifiers. In this study, we comparee the performance of deep intermediate level integration against shallow feature and decision levels integrations for the prediction of Alzheimer’s stages.CC BY · PubMed Central
Figure 3 Intermediate-feature-level combination deep models for multimodality data integration for clinical decision support. Data from diverse sources, imaging, EHR and SNP are combined using novel deep architectures. 3D convolutional neural network architectures used on 3D MR image regions to obtain intermediate imaging features. Deep stacked denoising autoencoders are used to obtain intermediate EHR features. Deep stacked denoising autoencoders are used obtain intermediate SNP features. The 3 types of intermediate features are passed into a classification layer for classification into Alzheimer’s stages (CN, MCI and AD).
Figure 3 Intermediate-feature-level combination deep models for multimodality data integration for clinical decision support. Data from diverse sources, imaging, EHR and SNP are combined using novel deep architectures. 3D convolutional neural network architectures used on 3D MR image regions to obtain intermediate imaging features. Deep stacked denoising autoencoders are used to obtain intermediate EHR features. Deep stacked denoising autoencoders are used obtain intermediate SNP features. The 3 types of intermediate features are passed into a classification layer for classification into Alzheimer’s stages (CN, MCI and AD).CC BY · PubMed Central
Figure 4 Internal cross validation results for individual data modality to predict Alzheimer’s stage (a) Imaging results: deep learning prediction performs better than shallow learning predictions (b) EHR results: deep learning outperforms shallow models kNN and SVM and is comparable to decision trees and random forests (c) SNP results: deep learning outperforms shallow models. The kNN, SVM, RF and decision trees are shallow models. (kNN k-Nearest Neighbors, SVM support vector machines, and RF random forests).
Figure 4 Internal cross validation results for individual data modality to predict Alzheimer’s stage (a) Imaging results: deep learning prediction performs better than shallow learning predictions (b) EHR results: deep learning outperforms shallow models kNN and SVM and is comparable to decision trees and random forests (c) SNP results: deep learning outperforms shallow models. The kNN, SVM, RF and decision trees are shallow models. (kNN k-Nearest Neighbors, SVM support vector machines, and RF random forests).CC BY · PubMed Central
Figure 5 Internal cross validation results for integration of data modalities to predict Alzheimer’s stage (a) Imaging + EHR + SNP. Deep learning prediction performs better than shallow learning predictions (b) EHR + SNP Deep learning prediction performs better than shallow learning predictions (c) Imaging + EHR deep learning prediction performs better than shallow learning predictions (d) Imaging + SNP results. Shallow learning gave a better prediction than deep learning due to small sample sizes. (kNN k-Nearest Neighbors, SVM support vector machines, RF random forests, SM shallow models, and DL deep learning).
Figure 5 Internal cross validation results for integration of data modalities to predict Alzheimer’s stage (a) Imaging + EHR + SNP. Deep learning prediction performs better than shallow learning predictions (b) EHR + SNP Deep learning prediction performs better than shallow learning predictions (c) Imaging + EHR deep learning prediction performs better than shallow learning predictions (d) Imaging + SNP results. Shallow learning gave a better prediction than deep learning due to small sample sizes. (kNN k-Nearest Neighbors, SVM support vector machines, RF random forests, SM shallow models, and DL deep learning).CC BY · PubMed Central
Figure 6 Feature extraction for deep model interpretation. Novel feature interpretation methodology where features are masked one at a time and the effect on the classification is observed. The feature which gives the highest drop in accuracy is ranked the highest. Once we ranked the features, we checked if the intermediate features picked associations different from raw data using cluster analysis. Deep models show associations which are different from shallow models, which accounts for superior performance.
Figure 6 Feature extraction for deep model interpretation. Novel feature interpretation methodology where features are masked one at a time and the effect on the classification is observed. The feature which gives the highest drop in accuracy is ranked the highest. Once we ranked the features, we checked if the intermediate features picked associations different from raw data using cluster analysis. Deep models show associations which are different from shallow models, which accounts for superior performance.CC BY · PubMed Central
Figure 7 Data pre-processing pipeline for three data modalities: (a) Imaging data is first skull stripped, segmented into white matter, grey matter, and cerebrospinal fluid. Then the images are registered to a standard space, prior to extracting 21 brain regions using anatomical automatic labeling atlases. (b) Clinical data is normalized between 1–2 or encoded as 1–2. Then we discard features with values missing values > 70% to obtain 1680 features for 204 patients. (c) SNP data is first filtered, error corrected, feature selection using known genes and then followed by maximum relevance (maxrel) based methods, to obtain 500 SNPS for 808 patients.
Figure 7 Data pre-processing pipeline for three data modalities: (a) Imaging data is first skull stripped, segmented into white matter, grey matter, and cerebrospinal fluid. Then the images are registered to a standard space, prior to extracting 21 brain regions using anatomical automatic labeling atlases. (b) Clinical data is normalized between 1–2 or encoded as 1–2. Then we discard features with values missing values > 70% to obtain 1680 features for 204 patients. (c) SNP data is first filtered, error corrected, feature selection using known genes and then followed by maximum relevance (maxrel) based methods, to obtain 500 SNPS for 808 patients.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 5

  1. Schulam, P., Wigley, F. & Saria, S. In AAAI, 2956–2964 (2015).
  2. Suk, H.-I. & Shen, D. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 583–590 (Springer, New York, 2013).
  3. Mhaskar, H., Liao, Q. & Poggio, T. Learning functions: when is deep better than shallow. arXiv preprint arXiv:1603.00988 (2016). · arXiv
  4. Che, Z., Purushotham, S., Khemani, R. & Liu, Y. Distilling knowledge from deep networks with applications to healthcare domain. arXiv preprint arXiv:1512.03542 (2015). · arXiv
  5. Kingma, D. & Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). · arXiv

Cited by

Sorted by citations

See all 732

Related

See all

Metadata

Authors
Venugopalan, J., Tong, L., Hassanzadeh, H. R., & Wang, M. D.
Affiliations
Georgia Institute of Technology (US)
Corresponding author
Wang, May D.
ORCID
Venue
Scientific Reports
Publisher
Springer Science and Business Media LLC
Published
2021-02-05
Volume
11
Issue
1
Pages
3254-3254
ISSN
2045-2322 (ISSN-L)
DOI
10.1038/s41598-020-74399-w
PMC
PMC7864942
OpenAlex
W3127597934
Cited by
732
References
57
Citations by year
2026: 113 · 2025: 181 · 2024: 162 · 2023: 141 · 2022: 108 · 2021: 27
FWCI
78.57
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • U.S. Department of Health & Human Services | NIH | National Center for Advancing Translational Sciences (UL1TR000454)
  • Microsoft Research
  • Hewlett Packard | HP Labs
  • U.S. Department of Health & Human Services | NIH | Center for Information Technology (UL1TR000454)
Topics
  • Health Sciences › Medicine › Psychiatry and Mental health › Dementia and Cognitive Impairment Research
  • Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Alzheimer's disease research and treatments
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
Alzheimer's disease, mild cognitive impairment, disease staging, multimodal fusion, magnetic resonance imaging, single nucleotide polymorphisms, genetic data, stacked denoising autoencoder, 3D convolutional neural networks, feature interpretation, hippocampus, amygdala, Rey Auditory Verbal Learning Test, ADNI
MeSH terms
Deep Learning, Alzheimer Disease, Diagnosis, Computer-Assisted, Humans, Early Diagnosis, Data Mining

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review