CoChat
Sign Up
Cited by 223Dementia and Cognitive Impairment ResearchMachine Learning in HealthcareArtificial Intelligence in Healthcare

The DOI matches its Crossref record; 64 of the 66 references it cites were found in a registry or an index.

Machine learning for comprehensive forecasting of Alzheimer’s Disease progression

Open access
Fisher, Charles K., Smith, Aaron M., Walsh, Jonathan R., Coalition Against Major Diseases, Simon, Adam J., Edgar, Chris, Jack, Clifford R., Holtzman, David, Russell, David, Hill, Derek, Grosset, Donald, Wood, Fred, Vanderstichele, Hugo, Morris, John, Blennow, Kaj, Marek, Ken, Shaw, Leslie M, Albert, Marilyn, Weiner, Michael, Fox, Nick, Aisen, Paul, Cole, Patricia E., Petersen, Ronald, Sherer, Todd, Kubick, Wayne, Abbott, Alliance for Aging Research, Alzheimer’s Association, Alzheimer’s Foundation of America, AstraZeneca Pharmaceuticals LP, Bristol-Myers Squibb Company, Critical Path Institute, CHDI Foundation, Inc., Eli Lilly and Company, F. Hoffmann-La Roche Ltd, Forest Research Institute, Genentech, Inc., GlaxoSmithKline, Johnson & Johnson, National Health Council, Novartis Pharmaceuticals Corporation, Parkinson’s Action Network, Parkinson’s Disease Foundation, Pfizer, Inc., sanofi-aventis. Collaborating Organiza·2019·Scientific Reports
DOI 10.1038/s41598-019-49656-2PMC PMC6754403
Open this paper in CoChat — free accountOpen PDF

Abstract

Most approaches to machine learning from electronic health data can only predict a single endpoint. The ability to simultaneously simulate dozens of patient characteristics is a crucial step towards personalized medicine for Alzheimer’s Disease. Here, we use an unsupervised machine learning model called a Conditional Restricted Boltzmann Machine (CRBM) to simulate detailed patient trajectories. We use data comprising 18-month trajectories of 44 clinical variables from 1909 patients with Mild Cognitive Impairment or Alzheimer’s Disease to train a model for personalized forecasting of disease progression. We simulate synthetic patient data including the evolution of each sub-component of cognitive exams, laboratory tests, and their associations with baseline clinical characteristics. Synthetic patient data generated by the CRBM accurately reflect the means, standard deviations, and correlations of each variable over time to the extent that synthetic data cannot be distinguished from actual data by a logistic regression. Moreover, our unsupervised model predicts changes in total ADAS-Cog scores with the same accuracy as specifically trained supervised models, additionally capturing the correlation structure in the components of ADAS-Cog, and identifies sub-components associated with word recall as predictive of progression.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-019-49656-2

Cites

64 of 64 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Overview of the data and model. (A) Study data built from the CAMD database consists of 18-month longitudinal trajectories of 1909 patients with MCI or AD. Our model uses 44 variables, including the individual components of the ADAS-Cog and MMSE scores, laboratory tests, and background information. (B) To capture time dependence, we model the joint distribution of the data at time t + 1 and the data at time t using a Conditional Restricted Boltzmann Machine (CRBM) with ReLU hidden units. Multimodal observations are modeled with different types of units in the visible layer and missing observations are automatically imputed.
Figure 1 Overview of the data and model. (A) Study data built from the CAMD database consists of 18-month longitudinal trajectories of 1909 patients with MCI or AD. Our model uses 44 variables, including the individual components of the ADAS-Cog and MMSE scores, laboratory tests, and background information. (B) To capture time dependence, we model the joint distribution of the data at time t + 1 and the data at time t using a Conditional Restricted Boltzmann Machine (CRBM) with ReLU hidden units. Multimodal observations are modeled with different types of units in the visible layer and missing observations are automatically imputed.CC BY · PubMed Central
Figure 2 The model accurately simulates individual patient trajectories. The mean and variance over all patients of the per-patient z-score distribution is shown for every time-dependent variable (except dropout) at every time point beyond baseline. Results are shown for all CV models. For each patient, the z-score is calculated from 100 simulations of that patient conditioned on the baseline data (the synthetic subjects are of type (i) described above). The first (mean) and second (standard deviation) moments of the distribution of z-scores over all patients for each variable and time point is computed and displayed with a point (mean) and error bar (standard deviation). Under the assumption that a variable is normal, this z-score distribution should be a standard normal, with mean 0 and standard deviation 1. For each variable at each time point, we use the Kolmogorov-Smirnov test to evaluate whether the mean and standard deviation of the z-score distribution are significantly different from a standard normal. Any significantly different (p < 0.05) cases remaining so after a Bonferroni correction are labeled in red.
Figure 2 The model accurately simulates individual patient trajectories. The mean and variance over all patients of the per-patient z-score distribution is shown for every time-dependent variable (except dropout) at every time point beyond baseline. Results are shown for all CV models. For each patient, the z-score is calculated from 100 simulations of that patient conditioned on the baseline data (the synthetic subjects are of type (i) described above). The first (mean) and second (standard deviation) moments of the distribution of z-scores over all patients for each variable and time point is computed and displayed with a point (mean) and error bar (standard deviation). Under the assumption that a variable is normal, this z-score distribution should be a standard normal, with mean 0 and standard deviation 1. For each variable at each time point, we use the Kolmogorov-Smirnov test to evaluate whether the mean and standard deviation of the z-score distribution are significantly different from a standard normal. Any significantly different (p < 0.05) cases remaining so after a Bonferroni correction are labeled in red.CC BY · PubMed Central
Figure 3 The CRBM models correlations well. (A) Correlations between variables as predicted by the model (below the diagonal) and calculated from the data (above the diagonal). Components of the cognitive scores are strongly correlated with each other, but not with other clinical data. (B) Scatterplot of observed vs predicted correlations for each time point, over all times. (C) Scatterplot of observed vs predicted autocorrelations with time lag of 3 months. (D) Scatterplot of observed vs predicted autocorrelations with time lag of 6 months. The color gradient in (B–D) represents the fraction of observations for which the variables used to compute the correlation were present; lighter colors mean more of the data was missing. In all cases, synthetic patients conditioned on baseline data from actual patients is used (synthetic patients of type (i) above). In (A), the correlation coefficients shown are averaged over the 5 CV models. In (B–D), the correlation coefficients for each of the 5 CV folds are shown, and the R2 values shown are the mean and standard deviation over the 5 CV folds, computed from a least squares fit weighted by the fraction of data present when computing the correlations. In all cases the correlations for data are only computed on samples for which the relevant variables are both present (i.e., missing data is ignored).
Figure 3 The CRBM models correlations well. (A) Correlations between variables as predicted by the model (below the diagonal) and calculated from the data (above the diagonal). Components of the cognitive scores are strongly correlated with each other, but not with other clinical data. (B) Scatterplot of observed vs predicted correlations for each time point, over all times. (C) Scatterplot of observed vs predicted autocorrelations with time lag of 3 months. (D) Scatterplot of observed vs predicted autocorrelations with time lag of 6 months. The color gradient in (B–D) represents the fraction of observations for which the variables used to compute the correlation were present; lighter colors mean more of the data was missing. In all cases, synthetic patients conditioned on baseline data from actual patients is used (synthetic patients of type (i) above). In (A), the correlation coefficients shown are averaged over the 5 CV models. In (B–D), the correlation coefficients for each of the 5 CV folds are shown, and the R2 values shown are the mean and standard deviation over the 5 CV folds, computed from a least squares fit weighted by the fraction of data present when computing the correlations. In all cases the correlations for data are only computed on samples for which the relevant variables are both present (i.e., missing data is ignored).CC BY · PubMed Central
Figure 4 Synthetic data is challenging to distinguish from real data. The AUC of a logistic regression model trained at each time point to distinguish between real and synthetic data is shown. At each time point a dataset is formed comprised of real and synthetic patient data in which the baseline data for each group is the same. A logistic regression model is trained to separate these two groups, and the performance using the AUC metric is estimated with 5-fold cross validation. This procedure is repeated many (100) times and the average AUC is shown along with the standard deviation. Finally, this entire method is repeated for each CV fold (which are all shown). To handle missing data mean imputation is used, with the corresponding entries in the synthetic data also assigned the same values. The performance of the logistic regression at each time point is consistent with statistically indistinguishable real and synthetic data.
Figure 4 Synthetic data is challenging to distinguish from real data. The AUC of a logistic regression model trained at each time point to distinguish between real and synthetic data is shown. At each time point a dataset is formed comprised of real and synthetic patient data in which the baseline data for each group is the same. A logistic regression model is trained to separate these two groups, and the performance using the AUC metric is estimated with 5-fold cross validation. This procedure is repeated many (100) times and the average AUC is shown along with the standard deviation. Finally, this entire method is repeated for each CV fold (which are all shown). To handle missing data mean imputation is used, with the corresponding entries in the synthetic data also assigned the same values. The performance of the logistic regression at each time point is consistent with statistically indistinguishable real and synthetic data.CC BY · PubMed Central
Figure 5 The model accurately forecasts across variables. Relative errors of the model (CRBM) and a random forest (RF) specifically trained to predict the value of a single variable at a single time point. The root mean square (RMS) errors are scaled by the standard deviation of the data to be predicted. The mean (top number) and standard deviation (bottom number, in parentheses) of these scaled errors over the 5 CV folds are shown. Predictions are shown for every time-dependent variable except dropout. At each time point and for each variable, the better of the random forest and CRBM predictions is shown in bold.
Figure 5 The model accurately forecasts across variables. Relative errors of the model (CRBM) and a random forest (RF) specifically trained to predict the value of a single variable at a single time point. The root mean square (RMS) errors are scaled by the standard deviation of the data to be predicted. The mean (top number) and standard deviation (bottom number, in parentheses) of these scaled errors over the 5 CV folds are shown. Predictions are shown for every time-dependent variable except dropout. At each time point and for each variable, the better of the random forest and CRBM predictions is shown in bold.CC BY · PubMed Central
Figure 6 The model accurately captures statistics of variables for which supervised methods do not. (A) The covariance values between the model (CRBM) and the random forests (RF) compared to the data. Theil-Sen estimators for the slope of the covariance values in the data relative to the CRBM and RF are shown. Large covariance values are off-scale and not shown on the figure. (B) The distribution of ADAS-Cog scores for the data, the model (CRBM), and the random forests (RF). The random forest models shown are the same models trained to predict single variables at single time points, shown in Fig. 5. The CRBM is conditioned on the baseline data and simulates the 18-month data, while the RF models predict the 18-month data from the baseline data. In both cases the CRBM accurately captures the statistics of the data, while the RF under-predicts the covariance values and the extent of the ADAS-Cog distribution. In both (A) and (B), the data from all 5 CV folds are shown together, and in (A) the Theil-Sen slope is computed on this combined dataset. The errors in the Theil-Sen slope and the error bars in (B) are standard deviations across the 5 CV folds.
Figure 6 The model accurately captures statistics of variables for which supervised methods do not. (A) The covariance values between the model (CRBM) and the random forests (RF) compared to the data. Theil-Sen estimators for the slope of the covariance values in the data relative to the CRBM and RF are shown. Large covariance values are off-scale and not shown on the figure. (B) The distribution of ADAS-Cog scores for the data, the model (CRBM), and the random forests (RF). The random forest models shown are the same models trained to predict single variables at single time points, shown in Fig. 5. The CRBM is conditioned on the baseline data and simulates the 18-month data, while the RF models predict the 18-month data from the baseline data. In both cases the CRBM accurately captures the statistics of the data, while the RF under-predicts the covariance values and the extent of the ADAS-Cog distribution. In both (A) and (B), the data from all 5 CV folds are shown together, and in (A) the Theil-Sen slope is computed on this combined dataset. The errors in the Theil-Sen slope and the error bars in (B) are standard deviations across the 5 CV folds.CC BY · PubMed Central
Figure 7 The model accurately forecasts progression and allows for interpretation. (A) Violin plot of the ADAS-Cog score over time computed from the data and the model. The data from all 5 CV folds are shown together. (B) Out-of-sample predictive accuracy for the change in ADAS-Cog score from baseline (i.e., t = 0) for different study durations. Separate neural network, random forest, and linear regression models were trained to predict the change in ADAS-Cog score from baseline for each study duration. The points (errors) are the means (standard deviations) over the 5 CV folds. (C) We created a simulated patient population with MCI and an initial ADAS-Cog score of 10, and simulated the evolution of each synthetic patient for 18 months. The 5% of synthetic patients with the largest ADAS-Cog score increase were designated “fast progressors” and the bottom 5% of patients with the smallest ADAS-Cog score increase were designated “slow progressors”. Differences between the fast and slow progressors (the “absolute effect size”) were quantified using the absolute value of Cohen’s d-statistic, which measures the mean difference divided by a pooled standard deviation47. The average effect size over the 5 CV folds is shown for each variable.
Figure 7 The model accurately forecasts progression and allows for interpretation. (A) Violin plot of the ADAS-Cog score over time computed from the data and the model. The data from all 5 CV folds are shown together. (B) Out-of-sample predictive accuracy for the change in ADAS-Cog score from baseline (i.e., t = 0) for different study durations. Separate neural network, random forest, and linear regression models were trained to predict the change in ADAS-Cog score from baseline for each study duration. The points (errors) are the means (standard deviations) over the 5 CV folds. (C) We created a simulated patient population with MCI and an initial ADAS-Cog score of 10, and simulated the evolution of each synthetic patient for 18 months. The 5% of synthetic patients with the largest ADAS-Cog score increase were designated “fast progressors” and the bottom 5% of patients with the smallest ADAS-Cog score increase were designated “slow progressors”. Differences between the fast and slow progressors (the “absolute effect size”) were quantified using the absolute value of Cohen’s d-statistic, which measures the mean difference divided by a pooled standard deviation47. The average effect size over the 5 CV folds is shown for each variable.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 66 references

Research works printed but not linked: 2

  1. th International Symposium on, 1015-1018 (IEEE, 2014).
  2. Dankar, F. K. & El Emam, K. Practicing differential privacy in health care: A review. Transactions on Data Priv. 6, 35–67 (2013).

Cited by

Sorted by citations

See all 223

Related

Metadata

Authors
Fisher, C. K., Smith, A. M., Walsh, J. R., Diseases, C. A. M., Simon, A. J., Edgar, C., Jack, C. R., Holtzman, D., Russell, D., Hill, D., Grosset, D., Wood, F., Vanderstichele, H., Morris, J., Blennow, K., Marek, K., Shaw, L. M., Albert, M., Weiner, M., Fox, N., Aisen, P., Cole, P. E., Petersen, R., Sherer, T., Kubick, W., & Abbott, A. F. A. R.
Affiliations
  • Mayo Clinic (US)
  • Washington University in St. Louis (US)
  • Institute for Neurodegenerative Disorders (US)
  • University of Glasgow (GB)
  • Fujirebio (Belgium) (BE)
  • Johns Hopkins University (US)
  • University of California, San Francisco (US)
  • University College London (GB)
  • Michael J. Fox Foundation (US)
Corresponding author
Fisher, Charles K.
ORCID
Venue
Scientific Reports
Publisher
Springer Science and Business Media LLC
Published
2019-09-20
Volume
9
Issue
1
Pages
13622-13622
ISSN
2045-2322 (ISSN-L)
DOI
10.1038/s41598-019-49656-2
PMC
PMC6754403
OpenAlex
W2974138729
Cited by
223
References
66
Citations by year
2026: 22 · 2025: 37 · 2024: 30 · 2023: 39 · 2022: 33 · 2021: 27 · 2020: 32 · 2019: 3
FWCI
18.92
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Topics
  • Health Sciences › Medicine › Psychiatry and Mental health › Dementia and Cognitive Impairment Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
Alzheimer's disease progression, mild cognitive impairment, conditional restricted Boltzmann machine, ADAS-Cog, word recall, cognitive decline prediction
MeSH terms
Machine Learning, Aged, Aged, 80 and over, Alzheimer Disease, Cognition, Female, Forecasting, Humans, Male, Middle Aged, Models, Theoretical, Neuropsychological Tests, Models, Statistical, Disease Progression, Cognitive Dysfunction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review