CoChat
Sign Up
Cited by 229Machine Learning in HealthcareMental Health via WritingArtificial Intelligence in Healthcare

The DOI matches its Crossref record; 55 of the 56 references it cites were found in a registry or an index.

Machine learning model to predict mental health crises from electronic health records

Open access
Garriga, Roger, Mas, Javier, Abraha, Semhar, Nolan, Jon, Harrison, Oliver, Tadros, George, Matic, Aleksandar·2022·Nature Medicine
DOI 10.1038/s41591-022-01811-5PMC PMC9205775

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

The timely identification of patients who are at risk of a mental health crisis can lead to improved outcomes and to the mitigation of burdens and costs. However, the high prevalence of mental health problems means that the manual review of complex patient records to make proactive care decisions is not feasible in practice. Therefore, we developed a machine learning model that uses electronic health records to continuously monitor patients for risk of a mental health crisis over a period of 28 days. The model achieves an area under the receiver operating characteristic curve of 0.797 and an area under the precision-recall curve of 0.159, predicting crises with a sensitivity of 58% at a specificity of 85%. A follow-up 6-month prospective study evaluated our algorithm's use in clinical practice and observed predictions to be clinically valuable in terms of either managing caseloads or mitigating the risk of crisis in 64% of cases. To our knowledge, this study is the first to continuously predict the risk of a wide range of mental health crises and to explore the added value of such predictions in clinical practice.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41591-022-01811-5

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Crisis episode example. Example of a crisis episode timeline: crisis onset is the first crisis event of a crisis episode that follows a stable week (that is, a week without crisis events).
Fig. 1 Crisis episode example. Example of a crisis episode timeline: crisis onset is the first crisis event of a crisis episode that follows a stable week (that is, a week without crisis events).CC BY · PubMed Central
Fig. 2 System diagram. Time series of events are represented with the timestamps and event characteristics in different SQL tables in the hospital’s database. These tables are processed and converted into features for the modeling task. Models are trained, tuned and selected based on the data for the period 2012–2019. The system predicts the risk of crisis onset within the next 28 days (whereby the algorithm is queried every week for every patient). The patients with the highest predicted risk are displayed on the dashboard delivered to clinicians alongside key indicators, patient notes and a questionnaire form about each patient, which the clinician fills out. The icons in this figure were made by Freepik from www.flaticon.com. IP, inpatient.
Fig. 2 System diagram. Time series of events are represented with the timestamps and event characteristics in different SQL tables in the hospital’s database. These tables are processed and converted into features for the modeling task. Models are trained, tuned and selected based on the data for the period 2012–2019. The system predicts the risk of crisis onset within the next 28 days (whereby the algorithm is queried every week for every patient). The patients with the highest predicted risk are displayed on the dashboard delivered to clinicians alongside key indicators, patient notes and a questionnaire form about each patient, which the clinician fills out. The icons in this figure were made by Freepik from www.flaticon.com. IP, inpatient.CC BY · PubMed Central
Fig. 3 Final model performance. a, ROC curve for the crisis prediction task. Comparison among the proposed final model (XGBoost general), a proposed diagnosis-specific model (XGBoost per diagnosis) and two baseline models. The solid lines and lighter-colored envelopes around each line were derived from the test evaluations (n = 25) as the mean and 95% CI, respectively. b, Precision-recall curve for the crisis prediction task with the same characteristics as a. c, Box plot of the AUROC curve evaluated per diagnosis. Comparison among the four models considered as in a and b. The solid line corresponds to the median value; the box limits correspond to the first Q1 (left limit) and third Q3 (right limit) quartiles; the whiskers denote the rest of the distribution range from Q1–1.5 (Q3–Q1) (left whisker) to Q3 + 1.5 (Q3–Q1) (right whisker); and the points displayed correspond to the outliers. d–f, AUROC curve evaluated for different subsets of the study cohort based on age group (d), time since the patient first visited the hospital (e) and time since the patient’s last crisis episode (f). The dots and bars derive from the test evaluations (n = 25) as the mean and 95% CI, respectively.
Fig. 3 Final model performance. a, ROC curve for the crisis prediction task. Comparison among the proposed final model (XGBoost general), a proposed diagnosis-specific model (XGBoost per diagnosis) and two baseline models. The solid lines and lighter-colored envelopes around each line were derived from the test evaluations (n = 25) as the mean and 95% CI, respectively. b, Precision-recall curve for the crisis prediction task with the same characteristics as a. c, Box plot of the AUROC curve evaluated per diagnosis. Comparison among the four models considered as in a and b. The solid line corresponds to the median value; the box limits correspond to the first Q1 (left limit) and third Q3 (right limit) quartiles; the whiskers denote the rest of the distribution range from Q1–1.5 (Q3–Q1) (left whisker) to Q3 + 1.5 (Q3–Q1) (right whisker); and the points displayed correspond to the outliers. d–f, AUROC curve evaluated for different subsets of the study cohort based on age group (d), time since the patient first visited the hospital (e) and time since the patient’s last crisis episode (f). The dots and bars derive from the test evaluations (n = 25) as the mean and 95% CI, respectively.CC BY · PubMed Central
Fig. 4 Most predictive features. a, Complete distribution of the SHAP values for the top 20 features based on the highest mean absolute SHAP value. Each sample of the test set is represented as a data point per feature, and the x axis shows the positive or negative effect on the model’s prediction of the feature. The color coding depicts the value of the feature and is scaled independently based on the range observed in the data. b, Absolute feature contribution of the 20 features with the highest mean absolute SHAP value. c–h, Six examples of dependence plots showing the effect on the PRS with respect to the feature value. Each data point (n = 371,010) represents a sample in the test set, with the solid lines and the lighter-colored envelopes representing, respectively, the mean effect and its standard deviation per feature value. The variability at each feature value corresponds to interaction with the rest of the features. Missing values (representing the absence of events) are colored gray. GP, general practitioner; MHA, mental health act; std, standard deviation.
Fig. 4 Most predictive features. a, Complete distribution of the SHAP values for the top 20 features based on the highest mean absolute SHAP value. Each sample of the test set is represented as a data point per feature, and the x axis shows the positive or negative effect on the model’s prediction of the feature. The color coding depicts the value of the feature and is scaled independently based on the range observed in the data. b, Absolute feature contribution of the 20 features with the highest mean absolute SHAP value. c–h, Six examples of dependence plots showing the effect on the PRS with respect to the feature value. Each data point (n = 371,010) represents a sample in the test set, with the solid lines and the lighter-colored envelopes representing, respectively, the mean effect and its standard deviation per feature value. The variability at each feature value corresponds to interaction with the rest of the features. Missing values (representing the absence of events) are colored gray. GP, general practitioner; MHA, mental health act; std, standard deviation.CC BY · PubMed Central
Extended Data Fig. 1 Demographics and patient’s characteristics. Summary of the retrospective cohort per gender, age group, ethnic group, marital status and primary diagnosed disorder category; including the number and percentage of patients, crisis episodes in train and test per each group category. No major differences in the distribution of crisis episodes in train and test were observed between group categories.
Extended Data Fig. 1 Demographics and patient’s characteristics. Summary of the retrospective cohort per gender, age group, ethnic group, marital status and primary diagnosed disorder category; including the number and percentage of patients, crisis episodes in train and test per each group category. No major differences in the distribution of crisis episodes in train and test were observed between group categories.CC BY · PubMed Central
Extended Data Fig. 2 Prevalence of the target variable (a crisis episode within the next 28 days) per disorder type. Prevalence of the target variable for different disorders. Adult personality and behaviour (F6 in ICD-10 categorisation) and Psychoactive substance use (F1 in ICD-10 categorisation) show a slightly greater prevalence of crisis episodes, whereas the prevalence was lower for Organic including symptomatic mental disorders (F0 in ICD-10 categorisation) and Not diagnosed patients. A small difference was observed between train and test, with a lower prevalence in the test set overall.
Extended Data Fig. 2 Prevalence of the target variable (a crisis episode within the next 28 days) per disorder type. Prevalence of the target variable for different disorders. Adult personality and behaviour (F6 in ICD-10 categorisation) and Psychoactive substance use (F1 in ICD-10 categorisation) show a slightly greater prevalence of crisis episodes, whereas the prevalence was lower for Organic including symptomatic mental disorders (F0 in ICD-10 categorisation) and Not diagnosed patients. A small difference was observed between train and test, with a lower prevalence in the test set overall.CC BY · PubMed Central
Extended Data Fig. 3 Evaluation of multiple Machine Learning models to predict the risk of crisis onset during the following 28 days. Values in bold denote the model with the highest performance.
Extended Data Fig. 3 Evaluation of multiple Machine Learning models to predict the risk of crisis onset during the following 28 days. Values in bold denote the model with the highest performance.CC BY · PubMed Central
Extended Data Fig. 4 Statistical significance analysis comparing the AUROC of XGBoost to the other models. Statistical significance analysis was done using the Mann-Whitney U test. The two-stage step-up method of Benjamini, Krieger and Yekutieli was used to correct the p-values of the multiple tests performed.
Extended Data Fig. 4 Statistical significance analysis comparing the AUROC of XGBoost to the other models. Statistical significance analysis was done using the Mann-Whitney U test. The two-stage step-up method of Benjamini, Krieger and Yekutieli was used to correct the p-values of the multiple tests performed.CC BY · PubMed Central
Extended Data Fig. 5 Features used in each baseline.
Extended Data Fig. 5 Features used in each baseline.CC BY · PubMed Central
Extended Data Fig. 6 Model calibration and net benefit. (a), (b) Calibration curves of the general model (XGBoost general) and diagnosis specific models (XGBoost per diagnosis). Yellow and blue lines represent the non calibrated and calibrated curves for both models, respectively. The diagonal dotted line shows the ideal calibration reference curve. (c) Decision curve shows the net benefit versus the threshold probability, for the proposed models and baselines. The general model (XGBoost general) outperforms the baselines and the diagnosis specific model (XGBoost per diagnosis) at all thresholds. The solid lines and lighter-coloured envelopes around each line were derived from the test evaluations (n = 25) as the mean and 95% confidence interval respectively.
Extended Data Fig. 6 Model calibration and net benefit. (a), (b) Calibration curves of the general model (XGBoost general) and diagnosis specific models (XGBoost per diagnosis). Yellow and blue lines represent the non calibrated and calibrated curves for both models, respectively. The diagonal dotted line shows the ideal calibration reference curve. (c) Decision curve shows the net benefit versus the threshold probability, for the proposed models and baselines. The general model (XGBoost general) outperforms the baselines and the diagnosis specific model (XGBoost per diagnosis) at all thresholds. The solid lines and lighter-coloured envelopes around each line were derived from the test evaluations (n = 25) as the mean and 95% confidence interval respectively.CC BY · PubMed Central
Extended Data Fig. 7 Model performance per diagnosis. (a) Box-plot of the average precision (area under the precision recall curve) evaluated per diagnosis. Comparison between the final model (XGBoost general), a diagnosis specific model (XGBoost per diagnosis) and two baseline models. Dotted line marks the mean average precision of the general model (n=25).(b), (c) Box-plot of the precision and recall respectively per diagnosis with a threshold corresponding to 15% of false positive rate (obtained with evaluationson patients in the test set (n=25)). (c) Box-plot of the recall per diagnosis with a threshold corresponding to 15% of false positive rate. In all Box-plots the solid line corresponds to the median value, the box limits to the first Q1 (left limit) and third (right limit) quartiles, the whiskers denote the rest of the distribution range from Q1-1.5(Q3-Q1) (left whisker) to Q3+1.5(Q3-Q1) (right whisker) and the points displayed correspond to the outliers.
Extended Data Fig. 7 Model performance per diagnosis. (a) Box-plot of the average precision (area under the precision recall curve) evaluated per diagnosis. Comparison between the final model (XGBoost general), a diagnosis specific model (XGBoost per diagnosis) and two baseline models. Dotted line marks the mean average precision of the general model (n=25).(b), (c) Box-plot of the precision and recall respectively per diagnosis with a threshold corresponding to 15% of false positive rate (obtained with evaluationson patients in the test set (n=25)). (c) Box-plot of the recall per diagnosis with a threshold corresponding to 15% of false positive rate. In all Box-plots the solid line corresponds to the median value, the box limits to the first Q1 (left limit) and third (right limit) quartiles, the whiskers denote the rest of the distribution range from Q1-1.5(Q3-Q1) (left whisker) to Q3+1.5(Q3-Q1) (right whisker) and the points displayed correspond to the outliers.CC BY · PubMed Central
Extended Data Fig. 8 Precision and recall per cohort. Precision evaluated with respect to (a) different age groups; (b) time since the first hospital visit; (c) time since the last hospital visit; with a threshold corresponding to 15% of false positive rate. Recall evaluated with respect to (d) different age groups; (e) time since the first hospital visit; (f) time since the last visit; with a threshold corresponding to 15% of false positive rate obtained with evaluations in the test set (n=25). The dots and bars were derived from the test evaluations (n = 25) as the mean and 95% confidence interval respectively.
Extended Data Fig. 8 Precision and recall per cohort. Precision evaluated with respect to (a) different age groups; (b) time since the first hospital visit; (c) time since the last hospital visit; with a threshold corresponding to 15% of false positive rate. Recall evaluated with respect to (d) different age groups; (e) time since the first hospital visit; (f) time since the last visit; with a threshold corresponding to 15% of false positive rate obtained with evaluations in the test set (n=25). The dots and bars were derived from the test evaluations (n = 25) as the mean and 95% confidence interval respectively.CC BY · PubMed Central
Extended Data Fig. 9 Examples of features contribution to the predicted risk score. Four representative force plots, depicting how the features contributed to the prediction for four specific data points. From top to bottom: Patient not going to have a crisis during the next four weeks (target=0), the model assigned a prediction value of 0.178. Patient not going to have a crisis during the next four weeks (target=0), the model assigned a prediction value of 0.129. Patient going to have a crisis during the next four weeks (target=1) the model assigned a prediction value of 0.792. Patient going to have a crisis during the next four weeks (target=1) the model assigned a prediction value of 0.725.
Extended Data Fig. 9 Examples of features contribution to the predicted risk score. Four representative force plots, depicting how the features contributed to the prediction for four specific data points. From top to bottom: Patient not going to have a crisis during the next four weeks (target=0), the model assigned a prediction value of 0.178. Patient not going to have a crisis during the next four weeks (target=0), the model assigned a prediction value of 0.129. Patient going to have a crisis during the next four weeks (target=1) the model assigned a prediction value of 0.792. Patient going to have a crisis during the next four weeks (target=1) the model assigned a prediction value of 0.725.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 3

  1. Nielsen, D. Tree Boosting with XGBoost: Why Does XGBoost Win ‘Every’ Machine Learning Competition? Master’s thesis, Norwegian University of Science and Technology (2016).
  2. Bergstra, J., Yamins, D. & Cox, D. Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures. In: Dasgupta, S. & McAllester, D. (eds) Proc. 30th International Conference on Machine Learning, 115–123 (PMLR, 2013).
  3. Bergstra, J. S., Bardenet, R., Bengio, Y. & Kégl, B. Algorithms for hyper-parameter optimization. In: Shawe-Taylor, J., Zemel, R. S., Bartlett, P. L., Pereira, F. & Weinberger, K. Q. (eds) Advances in Neural Information Processing Systems 24, 2546–2554 (Curran Associates, 2011).

Cited by

Sorted by citations

See all 229

Related

See all

Metadata

Authors
Garriga, R., Mas, J., Abraha, S., Nolan, J., Harrison, O., Tadros, G., & Matic, A.
Affiliations
  • Universitat Pompeu Fabra (ES)
  • Birmingham and Solihull Mental Health NHS Foundation Trust (GB)
  • Aston University (GB)
Corresponding author
  • Garriga, Roger
  • Matic, Aleksandar
ORCID
Venue
Nature Medicine
Publisher
Springer Science and Business Media LLC
Published
2022-05-16
Volume
28
Issue
6
Pages
1240-1248
ISSN
  • 1078-8956 (ISSN-L)
  • 1546-170X
DOI
10.1038/s41591-022-01811-5
PMC
PMC9205775
OpenAlex
W4280536523
Cited by
229
References
56
Citations by year
2026: 56 · 2025: 90 · 2024: 46 · 2023: 33 · 2022: 4
FWCI
14.18
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Medicine (publishedVersion, cc-by)
  • Research Square (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Aston Publications Explorer (Aston University) (acceptedVersion)
  • Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
Funders
Health Foundation (709246)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Social Sciences › Psychology › Social Psychology › Mental Health via Writing
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
electronic health records, risk prediction, clinical decision support, continuous patient monitoring, clinical utility, mental health services
MeSH terms
Machine Learning, Humans, Mental Health, Prospective Studies, ROC Curve, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review