Development and Validation of an Electronic Health Record–Based Machine Learning Model to Estimate Delirium Risk in Newly Hospitalized Patients Without Known Cognitive Impairment
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Importance: Current methods for identifying hospitalized patients at increased risk of delirium require nurse-administered questionnaires with moderate accuracy. Objective: To develop and validate a machine learning model that predicts incident delirium risk based on electronic health data available on admission. Design, Setting, and Participants: Retrospective cohort study evaluating 5 machine learning algorithms to predict delirium using 796 clinical variables identified by an expert panel as relevant to delirium prediction and consistently available in electronic health records within 24 hours of admission. The training set comprised 14 227 adult patients with non-intensive care unit hospital stays and no delirium on admission who were discharged between January 1, 2016, and August 31, 2017, from UCSF Health, a large academic health institution. The test set comprised 3996 patients with hospital stays who were discharged between August 1, 2017, and November 30, 2017. Exposures: Patient demographic characteristics, diagnoses, nursing records, laboratory results, and medications available in electronic health records during hospitalization. Main Outcomes and Measures: Delirium was defined as a positive Nursing Delirium Screening Scale or Confusion Assessment Method for the Intensive Care Unit score. Models were assessed using the area under the receiver operating characteristic curve (AUC) and compared against the 4-point scoring system AWOL (age >79 years, failure to spell world backward, disorientation to place, and higher nurse-rated illness severity), a validated delirium risk-assessment tool routinely administered in this cohort. Results: The training set included 14 227 patients (5113 [35.9%] aged >64 years; 7335 [51.6%] female; 687 [4.8%] with delirium), and the test set included 3996 patients (1491 [37.3%] aged >64 years; 1966 [49.2%] female; 191 [4.8%] with delirium). In total, the analysis included 18 223 hospital admissions (6604 [36.2%] aged >64 years; 9301 [51.0%] female; 878 [4.8%] with delirium). The AWOL system achieved a baseline AUC of 0.678. The gradient boosting machine model performed best, with an AUC of 0.855. Setting specificity at 90%, the model had a 59.7% (95% CI, 52.4%-66.7%) sensitivity, 23.1% (95% CI, 20.5%-25.9%) positive predictive value, 97.8% (95% CI, 97.4%-98.1%) negative predictive value, and a number needed to screen of 4.8. Penalized logistic regression and random forest models also performed well, with AUCs of 0.854 and 0.848, respectively. Conclusions and Relevance: Machine learning can be used to estimate hospital-acquired delirium risk using electronic health record data available within 24 hours of hospital admission. Such a model may allow more precise targeting of delirium prevention resources without increasing the burden on health care professionals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data
Quan, H., Sundararajan, V., Halfon, P., Fong, A., Burnand, B., Luthi, J.-C., Saunders, L. D., Beck, C. A., Feasby, T. E., & Ghali, W. A. (2005) · Medical Care · doi:10.1097/01.mlr.0000182534.19832.83 · Cited by 10,669
Inouye, S. K., Westendorp, R. G., & Saczynski, J. S. (2014) · The Lancet · doi:10.1016/s0140-6736(13)60688-1 · Cited by 3,722
Ely, E. W., Inouye, S. K., Bernard, G. R., Gordon, S., Francis, J., May, L., Truman, B., Speroff, T., Gautam, S., Margolin, R., et al. (2001) · JAMA · doi:10.1001/jama.286.21.2703 · Cited by 3,210
Inouye, S. K., Viscoli, C. M., Horwitz, R. I., Hurst, L. D., & Tinetti, M. E. (1993) · Annals of Internal Medicine · doi:10.7326/0003-4819-119-6-199309150-00005 · Cited by 854
Effectiveness of Multicomponent Nonpharmacological Delirium Interventions: A Meta-analysis
Hshieh, T. T., Yue, J., Oh, E., Puelle, M., Dowal, S., Travison, T., & Inouye, S. K. (2015) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2014.7779 · Cited by 801
Pirracchio, R., Petersen, M. L., Carone, M., Rigon, M. R., Chevret, S., & van der Laan, M. J. (2015) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(14)70239-5 · Cited by 374
Mortality Risk Score Prediction in an Elderly Population Using Machine Learning
Rose, S. (2013) · American Journal of Epidemiology · doi:10.1093/aje/kws241 · Cited by 193
Cited by
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
Artificial Intelligence -based technologies in nursing: A scoping literature review of the evidence
von Gerich, H., Moen, H., Block, L. J., Chu, C. H., DeForest, H., Hobensack, M., Michalowski, M., Mitchell, J., Nibber, R., Olalia, M. A., et al. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2021.104153 · Cited by 360
“…, Martins et al., 2016 , Moskowitz et al., 2020 , Safavi et al., 2019 , Setoguchi et al., 2016 , Subramaniam and Dass, 2021 , Unger et al., 2019 , Wong et al., 2018 , Zachariah et al., 2020 16.1 Level V (e.g. qualitative meta-synthesis) 0 Level VI (e.g. descriptive study) Aldaz et al., 2015 ,…”
InfluentialPredictive analytics with gradient boosting in clinical medicine
written on behalf of AME Big-Data Clinical Trial Collaborative Group, Zhang, Z., Zhao, Y., Canes, A., Steinberg, D., & Lyashevska, O. (2019) · Annals of Translational Medicine · doi:10.21037/atm.2019.03.29 · Cited by 312
Shung, D. L., Au, B., Taylor, R. A., Tay, J. K., Laursen, S. B., Stanley, A. J., Dalton, H. R., Ngu, J., Schultz, M., & Laine, L. (2020) · Gastroenterology · doi:10.1053/j.gastro.2019.09.009 · Cited by 227
“47-49 The next steps would be an electronic health record based study that would reliably identify patients presenting with acute UGIB, use structured datafields as predictive variables to develop models based on local patterns of disease and outcomes, and then prospectively validate the models in…”
Lynam, A. L., Dennis, J. M., Owen, K. R., Oram, R. A., Jones, A. G., Shields, B. M., & Ferrat, L. A. (2020) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00075-2 · Cited by 142
Metadata
- Authors
- Wong, A., Young, A. T., Liang, A. S., Gonzales, R., Douglas, V. C., & Hadley, D.
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2018-08-03
- Volume
- 1
- Issue
- 4
- Pages
- e181018
- ISSN
- 2574-3805 (ISSN-L)
- OpenAlex
- W2885408863
- Cited by
- 152
- References
- 8
- FWCI
- 10.97
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Development and Validation of an Electronic Health Record–Based Machine Learning Model to Estimate Delirium Risk in Newly Hospitalized Patients Without Known Cognitive Impairment. https://app.cochat.ai/papers/10.1001/jamanetworkopen.2018.1018
- The paper
- Wong, A., Young, A. T., Liang, A. S., Gonzales, R., Douglas, V. C., & Hadley, D. (2018). Development and Validation of an Electronic Health Record–Based Machine Learning Model to Estimate Delirium Risk in Newly Hospitalized Patients Without Known Cognitive Impairment. JAMA Network Open. https://doi.org/10.1001/jamanetworkopen.2018.1018
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.