CoChat
Sign Up
Cited by 14Machine Learning in HealthcareHealthcare Technology and Patient MonitoringArtificial Intelligence in Healthcare

Utilizing timestamps of longitudinal electronic health record data to classify clinical deterioration events

Open access
Fu, Li-Heng, Knaplund, Chris, Cato, Kenrick, Perotte, Adler, Kang, Min-Jeoung, Dykes, Patricia C, Albers, David, Collins Rossetti, Sarah·2021·Journal of the American Medical Informatics Association
DOI 10.1093/jamia/ocab111PMC PMC8363809
Open this paper in CoChat — free account

Abstract

Objective: To propose an algorithm that utilizes only timestamps of longitudinal electronic health record data to classify clinical deterioration events. Materials and methods: This retrospective study explores the efficacy of machine learning algorithms in classifying clinical deterioration events among patients in intensive care units using sequences of timestamps of vital sign measurements, flowsheets comments, order entries, and nursing notes. We design a data pipeline to partition events into discrete, regular time bins that we refer to as timesteps. Logistic regressions, random forest classifiers, and recurrent neural networks are trained on datasets of different length of timesteps, respectively, against a composite outcome of death, cardiac arrest, and Rapid Response Team calls. Then these models are validated on a holdout dataset. Results: A total of 6720 intensive care unit encounters meet the criteria and the final dataset includes 830 578 timestamps. The gated recurrent unit model utilizes timestamps of vital signs, order entries, flowsheet comments, and nursing notes to achieve the best performance on the time-to-outcome dataset, with an area under the precision-recall curve of 0.101 (0.06, 0.137), a sensitivity of 0.443, and a positive predictive value of 0. 092 at the threshold of 0.6. Discussion and Conclusion: This study demonstrates that our recurrent neural network models using only timestamps of longitudinal electronic health record data that reflect healthcare processes achieve well-performing discriminative power.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocab111

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Through experimentation on retrospective datasets of hospitalized patients' encounters in the EHR, the authors refined their machine learning modeling approach, focused on temporal patterns of nursing observations, using ensemble learning to identify the best performing model for a patient's specific clinical situation for each hour of the day.

    “Through experimentation on retrospective datasets of hospitalized patients’ encounters in the EHR, we refined our machine learning modeling approach, which is focused on temporal patterns of nursing observations, using ensemble learning to identify the best performing model for a patient’s specific clinical situation for each hour of the day58.”

    From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [58]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 53 · Cited by 14

Research works printed but not linked: 48

  1. [1]1. McQuillan P, Pilkington S, Allan A, et al. Confidential inquiry into quality of care before admission to intensive care. BMJ. Clin Res Ed . 1998; 316 (7148): 1853-8.
  2. [2]2. Beaumont K, Luettel D, Thomson R. Deterioration in hospital patients: early signs and appropriate actions. Nurs Stand (through 2013) 2008; 23 (1): 43-8.
  3. [3]3. Hillman KM, Bristow PJ, Chey T, et al. Duration of life-threatening antecedents prior to intensive care admission. Intensive Care Med 2002; 28 (11): 1629-34.
  4. [4]4. Bapoje SR, Gaudiani JL, Narayanan V, Albert RK. Unplanned transfers to a medical intensive care unit: causes and relationship to preventable errors in care. J Hosp Med 2011; 6 (2): 68-72.
  5. [5]5. Goldhill DR, White SA, Sumner A. Physiological values and procedures in the 24 h before ICU admission from the ward. Anaesthesia 1999; 54 (6): 529-34.
  6. [6]6. Schein RM, Hazday N, Pena M, Ruben BH, Sprung CL. Clinical antecedents to in-hospital cardiopulmonary arrest. Chest 1990; 98 (6): 1388-92.
  7. [7]7. Morgan RJMW, F, Wright, MM. An Early Warning Scoring System for detecting developing critical illness. Clin Intens Care 1997; 8: 100.
  8. [8]8. Prytherch DR, Smith GB, Schmidt PE, Featherstone PI. ViEWS-Towards a national early warning score for detecting adult inpatient deterioration. Resuscitation 2010; 81 (8): 932-7.
  9. [9]9. Subbe CP, Kruger M, Rutherford P, Gemmel L. Validation of a modified Early Warning Score in medical admissions. QJM 2001; 94 (10): 521-6.
  10. [10]10. Churpek MM, Yuen TC, Park SY, Gibbons R, Edelson DP. Using electronic health record data to develop and validate a prediction model for adverse outcomes in the wards. Crit Care Med 2014; 42 (4): 841-8.
  11. [11]11. Escobar GJ, LaGuardia JC, Turk BJ, Ragins A, Kipnis P, Draper D. Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record. J Hosp Med 2012; 7 (5): 388-95.
  12. [12]12. Churpek MM, Yuen TC, Park SY, Meltzer DO, Hall JB, Edelson DP. Derivation of a cardiac arrest prediction model using ward vital signs. Crit Care Med 2012; 40 (7): 2102-8.
  13. [14]14. Devita MA, Bellomo R, Hillman K, et al. Findings of the first consensus conference on medical emergency teams. Crit Care Med 2006; 34 (9): 2463-78.
  14. [15]15. Jones DA, DeVita MA, Bellomo R. Rapid-response teams. N Engl J Med 2011; 365 (2): 139-46.
  15. [17]17. charts, of patients admitted to a tertiary referral intensive care unit after CPR. Resuscitation 2011; 82 (2): 150-4.
  16. [18]18. Maharaj R, Raffaele I, Wendon J. Rapid response systems: a systematic review and meta-analysis. Crit Care 2015; 19 (1): 254.
  17. [19]19. Teuma Custo R, Trapani J. The impact of rapid response systems on mortality and cardiac arrests: a literature review. Intensive Crit Care Nurs 2020; 59: 102848.
  18. [20]20. Smith ME, Chiovaro JC, O'Neil M, et al. Early warning system scores for clinical deterioration in hospitalized patients: a systematic review. Ann AmThorac Soc 2014; 11 (9): 1454-65.
  19. [21]21. Fu LH, Schwartz J, Moy A, et al. Development and validation of early warning score system: a systematic literature review. J Biomed Inform 2020; 105: 103410.
  20. [22]22. Rothman MJ, Rothman SI, Beals J. Development and validation of a continuous measure of patient condition using the electronic medical record. J Biomed Inform 2013; 46 (5): 837-48.
  21. [23]23. Rapid Response Team composition, resourcing and calling criteria in Australia. Resuscitation 2012; 83 (5): 563-7.
  22. [24]24. McGaughey J, O'Halloran P, Porter S, Blackwood B. Early warning systems and rapid response to the deteriorating patient in hospital: A systematic realist review. J Adv Nurs 2017; 73 (12): 2877-91.
  23. [25]25. Collins SA, Vawdrey DK. 'Reading between the lines' of flow sheet data: nurses' optional documentation associated with cardiac arrest outcomes. Appl Nurs Res 2012; 25 (4): 251-7.
  24. [26]26. Odell M, Victor C, Oliver D. Nurses' role in detecting deterioration in ward patients: systematic literature review. J Adv Nurs 2009; 65 (10): 1992-2006.
  25. [27]27. Collins SA, Cato K, Albers D, et al. Relationship between nursing documentation and patients' mortality.. AmJCrit Care 2013; 22 (4): 306-13.
  26. [28]28. Collins SA, Fred M, Wilcox L, Vawdrey DK. Workarounds used by nurses to overcome design constraints of electronic health records. In: proceedings of the 11th International Congress on Nursing Informatics ; June 2327, 2012; Montreal, Canada.
  27. [29]29. Douw G, Schoonhoven L, Holwerda T, et al. Nurses' worry or concern and early recognition of deteriorating patients on general wards in acute care hospitals: a systematic review. Crit Care 2015; 19 (1): 230.
  28. [30]30. Hripcsak G, Albers DJ. Next-generation phenotyping of electronic health records. J Am Med Inform Assoc 2013; 20 (1): 117-21.
  29. [32]32. Rossetti SC, Knaplund C, Albers D, et al. Leveraging clinical expertise as a feature - not an outcome - of predictive models: evaluation of an early warning system use case. AMIA Annu Symp Proc 2019; 2019: 323-332.
  30. [34]34. Chung J, Gulcehre C, Cho K, Bengio Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555; 2014. · arXiv
  31. [35]35. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput 1997; 9 (8): 1735-80.
  32. [36]36. Tibshirani R. Regression shrinkage and selection via the Lasso. J Roy Stat Soc Ser B (Methodol) 1996; 58 (1): 267-88.
  33. [37]37. Hoerl AE, Kennard RW. Ridge regression: biased estimation for nonorthogonal problems. Technometrics 1970; 12 (1): 55-67.
  34. [38]38. Li L, Jamieson K, DeSalvo G, Rostamizadeh A, Talwalkar A. Hyperband: a novel bandit-based approach to hyperparameter optimization. J Mach Learn Res 2017; 18 (1): 6765-816.
  35. [39]39. Dziadzko MA, Novotny PJ, Sloan J, et al. Multicenter derivation and validation of an early warning score for acute respiratory failure or death in the hospital. Crit Care 2018; 22 (1): 286.
  36. [40]40. Kipnis P, Turk BJ, Wulf DA, et al. Development and validation of an electronic medical record-based alert score for detection of inpatient deterioration outside the ICU. J Biomed Inform 2016; 64: 10-9.
  37. [41]41. Alvarez CA, Clark CA, Zhang S, et al. Predicting out of intensive care unit cardiopulmonary arrest or death using electronic medical record data. BMCMedInform Decis Mak 2013; 13: 28.
  38. [42]42. Collins SA, Fred M, Wilcox L, Vawdrey DK. Workarounds used by nurses to overcome design constraints of electronic health records. Ni 2012 (2012) 2012; 2012: 93.
  39. [43]43. Ibrahim JG, Molenberghs G. Missing data methods in longitudinal studies: a review. Test (Madr) 2009; 18 (1): 1-43.
  40. [44]44. Churpek MM, Yuen TC, Winslow C, Meltzer DO, Kattan MW, Edelson DP. Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards. Crit Care Med 2016; 44 (2): 368-74.
  41. [45]45. Churpek MM, Adhikari R, Edelson DP. The value of vital sign trends for detecting clinical deterioration on the wards. Resuscitation 2016; 102: 1-5. doi: 10.1016/j.resuscitation.2016.02.005[published Online First: Epub Date] j . · DOI
  42. [46]46. Churpek MM, Yuen TC, Winslow C, et al. Multicenter development and validation of a risk stratification tool for ward patients. Am J Respir Crit Care Med 2014; 190 (6): 649-55.
  43. [47]47. Harutyunyan H, Khachatrian H, Kale DC, Ver Steeg G, Galstyan A. Multitask learning and benchmarking with clinical time series data. Sci Data 2019; 6 (1): 96.
  44. [48]48. Ge W, Huh JW, Park YR, Lee JH, Kim YH, Turchin A. An interpretable ICU mortality prediction model based on logistic regression and recurrent neural networks with LSTM units. AMIA Annu Symp Proc 2018; 2018: 460-9.
  45. [49]49. Che Z, Purushotham S, Cho K, Sontag D, Liu Y. Recurrent neural networks for multivariate time series with missing values. Sci Rep 2018; 8 (1): 6085.
  46. [50]50. Lipton ZC, Kale DC, Wetzel R. Modeling missing data in clinical time series with rnns. arXiv preprint arXiv:1606.04130; 2016. · arXiv
  47. [51]51. Hripcsak G, Albers DJ. High-fidelity phenotyping: richness and freedom from bias. J Am Med Inform Assoc 2018; 25 (3): 289-94.
  48. [52]52. Agniel D, Kohane IS, Weber GM. Biases in electronic health record data due to processes within the healthcare system: retrospective observational study. BMJ (Clin Res Ed) 2018; 361: k1479.

Authors
Fu, L.-H., Knaplund, C., Cato, K., Perotte, A., Kang, M.-J., Dykes, P. C., Albers, D., & Collins Rossetti, S.
Affiliations
  • Columbia University (US)
  • Department of Biomedical Informatics, Columbia University, New York, New York, USA
  • Catholic University of Korea (KR)
  • Brigham and Women's Hospital (US)
  • University of Colorado Anschutz (US)
Corresponding author
Fu, Li-Heng
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2021-07-16
Volume
28
Issue
9
Pages
1955-1963
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1093/jamia/ocab111
PMC
PMC8363809
OpenAlex
W3177640789
Cited by
14
References
53
Citations by year
2026: 2 · 2025: 4 · 2024: 4 · 2023: 1 · 2022: 2 · 2021: 1
FWCI
0.70
Type
journal article
Language
en
Open access
Licence
academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
Locations
  • Journal of the American Medical Informatics Association (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
National Institute for Nursing Research (NINR) funded CONCERN Study (1R01NR016941)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
clinical deterioration detection, intensive care unit, recurrent neural networks, gated recurrent unit, vital signs, nursing notes, order entry, cardiac arrest, mortality
MeSH terms
Machine Learning, Clinical Deterioration, Humans, Retrospective Studies, Vital Signs, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review