CoChat
Sign Up
Cited by 133Electronic Health Records SystemsMachine Learning in HealthcareArtificial Intelligence in Healthcare and Education

Clinical Implementation of Predictive Models Embedded within Electronic Health Record Systems: A Systematic Review

Open access
Lee, Terrence C., Shah, Neil U., Haack, Alyssa, Baxter, Sally L.·2020·Informatics
DOI 10.3390/informatics7030025PMC PMC7710328
Open this paper in CoChat — free accountOpen PDF

Abstract

Predictive analytics using electronic health record (EHR) data have rapidly advanced over the last decade. While model performance metrics have improved considerably, best practices for implementing predictive models into clinical settings for point-of-care risk stratification are still evolving. Here, we conducted a systematic review of articles describing predictive models integrated into EHR systems and implemented in clinical practice. We conducted an exhaustive database search and extracted data encompassing multiple facets of implementation. We assessed study quality and level of evidence. We obtained an initial 3393 articles for screening, from which a final set of 44 articles was included for data extraction and analysis. The most common clinical domains of implemented predictive models were related to thrombotic disorders/anticoagulation (25%) and sepsis (16%). The majority of studies were conducted in inpatient academic settings. Implementation challenges included alert fatigue, lack of training, and increased work burden on the care team. Of 32 studies that reported effects on clinical outcomes, 22 (69%) demonstrated improvement after model implementation. Overall, EHR-based predictive models offer promising results for improving clinical outcomes, although several gaps in the literature remain, and most study designs were observational. Future studies using randomized controlled trials may help improve the generalizability of findings.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/informatics7030025

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most published studies target a particular condition or hospital setting, and only a minority of EWSs have been evaluated in randomized controlled trials.

    “Most published studies target a particular condition (for example, sepsis) or hospital setting (for example, ICU)4 and only a minority of EWSs have been evaluated in randomized controlled trials4,6.”

    From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [6]

    The Results (Study Characteristics; page not provided) describe studies focused on clinical domains including sepsis and report that 28 of 44 centered on inpatient populations. The Results (Quality Assessment; page not provided) find that randomized controlled trials were a minority (5 of 44). However, that figure covers studies of EHR-embedded predictive models generally, not early warning systems (EWSs) specifically, and the copy does not establish that most studies targeted a particular hospital setting such as an ICU. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 76 · Cited by 133

Research works printed but not linked: 65

  1. [1]Parikh RB; Kakad M; Bates DW Integrating Predictive Analytics into High-Value Care: The Dawn of Precision Delivery. JAMA 2016, 315, 651– 652. 26881365
  2. [2]Henry J; Pylypchuk Y; Searcy T; Patel V Adoption of Electronic Health Record Systems among U.S. Non-Federal Acute Care Hospitals: 2008–2015. Available online: /evaluations/data-briefs/non-federal-acute-care-hospital-ehr-adoption-2008-2015.php (accessed on 18 September 2019). 40127322
  3. [3]Cohen MF Impact of the HITECH financial incentives on EHR adoption in small, physician-owned practices. Int. J. Med. Inform. 2016, 94, 143– 154. 27573322
  4. [4]Mennemeyer ST; Menachemi N; Rahurkar S; Ford EW Impact of the HITECH Act on physicians’ adoption of electronic health records. J. Am. Med. Inform. Assoc. 2016, 23, 375– 379. 26228764 PMC7784315
  5. [6]Weller GB; Lovely J; Larson DW; Earnshaw BA; Huebner M Leveraging electronic health records for predictive modeling of post-surgical complications. Stat. Methods Med. Res. 2017, 27, 3271– 3285. 29298612
  6. [8]Nemati S; Holder A; Razmi F; Stanley MD; Clifford GD; Buchman TG An Interpretable Machine Learning Model for Accurate Prediction of Sepsis in the ICU. Crit. Care Med. 2018, 46, 547– 553. 29286945 PMC5851825
  7. [10]Xue Y; Liang H; Norbury J; Gillis R; Killingworth B Predicting the risk of acute care readmissions among rehabilitation inpatients: A machine learning approach. J. Biomed. Inform. 2018, 86, 143– 148. 30237014
  8. [11]Artetxe A; Beristain A; Graña M Predictive models for hospital readmission risk: A systematic review of methods. Comput. Methods Programs Biomed. 2018, 164, 49– 64. 30195431
  9. [12]Byrd RJ; Steinhubl SR; Sun J; Ebadollahi S; Stewart WF Automatic identification of heart failure diagnostic criteria, using text analysis of clinical notes from electronic health records. Int. J. Med. Inform. 2014, 83, 983– 992. 23317809 PMC5234735
  10. [13]Alemi F; Avramovic S; Schwartz MD Electronic Health Record-Based Screening for Substance Abuse. Big Data 2018, 6, 214– 224. 30283729 PMC6154440
  11. [14]Taylor RA; Pare JR; Venkatesh AK; Mowafi H; Melnick ER; Fleischman W; Hall MK Prediction of In-hospital Mortality in Emergency Department Patients with Sepsis: A Local Big Data-Driven, Machine Learning Approach. Acad. Emerg. Med. 2016, 23, 269– 278. 26679719 PMC5884101
  12. [15]Hoogendoorn M; El Hassouni A; Mok K; Ghassemi M; Szolovits P Prediction using patient comparison vs. modeling: A case study for mortality prediction. In Proceedings of the 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA, 16–20 August 2016; pp. 2464– 2467. 28268823
  13. [16]Shickel B; Tighe PJ; Bihorac A; Rashidi P Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis. IEEE J. Biomed. Health Inform. 2018, 22, 1589– 1604. 29989977 PMC6043423
  14. [17]Goldstein BA; Navar AM; Pencina MJ; Ioannidis JPA Opportunities and challenges in developing risk prediction models with electronic health records data: A systematic review. J. Am. Med. Inform. Assoc. 2017, 24, 198– 208. 27189013 PMC5201180
  15. [18]Kruse CS; Goswamy R; Raval Y; Marawi S Challenges and Opportunities of Big Data in Health Care: A Systematic Review. JMIR Med. Inform. 2016, 4, e38. 27872036 PMC5138448
  16. [19]Agrawal S; Vaidya V; Chandrashekaraiah P; Kulkarni H; Chen L; Rudeen K; Narayanan B; Inbar O; Hyde B Development of an artificial intelligence model to predict survival at specific time intervals for lung cancer patients. J. Clin. Oncol. 2019, 37, 6556.
  17. [20]Shaw J; Rudzicz F; Jamieson T; Goldfarb A Artificial Intelligence and the Implementation Challenge. J. Med. Internet Res. 2019, 21, e13659. 31293245 PMC6652121
  18. [22]Liberati A; Altman DG; Tetzlaff J; Mulrow C; Gøtzsche PC; Ioannidis JPA; Clarke M; Devereaux PJ; Kleijnen J; Moher D The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration. PLoS Med. 2009, 6, e1000100. 19621070 PMC2707010
  19. [23]Moher D; Liberati A; Tetzlaff J; Altman DG The PRISMA Group Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 2009, 6, e1000097. 19621072 PMC2707599
  20. [26]Maynard GA; Morris TA; Jenkins IH; Stone S; Lee J; Renvall M; Fink E; Schoenhaus R Optimizing Prevention of Hospital-acquired Venous Thromboembolism (VTE): Prospective Validation of a VTE Risk Assessment Model. J. Hosp. Med. 2010, 5, 10– 18. 19753640
  21. [27]Novis SJ; Havelka GE; Ostrowski D; Levin B; Blum-Eisa L; Prystowsky JB; Kibbe MR Prevention of thromboembolic events in surgical patients through the creation and implementation of a computerized risk assessment program. J. Vasc. Surg. 2010, 51, 648– 654. 20022209
  22. [28]Fossum M; Alexander GL; Ehnfors M; Ehrenberg A Effects of a computerized decision support system on pressure ulcers and malnutrition in nursing homes for the elderly. Int. J. Med. Inform. 2011, 80, 607– 617. 21783409
  23. [29]Herasevich V; Tsapenko M; Kojicic M; Ahmed A; Kashyap R; Venkata C; Shahjehan K; Thakur SJ; Pickering BW; Zhang J; Limiting ventilator-induced lung injury through individual electronic medical record surveillance. Crit. Care Med. 2011, 39, 34– 39. 20959788
  24. [30]Nelson JL; Smith BL; Jared JD; Younger JG Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann. Emerg. Med. 2011, 57, 500– 504. 21227543
  25. [31]Umscheid CA; Hanish A; Chittams J; Weiner MG; Hecht TEH Effectiveness of a novel and scalable clinical decision support intervention to improve venous thromboembolism prophylaxis: A quasi-experimental study. BMC Med. Inform. Decis. Mak. 2012, 12, 92. 22938083 PMC3502442
  26. [32]Baillie CA; Vanzandbergen C; Tait G; Hanish A; Leas B; French B; William Hanson C; Behta M; Umscheid CA The readmission risk flag: Using the electronic health record to automatically identify patients at risk for 30-day readmission. J. Hosp. Med. 2013, 8, 689– 695. 24227707 PMC4407637
  27. [33]Amarasingham R; Patel PC; Toto K; Nelson LL; Swanson TS; Moore BJ; Xie B; Zhang S; Alvarez KS; Ma Y; Allocating scarce resources in real-time to reduce heart failure readmissions: A prospective, controlled study. BMJ Qual. Saf. 2013, 22, 998– 1005. PMC3888600 23904506
  28. [34]Litvin C; Ornstein SM Use of clinical decision support tools to improve primary care chronic kidney disease management. J. Gen. Intern. Med. 2013, 28, S448.
  29. [35]Oh S-H; Park E-J; Jin Y; Piao J; Lee S-M Automatic delirium prediction system in a Korean surgical intensive care unit. Nurs. Crit. Care 2014, 19, 281– 291. 24165109
  30. [36]Resetar E; McMullen KM; Russo AJ; Doherty JA; Gase KA; Woeltje KF Development, Implementation and Use of Electronic Surveillance for Ventilator-Associated Events (VAE) in Adults. AMIA Annu. Symp. Proc. 2014, 2014, 1010– 1017. 25954410 PMC4419870
  31. [37]Amland RC; Dean BB; Yu H; Ryan H; Orsund T; Hackman JL; Roberts SR Computerized Clinical Decision Support to Prevent Venous Thromboembolism Among Hospitalized Patients: Proximal Outcomes from a Multiyear Quality Improvement Project. J. Healthc Qual. 2015, 37, 221– 231. 26151096
  32. [38]Faerber AE; Horvath R; Stillman C; O’Connell ML; Hamilton AL; Newhall KA; Likosky DS; Goodney PP Development and pilot feasibility study of a health information technology tool to calculate mortality risk for patients with asymptomatic carotid stenosis: The Carotid Risk Assessment Tool (CARAT). BMC Med. Inform. Decis. Mak. 2015, 15, 20. 25890090 PMC4404562
  33. [39]Hao S; Wang Y; Jin B; Shin AY; Zhu C; Huang M; Zheng L; Luo J; Hu Z; Fu C; Development, Validation and Deployment of a Real Time 30 Day Hospital Readmission Risk Assessment Tool in the Maine Healthcare Information Exchange. PLoS ONE 2015, 10, e0140271. 26448562 PMC4598005
  34. [40]Kharbanda EO; Nordin JD; Sinaiko AR; Ekstrom HL; Stultz JM; Sherwood NE; Fontaine PL; Asche SE; Dehmer SP; Amundson JH; TeenBP: Development and Piloting of an EHR-Linked Clinical Decision Support System to Improve Recognition of Hypertension in Adolescents. EGEMS 2015, 3, 1142. 26290886 PMC4537153
  35. [41]Lustig DB; Rodriguez R; Wells PS Implementation and validation of a risk stratification method at The Ottawa Hospital to guide thromboprophylaxis in ambulatory cancer patients at intermediate-high risk for venous thrombosis. Thromb. Res. 2015, 136, 1099– 1102. 26260645
  36. [42]Umscheid CA; Betesh J; VanZandbergen C; Hanish A; Tait G; Mikkelsen ME; French B; Fuchs BD Development, Implementation, and Impact of an Automated Early Warning and Response System for Sepsis. J. Hosp. Med. 2015, 10, 26– 31. 25263548 PMC4410778
  37. [43]Depinet H; von Allmen D; Towbin A; Hornung R; Ho M; Alessandrini E Risk Stratification to Decrease Unnecessary Diagnostic Imaging for Acute Appendicitis. Pediatrics 2016, 138, e20154031. 27553220
  38. [44]Narayanan N; Gross AK; Pintens M; Fee C; MacDougall C Effect of an electronic medical record alert for severe sepsis among ED patients. Am. J. Emerg. Med. 2016, 34, 185– 188. 26573784
  39. [45]Vinson DR; Ballard DW; Mark DG; Yealy DM; Reed ME; Huang J; Rauchwerger AS; Lin JS; Kene MV; Chettipally UK Impact of electronic clinical decision support on initial site of care for emergency department patients with acute pulmonary embolism. Ann. Emerg. Med. 2016, 68, S54.
  40. [46]Aakre C; Franco PM; Ferreyra M; Kitson J; Li M; Herasevich V Prospective validation of a near real-time EHR-integrated automated SOFA score calculator. Int. J. Med. Inform. 2017, 103, 1– 6. 28550994
  41. [47]Arts DL; Abu-Hanna A; Medlock SK; van Weert HCPM Effectiveness and usage of a decision support system to improve stroke prevention in general practice: A cluster randomized controlled trial. PLoS ONE 2017, 12, e0170974. 28245247 PMC5330455
  42. [48]Bookman K; West D; Ginde A; Wiler J; McIntyre R; Hammes A; Carlson N; Steinbruner D; Solley M; Zane R Embedded Clinical Decision Support in Electronic Health Record Decreases Use of High-cost Imaging in the Emergency Department: EmbED study. Acad. Emerg. Med. 2017, 24, 839– 845. 28391603 PMC5505794
  43. [49]Jin Y; Jin T; Lee S-M Automated Pressure Injury Risk Assessment System Incorporated Into an Electronic Health Record System. Nurs. Res. 2017, 66, 462– 472. 29095377
  44. [50]Samal L; D’Amore JD; Bates DW; Wright A Implementation of a scalable, web-based, automated clinical decision support risk-prediction tool for chronic kidney disease using C-CDA and application programming interfaces. J. Am. Med. Inform. Assoc. 2017, 24, 1111– 1115. 29016969 PMC6580936
  45. [51]Shimabukuro DW; Barton CW; Feldman MD; Mataraso SJ; Das R Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: A randomised clinical trial. BMJ Open Respir. Res. 2017, 4, e000234. PMC5687546 29435343
  46. [52]Chaturvedi S; Kelly A; Prabhakaran S; Saposnik G; Lee L; Malik A; Boerman C; Serlin G Electronic decision support for improvement of contemporary therapy for stroke prevention (EDICTS). Neurology 2018, 90, 569– 573. 30472172
  47. [53]Cherkin D; Balderson B; Wellman R; Hsu C; Sherman KJ; Evers SC; Hawkes R; Cook A; Levine MD; Piekara D; Effect of Low Back Pain Risk-Stratification Strategy on Patient Outcomes and Care Processes: The MATCH Randomized Trial in Primary Care. J. Gen. Intern. Med. 2018, 33, 1324– 1336. 29790073 PMC6082187
  48. [54]Ebinger JE; Strauss CE; Garberich RR; Bradley SM; Rush P; Chavez IJ; Poulose AK; Porten BR; Henry TD Value-Based ST-Segment-Elevation Myocardial Infarction Care Using Risk-Guided Triage and Early Discharge. Circ. Cardiovasc. Qual. Outcomes 2018, 11, e004553. 29654000 PMC5901756
  49. [55]Hebert C; Flaherty J; Smyer J; Ding J; Mangino JE Development and validation of an automated ventilator-associated event electronic surveillance system: A report of a successful implementation. Am. J. Infect. Control. 2018, 46, 316– 321. 29132696
  50. [56]Jung AD; Baker J; Droege CA; Nomellini V; Johannigman J; Holcomb JB; Goodman MD; Pritts TA Sooner is better: Use of a real-time automated bedside dashboard improves sepsis care. J. Surg. Res. 2018, 231, 373– 379. 30278956
  51. [57]Kang M-J; Jin Y; Jin T; Lee S-M Automated Medication Error Risk Assessment System (Auto-MERAS). J. Nurs. Care Qual. 2018, 33, 86– 93. 28505057
  52. [58]Karlsson L; Nilsson S; Bång M; Nilsson L; Charitakis E; Janzon M A clinical decision support tool for improving adherence to guidelines on anticoagulant therapy in patients with atrial fibrillation at risk of stroke: A cluster-randomized trial in a Swedish primary care setting (the CDS-AF study). PLoS Med. 2018, 15, e1002528. 29534063 PMC5849292
  53. [59]Moon K-J; Jin Y; Jin T; Lee S-M Development and validation of an automated delirium risk assessment system (Auto-DelRAS) implemented in the electronic health record system. Int. J. Nurs. Stud. 2018, 77, 46– 53. 29035732
  54. [60]Ridgway JP; Almirol EA; Bender A; Richardson A; Schmitt J; Friedman E; Lancki N; Leroux I; Pieroni N; Dehlin J; Which Patients in the Emergency Department Should Receive Preexposure Prophylaxis? Implementation of a Predictive Analytics Approach. AIDS Patient Care STDS 2018, 32, 202– 207. 29672136 PMC6939581
  55. [61]Turrentine FE; Sohn M-W; Wilson SL; Stanley C; Novicoff W; Sawyer RG; Williams MD Fewer thromboembolic events after implementation of a venous thromboembolism risk stratification tool. J. Surg. Res. 2018, 225, 148– 156. 29605025
  56. [62]Villa S; Weber EJ; Polevoi S; Fee C; Maruoka A; Quon T Decreasing triage time: Effects of implementing a step-wise ESI algorithm in an EHR. Int. J. Qual. Health Care 2018, 30, 375– 381. 29697806 PMC6005140
  57. [63]Vinson DR; Mark DG; Chettipally UK; Huang J; Rauchwerger AS; Reed ME; Lin JS; Kene MV; Wang DH; Sax DR; Increasing Safe Outpatient Management of Emergency Department Patients with Pulmonary Embolism A Controlled Pragmatic Trial. Ann. Intern. Med. 2018, 169, 855– 865. 30422263
  58. [64]Bedoya AD; Clement ME; Phelan M; Steorts RC; O’Brien C; Goldstein BA Minimal Impact of Implemented Early Warning Score and Best Practice Alert for Patient Deterioration. Crit. Care Med. 2019, 47, 49– 55. 30247239 PMC6298839
  59. [66]Ekstrom HL; Kharbanda EO; Ballard DW; Vinson DR; Vazquez-Benitez G; Chettipally UK; Dehmer SP; Kunisetty G; Sharma R; Rauchwerger AS; Development of a Clinical Decision Support System for Pediatric Abdominal Pain in Emergency Department Settings Across Two Health Systems Within the HCSRN. EGEMS 2019, 7, 15. 30993147 PMC6460497
  60. [67]Giannini HM; Ginestra JC; Chivers C; Draugelis M; Hanish A; Schweickert WD; Fuchs BD; Meadows L; Lynch M; Donnelly PJ; A Machine Learning Algorithm to Predict Severe Sepsis and Septic Shock: Development, Implementation, and Impact on Clinical Practice. Crit. Care Med. 2019, 47, 1485– 1492. 31389839 PMC8635476
  61. [68]Khoong EC; Karliner L; Lo L; Stebbins M; Robinson A; Pathak S; Santoyo-Olsson J; Scherzer R; Peralta CA A Pragmatic Cluster Randomized Trial of an Electronic Clinical Decision Support System to Improve Chronic Kidney Disease Management in Primary Care: Design, Rationale, and Implementation Experience. JMIR Res. Protoc. 2019, 8, e14022. 31199334 PMC6594214
  62. [69]Ogunwole SM; Phillips J; Gossett A; Downs JR Putting veterans with heart failure FIRST improves follow-up and reduces readmissions. BMJ Open Qual. 2019, 8, e000386. PMC6340603 30729191
  63. [70]Embi PJ; Leonard AC Evaluating alert fatigue over time to EHR-based clinical trial alerts: Findings from a randomized controlled study. J. Am. Med. Inform. Assoc. 2012, 19, e145– e148. 22534081 PMC3392862
  64. [71]Harris AH Path from Predictive Analytics to Improved Patient Outcomes: A Framework to Guide Use, Implementation, and Evaluation of Accurate Surgical Predictive Models. Ann. Surg. 2017, 265, 461– 463. 27735825 PMC5645012
  65. [72]Benda NC; Das LT; Abramson EL; Blackburn K; Thoman A; Kaushal R; Zhang Y; Ancker JS “ How did you get to this number?” Stakeholder needs for implementing predictive analytics: A pre-implementation qualitative study. J. Am. Med. Inform. Assoc. 2020, 27, 709– 716. PMC7647269 32159774

Authors
Lee, T. C., Shah, N. U., Haack, A., & Baxter, S. L.
Affiliations
University of California San Diego (US)
Corresponding author
Baxter, Sally L.
ORCID
Venue
Informatics
Publisher
MDPI AG
Published
2020-07-25
Volume
7
Issue
3
Pages
25
ISSN
2227-9709 (ISSN-L)
DOI
10.3390/informatics7030025
PMC
PMC7710328
OpenAlex
W3044246896
Cited by
133
References
76
Citations by year
2026: 32 · 2025: 29 · 2024: 35 · 2023: 18 · 2022: 13 · 2021: 6
FWCI
1.41
Type
journal article
Language
en
Open access
Licence
Locations
  • Informatics (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • U.S. National Library of Medicine (T15LM011271)
  • Heed Ophthalmic Foundation (NA)
Topics
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
prediction model, electronic health records, clinical implementation, thrombotic disorders, anticoagulation, sepsis, clinical decision support, alarm fatigue, implementation barriers, clinical outcomes

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review