CoChat
Sign Up
Cited by 128Electronic Health Records SystemsArtificial Intelligence in Healthcare and EducationHealthcare Technology and Patient Monitoring

Efficacy and unintended consequences of hard-stop alerts in electronic health record systems: a systematic review

Open access
Powers, Emily M, Shiffman, Richard N, Melnick, Edward R, Hickner, Andrew, Sharifi, Mona·2018·Journal of the American Medical Informatics Association
DOI 10.1093/jamia/ocy112PMC PMC6915824
Open this paper in CoChat — free account

Abstract

Objective: Clinical decision support (CDS) hard-stop alerts—those in which the user is either prevented from taking an action altogether or allowed to proceed only with the external override of a third party—are increasingly common but can be problematic. To understand their appropriate application, we asked 3 key questions: (1) To what extent are hard-stop alerts effective in improving patient health and healthcare delivery outcomes? (2) What are the adverse events and unintended consequences of hard-stop alerts? (3) How do hard-stop alerts compare to soft-stop alerts? Methods and Materials: Studies evaluating computerized hard-stop alerts in healthcare settings were identified from biomedical and computer science databases, gray literature sites, reference lists, and reviews. Articles were extracted for process outcomes, health outcomes, unintended consequences, user experience, and technical details. Results: Of 32 studies, 15 evaluated health outcomes, 16 process outcomes only, 10 user experience, and 4 compared hard and soft stops. Seventy-nine percent showed improvement in health outcomes and 88% in process outcomes. Studies reporting good user experience cited heavy user involvement and iterative design. Eleven studies reported on unintended consequences including avoidance of hard-stopped workflow, increased alert frequency, and delay to care. Hard stops were superior to soft stops in 3 of 4 studies. Conclusions: Hard stops can be effective and powerful tools in the CDS armamentarium, but they must be implemented judiciously with continuous user feedback informing rapid, iterative design. Investigators must report on associated health outcomes and unintended consequences when implementing IT solutions to clinical problems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocy112

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 55

  1. [1]1. QuickStats: Percentage of Office-Based Physicians with a Basic Electronic Health Record (EHR) System,* by State-National Electronic Health Records Survey,† United States, 2014§. 2015; 64 (34): 963-963. https:// www.cdc.gov/mmwr/preview/mmwrhtml/mm6434a10.htm.
  2. [2]2. Rosenberg SN, Sullivan M, Juster IA. Overrides of medication alerts in ambulatory care. Arch Intern Med 2009; 169 (14): 1337; author reply 1338.
  3. [4]4. Zenziper Straichman Y, Kurnik D, Matok I, et al . Prescriber response to computerized drug alerts for electronic prescriptions among hospitalized patients. Int J Med Inform 2017; 107: 70-5.
  4. [5]5. Wachter R. The Digital Doctor: Hope, Hype, and Harm at the Dawn of Medicine's Computer Age . New York: McGraw Hill Professional; 2015.
  5. [7]7. Paterno MD, Maviglia SM, Gorman PN, et al . Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc 2009; 16 (1): 40-6.
  6. [8]8. Strom BL, Schinnar R, Aberra F, et al . Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med 2010; 170 (17): 1578-83.
  7. [10]10. Evans AS, Lazar EJ, Tiase VL, et al . The role of housestaff in implementing medication reconciliation on admission at an academic medical center. AmJMedQual 2011; 26 (1): 39-42.
  8. [11]11. Goldzweig CL, Orshansky G, Paige NM, et al . Electronic health recordbased interventions for improving appropriate diagnostic imaging: a systematic review and meta-analysis. Ann Intern Med 2015; 162 (8): 557-65.
  9. [12]12. Helmons PJ, Coates CR, Kosterink JGW, et al . Decision support at the point of prescribing to increase formulary adherence. Am J Health Syst Pharm 2015; 72 (5): 408-13.
  10. [13]13. Hsu C-C, Chou C-Y, Chou C-L, et al . Impact of a warning CPOE system on the inappropriate pill splitting of prescribed medications in outpatients. PLoS One 2014; 9 (12): e114359.
  11. [14]14. Kilgore MR, McIlwain CA, Schmidt RA, et al . Reflex test reminders in required cancer synoptic templates decrease order entry error: an analysis of mismatch repair immunohistochemical orders to screen for Lynch syndrome. J Pathol Inform 2016; 7 (1): 48.
  12. [15]15. Loudon H, Nentin F, Silverman ME. Using clinical decision support as a means of implementing a universal postpartum depression screening program. Arch Womens Ment Health 2016; 19 (3): 501-5.
  13. [16]16. Nikolic D, Richter SS, Asamoto K, et al . Implementation of a clinical decision support tool for stool cultures and parasitological studies in hospitalized patients. J Clin Microbiol 2017; 55 (12): 3350-4.
  14. [17]17. Peterfreund RA, Driscoll WD, Walsh JL, et al . Evaluation of a mandatory quality assurance data capture in anesthesia: a secure electronic system to capture quality assurance information linked to an automated anesthesia record. Anesth Analg 2011; 112 (5): 1218-25.
  15. [18]18. Procop GW, Keating C, Stagno P, et al . Reducing duplicate testing: a comparison of two clinical decision support tools. AmJClin Pathol 2015; 143 (5): 623-6.
  16. [19]19. Rossi AP, Wellins CA, Savic M, et al . Use of computer alerts to prevent the inappropriate use of metformin in an inpatient setting. Qual Manag Health Care 2012; 21 (4): 235-9.
  17. [20]20. Schnall R, Sperling JD, Liu N, et al . The effect of an electronic 'hard-stop' alert on HIV testing rates in the emergency department. Stud Health Technol Inform 2013; 192: 432-6.
  18. [21]21. Vawdrey DK, Chang N, Compton A, et al . Impact of electronic medication reconciliation at hospital admission on clinician workflow. AMIA Annu Symp Proc 2010; 2010: 822-6.
  19. [22]22. Yousem DM. Combating overutilization: radiology benefits managers versus order entry decision support. Neuroimaging Clin N Am 2012; 22 (3): 497-509.
  20. [23]23. Zipkin R, Schrager SM, Keefer M, et al . Improving home management plan of care compliance rates through an electronic asthma action plan. J Asthma 2013; 50 (6): 664-71.
  21. [24]24. Felcher AH, Gold R, Mosen DM, et al . Decrease in unnecessary vitamin D testing using clinical decision support tools: making it harder to do the wrong thing. J Am Med Inform Assoc 2017; 24 (4): 776-80.
  22. [25]25. Cecchini M, Framski K, Lazette P, et al . Electronic intervention to improve structured cancer stage data capture. J Oncol Pract 2016; 12 (10): e949-56.
  23. [26]26. Stenner SP, Chakravarthy R, Johnson KB, et al . ePrescribing: reducing costs through in-class therapeutic interchange. Appl Clin Inform 2016; 7 (4): 1168-81.
  24. [27]27. MacKay M, Anderson C, Boehme S, et al . Frequency and severity of parenteral nutrition medication errors at a large children's hospital after implementation of electronic ordering and compounding. Nutr Clin Pract 2016; 31 (2): 195-206.
  25. [28]28. McNulty J, Donnelly E, Iorio K. Methodologies for sustaining barcode medication administration compliance. A multi-disciplinary approach. J Healthc Inf Manag 2009; 23: 30-3.
  26. [31]31. O'Connor SR, Tully MA, Ryan B, et al . Failure of a numerical quality assessment scale to identify potential risk of bias in a systematic review: a comparison study. BMCResNotes 2015; 8 (1): 224.
  27. [32]32. Bails D, Clayton K, Roy K, et al . Implementing online medication reconciliation at a large academic medical center. Jt Comm J Qual Patient Saf 2008; 34 (9): 499-508.
  28. [33]33. Balasuriya L, Vyles D, Bakerman P, et al . Computerized dose range checking using hard and soft stop alerts reduces prescribing errors in a pediatric intensive care unit. J Patient Saf 2017; 13 (3): 144-8.
  29. [34]34. Ban VS, Madden CJ, Browning T, et al . A novel use of the discrete templated notes within an electronic health record software to monitor resident supervision. J Am Med Inform Assoc 2017; 24: e2-8.
  30. [35]35. Bansal M, Abdallah A-O, Pennisi A, et al . Improving communication on intent of chemotherapy using QOPI scores and PDSA cycles. J Cancer Educ 2016; 31 (4): 736-41.
  31. [37]37. Galanter WL, Thambi M, Rosencranz H, et al . Effects of clinical decision support on venous thromboembolism risk assessment, prophylaxis, and prevention at a university teaching hospital. Am J Health Syst Pharm 2010; 67 (15): 1265-73.
  32. [38]38. Geeting GK, Beck M, Bruno MA, et al . Mandatory assignment of modified wells score before CT angiography for pulmonary embolism fails to improve utilization or percentage of positive cases. AJR Am J Roentgenol 2016; 207 (2): 442-9.
  33. [39]39. Griffey RT, Wittels K, Gilboy N, et al . Use of a computerized forcing function improves performance in ordering restraints. Ann Emerg Med 2009; 53 (4): 469-76.
  34. [40]40. Haut ER, Lau BD, Kraenzlin FS, et al . Improved prophylaxis and decreased rates of preventable harm with the use of a mandatory computerized clinical decision support tool for prophylaxis for venous thromboembolism in trauma. Arch Surg 2012; 147 (10): 901-7.
  35. [41]41. Heymann AD, Hoch I, Valinsky L, et al . Mandatory computer field for blood pressure measurement improves screening. FamPract 2005; 22 (2): 168-9.
  36. [42]42. Hildreth AN, Enniss T, Martin RS, et al . Surgical intensive care unit mobility is increased after institution of a computerized mobility order set and intensive care unit mobility protocol: a prospective cohort analysis. AmSurg 2010; 76: 818-22.
  37. [43]43. Hoo GWS, Wu CC, Vazirani S, et al . Does a clinical decision rule using Ddimer level improve the yield of pulmonary CT angiography? AJR Am J Roentgenol 2011; 196 (5): 1059-64.
  38. [44]44. Ip IK, Gershanik EF, Schneider LI, et al . Impact of IT-enabled intervention on MRI use for back pain. AmJMed 2014; 127 (6): 512-8.e1.
  39. [45]45. Khoury L, Dangodara AA, Lee J-A, et al . Implementation of a mandated venous thromboembolism clinical order set improves venous thromboembolism core measures. Hosp Pract 2014; 42 (5): 89-99.
  40. [46]46. Larson EA, Thompson PA, Anderson ZK, et al . Decreasing the critical value of hemoglobin required for physician notification reduces the rate of blood transfusions. Int J Gen Med 2016; 9: 133-6.
  41. [47]47. Lau BD, Haider AH, Streiff MB, et al . Eliminating health care disparities with mandatory clinical decision support: the venous thromboembolism (VTE) example. Med Care 2015; 53 (1): 18-24.
  42. [48]48. Park CS, Kim T-B, Kim SL, et al . The use of an electronic medical record system for mandatory reporting of drug hypersensitivity reactions has been shown to improve the management of patients in the university hospital in Korea. Pharmacoepidem Drug Saf 2008; 17 (9): 919-25.
  43. [49]49. Saigh O, Triola MM, Link RN. Brief report: failure of an electronic medical record tool to improve pain assessment documentation. J Gen Intern Med 2006; 21 (2): 185-8.
  44. [50]50. Tajmir S, Raja AS, Ip IK, et al . Impact of clinical decision support on radiography for acute ankle injuries: a randomized trial. West J Emerg Med 2017; 18 (3): 487-95.
  45. [51]51. Vartanians VM, Sistrom CL, Weilburg JB, et al . Increasing the appropriateness of outpatient imaging: effects of a barrier to ordering low-yield examinations. Radiology 2010; 255 (3): 842-9.
  46. [52]52. Wetmore DS, Gandhi MNA, Curatolo MC, et al . Simulation to test hardstop implementation of a pre-anesthetic induction checklist. In: 2014 IEEE 27th International Symposium on Computer-Based Medical Systems . 2014: 564-5. New York, NY.
  47. [53]53. Yu KH, Sweidan M, Williamson M, et al . Drug interaction alerts in software-what do general practitioners and pharmacists want? Med J Aust 2011; 195 (11-12): 676-80.
  48. [54]54. Bisantz AM, Wears RL. Forcing functions: the need for restraint. Ann Emerg Med 2009; 53 (4): 477-9.
  49. [55]55. Bates DW. Decision support in hospitals: getting the benefits: comment on 'unintended effects of a computerized physician order entry nearly hardstop alert to prevent a drug interaction'. Arch Intern Med 2010; 170 (17): 1583-4.
  50. [56]56. Page N, Baysari MT, Westbrook JI. A systematic review of the effectiveness of interruptive medication prescribing alerts in hospital CPOE systems to change prescriber behavior and improve patient safety. Int J Med Inform 2017; 105: 22-30.
  51. [57]57. Curtis CE, Al Bahar F, Marriott JF. The effectiveness of computerised decision support on antibiotic use in hospitals: a systematic review. PLoS One 2017; 12 (8): e0183062.
  52. [58]58. Beeler PE, Bates DW, Hug BL. Clinical decision support systems. Swiss Med Wkly 2014; 144: w14073.
  53. [59]59. McCoy AB, Thomas EJ, Krousel-Wood M, Sittig DF. Clinical decision support alert appropriateness: a review and proposal for improvement. Ochsner J 2014; 14 (2): 195-202.
  54. [60]60. Garg AX, Adhikari NKJ, McDonald H, et al . Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293 (10): 1223-38.
  55. [64]64. Sockolow P, Dowding D, Randell R, et al . Using mixed methods in health information technology evaluation. Stud Health Technol Inform 2016; 225: 83-7.

Cited by

Sorted by citations

Authors
Powers, E. M., Shiffman, R. N., Melnick, E. R., Hickner, A., & Sharifi, M.
Affiliations
  • Yale University (US)
  • Cushing/Whitney Medical Library, Yale University School of Medicine, New Haven, Connecticut, USA
Corresponding author
Powers, Emily M
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2018-09-18
Volume
25
Issue
11
Pages
1556-1566
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1093/jamia/ocy112
PMC
PMC6915824
OpenAlex
W2891105177
Cited by
128
References
3
Citations by year
2026: 20 · 2025: 15 · 2024: 16 · 2023: 22 · 2022: 21 · 2021: 11 · 2020: 17 · 2019: 6
FWCI
1.59
Type
journal article
Language
en
Open access
Licence
academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
Locations
  • Journal of the American Medical Informatics Association (publishedVersion)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • National Library of Medicine (T15 LM007056)
  • Agency for Healthcare Research and Quality (K08HS024332, K08HS021271)
Topics
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
Keywords
clinical decision support, electronic health records, alarm fatigue, unintended consequences, user experience, iterative design, health outcomes, process outcomes
MeSH terms
Humans, Therapy, Computer-Assisted, Medical Records Systems, Computerized, Decision Support Systems, Clinical, Medical Order Entry Systems, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review