Cited by 22Electronic Health Records SystemsPharmaceutical Practices and Patient OutcomesPatient Safety and Medication Errors

The effect of eliminating intermediate severity drug-drug interaction alerts on overall medication alert burden and acceptance rate

Open access
Knight, Amy M., Maygers, Joyce, Foltz, Kimberly A., John, Isha S., Yeh, Hsin Chieh, Brotman, Daniel J.·2019·Applied Clinical Informatics

Abstract

Objective This study aimed to determine the effects of reducing the number of drug-drug interaction (DDI) alerts in an order entry system. Methods Retrospective pre–post analysis at an urban medical center of the rates of medication alerts and alert acceptance during a 5-month period before and 5-month period after the threshold for firing DDI alerts was changed from “intermediate” to “severe.” To ensure that we could determine varying response to each alert type, we took an in-depth look at orders generating single alerts. Results Before the intervention, 241,915 medication orders were placed, of which 25.6% generated one or more medication alerts; 5.3% of the alerts were accepted. During the postintervention period, 245,757 medication orders were placed of which 16.0% generated one or more medication alerts, a 37.5% relative decrease in alert rate (95% confidence interval [CI]: −38.4 to −36.8%), but only a 9.6% absolute decrease (95% CI: −9.4 to −9.9%). 7.4% of orders generating alerts were accepted postintervention, a 39.6% relative increase in acceptance rate (95% CI: 33.2–47.2%), but only a 2.1% absolute increase (95% CI: 1.8–2.4%). When only orders generating a single medication alert were considered, there was a 69.1% relative decrease in the number of orders generating DDI alerts, and an 85.7% relative increase in the acceptance rate (95% CI: 58.6–126.2%), though only a 1.8% absolute increase (95% CI: 1.3–2.3%). Conclusion Eliminating intermediate severity DDI alerts resulted in a statistically significant decrease in alert burden and increase in the rate of medication alert acceptance, but alert acceptance remained low overall.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1055/s-0039-3400447

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Effective governance of CDS has been demonstrated in other institutions, resulting in the transition of alert and reminder content to other more appropriate areas in the EHR, downgrading the severity of alerts, and assessing the effect of CDS on clinician behavior and patient care.

    “Effective governance of CDS has been demonstrated in other institutions, resulting in the transition of alert and reminder content to other more appropriate areas in the EHR, downgrading the severity of alerts (often leading to them no longer firing), 38 39 and assessing the effect of CDS on clinician behavior and patient care. 40 41”

    From Qualitative Study of Inpatient Hospital Medicine Interruptive Alerts and Clinical Decision Support (2026) · cites this paper as [39]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 37

  1. Shortliffe EH. Computer programs to support clinical decision making. JAMA 1987;258(01):61-66 2 Clinical decision support. Available at: https://www.ahrq.gov/ cpi/about/otherwebsites/clinical-decision-support/index.html. Accessed August 13, 2019
  2. Bright TJ, Wong A, Dhurjati R, et al. Effect of clinical decision- support systems: a systematic review. Ann Intern Med 2012;157 (01):29-43
  3. Nuckols TK, Smith-Spangler C, Morton SC, et al. The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a system- atic review and meta-analysis. Syst Rev 2014;3:56
  4. Murphy DR, Reis B, Sittig DF, Singh H. Notifications received by primary care practitioners in electronic health records: a taxon- omy and time analysis. Am J Med 2012;125(02):209.e1-209.e7
  5. Weingart SN, Simchowitz B, Shiman L, et al. Clinicians' assess- ments of electronic medication safety alerts in ambulatory care. Arch Intern Med 2009;169(17):1627-1632
  6. Glassman PA, Simon B, Belperio P, Lanto A. Improving recognition of drug interactions: benefits and barriers to using automated drug alerts. Med Care 2002;40(12):1161-1171
  7. Ranji SR, Rennke S, Wachter RM. Computerised provider order entry combined with clinical decision support systems to improve medi- cation safety: a narrative review. BMJ Qual Saf 2014;23(09):773-780
  8. Kane-Gill SL, O'Connor MF, Rothschild JM, et al. Technologic distractions (part 1): Summary of approaches to manage alert quantity with intent to reduce alert fatigue and suggestions for alert fatigue metrics. Crit Care Med 2017;45(09):1481-1488
  9. van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13(02):138-147
  10. Isaac T, Weissman JS, Davis RB, et al. Overrides of medication alerts in ambulatory care. Arch Intern Med 2009;169(03):305-311
  11. Wachter R. The Digital Doctor: Hope, Hype, and Harm at the Dawn of Medicine's Computer Age. 1st ed. New York, NY: McGraw Hill Education; 2015
  12. Knight AM, Falade O, Maygers J, Sevransky JE. Factors associated with medication warning acceptance for hospitalized adults. J Hosp Med 2015;10(01):19-25
  13. Payne TH, Hines LE, Chan RC, et al. Recommendations to improve the usability of drug-drug interaction clinical decision support alerts. J Am Med Inform Assoc 2015;22(06):1243-1250
  14. Phansalkar S, van der Sijs H, Tucker AD, et al. Drug-drug inter- actions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc 2013; 20(03):489-493
  15. Bates DW. CPOE and clinical decision support in hospitals: getting the benefits: comment on "Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction". Arch Intern Med 2010;170(17):1583-1584
  16. Phansalkar S, Edworthy J, Hellier E, et al. A review of human factors principles for the design and implementation of medica- tion safety alerts in clinical information systems. J Am Med Inform Assoc 2010;17(05):493-501
  17. Classen DC, Phansalkar S, Bates DW. Critical drug-drug interactions for use in electronic health records systems with computerized physician order entry: review of leading approaches. J Patient Saf 2011;7(02):61-65
  18. van der Sijs H, Aarts J, van Gelder T, Berg M, Vulto A. Turning off frequently overridden drug alerts: limited opportunities for doing it safely. J Am Med Inform Assoc 2008;15(04):439-448
  19. van der Sijs H, Kowlesar R, Aarts J, Berg M, Vulto A, van Gelder T. Unintended consequences of reducing QT-alert overload in a computerized physician order entry system. Eur J Clin Pharmacol 2009;65(09):919-925
  20. Lee EK, Mejia AF, Senior T, Jose J. Improving patient safety through medical alert management: An automated decision tool to reduce alert fatigue. AMIA Annu Symp Proc 2010;2010:417-421
  21. Beccaro MA, Villanueva R, Knudson KM, Harvey EM, Langle JM, Paul W. Decision support alerts for medication ordering in a computer- ized provider order entry (CPOE) system: A systematic approach to decrease alerts. Appl Clin Inform 2010;1(03):346-362
  22. Schreiber R, Gregoire JA, Shaha JE, Shaha SH. Think time: a novel approach to analysis of clinicians' behavior after reduction of drug-drug interaction alerts. Int J Med Inform 2017;97:59-67 32
  23. Dexheimer JW, Kirkendall ES, Kouril M, et al. The effects of medication alerts on prescriber response in a pediatric hospital. Appl Clin Inform 2017;8(02):491-501
  24. Schiff GD, Hickman TT, Volk LA, Bates DW, Wright A. Computer- ised prescribing for safer medication ordering: still a work in progress. BMJ Qual Saf 2016;25(05):315-319
  25. Kawamanto K, Flynn MC, Kukhareva P, et al. A pragmatic guide to establishing clinical decision support governance and addressing decision support fatigue: A case study. AMIA Annu Symp Proc 2018;2018:624-633
  26. Rehr CA, Wong A, Seger DL, Bates DW. Determining inappropriate medication alerts from "inaccurate warning" overrides in the intensive care unit. Appl Clin Inform 2018;9(02):268-274
  27. Ko Y, Abarca J, Malone DC, et al. Practitioners' views on comput- erized drug-drug interaction alerts in the VA system. J Am Med Inform Assoc 2007;14(01):56-64 39
  28. Spina JR, Glassman PA, Simon B, et al. Potential safety gaps in order entry and automated drug alerts: a nationwide survey of VA physician self-reported practices with computerized order entry. Med Care 2011;49(10):904-910
  29. Kesselheim AS, Cresswell K, Phansalkar S, Bates DW, Sheikh A. Clinical decision support systems could be modified to reduce 'alert fatigue' while still minimizing the risk of litigation. Health Aff (Millwood) 2011;30(12):2310-2317
  30. Phansalkar S, Desai AA, Bell D, et al. High-priority drug-drug interactions for use in electronic health records. J Am Med Inform Assoc 2012;19(05):735-743
  31. McEvoy DS, Sittig DF, Hickman TT, et al. Variation in high- priority drug-drug interaction alerts across institutions and electronic health records. J Am Med Inform Assoc 2017;24 (02):331-338
  32. Tilson H, Hines LE, McEvoy G, et al. Recommendations for select- ing drug-drug interactions for clinical decision support. Am J Health Syst Pharm 2016;73(08):576-585
  33. Nanji KC, Seger DL, Slight SP, et al. Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc 2018;25(05):476-481
  34. Wong A, Amato MG, Seger DL, et al. Prospective evaluation of medication-related clinical decision support over-rides in the intensive care unit. BMJ Qual Saf 2018;27(09):718-724
  35. Wong A, Amato MG, Seger DL, et al. Evaluation of medication- related clinical decision support alert overrides in the intensive care unit. J Crit Care 2017;39:156-161
  36. Coleman JJ, van der Sijs H, Haefeli WE, et al. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC Med Inform Decis Mak 2013;13:111
  37. Paterno MD, Maviglia SM, Gorman PN, et al. Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc 2009;16(01):40-46

Cited by

Sorted by citations

Authors
Knight, A. M., Maygers, J., Foltz, K. A., John, I. S., Yeh, H. C., & Brotman, D. J.
Affiliations
  • Johns Hopkins University (US)
  • American Pharmacists Association (US)
Corresponding author
Knight, Amy M.
Venue
Applied Clinical Informatics
Published
2019-10
Volume
10
Issue
05
Pages
927-934
ISSN
1869-0327 (ISSN-L)
OpenAlex
W2994538615
Cited by
22
References
7
Citations by year
2026: 2 · 2025: 6 · 2024: 5 · 2023: 4 · 2022: 3 · 2021: 1 · 2020: 1
FWCI
1.32
Type
journal article
Language
en
Open access
Locations
  • Applied Clinical Informatics (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • Johns Hopkins Institute for Clinical and Translational Research (UL1 TR 001079)
  • National Center for Advancing Translational Sciences
  • National Institutes of Health
  • NIH Roadmap for Medical Research
Topics
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Medicine › Geriatrics and Gerontology › Pharmaceutical Practices and Patient Outcomes
  • Health Sciences › Health Professions › Emergency Medical Services › Patient Safety and Medication Errors
Keywords
drug-drug interactions, medication alerts, alert burden, electronic order entry, clinical decision support, drug safety
MeSH terms
Drug Interactions, Female, Humans, Male, Medication Errors, Middle Aged, Medical Order Entry Systems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review