Cited by 61Electronic Health Records SystemsPharmaceutical Practices and Patient OutcomesPatient Safety and Medication Errors

Structured override reasons for drug-drug interaction alerts in electronic health records

Open access
Wright, Adam, McEvoy, Dustin S, Aaron, Skye, McCoy, Allison B, Amato, Mary G, Kim, Hyun, Ai, Angela, Cimino, James J +11 more·2019·Journal of the American Medical Informatics Association

Abstract

Objective: The study sought to determine availability and use of structured override reasons for drug-drug interaction (DDI) alerts in electronic health records. Materials and Methods: We collected data on DDI alerts and override reasons from 10 clinical sites across the United States using a variety of electronic health records. We used a multistage iterative card sort method to categorize the override reasons from all sites and identified best practices. Results: Our methodology established 177 unique override reasons across the 10 sites. The number of coded override reasons at each site ranged from 3 to 100. Many sites offered override reasons not relevant to DDIs. Twelve categories of override reasons were identified. Three categories accounted for 78% of all overrides: “will monitor or take precautions,” “not clinically significant,” and “benefit outweighs risk.” Discussion: We found wide variability in override reasons between sites and many opportunities to improve alerts. Some override reasons were irrelevant to DDIs. Many override reasons attested to a future action (eg, decreasing a dose or ordering monitoring tests), which requires an additional step after the alert is overridden, unless the alert is made actionable. Some override reasons deferred to another party, although override reasons often are not visible to other users. Many override reasons stated that the alert was inaccurate, suggesting that specificity of alerts could be improved. Conclusions: Organizations should improve the options available to providers who choose to override DDI alerts. DDI alerting systems should be actionable and alerts should be tailored to the patient and drug pairs.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocz033

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. [1]Classen DC, Pestotnik SL, Evans RS, Lloyd JF, Burke JP. Adverse drug events in hospitalized patients. Excess length of stay, extra costs, and attributable mortality. JAMA 1997; 277 4: 301– 6. 9002492
  2. [2]Melmon KL. Preventable drug reactions–causes and cures. N Engl J Med 1971; 284 24: 1361– 8. 4930604
  3. [3]Bates DW, Cullen DJ, Laird N, et al Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group. JAMA 1995; 274 1: 29– 34. 7791255
  4. [4]Bates DW, Leape LL, Petrycki S. Incidence and preventability of adverse drug events in hospitalized adults. J Gen Intern Med 1993; 8 6: 289– 94. 8320571
  5. [6]Lesar TS, Briceland L, Stein DS. Factors related to errors in medication prescribing. JAMA 1997; 277 4: 312– 7. 9002494
  6. [7]Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003; 163 12: 1409– 16. 12824090
  7. [8]Kuperman GJ, Bobb A, Payne TH, et al Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc 2007; 14 1: 29– 40. 17068355 PMC2215064
  8. [9]Wright A, Aaron S, Seger DL, Samal L, Schiff GD, Bates DW. Reduced effectiveness of interruptive drug-drug interaction alerts after conversion to a commercial electronic health record. J Gen Intern Med 2018; 33 (11): 1868–76. PMC6206354 29766382
  9. [10]van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 2: 138– 47. 16357358 PMC1447540
  10. [11]Peterson JF, Bates DW. Preventable medication errors: identifying and eliminating serious drug interactions. J Am Pharm Assoc (Wash) 2001; 41 2: 159– 60. 11297326
  11. [13]McCoy AB, Thomas EJ, Krousel-Wood M, Sittig DF. Clinical decision support alert appropriateness: a review and proposal for improvement. Ochsner J 2014; 14 2: 195– 202. 24940129 PMC4052586
  12. [14]McCoy AB, Waitman LR, Lewis JB, et al A framework for evaluating the appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc 2012; 19 3: 346– 52. 21849334 PMC3341775
  13. [15]Rehr CA, Wong A, Seger DL, Bates DW. Determining Inappropriate Medication Alerts from “Inaccurate Warning” Overrides in the Intensive Care Unit. Appl Clin Inform 2018; 09 2: 268– 74. PMC5916798 29695013
  14. [17]Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003; 163 21: 2625– 31. 14638563
  15. [18]Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med 2010; 363 6: 501– 4. 20647183
  16. [19]Wright A, Henkin S, Feblowitz J, McCoy AB, Bates DW, Sittig DF. Early results of the meaningful use program for electronic health records. N Engl J Med 2013; 368 8: 779– 80. 23425186
  17. [20]Phansalkar S, van der Sijs H, Tucker AD, et al Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc 2013; 20 3: 489– 93. 23011124 PMC3628052
  18. [21]Peters LB, Bahr N, Bodenreider O. Evaluating drug-drug interaction information in NDF-RT and DrugBank. J Biomed Semantics 2015; 6: 19. 25964850 PMC4426554
  19. [23]Barrons R. Evaluation of personal digital assistant software for drug interactions. Am J Health Syst Pharm 2004; 61 4: 380– 5. 15011766
  20. [24]Page N, Baysari MT, Westbrook JI. A systematic review of the effectiveness of interruptive medication prescribing alerts in hospital CPOE systems to change prescriber behavior and improve patient safety. Int J Med Inform 2017; 105: 22– 30. 28750908
  21. [25]Wong A, Wright A, Seger DL, Amato MG, Fiskio JM, Bates D. Comparison of overridden medication-related clinical decision support in the intensive care unit between a commercial system and a legacy system. Appl Clin Inform 2017; 08 3: 866– 79. PMC6220698 28832067
  22. [26]McEvoy DS, Sittig DF, Hickman TT, et al Variation in high-priority drug-drug interaction alerts across institutions and electronic health records. J Am Med Inform Assoc 2017; 24 2: 331– 8. 27570216 PMC5391726
  23. [27]Phansalkar S, Desai AA, Bell D, et al High-priority drug-drug interactions for use in electronic health records. J Am Med Inform Assoc 2012; 19 5: 735– 43. 22539083 PMC3422823
  24. [28]Powers EM, Shiffman RN, Melnick ER, Hickner A, Sharifi M. Efficacy and unintended consequences of hard-stop alerts in electronic health record systems: a systematic review. J Am Med Inform Assoc 2018; 25 11: 1556–66. PMC6915824 30239810
  25. [29]Chused AE, Kuperman GJ, Stetson PD. Alert override reasons: a failure to communicate. AMIA Annu Symp Proc 2008; 2008: 111– 5. PMC2655991 18999082
  26. [30]Wright A, Ash JS, Aaron S, et al Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: results of a Delphi study. Int J Med Inform 2018; 118: 78– 85. 30153926 PMC6128667
  27. [35]Wright A, Goldberg H, Hongsermeier T, Middleton B. A description and functional taxonomy of rule-based decision support content at a large integrated delivery network. J Am Med Inform Assoc 2007; 14 4: 489– 96. 17460131 PMC2244910
  28. [36]Wright A, Sittig DF, Ash JS, et al Development and evaluation of a comprehensive clinical decision support taxonomy: comparison of front-end tools in commercial and internally developed electronic health record systems. J Am Med Inform Assoc 2011; 18 3: 232– 42. 21415065 PMC3078666
  29. [37]Dykstra RH, Ash JS, Campbell E, et al Persistent paper: the myth of “going paperless”. AMIA Annu Symp Proc 2009; 2009: 158– 62. 20351841 PMC2815440
  30. [39]Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. The unintended consequences of computerized provider order entry: findings from a mixed methods exploration. Int J Med Inform 2009; 78 (Suppl 1): S69– 76. 18786852 PMC2683676
  31. [40]Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. Exploring the unintended consequences of computerized physician order entry. Stud Health Technol Inform 2007; 129 (Pt 1): 198– 202. 17911706
  32. [41]Ai A, Wong A, Amato M, Wright A. Communication failure: analysis of prescribers' use of an internal free-text field on electronic prescriptions. J Am Med Inform Assoc 2018; 25 6: 709– 14. 29474659 PMC7647024
  33. [42]Wright A, Ai A, Ash J, et al Clinical decision support alert malfunctions: analysis and empirically derived taxonomy. J Am Med Inform Assoc 2018; 25 5: 496– 506. 29045651 PMC6019061
  34. [43]Grizzle AJ, Mahmood MH, Ko Y, et al Reasons provided by prescribers when overriding drug-drug interaction alerts. Am J Manag Care 2007; 13 10: 573– 8. 17927462
  35. [44]Renzi R, Finkbeiner S. Ciprofloxacin interaction with sodium warfarin: a potentially dangerous side effect. Am J Emerg Med 1991; 9 6: 551– 2. 1930395
  36. [45]Ahmed A, Stephens JC, Kaus CA, Fay WP. Impact of preemptive warfarin dose reduction on anticoagulation after initiation of trimethoprim-sulfamethoxazole or levofloxacin. J Thromb Thrombolysis 2008; 26 1: 44– 8. 17985084
  37. [46]Sittig DF, Belmont E, Singh H. Improving the safety of health information technology requires shared responsibility: It is time we all step up. Healthc (Amst) 2018; 6 1: 7– 12. 28716376
  38. [47]Ash JS, Sittig DF, McMullen CK, et al Studying the vendor perspective on clinical decision support. AMIA Annu Symp Proc 2011; 2011: 80– 7. 22195058 PMC3243293
  39. [48]Paterno MD, Maviglia SM, Gorman PN, et al Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc 2009; 16 1: 40– 6. 18952941 PMC2605599
  40. [49]Bates DW, Kuperman GJ, Wang S, et al Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 6: 523– 30. 12925543 PMC264429
  41. [50]Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 7494: 765. 15767266 PMC555881
  42. [51]Classen DC, Phansalkar S, Bates DW. Critical drug-drug interactions for use in electronic health records systems with computerized physician order entry: review of leading approaches. J Patient Saf 2011; 7 2: 61– 5. 21577077
  43. [52]Harper MB, Longhurst CA, McGuire TL, Tarrago R, Desai BR, Patterson A. Core drug-drug interaction alerts for inclusion in pediatric electronic health records with computerized prescriber order entry. J Patient Saf 2014; 10 1: 59– 63. 24522227
  44. [53]Leung AA, Keohane C, Lipsitz S, et al Relationship between medication event rates and the Leapfrog computerized physician order entry evaluation tool. J Am Med Inform Assoc 2013; 20 (e1): e85– 90. 23599225 PMC3715361
  45. [54]Metzger J, Welebob E, Bates DW, Lipsitz S, Classen DC. Mixed results in the safety performance of computerized physician order entry. Health Aff (Millwood) 2010; 29 4: 655– 63. 20368595
  46. [55]Chaparro JD, Classen DC, Danforth M, Stockwell DC, Longhurst CA. National trends in safety performance of electronic health record systems in children's hospitals. J Am Med Inform Assoc 2017; 24 2: 268– 74. 27638908 PMC7651940
  47. [56]Aaron S, McEvoy DS, Ray S, Hickman T-TT, Wright A. Cranky comments: detecting clinical decision support malfunctions through free-text override reasons. J Am Med Inform Assoc 2019; 26 1: 37– 43. 30590557 PMC6308015

Cited by

Sorted by citations

Authors
Wright, A., McEvoy, D. S., Aaron, S., McCoy, A. B., Amato, M. G., Kim, H., Ai, A., Cimino, J. J., Desai, B. R., El-Kareh, R., Galanter, W., Longhurst, C. A., Malhotra, S., Radecki, R. P., Samal, L., Schreiber, R., Shelov, E., Sirajuddin, A. M., & Sittig, D. F.
Affiliations
  • Brigham and Women's Hospital (US)
  • Partners eCare, Partners HealthCare, Boston, Massachusetts, USA
  • Vanderbilt University Medical Center (US)
  • Boston Children's Hospital (US)
  • University of Wisconsin–Madison (US)
  • University of Alabama at Birmingham (US)
  • Children's Hospital of Philadelphia (US)
  • UC San Diego Health System (US)
  • University of Illinois Chicago (US)
  • Department of Medicine, UC San Diego Health, University of California, San Diego, San Diego, California, USA
  • Weill Cornell Medicine (US)
  • Department of Emergency Medicine, Northwest Permanente, Portland, Oregon, USA
  • Geisinger Health System (US)
  • Memorial Hermann (US)
  • The University of Texas Health Science Center at Houston (US)
Corresponding author
Wright, Adam
Venue
Journal of the American Medical Informatics Association
Published
2019-04-26
Volume
26
Issue
10
Pages
934-942
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W2941666461
Cited by
61
References
5
Citations by year
2026: 1 · 2025: 8 · 2024: 3 · 2023: 10 · 2022: 24 · 2021: 4 · 2020: 8 · 2019: 2
FWCI
10.37
Type
journal article
Language
en
Open access
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by-nc)
Funders
  • National Library of Medicine
  • National Institutes of Health (R01LM011966)
Topics
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Medicine › Geriatrics and Gerontology › Pharmaceutical Practices and Patient Outcomes
  • Health Sciences › Health Professions › Emergency Medical Services › Patient Safety and Medication Errors
Keywords
drug-drug interaction alerts, electronic health records, clinical decision support
MeSH terms
Aged, Drug Interactions, Female, Humans, Male, Medication Errors, Middle Aged, Medical Order Entry Systems, Electronic Prescribing, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review