Structured override reasons for drug-drug interaction alerts in electronic health records
Open accessAbstract
Objective: The study sought to determine availability and use of structured override reasons for drug-drug interaction (DDI) alerts in electronic health records. Materials and Methods: We collected data on DDI alerts and override reasons from 10 clinical sites across the United States using a variety of electronic health records. We used a multistage iterative card sort method to categorize the override reasons from all sites and identified best practices. Results: Our methodology established 177 unique override reasons across the 10 sites. The number of coded override reasons at each site ranged from 3 to 100. Many sites offered override reasons not relevant to DDIs. Twelve categories of override reasons were identified. Three categories accounted for 78% of all overrides: “will monitor or take precautions,” “not clinically significant,” and “benefit outweighs risk.” Discussion: We found wide variability in override reasons between sites and many opportunities to improve alerts. Some override reasons were irrelevant to DDIs. Many override reasons attested to a future action (eg, decreasing a dose or ordering monitoring tests), which requires an additional step after the alert is overridden, unless the alert is made actionable. Some override reasons deferred to another party, although override reasons often are not visible to other users. Many override reasons stated that the alert was inaccurate, suggesting that specificity of alerts could be improved. Conclusions: Organizations should improve the options available to providers who choose to override DDI alerts. DDI alerting systems should be actionable and alerts should be tailored to the patient and drug pairs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
High override rates of 56.3%-95.7% have already been described for DDI alerts.
“High override rates of 56.3%-95.7% have already been described for DDI alerts [13, 21, 27–32].”
From Overall performance of a drug–drug interaction clinical decision support system: quantitative evaluation and end-user survey (2022) · cites this paper as [28]
The Results report a high overall DDI-alert override rate of 91%. Table 2 gives site-level rates of 65%–95%, but does not report the claimed 56.3%–95.7% range. This source supports the general point about high override rates, not those exact endpoints. (LLM, conf=0.88)
Experimental — accuracy not yet measuredOur study results are in agreement with previous reports.
“Our study results are in agreement with previous reports.”
From Overall performance of a drug–drug interaction clinical decision support system: quantitative evaluation and end-user survey (2022) · cites this paper as [28]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Relationship between medication errors and adverse drug events
Bates, D. W., Boyle, D. L., Vliet, M. B. V., Schneider, J., & Leape, L. (1995) · Journal of General Internal Medicine · doi:10.1007/bf02600255 · Cited by 981 · see citing works
Types of Unintended Consequences Related to Computerized Provider Order Entry
Campbell, E. M., Sittig, D. F., Ash, J. S., Guappone, K. P., & Dykstra, R. H. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2042 · Cited by 823 · see citing works
Self-citationComputerized Provider Order Entry Adoption: Implications for Clinical Workflow
Campbell, E. M., Guappone, K. P., Sittig, D. F., Dykstra, R. H., & Ash, J. S. (2009) · Journal of General Internal Medicine · doi:10.1007/s11606-008-0857-9 · Cited by 139 · see citing works
Self-citationSlight, S. P., Seger, D. L., Nanji, K. C., Cho, I., Maniam, N., Dykes, P. C., & Bates, D. W. (2013) · PLoS ONE · doi:10.1371/journal.pone.0085071 · Cited by 86 · see citing works
Wong, A., Rehr, C., Seger, D. L., Amato, M. G., Beeler, P. E., Slight, S. P., Wright, A., & Bates, D. W. (2019) · Drug Safety · doi:10.1007/s40264-018-0756-x · Cited by 21 · cited 1× · see citing works
“However, medication-related CDS alerts are frequently overridden,(9,10) sometimes with good cause, as adverse events rarely occur when prescribers override even the highest-level alerts.(11,12) Historically, the rationale for high override rates was alert fatigue and poor usability, but recent research suggests a more…”
BackgroundSelf-citation
Flagged first, then by citations
Research works printed but not linked: 47
- [1]Classen DC, Pestotnik SL, Evans RS, Lloyd JF, Burke JP. Adverse drug events in hospitalized patients. Excess length of stay, extra costs, and attributable mortality. JAMA 1997; 277 4: 301– 6. 9002492
- [2]Melmon KL. Preventable drug reactions–causes and cures. N Engl J Med 1971; 284 24: 1361– 8. 4930604
- [3]Bates DW, Cullen DJ, Laird N, et al Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group. JAMA 1995; 274 1: 29– 34. 7791255
- [4]Bates DW, Leape LL, Petrycki S. Incidence and preventability of adverse drug events in hospitalized adults. J Gen Intern Med 1993; 8 6: 289– 94. 8320571
- [6]Lesar TS, Briceland L, Stein DS. Factors related to errors in medication prescribing. JAMA 1997; 277 4: 312– 7. 9002494
- [7]Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003; 163 12: 1409– 16. 12824090
- [8]Kuperman GJ, Bobb A, Payne TH, et al Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc 2007; 14 1: 29– 40. 17068355 PMC2215064
- [9]Wright A, Aaron S, Seger DL, Samal L, Schiff GD, Bates DW. Reduced effectiveness of interruptive drug-drug interaction alerts after conversion to a commercial electronic health record. J Gen Intern Med 2018; 33 (11): 1868–76. PMC6206354 29766382
- [10]van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 2: 138– 47. 16357358 PMC1447540
- [11]Peterson JF, Bates DW. Preventable medication errors: identifying and eliminating serious drug interactions. J Am Pharm Assoc (Wash) 2001; 41 2: 159– 60. 11297326
- [13]McCoy AB, Thomas EJ, Krousel-Wood M, Sittig DF. Clinical decision support alert appropriateness: a review and proposal for improvement. Ochsner J 2014; 14 2: 195– 202. 24940129 PMC4052586
- [14]McCoy AB, Waitman LR, Lewis JB, et al A framework for evaluating the appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc 2012; 19 3: 346– 52. 21849334 PMC3341775
- [15]Rehr CA, Wong A, Seger DL, Bates DW. Determining Inappropriate Medication Alerts from “Inaccurate Warning” Overrides in the Intensive Care Unit. Appl Clin Inform 2018; 09 2: 268– 74. PMC5916798 29695013
- [17]Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003; 163 21: 2625– 31. 14638563
- [18]Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med 2010; 363 6: 501– 4. 20647183
- [19]Wright A, Henkin S, Feblowitz J, McCoy AB, Bates DW, Sittig DF. Early results of the meaningful use program for electronic health records. N Engl J Med 2013; 368 8: 779– 80. 23425186
- [20]Phansalkar S, van der Sijs H, Tucker AD, et al Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc 2013; 20 3: 489– 93. 23011124 PMC3628052
- [21]Peters LB, Bahr N, Bodenreider O. Evaluating drug-drug interaction information in NDF-RT and DrugBank. J Biomed Semantics 2015; 6: 19. 25964850 PMC4426554
- [23]Barrons R. Evaluation of personal digital assistant software for drug interactions. Am J Health Syst Pharm 2004; 61 4: 380– 5. 15011766
- [24]Page N, Baysari MT, Westbrook JI. A systematic review of the effectiveness of interruptive medication prescribing alerts in hospital CPOE systems to change prescriber behavior and improve patient safety. Int J Med Inform 2017; 105: 22– 30. 28750908
- [25]Wong A, Wright A, Seger DL, Amato MG, Fiskio JM, Bates D. Comparison of overridden medication-related clinical decision support in the intensive care unit between a commercial system and a legacy system. Appl Clin Inform 2017; 08 3: 866– 79. PMC6220698 28832067
- [26]McEvoy DS, Sittig DF, Hickman TT, et al Variation in high-priority drug-drug interaction alerts across institutions and electronic health records. J Am Med Inform Assoc 2017; 24 2: 331– 8. 27570216 PMC5391726
- [27]Phansalkar S, Desai AA, Bell D, et al High-priority drug-drug interactions for use in electronic health records. J Am Med Inform Assoc 2012; 19 5: 735– 43. 22539083 PMC3422823
- [28]Powers EM, Shiffman RN, Melnick ER, Hickner A, Sharifi M. Efficacy and unintended consequences of hard-stop alerts in electronic health record systems: a systematic review. J Am Med Inform Assoc 2018; 25 11: 1556–66. PMC6915824 30239810
- [29]Chused AE, Kuperman GJ, Stetson PD. Alert override reasons: a failure to communicate. AMIA Annu Symp Proc 2008; 2008: 111– 5. PMC2655991 18999082
- [30]Wright A, Ash JS, Aaron S, et al Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: results of a Delphi study. Int J Med Inform 2018; 118: 78– 85. 30153926 PMC6128667
- [35]Wright A, Goldberg H, Hongsermeier T, Middleton B. A description and functional taxonomy of rule-based decision support content at a large integrated delivery network. J Am Med Inform Assoc 2007; 14 4: 489– 96. 17460131 PMC2244910
- [36]Wright A, Sittig DF, Ash JS, et al Development and evaluation of a comprehensive clinical decision support taxonomy: comparison of front-end tools in commercial and internally developed electronic health record systems. J Am Med Inform Assoc 2011; 18 3: 232– 42. 21415065 PMC3078666
- [37]Dykstra RH, Ash JS, Campbell E, et al Persistent paper: the myth of “going paperless”. AMIA Annu Symp Proc 2009; 2009: 158– 62. 20351841 PMC2815440
- [39]Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. The unintended consequences of computerized provider order entry: findings from a mixed methods exploration. Int J Med Inform 2009; 78 (Suppl 1): S69– 76. 18786852 PMC2683676
- [40]Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. Exploring the unintended consequences of computerized physician order entry. Stud Health Technol Inform 2007; 129 (Pt 1): 198– 202. 17911706
- [41]Ai A, Wong A, Amato M, Wright A. Communication failure: analysis of prescribers' use of an internal free-text field on electronic prescriptions. J Am Med Inform Assoc 2018; 25 6: 709– 14. 29474659 PMC7647024
- [42]Wright A, Ai A, Ash J, et al Clinical decision support alert malfunctions: analysis and empirically derived taxonomy. J Am Med Inform Assoc 2018; 25 5: 496– 506. 29045651 PMC6019061
- [43]Grizzle AJ, Mahmood MH, Ko Y, et al Reasons provided by prescribers when overriding drug-drug interaction alerts. Am J Manag Care 2007; 13 10: 573– 8. 17927462
- [44]Renzi R, Finkbeiner S. Ciprofloxacin interaction with sodium warfarin: a potentially dangerous side effect. Am J Emerg Med 1991; 9 6: 551– 2. 1930395
- [45]Ahmed A, Stephens JC, Kaus CA, Fay WP. Impact of preemptive warfarin dose reduction on anticoagulation after initiation of trimethoprim-sulfamethoxazole or levofloxacin. J Thromb Thrombolysis 2008; 26 1: 44– 8. 17985084
- [46]Sittig DF, Belmont E, Singh H. Improving the safety of health information technology requires shared responsibility: It is time we all step up. Healthc (Amst) 2018; 6 1: 7– 12. 28716376
- [47]Ash JS, Sittig DF, McMullen CK, et al Studying the vendor perspective on clinical decision support. AMIA Annu Symp Proc 2011; 2011: 80– 7. 22195058 PMC3243293
- [48]Paterno MD, Maviglia SM, Gorman PN, et al Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc 2009; 16 1: 40– 6. 18952941 PMC2605599
- [49]Bates DW, Kuperman GJ, Wang S, et al Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 6: 523– 30. 12925543 PMC264429
- [50]Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 7494: 765. 15767266 PMC555881
- [51]Classen DC, Phansalkar S, Bates DW. Critical drug-drug interactions for use in electronic health records systems with computerized physician order entry: review of leading approaches. J Patient Saf 2011; 7 2: 61– 5. 21577077
- [52]Harper MB, Longhurst CA, McGuire TL, Tarrago R, Desai BR, Patterson A. Core drug-drug interaction alerts for inclusion in pediatric electronic health records with computerized prescriber order entry. J Patient Saf 2014; 10 1: 59– 63. 24522227
- [53]Leung AA, Keohane C, Lipsitz S, et al Relationship between medication event rates and the Leapfrog computerized physician order entry evaluation tool. J Am Med Inform Assoc 2013; 20 (e1): e85– 90. 23599225 PMC3715361
- [54]Metzger J, Welebob E, Bates DW, Lipsitz S, Classen DC. Mixed results in the safety performance of computerized physician order entry. Health Aff (Millwood) 2010; 29 4: 655– 63. 20368595
- [55]Chaparro JD, Classen DC, Danforth M, Stockwell DC, Longhurst CA. National trends in safety performance of electronic health record systems in children's hospitals. J Am Med Inform Assoc 2017; 24 2: 268– 74. 27638908 PMC7651940
- [56]Aaron S, McEvoy DS, Ray S, Hickman T-TT, Wright A. Cranky comments: detecting clinical decision support malfunctions through free-text override reasons. J Am Med Inform Assoc 2019; 26 1: 37– 43. 30590557 PMC6308015
Cited by
Sorted by citations
Appropriateness of Overridden Alerts in Computerized Physician Order Entry: Systematic Review
Poly, T. N., Islam, M. M., Yang, H.-C., & Li, Y.-C. (. (2020) · JMIR Medical Informatics · doi:10.2196/15653 · Cited by 118 · see citing works
Van De Sijpe, G., Quintens, C., Walgraeve, K., Van Laer, E., Penny, J., De Vlieger, G., Schrijvers, R., De Munter, P., Foulon, V., Casteels, M., et al. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01783-z · Cited by 54 · see citing works
- Authors
- Wright, A., McEvoy, D. S., Aaron, S., McCoy, A. B., Amato, M. G., Kim, H., Ai, A., Cimino, J. J., Desai, B. R., El-Kareh, R., Galanter, W., Longhurst, C. A., Malhotra, S., Radecki, R. P., Samal, L., Schreiber, R., Shelov, E., Sirajuddin, A. M., & Sittig, D. F.
- Affiliations
- Brigham and Women's Hospital (US)
- Partners eCare, Partners HealthCare, Boston, Massachusetts, USA
- Vanderbilt University Medical Center (US)
- Boston Children's Hospital (US)
- University of Wisconsin–Madison (US)
- University of Alabama at Birmingham (US)
- Children's Hospital of Philadelphia (US)
- UC San Diego Health System (US)
- University of Illinois Chicago (US)
- Department of Medicine, UC San Diego Health, University of California, San Diego, San Diego, California, USA
- Weill Cornell Medicine (US)
- Department of Emergency Medicine, Northwest Permanente, Portland, Oregon, USA
- Geisinger Health System (US)
- Memorial Hermann (US)
- The University of Texas Health Science Center at Houston (US)
- Corresponding author
- Wright, Adam
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-04-26
- Volume
- 26
- Issue
- 10
- Pages
- 934-942
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC6748816
- OpenAlex
- W2941666461
- Cited by
- 61
- References
- 5
- Citations by year
- 2026: 1 · 2025: 8 · 2024: 3 · 2023: 10 · 2022: 24 · 2021: 4 · 2020: 8 · 2019: 2
- FWCI
- 10.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by-nc)
- Funders
- National Library of Medicine
- National Institutes of Health (R01LM011966)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Geriatrics and Gerontology › Pharmaceutical Practices and Patient Outcomes
- Health Sciences › Health Professions › Emergency Medical Services › Patient Safety and Medication Errors
- Keywords
- drug-drug interaction alerts, electronic health records, clinical decision support
- MeSH terms
- Aged, Drug Interactions, Female, Humans, Male, Medication Errors, Middle Aged, Medical Order Entry Systems, Electronic Prescribing, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review