Optimization of drug–drug interaction alert rules in a pediatric hospital's electronic health record system using a visual analytics dashboard
Open accessAbstract
Objective To develop and evaluate an electronic dashboard of hospital-wide electronic health record medication alerts for an alert fatigue reduction quality improvement project. Methods We used visual analytics software to develop the dashboard. We collaborated with the hospital-wide Clinical Decision Support committee to perform three interventions successively deactivating clinically irrelevant drug–drug interaction (DDI) alert rules. We analyzed the impact of the interventions on care providers’ and pharmacists’ alert and override rates using an interrupted time series framework with piecewise regression. Results We evaluated 2 391 880 medication alerts between January 31, 2011 and January 26, 2014. For pharmacists, the median alert rate prior to the first DDI deactivation was 58.74 alerts/100 orders (IQR 54.98–60.48) and 25.11 alerts/100 orders (IQR 23.45–26.57) following the three interventions (p<0.001). For providers, baseline median alert rate prior to the first round of DDI deactivation was 19.73 alerts/100 orders (IQR 18.66–20.24) and 15.11 alerts/100 orders (IQR 14.44–15.49) following the three interventions (p<0.001). In a subgroup analysis, we observed a decrease in pharmacists’ override rates for DDI alerts that were not modified in the system from a median of 93.06 overrides/100 alerts (IQR 91.96–94.33) to 85.68 overrides/100 alerts (IQR 84.29–87.15, p<0.001). The medication serious safety event rate decreased during the study period, and there were no serious safety events reported in association with the deactivated alert rules. Conclusions An alert dashboard facilitated safe rapid-cycle reductions in alert burden that were temporally associated with lower pharmacist override rates in a subgroup of DDIs not directly affected by the interventions; meanwhile, the pharmacists’ frequency of selecting the ‘cancel’ option increased. We hypothesize that reducing the alert burden enabled pharmacists to devote more attention to clinically relevant alerts.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A pre-post study showed that after irrelevant alerts were retired, pharmacist alert override rates decreased from 93% to 86%.
“A pre-post study showed that after irrelevant alerts were retired, pharmacist alert override rates decreased from 93% to 86% [21].”
From Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system (2017) · cites this paper as [21]
Some organizations have successfully implemented these tools and reduced the number of alerts and overrides.
“Although some organizations have successfully implemented these tools and reduced the number of alerts and overrides,38 other reports have found that many organizations are not following these recommendations, in part due to a lack of consensus for whether or not to turn off or modify, or how to modify, alerts.27,28,39”
From Clinician collaboration to improve clinical decision support: the Clickbusters initiative (2022) · cites this paper as [38]
A frequently described approach in prior literature involves individuals or committees reviewing EHR utilization logs or dashboards, which are effective at identifying individual alerts with high override rates and alert trends across departments or provider type.
“One of the most frequently described approaches described in prior literature has involved individuals or committees reviewing EHR utilization logs or dashboards, as these are very effective at identifying individual alerts with high override rates, as well as alert trends across departments or provider type.27,28,34,38”
From Clinician collaboration to improve clinical decision support: the Clickbusters initiative (2022) · cites this paper as [38]
Cites
The visual display of quantitative information
(1985) · General Pharmacology The Vascular System · doi:10.1016/0306-3623(85)90230-7 · Cited by 5,090 · see citing works
The Visual Display of Quantitative Information.
Fayers, P. M., & Tufte, E. R. (1990) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.2307/2983122 · Cited by 1,951 · see citing works
Ash, J. S. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1471 · Cited by 1,789 · see citing works
The Visual Display of Quantitative Information
Mulrow, E. (2002) · Technometrics · doi:10.1198/tech.2002.s78 · Cited by 1,546 · see citing works
The Data Warehouse Lifecycle Toolkit
Kimball, R., Ross, M., Thornthwaite, W., Mundy, J., & Becker, B. (2009) · Cited by 697 · see citing works
The Incident Reporting System Does Not Detect Adverse Drug Events: A Problem for Quality Improvement
Cullen, D. J., Bates, D. W., Small, S., Cooper, J. B., Nemeskal, A. R., & Leape, L. L. (1995) · The Joint Commission Journal on Quality Improvement · doi:10.1016/s1070-3241(16)30180-8 · Cited by 601 · see citing works
Thomas, J. J., & Cook, K. A. (2006) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2006.5 · Cited by 565 · see citing works
Physicians' Decisions to Override Computerized Drug Alerts in Primary Care
Weingart, S. N., Toth, M., Sands, D. Z., Aronson, M. D., Davis, R. B., & Phillips, R. S. (2003) · Archives of Internal Medicine · doi:10.1001/archinte.163.21.2625 · Cited by 487 · see citing works
Fortescue, E. B., Kaushal, R., Landrigan, C. P., McKenna, K. J., Clapp, M., Federico, F. A., Goldmann, D. A., & Bates, D. W. (2003) · PEDIATRICS · doi:10.1542/peds.111.4.722 · Cited by 462 · see citing works
Phansalkar, S., van der Sijs, H., Tucker, A. D., Desai, A. A., Bell, D. S., Teich, J. M., Middleton, B., & Bates, D. W. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001089 · Cited by 239 · see citing works
High-priority drug–drug interactions for use in electronic health records
Phansalkar, S., Desai, A. A., Bell, D., Yoshida, E., Doole, J., Czochanski, M., Middleton, B., & Bates, D. W. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000612 · Cited by 134 · see citing works
Seidling, H. M., Phansalkar, S., Seger, D. L., Paterno, M. D., Shaykevich, S., Haefeli, W. E., & Bates, D. W. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2010-000039 · Cited by 129 · see citing works
A critical evaluation of clinical decision support for the detection of drug–drug interactions
Smithburger, P. L., Buckley, M. S., Bejian, S. M., Burenheide, K., & Kane‐Gill, S. L. (2011) · Expert Opinion on Drug Safety · doi:10.1517/14740338.2011.583916 · Cited by 114 · see citing works
Kuperman, G. J., Reichley, R. M., & Bailey, T. C. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2055 · Cited by 78 · see citing works
Smithburger, P. L., Kane‐Gill, S. L., Benedict, N. J., Falcione, B. A., & Seybert, A. L. (2010) · Annals of Pharmacotherapy · doi:10.1345/aph.1p377 · Cited by 62 · see citing works
Jha, A. K., Laguette, J., Seger, A. C., & Bates, D. W. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2634 · Cited by 60 · see citing works
Chui, K. K. H., Wenger, J., Cohen, S. A., et al. (2011) · PLoS ONE · doi:10.1371/journal.pone.0014683 · Cited by 44 · see citing works
Computerized Provider Order Entry and Patient Safety
Abramson, E. L., & Kaushal, R. (2012) · Pediatric Clinics of North America · doi:10.1016/j.pcl.2012.08.001 · Cited by 38 · see citing works
Taegtmeyer, A. B., Kullak-Ublick, G. A., Widmer, N., Falk, V., & Jetter, A. (2012) · Cardiology · doi:10.1159/000343272 · Cited by 31 · see citing works
Harper, M. B., Longhurst, C., McGuire, T. L., et al. (2014) · Journal of Patient Safety · doi:10.1097/pts.0000000000000050 · Cited by 24 · see citing works
Kumasaka, N., Nakamura, Y., & Kamatani, N. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010207 · Cited by 23 · see citing works
Guzek, M., Zorina, O. I., Semmler, A., Gonzenbach, R. R., Huber, M., Kullak-Ublick, G. A., Weller, M., & Russmann, S. (2011) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.2197 · Cited by 18 · see citing works
Contradictions in Contraindications for Drug-Drug Interactions
Shah, V. S., Weber, R. J., & Nahata, M. C. (2011) · Annals of Pharmacotherapy · doi:10.1345/aph.1p792 · Cited by 12 · see citing works
Customizing a commercial rule base for detecting drug-drug interactions.
Resetar, E., Reichley, R. M., Noirot, L. A., et al. (2005) · PubMed · Cited by 11 · see citing works
Visual analytics for immunologists: Data compression and fractal distributions
Naumova, E. N. (2010) · Self/Nonself · doi:10.4161/self.1.3.12876 · Cited by 7 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
with the HITEC Investigators, Ancker, J. S., Edwards, A., Nosal, S., Hauser, D., Mauer, E., & Kaushal, R. (2017) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-017-0430-8 · Cited by 709 · see citing works
“Examples of quantitative research supporting the overload explanation include studies finding that override rates decreased overall after irrelevant alerts were discontinued [21], that ability to remember alerts declined as number of different alerts rose [22], and that inappropriate overrides were associated with num…”
BackgroundAn integrated big data analytics-enabled transformation model: application to health care
Wang, Y., Kung, L., Wang, W. Y. C., & Cegielski, C. G. (2018) · Information & Management · doi:10.1016/j.im.2017.04.001 · Cited by 508 · see citing works
Croux, C., Nguyen, N.-P.-T., Lee, J., Raynaud, C., Saint-Prix, F., Gonzalez-Pajuelo, M., Meynial-Salles, I., & Soucaille, P. (2016) · Biotechnology for Biofuels · doi:10.1186/s13068-016-0432-2 · Cited by 41 · see citing works
Clinician collaboration to improve clinical decision support: the Clickbusters initiative
McCoy, A. B., Russo, E. M., Johnson, K. B., Addison, B., Patel, N., Wanderer, J. P., Mize, D. E., Jackson, J. G., Reese, T. J., Littlejohn, S., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac027 · Cited by 41 · see citing works
- Authors
- Simpao, A. F., Ahumada, L. M., Desai, B. R., Bonafide, C. P., Gálvez, J. A., Rehman, M. A., Jawad, A. F., Palma, K. L., & Shelov, E. D.
- Affiliations
- Department of Anesthesiology and Critical Care Medicine, Perelman School of Medicine at the University of Pennsylvania and the Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, USA
- Children's Hospital of Philadelphia (US)
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2014-10-15
- Volume
- 22
- Issue
- 2
- Pages
- 361-369
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC11749158
- OpenAlex
- W2115513526
- Cited by
- 94
- References
- 25
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 9 · 2023: 6 · 2022: 12 · 2021: 12 · 2020: 14 · 2019: 8 · 2018: 4 · 2017: 5 · 2016: 9 · 2015: 7 · 2014: 1
- FWCI
- 13.41
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Geriatrics and Gerontology › Pharmaceutical Practices and Patient Outcomes
- Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
- Keywords
- drug-drug interaction alerts, alarm fatigue, clinical decision support, electronic health record alerts, visual analytics dashboard, children's hospitals, drug safety
- MeSH terms
- Audiovisual Aids, Child, Drug Interactions, Drug Therapy, Computer-Assisted, Hospitals, Pediatric, Humans, Medication Errors, Pharmacists, Software, User-Computer Interface, Medical Records Systems, Computerized, Decision Support Systems, Clinical, Medical Order Entry Systems, Interrupted Time Series Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
