CoChat
Sign Up
Cited by 76Electronic Health Records SystemsHealthcare Technology and Patient MonitoringArtificial Intelligence in Healthcare and Education

Clinical decision support stewardship: best practices and techniques to monitor and improve interruptive alerts

Open access
Chaparro, Juan D., Beus, Jonathan M., Dziorny, Adam C., Hagedorn, Philip A., Hernandez, Sean, Kandaswamy, Swaminathan, Kirkendall, Eric S., McCoy, Allison B. +2 more·2022·Applied Clinical Informatics
DOI 10.1055/s-0042-1748856PMC PMC9132737
Open this paper in CoChat — free accountOpen PDF

Abstract

Interruptive clinical decision support systems, both within and outside of electronic health records, are a resource that should be used sparingly and monitored closely. Excessive use of interruptive alerting can quickly lead to alert fatigue and decreased effectiveness and ignoring of alerts. In this review, we discuss the evidence for effective alert stewardship as well as practices and methods we have found useful to assess interruptive alert burden, reduce excessive firings, optimize alert effectiveness, and establish quality governance at our institutions. We also discuss the importance of a holistic view of the alerting ecosystem beyond the electronic health record.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1055/s-0042-1748856

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Our findings assessing interruptive alert helpfulness and recommendations to change interruptive alert presentation, timing, and targeting were also like those found in previous studies.

    “Our findings assessing interruptive alert helpfulness and recommendations to change interruptive alert presentation, timing, and targeting were also like those found in previous studies. 35 36 40”

    From Qualitative Study of Inpatient Hospital Medicine Interruptive Alerts and Clinical Decision Support (2026) · cites this paper as [36]

  • Previous research has shown interruptive alert optimization by using strategies of eliciting end-user feedback, reducing alert volume, and reducing inappropriate firing of alerts, similar to the themes identified by participants in our study.

    “Previous research has shown interruptive alert optimization by using strategies of eliciting end-user feedback, reducing alert volume, and reducing inappropriate firing of alerts, similar to the themes identified by participants in our study. 34 35 36 37”

    From Qualitative Study of Inpatient Hospital Medicine Interruptive Alerts and Clinical Decision Support (2026) · cites this paper as [36]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 58

  1. Kwan JL, Lo L, Ferguson J, et al. Computerised clinical decision support systems and absolute improvements in care: meta- analysis of controlled clinical trials. BMJ 2020;370:m3216
  2. Varghese J, Kleine M, Gessner SI, Sandmann S, Dugas M. Effects of computerized decision support system implementations on pa- tient outcomes in inpatient care: a systematic review. J Am Med Inform Assoc 2018;25(05):593-602
  3. Bonafide CP, Localio AR, Holmes JH, et al. Video analysis of factors associated with response time to physiologic monitor alarms in a children's hospital. JAMA Pediatr 2017;171(06):524-531
  4. Escovedo C, Bell D, Cheng E, et al. Noninterruptive clinical decision support decreases ordering of respiratory viral panels during influenza season. Appl Clin Inform 2020;11(02):315-322
  5. Gouveia WA, Diamantis C, Barnett GO. Computer applications in the hospital medication system. Am J Hosp Pharm 1969;26(03): 141-150
  6. Chaparro JD, Hussain C, Lee JA, Hehmeyer J, Nguyen M, Hoffman J. Reducing interruptive alert burden using quality improvement methodology. Appl Clin Inform 2020;11(01):46-58 10
  7. Westbrook JI, Coiera E, Dunsmuir WT, et al. The impact of interruptions on clinical task completion. Qual Saf Health Care 2010;19(04):284-289
  8. Grundgeiger T, Sanderson P. Interruptions in healthcare: theoret- ical views. Int J Med Inform 2009;78(05):293-307
  9. Ashcroft DM, Quinlan P, Blenkinsopp A. Prospective study of the incidence, nature and causes of dispensing errors in community pharmacies. Pharmacoepidemiol Drug Saf 2005;14(05):327-332
  10. Westbrook JI, Woods A, Rob MI, Dunsmuir WT, Day RO. Associa- tion of interruptions with an increased risk and severity of medication administration errors. Arch Intern Med 2010;170 (08):683-690
  11. Ariosto D. Factors contributing to CPOE opiate allergy alert over- rides. AMIA Annu Symp Proc 2014;2014:256-265
  12. Humphrey K, Jorina M, Harper M, Dodson B, Kim SY, Ozonoff A. An investigation of drug-drug interaction alert overrides at a pediat- ric hospital. Hosp Pediatr 2018;8(05):293-299
  13. van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13(02):138-147 17
  14. Muthu N, Shelov E, Tobias MC, Karavite DJ, Orenstein EW, Grund- meier RW. Variability in user response to custom alerts in the electronic health record: an observational study. Paper presented at: the AMIA 2019 National Symposium Washington DC; Novem- ber 16-20, 2019
  15. Bliss JP, Gilson RD, Deaton JE. Human probability matching behaviour in response to alarms of varying reliability. Ergonomics 1995;38(11):2300-2312
  16. Bliss JP, Dunn MC. Behavioural implications of alarm mistrust as a function of task workload. Ergonomics 2000;43(09):1283-1300
  17. Baysari MT, Tariq A, Day RO, Westbrook JI. Alert override as a habitual behavior -a new perspective on a persistent problem. J Am Med Inform Assoc 2017;24(02):409-412
  18. Simpao AF, Ahumada LM, Desai BR, et al. Optimization of drug- drug interaction alert rules in a pediatric hospital's electronic health record system using a visual analytics dashboard. J Am Med Inform Assoc 2015;22(02):361-369
  19. Greenes RA, Bates DW, Kawamoto K, Middleton B, Osheroff J, Shahar Y. Clinical decision support models and frameworks: seeking to address research issues underlying implementation successes and failures. J Biomed Inform 2018;78:134-143
  20. Wright A, Hickman TT, McEvoy D, et al. Analysis of clinical decision support system malfunctions: a case series and survey. J Am Med Inform Assoc 2016;23(06):1068-1076
  21. McGreevey JD III, Mallozzi CP, Perkins RM, Shelov E, Schreiber R. Reducing alert burden in electronic health records: state of the art recommendations from four health systems. Appl Clin Inform 2020;11(01):1-12
  22. Schreiber R, Gregoire JA, Shaha JE, Shaha SH. Think time: a novel approach to analysis of clinicians' behavior after reduction of drug-drug interaction alerts. Int J Med Inform 2017;97:59-67 26
  23. Elias P, Peterson E, Wachter B, Ward C, Poon E, Navar AM. Evaluating the impact of interruptive alerts within a health system: use, response time, and cumulative time burden. Appl Clin Inform 2019;10(05):909-917
  24. McCoy AB, Thomas EJ, Krousel-Wood M, Sittig DF. Clinical deci- sion support alert appropriateness: a review and proposal for improvement. Ochsner J 2014;14(02):195-202
  25. McCoy AB, Waitman LR, Lewis JB, et al. A framework for evaluating the appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc 2012;19(03):346-352 30 Strategies to the five rights of clinical decision support. Janu- ary 23, 2019. Accessed January 11, 2022 at: https://userweb. epic.com/Thread/82831/Strategies-to-the-Five-Rights-of-Clin- cial-Decision-Support/?reply=385118
  26. Orenstein EW, Kandaswamy S, Muthu N, et al. Alert burden in pediatric hospitals: a cross-sectional analysis of six academic pediatric health systems using novel metrics. J Am Med Inform Assoc 2021;28(12):2654-2660
  27. Sinha A, Stevens LA, Su F, Pageler NM, Tawfik DS. Measuring electronic health record use in the pediatric ICU using audit-logs and screen recordings. Appl Clin Inform 2021;12(04):737-744
  28. Bonafide CP, Lin R, Zander M, et al. Association between exposure to nonactionable physiologic monitor alarms and response time in a children's hospital. J Hosp Med 2015;10(06):345-351
  29. Bonafide CP, Miller JM, Localio AR, et al. Association between mobile telephone interruptions and medication administration errors in a pediatric intensive care unit. JAMA Pediatr 2020;174 (02):162-169
  30. Kandaswamy S, Karavite DJ, Muthu N, et al. User and task analysis for evaluation of clinical decision support for quality improvement. Proc Hum Factors Ergon Soc Annu Meet 2020;64(01):750-754
  31. Wright A, Ash JS, Aaron S, et al. Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: results of a Delphi study. Int J Med Inform 2018; 118:78-85 40
  32. Aaron S, McEvoy DS, Ray S, Hickman TT, Wright A. Cranky com- ments: detecting clinical decision support malfunctions through free-text override reasons. J Am Med Inform Assoc 2019;26(01): 37-43
  33. Ray S, McEvoy DS, Aaron S, Hickman TT, Wright A. Using statistical anomaly detection models to find clinical decision support mal- functions. J Am Med Inform Assoc 2018;25(07):862-871
  34. Kassakian SZ, Yackel TR, Gorman PN, Dorr DA. Clinical decisions support malfunctions in a commercial electronic health record. Appl Clin Inform 2017;8(03):910-923
  35. Yoshida E, Fei S, Bavuso K, Lagor C, Maviglia S. The value of monitoring clinical decision support interventions. Appl Clin Inform 2018;9(01):163-173
  36. Seidling HM, Phansalkar S, Seger DL, et al. Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc 2011;18(04):479-484
  37. Kawamanto K, Flynn MC, Kukhareva P, et al. A pragmatic guide to establishing clinical decision support governance and addressing decision support fatigue: a case study. AMIA Annu Symp Proc 2018;2018:624-633
  38. DeRosier J, Stalhandske E, Bagian JP, Nudell T. Using health care failure mode and effect analysis: the VA National Center for Patient Safety's prospective risk analysis system. Jt Comm J Qual Improv 2002;28(05):248-267 47 Electronic Health Record Association. Electronic health record design patterns for patient safety. 2017. Accessed January 11, 2022 at: https://www.ehra.org/sites/ehra.org/files/docs/ehra-de- sign-patterns-for-safety.pdf
  39. Yu D, Obuseh M, DeLaurentis P. Quantifying the impact of infusion alerts and alarms on nursing workflows: a retrospective analysis. Appl Clin Inform 2021;12(03):528-538 50
  40. Ruskin KJ, Hueske-Kraus D. Alarm fatigue: impacts on patient safety. Curr Opin Anaesthesiol 2015;28(06):685-690
  41. Hagedorn PA, Singh A, Luo B, Bonafide CP, Simmons JM. Secure text messaging in healthcare: latent threats and opportunities to improve patient safety. J Hosp Med 2020;15(06):378-380
  42. Pater CM, Sosa TK, Boyer J, et al. Time series evaluation of improvement interventions to reduce alarm notifications in a paediatric hospital. BMJ Qual Saf 2020;29(09):717-726
  43. Tchou MJ, Andersen H, Robinette E, et al. Accelerating initiation of adequate antimicrobial therapy using real-time decision support and microarray testing. Pediatr Qual Saf 2019;4(04):e191
  44. Singh A, Haque A, Alahi A, et al. Automatic detection of hand hygiene using computer vision technology. J Am Med Inform Assoc 2020;27(08):1316-1320
  45. Redelmeier DA, Tibshirani RJ. Association between cellular-tele- phone calls and motor vehicle collisions. N Engl J Med 1997;336 (07):453-458
  46. Orenstein EW, Muthu N, Weitkamp AO, et al. Towards a maturity model for clinical decision support operations. Appl Clin Inform 2019;10(05):810-819
  47. Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing inter- ventions. J Biomed Inform 2012;45(06):1202-1216
  48. Miller K, Capan M, Weldon D, et al. The design of decisions: matching clinical decision support recommendations to Nielsen's design heuristics. Int J Med Inform 2018;117:19-25 61
  49. Kannampallil TG, Kaufman DR, Patel VL. Cognitive Informatics for Biomedicine: Human Computer Interaction in Healthcare. 1st ed. Springer International Publishing; 2015
  50. Rodriguez JA, Clark CR, Bates DW. Digital health equity as a necessity in the 21st century Cures Act era. JAMA 2020;323 (23):2381-2382 63
  51. Majumder MA, Guerrini CJ, Bollinger JM, Cook-Deegan R, McGuire AL. Sharing data under the 21st Century Cures Act. Genet Med 2017;19(12):1289-1294 64
  52. Pageler NM, Webber EC, Lund DP. Implications of the 21st Century Cures Act in pediatrics. Pediatrics 2021;147(03):e2020034199 65
  53. Gordon WJ, Mandl KD. The 21st Century Cures Act: a competitive apps market and the risk of innovation blocking. J Med Internet Res 2020;22(12):e24824 66
  54. Dzau VJ, Cho A, Ellaissi W, et al. Transforming academic health centers for an uncertain future. N Engl J Med 2013;369(11): 991-993
  55. Horwitz LI, Kuznetsova M, Jones SA. Creating a learning health system through rapid-cycle, randomized testing. N Engl J Med 2019;381(12):1175-1179
  56. Friedman C, Rubin J, Brown J, et al. Toward a science of learning systems: a research agenda for the high-functioning Learning Health System. J Am Med Inform Assoc 2015;22(01): 43-50 69
  57. Crandall W, Kappelman MD, Colletti RB, et al. ImproveCareNow: The development of a pediatric inflammatory bowel disease improvement network. Inflamm Bowel Dis 2011;17(01): 450-457
  58. Marsolo K, Margolis PA, Forrest CB, Colletti RB, Hutton JJ. A digital architecture for a network-based learning health system: inte- grating chronic care management, quality improvement, and research. EGEMS (Wash DC) 2015;3(01):1168 Applied Clinical Informatics Vol. 13 No. 3/2022 © 2022. Thieme. All rights reserved.

Cited by

Sorted by citations

Authors
Chaparro, J. D., Beus, J. M., Dziorny, A. C., Hagedorn, P. A., Hernandez, S., Kandaswamy, S., Kirkendall, E. S., McCoy, A. B., Muthu, N., & Orenstein, E. W.
Affiliations
  • Nationwide Children's Hospital (US)
  • Emory University (US)
  • University of Rochester Medicine (US)
  • Cincinnati Children's Hospital Medical Center (US)
  • Wake Forest University Health Sciences (US)
  • Vanderbilt University Medical Center (US)
  • Children's Hospital of Philadelphia (US)
Corresponding author
Chaparro, Juan D.
ORCID
Venue
Applied Clinical Informatics
Publisher
Georg Thieme Verlag KG
Published
2022-05
Volume
13
Issue
03
Pages
560-568
ISSN
1869-0327 (ISSN-L)
DOI
10.1055/s-0042-1748856
PMC
PMC9132737
OpenAlex
W4281557486
Cited by
76
References
5
Citations by year
2026: 32 · 2025: 21 · 2024: 11 · 2023: 10 · 2022: 2
FWCI
1.60
Type
journal article
Language
en
Open access
Locations
  • Applied Clinical Informatics (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • National Institutes of Health (UL1TR001420)
  • National Center for Advancing Translational Sciences
Topics
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
interruptive alerts, clinical decision support, alarm fatigue, electronic health records, alert burden, quality governance, clinical decision support systems
MeSH terms
Ecosystem, Decision Support Systems, Clinical, Medical Order Entry Systems, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review