A framework for evaluating the appropriateness of clinical decision support alerts and responses
Open accessAbstract
OBJECTIVE: Alerting systems, a type of clinical decision support, are increasingly prevalent in healthcare, yet few studies have concurrently measured the appropriateness of alerts with provider responses to alerts. Recent reports of suboptimal alert system design and implementation highlight the need for better evaluation to inform future designs. The authors present a comprehensive framework for evaluating the clinical appropriateness of synchronous, interruptive medication safety alerts. METHODS: Through literature review and iterative testing, metrics were developed that describe successes, justifiable overrides, provider non-adherence, and unintended adverse consequences of clinical decision support alerts. The framework was validated by applying it to a medication alerting system for patients with acute kidney injury (AKI). RESULTS: Through expert review, the framework assesses each alert episode for appropriateness of the alert display and the necessity and urgency of a clinical response. Primary outcomes of the framework include the false positive alert rate, alert override rate, provider non-adherence rate, and rate of provider response appropriateness. Application of the framework to evaluate an existing AKI medication alerting system provided a more complete understanding of the process outcomes measured in the AKI medication alerting system. The authors confirmed that previous alerts and provider responses were most often appropriate. CONCLUSION: The new evaluation model offers a potentially effective method for assessing the clinical appropriateness of synchronous interruptive medication alerts prior to evaluating patient outcomes in a comparative trial. More work can determine the generalizability of the framework for use in other settings and other alert types.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Few studies have evaluated the appropriateness of alert overrides, and little prior data exist about the appropriateness of overrides by alert type.
“Few studies have evaluated the appropriateness of alert overrides, 24 and little prior data exist about the appropriateness of overrides by alert type.”
From Overrides of medication-related clinical decision support alerts in outpatients (2014) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Human error: models and management
Reason, J. (2000) · BMJ · doi:10.1136/bmj.320.7237.768 · Cited by 5,665 · see citing works
The American Journal of Managed Care
(2020) · The American Journal of Managed Care · doi:10.37765/ajmc · Cited by 3,238 · see citing works
Role of Computerized Physician Order Entry Systems in Facilitating Medication Errors
Koppel, R. (2005) · JAMA · doi:10.1001/jama.293.10.1197 · Cited by 2,126 · see citing works
Ash, J. S. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1471 · Cited by 1,789 · see citing works
Systems Analysis of Adverse Drug Events
Leape, L. L. (1995) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1995.03530010049034 · Cited by 1,404 · see citing works
Overriding of Drug Safety Alerts in Computerized Physician Order Entry
van der Sijs, H., Aarts, J., Vulto, A., & Berg, M. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1809 · Cited by 1,117 · see citing works
Medication-related clinical decision support in computerized provider order entry systems: a review
Kuperman, G. J., Bobb, A., Payne, T. H., Avery, A. J., Gandhi, T. K., Burns, G., Classen, D. C., & Bates, D. W. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2170 · Cited by 763 · see citing works
Physicians' Decisions to Override Computerized Drug Alerts in Primary Care
Weingart, S. N., Toth, M., Sands, D. Z., Aronson, M. D., Davis, R. B., & Phillips, R. S. (2003) · Archives of Internal Medicine · doi:10.1001/archinte.163.21.2625 · Cited by 487 · see citing works
Incidence and preventability of adverse drug events in hospitalized adults
Bates, D. W., Leape, L. L., & Petrycki, S. (1993) · Journal of General Internal Medicine · doi:10.1007/bf02600138 · Cited by 373 · see citing works
Improving Acceptance of Computerized Prescribing Alerts in Ambulatory Care
Shah, N. R., Seger, A. C., Seger, D. L., Fiskio, J. M., Kuperman, G. J., Blumenfeld, B., Recklet, E. G., Bates, D. W., & Gandhi, T. K. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1868 · Cited by 353 · see citing works
Overrides of Medication Alerts in Ambulatory Care
Isaac, T., Weissman, J. S., Davis, R. B., Massagli, M., Cyrulik, A., Sands, D. Z., & Weingart, S. N. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2008.551 · Cited by 309 · see citing works
Schedlbauer, A., Prasad, V., Mulvaney, C., Phansalkar, S., Stanton, W., Bates, D. W., & Avery, A. J. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2910 · Cited by 262 · see citing works
Hsieh, T. C., Kuperman, G. J., Jaggi, T., Hojnowski-Diaz, P., Fiskio, J., Williams, D. H., Bates, D. W., & Gandhi, T. K. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1556 · Cited by 232 · see citing works
Phansalkar, S., Edworthy, J., Hellier, E., Seger, D. L., Schedlbauer, A., Avery, A. J., & Bates, D. W. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.005264 · Cited by 226 · see citing works
Strom, B. L., Schinnar, R., Aberra, F., Bilker, W., Hennessy, S., Leonard, C. E., & Pifer, E. (2010) · Archives of Internal Medicine · doi:10.1001/archinternmed.2010.324 · Cited by 217 · see citing works
Bloomrosen, M., Starren, J. B., Lorenzi, N. M., Ash, J. S., Patel, V. L., & Shortliffe, E. H. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.007567 · Cited by 181 · see citing works
Turning Off Frequently Overridden Drug Alerts: Limited Opportunities for Doing It Safely
van der Sijs, H., Aarts, J., van Gelder, T., Berg, M., & Vulto, A. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2311 · Cited by 175 · see citing works
Miller, R. A., Waitman, L. R., Chen, S., & Rosenbloom, S. T. (2005) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.08.009 · Cited by 144 · see citing works
Self-citationMcCoy, A. B., Waitman, L. R., Gadd, C. S., Danciu, I., Smith, J. P., Lewis, J. B., Schildcrout, J. S., & Peterson, J. F. (2010) · American Journal of Kidney Diseases · doi:10.1053/j.ajkd.2010.05.024 · Cited by 124 · see citing works
Self-citationDrug safety alert generation and overriding in a large Dutch university medical centre
van der Sijs, H., Mulder, A., van Gelder, T., Aarts, J., Berg, M., & Vulto, A. (2009) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.1800 · Cited by 119 · see citing works
Tamblyn, R. M., Huang, A. R., Taylor, L., Kawasumi, Y., Bartlett, G., Grad, R. M., Jacques, A., Dawes, M. G., Abrahamowicz, M., Perreault, R. B., et al. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2606 · Cited by 110 · see citing works
Gurwitz, J. H., Field, T. S., Rochon, P., Judge, J., Harrold, L. R., Bell, C. M., Lee, M., White, K., LaPrino, J., Erramuspe‐Mainard, J., et al. (2008) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.2008.02004.x · Cited by 106 · see citing works
Practitioners’ views on computerized drug-drug interaction alerts in the VA system
Ko, Y., Abarca, J., Malone, D. C., Dare, D. C., Geraets, D., Houranieh, A., Jones, W. N., Nichol, W. P., Schepers, G. P., & Wilhardt, M. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2224 · Cited by 102 · see citing works
(2007) · PubMed · Cited by 93 · see citing works
Evaluation and Certification of Computerized Provider Order Entry Systems
Classen, D. C., Avery, A. J., & Bates, D. W. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2248 · Cited by 73 · see citing works
Understanding handling of drug safety alerts: a simulation study
van der Sijs, H., van Gelder, T., Vulto, A., Berg, M., & Aarts, J. (2010) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2010.01.008 · Cited by 72 · see citing works
Frequency of inappropriate medical exceptions to quality measures
Persell, S. D., Dolan, N. C., Friesema, E. M., Thompson, J. A., Kaiser, D., & Baker, D. W. (2010) · Annals of Internal Medicine · doi:10.7326/0003-4819-152-4-201002160-00007 · Cited by 66 · see citing works
Supporting communication in an integrated patient record system.
Giuse, D. A. (2003) · PubMed · Cited by 59 · see citing works
A new approach to the implementation of direct care-provider order entry.
Geissbuhler, A. R., & Miller, R. A. (1996) · PubMed · Cited by 59 · see citing works
Self-citationPhysicians’ reasons for failing to comply with computerized preventive care guidelines
Litzelman, D. K., & Tierney, W. M. (1996) · Journal of General Internal Medicine · doi:10.1007/bf02599049 · Cited by 56 · see citing works
Lee, E. K., Mejia, A. F., Senior, T., & Jose, J. H. (2010) · PubMed Central · Cited by 54 · see citing works
Strom, B. L., Schinnar, R., Bilker, W., Hennessy, S., Leonard, C. E., & Pifer, E. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.000695 · Cited by 51 · see citing works
Russ, A. L., Zillich, A. J., McManus, M. S., Doebbeling, B. N., & Saleem, J. J. (2009) · PubMed Central · Cited by 45 · see citing works
Tamblyn, R., Reidel, K., Huang, A., Taylor, L., Winslade, N., Bartlett, G., Grad, R., Jacques, A., Dawes, M., Larochelle, P., et al. (2010) · Medical Decision Making · doi:10.1177/0272989x09342752 · Cited by 44 · see citing works
Decision Support Alerts for Medication Ordering in a Computerized Provider Order Entry (CPOE) System
Villanueva, R., Knudson, K. M., Harvey, E. M., Langle, J. M., Paul, W., & Beccaro, M. A. D. (2010) · Applied Clinical Informatics · doi:10.4338/aci-2009-11-ra-0014 · Cited by 26 · see citing works
Functionality test for drug safety alerting in computerized physician order entry systems
van der Sijs, H., Bouamar, R., van Gelder, T., Aarts, J. E. C. M., Berg, M., & Vulto, A. G. (2010) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2010.01.005 · Cited by 19 · see citing works
Unintended consequences of reducing QT-alert overload in a computerized physician order entry system
van der Sijs, H., Kowlesar, R., Aarts, J., Berg, M., Vulto, A. G., & van Gelder, T. (2009) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-009-0654-3 · Cited by 18 · see citing works
Swiderski, S. M., Pedersen, C. A., Schneider, P. J., & Miller, A. S. (2007) · Journal of Patient Safety · doi:10.1097/01.jps.0000242991.86165.80 · Cited by 11 · see citing works
Interpreting nurses' responses to clinical documentation alerts.
Cimino, J. J., Farnum, L., Cochran, K. A., Moore, S. D., Sengstack, P. R., & McKeeby, J. W. (2010) · PubMed Central · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The REDCap consortium: building an international community of software platform partners
Harris, P. A., Taylor, R., Minor, B. L., Elliott, V., Fernandez, M., O'Neal, L., McLeod, L., Delacqua, G., Delacqua, F., Kirby, J., et al. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103208 · Cited by 24,013 · see citing works
Overrides of medication-related clinical decision support alerts in outpatients
Nanji, K. C., Slight, S. P., Seger, D. L., Cho, I., Fiskio, J. M., Redden, L. M., Volk, L. A., & Bates, D. W. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001813 · Cited by 219 · see citing works
Electronic health records and national patient-safety goals
Sittig, D. F., & Singh, H. (2012) · New England Journal of Medicine · doi:10.1056/nejmsb1205420 · Cited by 204 · see citing works
Appropriateness of Overridden Alerts in Computerized Physician Order Entry: Systematic Review
Poly, T. N., Islam, M. M., Yang, H.-C., & Li, Y.-C. (. (2020) · JMIR Medical Informatics · doi:10.2196/15653 · Cited by 118 · see citing works
Mahadevaiah, G., RV, P., Bermejo, I., Jaffray, D., Dekker, A., & Wee, L. (2020) · Medical Physics · doi:10.1002/mp.13562 · Cited by 114 · see citing works
Stultz, J. S., & Nahata, M. C. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001725 · Cited by 28 · see citing works
- Authors
- McCoy, A. B., Waitman, L. R., Lewis, J. B., Wright, J. A., Choma, D. P., Miller, R. A., & Peterson, J. F.
- Affiliations
- Vanderbilt University (US)
- University of Kansas Medical Center (US)
- Vanderbilt University Medical Center (US)
- Corresponding author
- McCoy, A. B.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2012-05-01
- Volume
- 19
- Issue
- 3
- Pages
- 346-352
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC3341775
- OpenAlex
- W1833822692
- Cited by
- 138
- References
- 39
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 6 · 2023: 5 · 2022: 14 · 2021: 9 · 2020: 10 · 2019: 12 · 2018: 11 · 2017: 6 · 2016: 9 · 2015: 15 · 2014: 10 · 2013: 14 · 2012: 5
- FWCI
- 15.38
- Type
- journal article
- Language
- en
- Open access
- green
- europepmc.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Vanderbilt University
- U.S. National Library of Medicine (R01LM009965, T15LM007450)
- National Center for Research Resources (KL2RR024977, TL1RR024978, UL1RR024975)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
- Health Sciences › Health Professions › Emergency Medical Services › Patient Safety and Medication Errors
- Keywords
- clinical decision support alerts, interruptive alerts, medication safety alerts, alert override, false positive alerts, acute kidney injury
- MeSH terms
- Drug Therapy, Computer-Assisted, Humans, Medical Audit, Medication Errors, Medication Systems, Hospital, Models, Theoretical, Retrospective Studies, Tennessee, User-Computer Interface, Single-Blind Method, Reminder Systems, Decision Support Systems, Clinical, Medical Order Entry Systems, Electronic Health Records, Acute Kidney Injury
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
