Prospective evaluation of medication-related clinical decision support over-rides in the intensive care unit
Open accessAbstract
Background: Clinical decision support (CDS) displayed in electronic health records has been found to reduce the incidence of medication errors and adverse drug events (ADE). Recent data suggested that medication-related CDS alerts were frequently over-ridden, often inappropriately. Patients in the intensive care unit (ICU) are at an increased risk of ADEs; however, limited data exist on the benefits of CDS in the ICU. This study aims to evaluate potential harm associated with medication-related CDS over-rides in the ICU. Methods: This was a prospective observational study of adults admitted to any of six ICUs between July 2016 and April 2017 at our institution. Patients with provider-overridden CDS for dose (orders for scheduled frequency and not pro re nata), drug allergy, drug–drug interaction, geriatric and renal alerts (contraindicated medications for renal function or renal dosing) were included. The primary outcome was the appropriateness of over-rides, which were evaluated by two independent reviewers. Secondary outcomes included incidence of ADEs following alert over-ride and risk of ADEs based on over-ride appropriateness. Results: A total of 2448 over-ridden alerts from 712 unique patient encounters met inclusion criteria. The overall appropriateness rate for over-rides was 81.6% and varied by alert type. More ADEs (potential and definite) were identified following inappropriate over-rides compared with appropriate over-rides (16.5 vs 2.74 per 100 over-ridden alerts, Fisher’s exact test P<0.001). An adjusted logistic regression model showed that inappropriate over-rides were associated with an increased risk of ADEs (OR 6.14, 95% CI 4.63 to 7.71, P<0.001). Conclusions: Approximately four of five identified CDS over-rides were appropriately over-ridden, with the rate varying by alert type. However, inappropriate over-rides were six times as likely to be associated with potential and definite ADEs, compared with appropriate over-rides. Further efforts should be targeted at improving the positive predictive value of CDS such as by suppressing alerts that are appropriately over-ridden.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Wong et al. showed that most alerts (86.9%) were triggered by medication combinations that increase the risk of QTc prolongation.
“Also Wong et al. [31] showed that most alerts (86.9%) were triggered by medication combinations that increase the risk of QTc prolongation.”
From Overall performance of a drug–drug interaction clinical decision support system: quantitative evaluation and end-user survey (2022) · cites this paper as [31]
The Characteristics of alert types section reports that 86.9% (1,569) of the evaluated drug–drug interaction (DDI) over-rides involved combinations that increase QTc-prolongation risk. The citing sentence calls these “most alerts” without specifying that the percentage applies to DDI over-rides, not all alerts studied. (LLM, conf=0.87)
Experimental — accuracy not yet measuredHigh override rates of 56.3%-95.7% have already been described for DDI alerts.
“High override rates of 56.3%-95.7% have already been described for DDI alerts [13, 21, 27–32].”
From Overall performance of a drug–drug interaction clinical decision support system: quantitative evaluation and end-user survey (2022) · cites this paper as [31]
Edrees et al., Nanji et al. and Wong et al. found override appropriateness rates of 45.4%, 62% and 82%, respectively.
“Edrees et al. [27], Nanji et al. [29] and Wong et al. [31] found override appropriateness rates of 45.4%, 62% and 82%, respectively.”
From Overall performance of a drug–drug interaction clinical decision support system: quantitative evaluation and end-user survey (2022) · cites this paper as [31]
Our study results are in agreement with previous reports.
“Our study results are in agreement with previous reports.”
From Overall performance of a drug–drug interaction clinical decision support system: quantitative evaluation and end-user survey (2022) · cites this paper as [31]
The source reports findings on clinical decision-support alert over-rides and adverse drug events (Conclusion; Adverse drug events). The citing sentence does not identify which of its study results allegedly agree with this report, so the agreement cannot be checked against our copy. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vincent, J. L., Moreno, R., Takala, J., Willatts, S., De Mendonça, A., Bruining, H., Reinhart, C. K., Suter, P. M., & Thijs, L. G. (1996) · Intensive Care Medicine · doi:10.1007/bf01709751 · Cited by 11,674 · see citing works
(2012) · Kidney International Supplements · doi:10.1038/kisup.2012.1 · Cited by 4,326 · see citing works
The Nature of Adverse Events in Hospitalized Patients
Leape, L. L., Brennan, T. A., Laird, N. M., Lawthers, A. G., Localio, A. R., Barnes, B. A., Hebert, L. E., Newhouse, J. P., Weiler, P. C., & Hiatt, H. H. (1991) · New England Journal of Medicine · doi:10.1056/nejm199102073240605 · Cited by 3,766 · see citing works
By the American Geriatrics Society 2015 Beers Criteria Update Expert Panel (2015) · Journal of the American Geriatrics Society · doi:10.1111/jgs.13702 · Cited by 2,609 · see citing works
Bates, D. W. (1998) · JAMA · doi:10.1001/jama.280.15.1311 · Cited by 2,107 · see citing works
Self-citationClassen, D. C. (1997) · JAMA · doi:10.1001/jama.277.4.301 · Cited by 1,781 · see citing works
Bates, D. W. (1997) · JAMA · doi:10.1001/jama.277.4.307 · Cited by 1,678 · see citing works
Self-citationThe Impact of Computerized Physician Order Entry on Medication Error Prevention
Bates, D. W., Teich, J. M., Lee, J., Seger, D., Kuperman, G. J., Ma'Luf, N., Boyle, D., & Leape, L. (1999) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1999.00660313 · Cited by 1,260 · see citing works
Self-citationClassen, D. C. (1997) · JAMA · doi:10.1001/jama.1997.03540280039031 · Cited by 1,137 · see citing works
The Costs of Adverse Drug Events in Hospitalized Patients
Bates, D. W. (1997) · JAMA · doi:10.1001/jama.1997.03540280045032 · Cited by 1,072 · see citing works
Self-citationRothschild, J. M., Landrigan, C. P., CRONIN, J. W., Kaushal, R., Lockley, S. W., Burdick, E., Stone, P. H., Lilly, C. M., Katz, J. T., Czeisler, C. A., et al. (2005) · Critical Care Medicine · doi:10.1097/01.ccm.0000171609.91035.bd · Cited by 925 · see citing works
Self-citationPreventable adverse drug events in hospitalized patients
Cullen, D. J., Sweitzer, B. J., Bates, D. W., Burdick, E., Edmondson, A. C., & Leape, L. L. (1997) · Critical Care Medicine · doi:10.1097/00003246-199708000-00014 · Cited by 611 · see citing works
Self-citationAdverse drug events and medication errors: detection and classification methods
Morimoto, T. (2004) · BMJ Quality & Safety · doi:10.1136/qhc.13.4.306 · Cited by 503 · see citing works
Guided Medication Dosing for Inpatients With Renal Insufficiency
Chertow, G. M. (2001) · JAMA · doi:10.1001/jama.286.22.2839 · Cited by 447 · see citing works
Adverse drug events and medication errors: detection and classification methods
Morimoto, T. (2004) · BMJ Quality & Safety · doi:10.1136/qshc.2004.010611 · Cited by 402 · see citing works
Overrides of medication-related clinical decision support alerts in outpatients
Nanji, K. C., Slight, S. P., Seger, D. L., Cho, I., Fiskio, J. M., Redden, L. M., Volk, L. A., & Bates, D. W. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001813 · Cited by 219 · see citing works
Self-citationIncidence of Adverse Drug Events and Medication Errors in Japan: the JADE Study
Morimoto, T., Sakuma, M., Matsui, K., et al. (2010) · Journal of General Internal Medicine · doi:10.1007/s11606-010-1518-3 · Cited by 162 · see citing works
Guided Prescription of Psychotropic Medications for Geriatric Inpatients
Peterson, J. F., Kuperman, G. J., Shek, C., Patel, M., Avorn, J., & Bates, D. W. (2005) · Archives of Internal Medicine · doi:10.1001/archinte.165.7.802 · Cited by 132 · see citing works
Self-citationAnalysis of clinical decision support system malfunctions: a case series and survey
Wright, A., Hickman, T.-T. T., McEvoy, D., Aaron, S., Ai, A., Andersen, J. M., Hussain, S., Ramoni, R., Fiskio, J., Sittig, D. F., et al. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw005 · Cited by 132 · see citing works
Self-citationLin, C.-P., Payne, T. H., Nichol, W. P., Hoey, P. J., Anderson, C. L., & Gennari, J. H. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2453 · Cited by 110 · see citing works
Prediction of Adverse Drug Reactions Using Decision Tree Modeling
Hammann, F., Gutmann, H., Vogt, N., et al. (2010) · Clinical Pharmacology & Therapeutics · doi:10.1038/clpt.2009.248 · Cited by 108 · see citing works
Influence of adverse drug events on morbidity and mortality in intensive care units: the JADE study
Ohta, Y., Sakuma, M., Koike, K., Bates, D. W., & Morimoto, T. (2014) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzu081 · Cited by 42 · see citing works
Self-citationWong, A., Amato, M. G., Seger, D. L., Slight, S. P., Beeler, P. E., Dykes, P. C., Fiskio, J. M., Silvers, E. R., Orav, E. J., Eguale, T., et al. (2017) · Journal of Critical Care · doi:10.1016/j.jcrc.2017.02.027 · Cited by 41 · see citing works
Self-citationGoss, F. R., Plasek, J. M., Lau, J. J., et al. (2014) · PubMed · Cited by 37 · see citing works
Wong, A., Wright, A. T., Seger, D. L., Amato, M. G., Fiskio, J. M., & Bates, D. W. (2017) · Applied Clinical Informatics · doi:10.4338/aci-2017-04-ra-0059 · Cited by 32 · see citing works
Self-citationAdverse Drug Events in Critically Ill Patients With Cancer
Nazer, L. H., Hawari, F. I., & Alnajjar, T. (2013) · Journal of Pharmacy Practice · doi:10.1177/0897190013513302 · Cited by 17 · see citing works
Developing strategies for predicting hyperkalemia in potassium-increasing drug-drug interactions
Eschmann, E., Beeler, P. E., Schneemann, M., & Blaser, J. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw050 · Cited by 13 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Kidney Disease: Improving Global Outcomes (KDIGO) Acute Kidney Injury Work Group: KDIGO clinical practice guideline for acute kidney injury. Kidney Int 2012;2:1-138.
Cited by
Sorted by citations
Appropriateness of Overridden Alerts in Computerized Physician Order Entry: Systematic Review
Poly, T. N., Islam, M. M., Yang, H.-C., & Li, Y.-C. (. (2020) · JMIR Medical Informatics · doi:10.2196/15653 · Cited by 118 · see citing works
Scott, I. A., Pillans, P. I., Barras, M., & Morris, C. (2018) · Therapeutic Advances in Drug Safety · doi:10.1177/2042098618784809 · Cited by 114 · see citing works
Edrees, H., Amato, M. G., Wong, A., Seger, D. L., & Bates, D. W. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa034 · Cited by 64 · see citing works
Self-citationVan De Sijpe, G., Quintens, C., Walgraeve, K., Van Laer, E., Penny, J., De Vlieger, G., Schrijvers, R., De Munter, P., Foulon, V., Casteels, M., et al. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01783-z · Cited by 54 · see citing works
“[31] found override appropriateness rates of 45.”
Horn, J., & Ueng, S. (2019) · Annals of Pharmacotherapy · doi:10.1177/1060028019863419 · Cited by 26 · see citing works
Wong, A., Rehr, C., Seger, D. L., Amato, M. G., Beeler, P. E., Slight, S. P., Wright, A., & Bates, D. W. (2019) · Drug Safety · doi:10.1007/s40264-018-0756-x · Cited by 21 · see citing works
Self-citation
- Authors
- Wong, A., Amato, M. G., Seger, D. L., Rehr, C., Wright, A., Slight, S. P., Beeler, P. E., Orav, E. J., & Bates, D. W.
- Affiliations
- Brigham and Women's Hospital (US)
- Clinical Quality Analysis, Partners Healthcare, Somerville, Massachusetts, USA
- University Hospital Zurich (CH)
- Harvard University (US)
- Corresponding author
- Wong, Adrian
- ORCID
- Venue
- BMJ Quality & Safety
- Publisher
- BMJ
- Published
- 2018-02-09
- Volume
- 27
- Issue
- 9
- Pages
- 718-724
- ISSN
- 1475-3898 (ISSN-L)
- 2044-5415
- 2044-5423
- OpenAlex
- W2789882727
- Cited by
- 77
- References
- 27
- Citations by year
- 2026: 5 · 2025: 7 · 2024: 5 · 2023: 6 · 2022: 25 · 2021: 9 · 2020: 8 · 2019: 6 · 2018: 6
- FWCI
- 9.21
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- BMJ Quality & Safety (publishedVersion)
- PubMed (publishedVersion)
- Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, other-oa)
- HighWire Press Open Archive (submittedVersion)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion, other-oa)
- Zurich Open Repository and Archive (University of Zurich)
- Funders
- CRICO
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Emergency Medical Services › Patient Safety and Medication Errors
- Health Sciences › Medicine › Geriatrics and Gerontology › Pharmaceutical Practices and Patient Outcomes
- Keywords
- clinical decision support, adverse drug reactions, intensive care unit, electronic health records, medication errors, drug-drug interactions, drug allergy alerts, positive predictive value
- MeSH terms
- Adult, Aged, Aged, 80 and over, Boston, Drug Hypersensitivity, Health Personnel, Humans, Intensive Care Units, Male, Medication Errors, Middle Aged, Pilot Projects, Prospective Studies, Logistic Models, Decision Support Systems, Clinical, Medical Order Entry Systems, Young Adult, Electronic Health Records, Drug-Related Side Effects and Adverse Reactions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
