On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Clinical decision support (CDS) for electronic prescribing systems (computerized physician order entry) should help prescribers in the safe and rational use of medicines. However, the best ways to alert users to unsafe or irrational prescribing are uncertain. Specifically, CDS systems may generate too many alerts, producing unwelcome distractions for prescribers, or too few alerts running the risk of overlooking possible harms. Obtaining the right balance of alerting to adequately improve patient safety should be a priority. METHODS: A workshop funded through the European Regional Development Fund was convened by the University Hospitals Birmingham NHS Foundation Trust to assess current knowledge on alerts in CDS and to reach a consensus on a future research agenda on this topic. Leading European researchers in CDS and alerts in electronic prescribing systems were invited to the workshop. RESULTS: We identified important knowledge gaps and suggest research priorities including (1) the need to determine the optimal sensitivity and specificity of alerts; (2) whether adaptation to the environment or characteristics of the user may improve alerts; and (3) whether modifying the timing and number of alerts will lead to improvements. We have also discussed the challenges and benefits of using naturalistic or experimental studies in the evaluation of alerts and suggested appropriate outcome measures. CONCLUSIONS: We have identified critical problems in CDS, which should help to guide priorities in research to evaluate alerts. It is hoped that this will spark the next generation of novel research from which practical steps can be taken to implement changes to CDS systems that will ultimately reduce alert fatigue and improve the design of future systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reason, J. (1990) · doi:10.1017/cbo9781139062367 · Cited by 5,108
Overriding of Drug Safety Alerts in Computerized Physician Order Entry
van der Sijs, H., Aarts, J., Vulto, A., & Berg, M. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1809 · Cited by 1,117
Self-citationThe benefits of imperfect diagnostic automation: a synthesis of the literature
Wickens, C. D., & Dixon, S. R. (2007) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220500370105 · Cited by 423
Overrides of Medication Alerts in Ambulatory Care
Isaac, T., Weissman, J. S., Davis, R. B., Massagli, M., Cyrulik, A., Sands, D. Z., & Weingart, S. N. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2008.551 · Cited by 309
Kesselheim, A. S., Cresswell, K., Phansalkar, S., Bates, D. W., & Sheikh, A. (2011) · Health Affairs · doi:10.1377/hlthaff.2010.1111 · Cited by 277 · cited 1×
“This has been, however, discussed in an American context [35], with particular emphasis on the liability implications of CDS with drug–drug interactions [36,37].”
BackgroundTiering Drug-Drug Interaction Alerts by Severity Increases Compliance Rates
Paterno, M. D., Maviglia, S. M., Gorman, P. N., Seger, D. L., Yoshida, E., Seger, A. C., Bates, D. W., & Gandhi, T. K. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2808 · Cited by 263
Phansalkar, S., Edworthy, J., Hellier, E., Seger, D. L., Schedlbauer, A., Avery, A. J., & Bates, D. W. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.005264 · Cited by 226 · cited 1×
“Certainly, the importance of applying human factors principles to matters such as placement, visibility, prioritization, and colour in the design of CDS alerts is well established [48-50].”
BackgroundTurning Off Frequently Overridden Drug Alerts: Limited Opportunities for Doing It Safely
van der Sijs, H., Aarts, J., van Gelder, T., Berg, M., & Vulto, A. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2311 · Cited by 175 · cited 2×
“For example, groups of clinicians have been asked to agree on which alerts could be turned off safely within a hospital system [32] and to assess the value of alerts for 120 drug–drug interactions [55].”
BackgroundSelf-citationA framework for evaluating the appropriateness of clinical decision support alerts and responses
McCoy, A. B., Waitman, L. R., Lewis, J. B., Wright, J. A., Choma, D. P., Miller, R. A., & Peterson, J. F. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000185 · Cited by 138
High-priority drug–drug interactions for use in electronic health records
Phansalkar, S., Desai, A. A., Bell, D., Yoshida, E., Doole, J., Czochanski, M., Middleton, B., & Bates, D. W. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000612 · Cited by 134 · cited 1×
“This method has also been used to identify and refine high-severity drug – drug interactions [56] and to identify low-priority drug – drug interactions that do not require interruptive alerts [57].”
Method
Cited by
Page, N., Baysari, M. T., & Westbrook, J. I. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.05.011 · Cited by 129
Appropriateness of Overridden Alerts in Computerized Physician Order Entry: Systematic Review
Poly, T. N., Islam, M. M., Yang, H.-C., & Li, Y.-C. (. (2020) · JMIR Medical Informatics · doi:10.2196/15653 · Cited by 118
“A CDSS with higher sensitivity and lower specificity could also contribute to the substantial number of inappropriate alerts [11,12].”
BackgroundThe Impact of Electronic Health Records on Diagnosis
Graber, M. L., Byrne, C., & Johnston, D. (2017) · Diagnosis · doi:10.1515/dx-2017-0012 · Cited by 108
Improving medication-related clinical decision support
Tolley, C. L., Slight, S. P., Husband, A. K., Watson, N., & Bates, D. W. (2018) · American Journal of Health-System Pharmacy · doi:10.2146/ajhp160830 · Cited by 105
Self-citationOlakotan, O. O., & Mohd Yusof, M. (2021) · Health Informatics Journal · doi:10.1177/14604582211007536 · Cited by 101
“52,54,57,83 Waiting (delays) Unnecessary delays caused by software malfunctions (e.g. poor computer response time) and system downtime Clinicians must wait for an alert before proceeding to other tasks because of slow processing time, computer crashes, and Internet connectivity problems.”
BackgroundInfluential
Metadata
- Authors
- Coleman, J. J., van der Sijs, H., Haefeli, W. E., Slight, S. P., McDowell, S. E., Seidling, H. M., Eiermann, B., Aarts, J., Ammenwerth, E., Ferner, R. E., & Slee, A.
- Venue
- BMC Medical Informatics and Decision Making
- Published
- 2013-10-01
- Volume
- 13
- Issue
- 1
- Pages
- 111-111
- ISSN
- 1472-6947 (ISSN-L)
- OpenAlex
- W2164590486
- Cited by
- 107
- References
- 11
- FWCI
- 14.13
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. https://app.cochat.ai/papers/10.1186/1472-6947-13-111
- The paper
- Coleman, J. J., van der Sijs, H., Haefeli, W. E., Slight, S. P., McDowell, S. E., Seidling, H. M., Eiermann, B., Aarts, J., Ammenwerth, E., Ferner, R. E., & Slee, A. (2013). On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC Medical Informatics and Decision Making. https://doi.org/10.1186/1472-6947-13-111
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.