The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 18 of the 56 sources its claims cite were checked for support.
Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system
Open accessAbstract
Background: Although alert fatigue is blamed for high override rates in contemporary clinical decision support systems, the concept of alert fatigue is poorly defined. We tested hypotheses arising from two possible alert fatigue mechanisms: (A) cognitive overload associated with amount of work, complexity of work, and effort distinguishing informative from uninformative alerts, and (B) desensitization from repeated exposure to the same alert over time. Methods: Retrospective cohort study using electronic health record data (both drug alerts and clinical practice reminders) from January 2010 through June 2013 from 112 ambulatory primary care clinicians. Results: On average, one-quarter of drug alerts received by a primary care clinician, and one-third of clinical reminders, were repeats for the same patient within the same year. Alert acceptance was associated with work complexity and repeated alerts, but not with the amount of work. Likelihood of reminder acceptance dropped by 30% for each additional reminder received per encounter, and by 10% for each five reminders received per day.
Overview
CoChat Summary
TL;DR
In a retrospective cohort of 112 primary care clinicians, alert acceptance fell with greater work complexity and more repeated alerts, but not with amount of workload or desensitization over time. Reducing within-patient repeat alerts may be a promising target for reducing alert fatigue.
Key findings
- Alert acceptance decreased with repeated alerts and complexity: for best-practice advisories (BPAs), each additional BPA per encounter reduced acceptance by 30% (IRR 0.70, p < .001), and each 5-percentage-point increase in repeated BPAs reduced acceptance by 10% (IRR 0.90, p < .001).
- Workload amount was not associated with acceptance: patients per year and encounters per year were not significant predictors. Patient comorbidity was associated with lower BPA acceptance in bivariate analysis (IRR 0.27, p = .008) but narrowly lost significance in the multivariable model (IRR 0.49, p = .06).
- No desensitization pattern was found: among 6 newly deployed BPAs, only 1 showed an early peak followed by decline (slopes +0.16 to −0.34; p = .03 to .24), likely due to flu seasonality; the others had no clear temporal pattern.
- Nurse practitioners were more than 4 times as likely as physicians to accept drug alerts (multivariable IRR 4.56, p = .002; bivariate IRR 3.97, p = .002). Repeated alerts were highly sticky: if the first BPA was overridden, 87.9% of subsequent instances were overridden; if the first was accepted, 51.9% of subsequent instances were overridden. For drug alerts, the corresponding override rates were 99.9% and 58.4%.
Methods
- Retrospective cohort using EHR data from January 2010 through June 2013 at the Institute for Family Health, including 112 ambulatory primary care clinicians (93 physicians, 19 nurse practitioners), 430,803 encounters, 99,649 patients, 1,266,325 BPAs, and 326,203 drug-drug interaction/drug-allergy interaction alerts.
- Clinician-level negative binomial models with an offset of log(number of alerts fired) and cluster-robust standard errors; predictors included clinician type, meaningful use attestation, sex, patients per year, encounters per year, alerts per encounter, patient comorbidity, percent within-patient repeats, order sets used, and labs ordered.
- Desensitization was tested on 6 newly deployed BPAs available continuously for ≥6 months and presented to ≥10 clinicians ≥5 times each; monthly acceptance rates for 46 clinicians were graphed with lowess smoothing and modeled with Poisson, zero-inflated Poisson, negative binomial, or zero-inflated negative binomial models selected by likelihood ratio and Vuong tests.
Limitations
- The paper states it was observational, used novel metrics because no validated metrics existed, and relied on a single commercial EHR with interruptive alerts, limiting generalizability. It had no DDI/DAI severity information and no manual review to distinguish appropriate from inappropriate overrides; repeats were assumed to be potentially uninformative, though relevance depends on the clinical situation.
- The paper also states that secondary EHR data lacked ambient noise, interruptions, time pressure, quality of patient-clinician interaction, and multitasking, likely biasing findings toward the null; high alert volume may have produced ceiling effects; and desensitization models used robust rather than cluster-robust errors.
- Evident from the text: drug alert acceptance was <1%, creating floor effects that weakened DDI/DAI analyses, and the clinician-level observational design cannot establish causality.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
11 Valid · 0 Not valid · 17 Not checked · 46 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“In practice, however, 49–96% of alerts are overridden [7], raising questions about the effectiveness of decision support.”
§ Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system > Background
What the sources say
- Supportedcites [7] van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13(2):138–47. source
The Results section states that drug safety alerts are overridden in 49% to 96% of cases. It notes an exception: serious overdose alerts were overridden in about one fourth of cases. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This was because a substantial body of work [7, 8, 32, 33] suggests that repeated CDSS alerts are less likely to be informative (that is, they are often equivalent to false alarms).”
§ Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system > Methods > Cognitive overload hypotheses
What the sources say
- Partly supportedcites [7] van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13(2):138–47. sourceExperimental — accuracy not yet measured
The Results describe many alerts as irrelevant or false positive and recommend preventing some repeat alerts, including interaction alerts previously found not to cause problems for a particular patient. But the paper does not establish that repeated alerts generally become less informative or are often false alarms; it also cautions that an alert a clinician already knows can still flag a potentially unsafe situation. Page numbers were not available in our copy. (LLM, conf=0.82)
- Partly supportedcites [8] Hsieh TC, Kuperman GJ, Jaggi T, et al. Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system. J Am Med Inform Assoc. 2004;11(6):482–91. sourceExperimental — accuracy not yet measured
The Abstract (page not specified) describes redundant drug-allergy alerts from repeated prescription renewals and alerts with low predictive value. The Discussion (page not specified) says frequent false alerts can obscure important warnings. The paper does not establish that repeated CDSS alerts generally are less informative or that repetition itself makes an alert a false alarm. (LLM, conf=0.84)
- Partly supportedcites [32] Stultz JS, Nahata MC. Appropriateness of commercially available and partially customized medication dosing alerts among pediatric patients. J Am Med Inform Assoc. 2014;21(e1):e35–42. sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) reports that most pediatric dosing alerts studied were inappropriate and that excessive inappropriate alerting may contribute to alert fatigue. This supports the concern that many alerts are uninformative, but the paper does not establish that an alert becomes less informative when repeated. (LLM, conf=0.83)
- Not checked — no full text of the sourcecites [33] Feldman PH, McDonald MV. Exploring the utility of automated drug alerts in home health care. J Healthc Qual. 2006;28(1):29–40. source
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“One study supporting the desensitization explanation showed that response to clinical trial recruiting alerts in the EHR declined by approximately 2.7% per week, falling from about 50% to about 35% over 36 weeks [19].”
§ Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system > Background
What the sources say
- Partly supportedcites [19] Embi PJ, Leonard AC. Evaluating alert fatigue over time to EHR-based clinical trial alerts: findings from a randomized controlled study. J Am Med Inform Assoc. 2012;19(e1):e145–8. sourceExperimental — accuracy not yet measured
The Results (page not provided) report that responses to EHR-based clinical trial alerts fell from about 50% initially to 35% at week 36. The reported decline was 2.7% per two-week period, not per week as the citing sentence states. (LLM, conf=0.92)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“a retrospective chart review in which expert reviewers identified inappropriate DDI overrides found the only factor associated with inappropriate overrides to be number of noncritical alerts received at the same time [23].”
§ Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system > Discussion
What the sources say
- Supportedcites [23] Miller AM, Boro MS, Korman NE, Davoren JB. Provider and pharmacist responses to warfarin drug–drug interaction alerts: a study of healthcare downstream of CPOE alerts. J Am Med Inform Assoc. 2011;18 Suppl 1:i45–50. source
The Data categorization section says raters classified override responses as appropriate or inappropriate, and the paper describes the study as a retrospective chart review. The Results section reports that the number of simultaneous non-critical alerts was the only factor associated with an inappropriate provider response, including in multivariate analysis. This finding concerns critical DDI alerts involving warfarin. (LLM, conf=0.91)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With every additional BPA per encounter, the likelihood a clinician would accept any BPA dropped by 30% (IRR = 0.70; p < .001; Table 2).”
§ Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system > Results > Cognitive overload hypothesis
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
29 more claims
Create a free account to see all 74 claims and to verify the 3 references still open.
- Probably not a substantial claim. 27
- It may state two separate claims as one. 14
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Complementary work on alert fatigue and automation bias shows that clinician interaction with decision support systems can recursively influence both inputs and outcomes, creating feedback loops that invalidate original performance assumptions [19,20].”
From Validation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems (2026) · cites this paper as [20]
Experimental — accuracy not yet measuredIn the citing paper
“In this framework, environmental factors (interruptions), along with the factors of task demand (task time, task difficulty) [19], technology (system usability) [20] and individual characteristics (professional competency, self-efficacy and mood), are the key variables that influence psychophysiological responses, workload modification and performance in human–machine interaction tasks [21].”
From Workflow interruption and nurses’ mental workload in electronic health record tasks: An observational study (2023) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Under time constraints, 8 of 211 alerts (13%) were handled incorrectly because of skill-based errors”
§ Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system > Discussion
Reported 13 · recomputed 3.791469
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system > Abstract > Results
“On average, one-quarter of drug alerts received by a primary care clinician, and one-third of clinical reminders, were repeats for the same patient within the same year.”
§ Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system > Results
“More than a quarter of BPAs were repeats for the same clinician for the same patient within the same year (median 26.2% per clinician; interquartile range [IQR] 16.5% to 35.1%). Similarly, of all DDI/DAI received by a clinician, almost 1/3 represented repeats for the same patient within the same year (median 31.8% per clinician; IQR 25.6% to 38.2%).”
Citations
Cites
(1967) · Project Euclid (Cornell University) · Cited by 4,720
Proceedings of the fifth Berkeley symposium on mathematical statistics and probability.
Kojima, K.-I. (1969) · Europe PMC (PubMed Central) · Cited by 3,648
Regression standard errors in clustered samples
Rogers, W. H. (1993) · Stata technical bulletin · Cited by 2,694
Bates, D. W. (1998) · JAMA · doi:10.1001/jama.280.15.1311 · Cited by 2,107
The “Meaningful Use” Regulation for Electronic Health Records
Blumenthal, D., & Tavenner, M. (2010) · New England Journal of Medicine · doi:10.1056/nejmp1006114 · Cited by 2,059 · cited 1×
“As a result of this promising history, as well as the federal EHR incentive program (the “meaningful use” program) [4], contemporary EHR products routinely integrate eprescribing with CDSS.”
BackgroundAmerican Academy of Family Physicians
(1966) · New England Journal of Medicine · doi:10.1056/nejm196611172752014 · Cited by 1,948
Ash, J. S. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1471 · Cited by 1,789
Proceedings of the Human Factors and Ergonomics Society 54th Annual Meeting
Kluge, A., Badura, B., Urbas, L., & Burkolter, D. (2010) · Cited by 1,127
Overriding of Drug Safety Alerts in Computerized Physician Order Entry
van der Sijs, H., Aarts, J., Vulto, A., & Berg, M. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1809 · Cited by 1,117 · cited 5×
“We considered alerts that were repeated the same year as potentially uninformative, following the work of previous researchers who have identified repeated alerts as duplicative and likely to have low relevance [7, 8, 32, 33].”
BackgroundInfluential(2003) · Cited by 969
Cited by
Giannini, H. M., Ginestra, J. C., Chivers, C., Draugelis, M., Hanish, A., Schweickert, W. D., Fuchs, B. D., Meadows, L., Lynch, M., Donnelly, P. J., et al. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003891 · Cited by 269
Digital health technologies: opportunities and challenges in rheumatology
Solomon, D. H., & Rudin, R. S. (2020) · Nature Reviews Rheumatology · doi:10.1038/s41584-020-0461-x · Cited by 228
Mangin, D., Bahat, G., Golomb, B. A., Mallery, L. H., Moorhouse, P., Onder, G., Petrovic, M., & Garfinkel, D. (2018) · Drugs & Aging · doi:10.1007/s40266-018-0554-2 · Cited by 226
Corny, J., Rajkumar, A., Martin, O., Dode, X., Lajonchère, J.-P., Billuart, O., Bézie, Y., & Buronfosse, A. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa154 · Cited by 225
Artificial intelligence-based clinical decision support in pediatrics
Ramgopal, S., Sanchez-Pinto, L. N., Horvat, C. M., Carroll, M. S., Luo, Y., & Florin, T. A. (2023) · Pediatric Research · doi:10.1038/s41390-022-02226-1 · Cited by 177
“One study of drug alerts in the primary care setting demonstrated a decline in the utilization of an alert-based CDS with additional reminders for the same patient.(19) In turn, ignored alerts from a CDS can result in adverse consequences.”
Background
Metadata
- Authors
- with the HITEC Investigators, Ancker, J. S., Edwards, A., Nosal, S., Hauser, D., Mauer, E., & Kaushal, R.
- Affiliations
- Weill Cornell Medicine (US)
- Institute for Family Health (US)
- Corresponding author
- Ancker, Jessica S.
- Venue
- BMC Medical Informatics and Decision Making
- Published
- 2017-04-10
- Volume
- 17
- Issue
- 1
- Pages
- 36-36
- ISSN
- 1472-6947 (ISSN-L)
- PMC
- PMC5387195
- OpenAlex
- W2605753964
- Cited by
- 709
- References
- 43
- Citations by year
- 2026: 160 · 2025: 87 · 2024: 76 · 2023: 82 · 2022: 90 · 2021: 74 · 2020: 73 · 2019: 38 · 2018: 26 · 2017: 2
- FWCI
- 52.24
- Type
- journal article
- Language
- en
- Update notices
- correction (2019-11-18)
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- New York State Department of Health (C025877)
- Agency for Healthcare Research and Quality (K01HS021531)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- alarm fatigue, clinical decision support, drug alerts, alert override, cognitive overload, workload, job complexity, desensitization, nurse practitioners, primary care providers, electronic health records
- MeSH terms
- Alert Fatigue, Health Personnel, Adult, Humans, Nurse Practitioners, Retrospective Studies, Workload, Decision Support Systems, Clinical, Electronic Health Records, Physicians, Primary Care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. https://app.cochat.ai/papers/10.1186/s12911-017-0430-8
- The paper
- with the HITEC Investigators, Ancker, J. S., Edwards, A., Nosal, S., Hauser, D., Mauer, E., & Kaushal, R. (2017). Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Medical Informatics and Decision Making. https://doi.org/10.1186/s12911-017-0430-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.