Clinician collaboration to improve clinical decision support: the Clickbusters initiative
Open accessAbstract
Objective: We describe the Clickbusters initiative implemented at Vanderbilt University Medical Center (VUMC), which was designed to improve safety and quality and reduce burnout through the optimization of clinical decision support (CDS) alerts. Materials and Methods: We developed a 10-step Clickbusting process and implemented a program that included a curriculum, CDS alert inventory, oversight process, and gamification. We carried out two 3-month rounds of the Clickbusters program at VUMC. We completed descriptive analyses of the changes made to alerts during the process, and of alert firing rates before and after the program. Results: Prior to Clickbusters, VUMC had 419 CDS alerts in production, with 488 425 firings (42 982 interruptive) each week. After 2 rounds, the Clickbusters program resulted in detailed, comprehensive reviews of 84 CDS alerts and reduced the number of weekly alert firings by more than 70 000 (15.43%). In addition to the direct improvements in CDS, the initiative also increased user engagement and involvement in CDS. Conclusions: At VUMC, the Clickbusters program was successful in optimizing CDS alerts by reducing alert firings and resulting clicks. The program also involved more users in the process of evaluating and improving CDS and helped build a culture of continuous evaluation and improvement of clinical content in the electronic health record.
AI snapshot
Written from the paper’s full text.
TL;DR
The Clickbusters initiative at Vanderbilt University Medical Center used a 10-step, gamified clinician-led process to review and optimize clinical decision support alerts. Across two 3-month rounds, it reviewed 84 alerts and reduced weekly alert firings/clicks by more than 70,000 (15.43%).
Key findings
- Before Clickbusters, VUMC had 419 CDS alerts in production, with 488,425 weekly firings, including 42,982 interruptive firings; 1.5% of total alerts and 9.1% of interruptive alerts were accepted.
- Two rounds reviewed 84 alerts: 42 were modified, 10 were turned off, and 32 needed no modifications; total weekly clicks busted were 71,227.
- Round 1 had 8 participants, 18 alert groups, 29 alerts analyzed, and 49,026 weekly clicks busted (10.38%); Round 2 had 20 participants, 24 alert groups, 55 alerts analyzed, and 22,201 weekly clicks busted (5.05%).
- A single unnecessary chronic care management alert turned off in Round 1 accounted for 40,813 weekly clicks reduced, or 83% of that round’s total.
Methods
- The program used a 10-step Clickbusting process aligned with Plan-Do-Study-Act, plus a curriculum, CDS alert inventory, oversight, wiki tracking, and gamification with points and awards.
- VUMC conducted two approximately 3-month rounds in 2020; participants selected alerts from prioritized groups based on burden and complexity scores.
- The study used descriptive analyses of alert logs before and after each round, including total alerts displayed, interruptive alerts, and acceptances; to control for confounding, reductions for modified alerts were adjusted using the firing reduction observed among unmodified alerts.
Limitations
- The paper states a critical limitation: Clickbusters requires an active Physician Builder program or another core group of nonoperational informatics team members able to select alerts and carry out the process; sites must judge whether safety and efficiency gains justify costly resources like a Physician Builder program.
- The paper notes it may still be possible in settings without such groups, such as nonacademic medical centers, but the process may take more time and operational effort.
- Evident from the text: this is a single-center, descriptive pre-post program evaluation without randomized comparison or reported patient outcome measures, and the adjustment for confounding relied on observed firing changes among unmodified alerts.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
21 of 53 cited sources checked · 10 supported · 0 flagged · accuracy not yet measured
- Premisecitation
Prior research has found that there are opportunities for improving or turning off CDS alerts to improve patient outcomes.
“Prior research has found that there are opportunities for improving or turning off CDS alerts to improve patient outcomes.25–28”
Quote in the paperSubstantial claim1 of 4 sources checked- cites [25] Phansalkar, 2012, High-priority drug–drug interactions for use in electronic health records, J Am Med Inform Assoc source
- cites [26] Phansalkar, 2013, Drug—drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records, J Am Med Inform Assoc source
The Results identify 33 drug–drug interaction alerts the panel judged safe to make non-interruptive, supporting an opportunity to improve CDS alerting. The Discussion explicitly advises against turning those alerts off completely and says further research is needed to determine whether the change improves patient safety; improved patient outcomes were not established. (LLM, conf=0.85)
Experimental — accuracy not yet measured - cites [27] van der Sijs, 2008, Turning off frequently overridden drug alerts: limited opportunities for doing it safely, J Am Med Inform Assoc source
- cites [28] Seidling, 2014, What, if all alerts were specific - estimating the potential impact on drug interaction alert burden, Int J Med Inform source
- Premisecitation
Evaluations of CDS have not consistently demonstrated improved patient outcomes.
“Despite initial promise of success and widespread CDS implementations, evaluations of CDS have not consistently demonstrated improved patient outcomes.9–12”
Quote in the paperSubstantial claim1 of 4 sources checked- cites [9] Gurwitz, 2008, Effect of computerized provider order entry with clinical decision support on adverse drug events in the long-term care setting, J Am Geriatr Soc source
- cites [10] Strom, 2010, Randomized clinical trial of a customized electronic alert requiring an affirmative response compared to a control group receiving a commercial passive CPOE alert: NSAID–warfarin co-prescribing as a test case, J Am Med Inform Assoc source
The Background (page not specified in our copy) says alerts are effective in some domains and ineffective in others, and that reviews find mixed effects of CPOE systems. The Results and Discussion report no advantage of the customized alert for prescribing behavior, but do not establish whether CDS evaluations consistently improve patient outcomes. (LLM, conf=0.84)
Experimental — accuracy not yet measured - cites [11] Tamblyn, 2010, Increasing the detection and response to adherence problems with cardiovascular medication in primary care through computerized drug management systems: a randomized controlled trial, Med Decis Making source
- cites [12] Strom, 2010, Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial, Arch Intern Med source
- Premisecitation
Additional efforts have focused on identifying opportunities for improvement by obtaining feedback from clinicians receiving alerts, such as sentiment analysis of comments entered when overriding alerts and feedback links within displayed alerts.
“Additional efforts have focused on identifying opportunities for improvement by obtaining feedback from clinicians receiving the alerts, such as sentiment analysis of comments entered by clinicians when overriding alerts52 and feedback links within displayed alerts.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites [52] Aaron, 2019, Cranky comments: detecting clinical decision support malfunctions through free-text override reasons, J Am Med Inform Assoc source
- Premisecitation
Another approach uses anomaly detection to identify alerts that are no longer performing as intended.
“Another approach uses anomaly detection to identify alerts that are no longer performing as intended.55,56”
Quote in the paperNot a substantial claim2 of 2 sources checked- cites [55] Ray, 2018, Using statistical anomaly detection models to find clinical decision support malfunctions, J Am Med Inform Assoc source
- cites [56] Wright, 2016, Analysis of clinical decision support system malfunctions: a case series and survey, J Am Med Inform Assoc source
The Discussion (Future Work; page unavailable) says anomaly detection methods are likely to help find CDSS malfunctions in real time. But it proposes investigating those methods rather than reporting an approach already in use, as the claim implies. The Methods describe a retrospective model for estimating the size of known alert anomalies, not an anomaly-detection system. (LLM, conf=0.87)
Experimental — accuracy not yet measured
- Premisecitation
Overrides may result from an excess of alerts that are repeated or deemed irrelevant (high alert burden).
“Overrides may result from an excess of alerts that are repeated or deemed irrelevant (ie, a high alert burden), causing alert fatigue and reducing the efficacy of CDS.13,24”
Quote in the paperSubstantial claim1 of 2 sources checked- cites [13] van der Sijs, 2006, Overriding of drug safety alerts in computerized physician order entry, J Am Med Inform Assoc source
- cites [24] Middleton, 2013, Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA, J Am Med Inform Assoc source
- Premisecitation
Alert nonadherence, or clinician overrides, occurs for 49%-96% of alerts.
“Alert nonadherence, or clinician overrides, is one key barrier that occurs for 49%–96% of alerts.13–23”
Quote in the paperSubstantial claim4 of 10 sources checked- cites [13] van der Sijs, 2006, Overriding of drug safety alerts in computerized physician order entry, J Am Med Inform Assoc source
The Results report that safety alerts were overridden in 49%–96% of cases, supporting the cited range. But the same passage excludes serious overdose alerts, which were overridden about one fourth of the time; the Discussion and Conclusion gives 27%. The citing sentence omits that qualification. (LLM, conf=0.86)
Experimental — accuracy not yet measured - cites [14] Weingart, 2003, Physicians’ decisions to override computerized drug alerts in primary care, Arch Intern Med source
- cites [15] Hsieh, 2004, Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry System, J Am Med Inform Assoc source
The Results report that 6,182 of 7,761 drug-allergy alerts (80%) were overridden, which falls within the cited range. The paper does not establish the 49%–96% range for alerts generally; its own measurement concerns drug-allergy alerts in one hospital's CPOE system. (LLM, conf=0.86)
Experimental — accuracy not yet measured - cites [16] Persell, 2010, Frequency of inappropriate medical exceptions to quality measures, Ann Intern Med source
- cites [17] Grizzle, 2007, Reasons provided by prescribers when overriding drug-drug interaction alerts, Am J Manag Care
- cites [18] van der Sijs, 2009, Drug safety alert generation and overriding in a large Dutch university medical centre, Pharmacoepidemiol Drug Saf source
- cites [19] Ko, 2007, Practitioners’ views on computerized drug-drug interaction alerts in the VA system, J Am Med Inform Assoc source
In the section labeled “2.6 (1.0) 18” (page unavailable), the paper says that the majority of prescription-order alerts are overridden and do not affect the final prescribing decision. It does not give the claimed 49%–96% range, so it supports frequent overrides but not that numerical estimate. (LLM, conf=0.86)
Experimental — accuracy not yet measured - cites [20] Isaac, 2009, Overrides of medication alerts in ambulatory care, Arch Intern Med source
- cites [21] Shah, 2006, Improving acceptance of computerized prescribing alerts in ambulatory care, J Am Med Inform Assoc source
Table 3 (Methods) reports clinician overrides, including rates of 58% for drug–drug alerts and 90% for drug-pregnancy alerts. But its overall override rate for interruptive alerts is 33%, and no reported category reaches 96%, so this paper does not support the stated 49%–96% range as a general rate. (LLM, conf=0.87)
Experimental — accuracy not yet measured - cites [22] Swiderski, 2007, A study of the frequency and rationale for overriding allergy warnings in a computerized prescriber order entry system, J Patient Saf source
- Findingown data
A single unnecessary alert for chronic care management turned off in the first round accounted for 40,813 weekly clicks reduced (83% of the total for that round).
“a single unnecessary alert for chronic care management turned off in the first round that accounted for 40 813 weekly clicks reduced (83% of the total for that round).”
Quote in the paperNot a substantial claim - Findingown data
Discussion with Clickbusters participants after completion of the program revealed that some participants spent as little as 2 hours completing their work, while others devoted 20 or more hours.
“Discussion with Clickbusters participants after completion of the program revealed that some participants spent as little as 2 hours completing their work, while others devoted 20 or more hours.”
Quote in the paperNot a substantial claim - Findingown data
Prior to Clickbusters, VUMC had 419 CDS alerts in production, with 488,425 firings (42,982 interruptive) each week.
“Prior to Clickbusters, VUMC had 419 CDS alerts in production, with 488 425 firings (42 982 interruptive) each week.”
Quote in the paperSubstantial claim - Findingown data
After 2 rounds, Clickbusters reduced the number of weekly clicks by more than 70,000 (15.43%).
“After 2 rounds, the Clickbusters program resulted in detailed, comprehensive reviews of 84 CDS alerts, which comprised 20% of the rule-based alerts implemented at VUMC, and reduced the number of weekly clicks by more than 70 000 (15.43%).”
Quote in the paperSubstantial claim- Rests on an assumption: The reduction in firings among unmodified alerts (20%) reflects confounding changes that would have affected modified alerts to the same degree.
- Findingown data
After 2 rounds, the Clickbusters program reduced the number of weekly alert firings by more than 70,000 (15.43%).
“After 2 rounds, the Clickbusters program resulted in detailed, comprehensive reviews of 84 CDS alerts and reduced the number of weekly alert firings by more than 70 000 (15.43%).”
Quote in the paperSubstantial claim- Rests on an assumption: The reduction in weekly alert firings observed after the Clickbusters program was attributable to the Clickbusters program rather than other changes.
- Findingown data
After 2 rounds, the Clickbusters program resulted in detailed, comprehensive reviews of 84 CDS alerts.
“After 2 rounds, the Clickbusters program resulted in detailed, comprehensive reviews of 84 CDS alerts and reduced the number of weekly alert firings by more than 70 000 (15.43%).”
Quote in the paperSubstantial claim - Findingown data
Most of these Physician Builders had not used the related tools since their training.
“We learned that while they had all received some training from Epic on alert build, most had not used the related tools since their training, and they felt that they needed a refresher.”
Quote in the paperSubstantial claim - Findingown data
After 2 rounds, Clickbusters resulted in reviews of 84 CDS alerts, comprising 20% of the rule-based alerts implemented at VUMC.
“After 2 rounds, the Clickbusters program resulted in detailed, comprehensive reviews of 84 CDS alerts, which comprised 20% of the rule-based alerts implemented at VUMC”
Quote in the paperNot a substantial claim - Findingnone given
Members of VCLIC or the operational informatics team subsequently made the changes suggested by nonbuilder participants.
“Members of VCLIC or the operational informatics team subsequently made the changes suggested by nonbuilder participants.”
Quote in the paperNot a substantial claim - Findingown data
The Clickbusters program succeeded in reducing alert firings and resulting clicks.
“The program succeeded in reducing alert firings and resulting clicks.”
Quote in the paperSubstantial claim - Interpretationreasoning
The Clickbusters program helped build a culture of continuous evaluation and improvement of clinical content in the electronic health record.
“The program also involved more users in the process of evaluating and improving CDS and helped build a culture of continuous evaluation and improvement of clinical content in the electronic health record.”
Quote in the paperSubstantial claim - Interpretationreasoning
Clickbusters helped build a culture of continuous evaluation and improvement of clinical content in the EHR.
“This program helped build a culture of continuous evaluation and improvement of clinical content in the EHR.”
Quote in the paperSubstantial claim - Study design and conductreasoning
A Clickbusters review satisfied the informatics team's annual or semiannual BPA review requirement in many cases.
“Further, our informatics team reviews each BPA on an annual or semiannual basis, and we determined that a Clickbusters review satisfied this requirement in many cases.”
Quote in the paperNot a substantial claim - Interpretationreasoning
The partnership created new relationships between Clickbusters participants and HealthIT staff that have led to subsequent collaborative efforts.
“it also created new relationships between Clickbusters participants and HealthIT staff that have led to subsequent collaborative efforts.”
Quote in the paperNot a substantial claim - Interpretationnone given
A key component of the Clickbusters program is partnership with the operational HealthIT department.
“A key component of the Clickbusters program is partnership with the operational HealthIT department.”
Quote in the paperSubstantial claim - Study design and conductown data
The analysis controlled for confounding changes to alerts by adjusting for the reduction in firings among unmodified alerts.
“we determined the reduction in firings for alerts not modified during Clickbusters after the round and multiplied the number of reduced clicks by that number (ie, unmodified alerts fired 20% less often after Clickbusters, so we determined the number of clicks actually busted to be 80% of the absolute difference in firings for modified alerts after Clickbusters).”
Quote in the paperSubstantial claim- Rests on an assumption: The reduction in firings among unmodified alerts (20%) reflects confounding changes that would have affected modified alerts to the same degree.
- Interpretationown data
At VUMC, the Clickbusters program was successful in optimizing CDS alerts by reducing alert firings and resulting clicks.
“At VUMC, the Clickbusters program was successful in optimizing CDS alerts by reducing alert firings and resulting clicks.”
Quote in the paperSubstantial claim- Rests on an assumption: The reduction in weekly alert firings observed after the Clickbusters program was attributable to the Clickbusters program rather than other changes.
- Rests on an assumption: Reducing alert firings and resulting clicks constitutes optimization of CDS alerts.
- Interpretationown data
The lower total number of clicks reduced in the second round was in large part due to that single alert turned off in the first round.
“Although more participants reviewed more alerts in the second round, the total number of clicks reduced was lower, in large part due to a single unnecessary alert for chronic care management turned off in the first round”
Quote in the paperNot a substantial claim - Interpretationreasoning
The partnership with HealthIT helped the Clickbusters program to succeed.
“This partnership helped the Clickbusters program to succeed”
Quote in the paperSubstantial claim - Interpretationreasoning
After Clickbusters, the VUMC CDS team benefited from a new corps of users with increased interest, engagement, and knowledge of CDS, who could serve as liaisons to clinical departments.
“After Clickbusters, the VUMC CDS team benefited from a new corps of users with increased interest, engagement, and knowledge of CDS, who could also serve as liaisons to clinical departments as diverse as pediatrics, general medicine, oncology, cardiology, surgery, nursing, and pharmacy.”
Quote in the paperSubstantial claim - Interpretationreasoning
Because the curriculum and alert data can be shared and have been made available, subsequent implementations of the program would require significantly less effort.
“Because this information can be shared and has been made available, subsequent implementations of the program would require significantly less effort.”
Quote in the paperSubstantial claim - Interpretationreasoning
Clickbusters is advantageous in that it can incorporate previously described approaches by providing that information (eg, utilization logs, dashboards, feedback) to participants to facilitate their evaluation.
“It is also advantageous in that it can incorporate the previously described approaches by providing that information (eg, utilization logs, dashboards, feedback) to the Clickbusters participants to facilitate their evaluation.”
Quote in the paperNot a substantial claim - Interpretationown data
A critical limitation of the Clickbusters approach is that it requires an active Physician Builder program or other core group of nonoperational informatics team members able to select alerts and carry out the Clickbusting process.
“A critical limitation of the Clickbusters approach is that it requires an active Physician Builder program or other core group of nonoperational informatics team members that are able to select alerts and carry out the Clickbusting process.”
Quote in the paperSubstantial claim
- May combine two claims. 24
- Classified as probably not a substantial claim. 14
May combine two claims.
PremisecitationMany organizations are not following these recommendations, in part due to a lack of consensus for whether or not to turn off or modify, or how to modify, alerts.
“Although some organizations have successfully implemented these tools and reduced the number of alerts and overrides,38 other reports have found that many organizations are not following these recommendations, in part due to a lack of consensus for whether or not to turn off or modify, or how to modify, alerts.27,28,39”
Quote in the paperSubstantial claim1 of 3 sources checked- cites [27] van der Sijs, 2008, Turning off frequently overridden drug alerts: limited opportunities for doing it safely, J Am Med Inform Assoc source
- cites [28] Seidling, 2014, What, if all alerts were specific - estimating the potential impact on drug interaction alert burden, Int J Med Inform source
- cites [39] McEvoy, 2017, Variation in high-priority drug-drug interaction alerts across institutions and electronic health records, J Am Med Inform Assoc source
The Discussion (‘A standard of care for DDI alerts?’) reports inconsistent adoption of the proposed high-priority DDI list and a lack of agreement about which alerts to implement and how to display them. It does not establish that organizations failed to follow recommendations because of that lack of agreement, or specifically assess decisions to turn off or modify alerts. (LLM, conf=0.82)
Experimental — accuracy not yet measured
May combine two claims.
PremisecitationHigh alert burden causes alert fatigue and reduces the efficacy of CDS.
“Overrides may result from an excess of alerts that are repeated or deemed irrelevant (ie, a high alert burden), causing alert fatigue and reducing the efficacy of CDS.13,24”
Quote in the paperSubstantial claim1 of 2 sources checked- cites [13] van der Sijs, 2006, Overriding of drug safety alerts in computerized physician order entry, J Am Med Inform Assoc source
- cites [24] Middleton, 2013, Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA, J Am Med Inform Assoc source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationInstitutions have implemented various dashboards that organize and present information from the EHR in a way that is easy to interpret.
“Similarly, to facilitate alert evaluations, institutions have implemented various dashboards that organize and present information from the EHR in a way that is easy to interpret.29–34”
Quote in the paperNot a substantial claim1 of 6 sources checked- cites [29] Simpao, 2014, A review of analytics and clinical informatics in health care, J Med Syst source
- cites [30] Zimmerman, 2007
- cites [31] Reynolds, 2008
- cites [32] McCoy, 2012, Real-time pharmacy surveillance and clinical decision support to reduce adverse drug events in acute kidney injury: a randomized, controlled trial, Appl Clin Inform source
- cites [33] Waitman, 2011, Adopting real-time surveillance dashboards as a component of an enterprisewide medication safety strategy, Jt Comm J Qual Patient Saf source
- cites [34] McCoy, 2014, Clinical decision support alert appropriateness: a review and proposal for improvement, Ochsner J
May combine two claims.
PremisecitationLike utilization logs, anomaly detection can successfully identify alerts that are no longer functioning as designed, but with less personnel effort due to the machine learning approach; however, it does not identify specific opportunities for improving alerts that were suboptimally designed at their onset.
“Like utilization logs, this approach can successfully identify alerts that are no longer functioning as designed, but with less personnel effort due to the machine learning approach; however, it does not identify specific opportunities for improving alerts that were suboptimally designed at their onset.”
Quote in the paperNot a substantial claim2 of 2 sources checked- cites [55] Ray, 2018, Using statistical anomaly detection models to find clinical decision support malfunctions, J Am Med Inform Assoc source
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured - cites [56] Wright, 2016, Analysis of clinical decision support system malfunctions: a case series and survey, J Am Med Inform Assoc source
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
May combine two claims.
PremisecitationA key challenge with clinical decision support is knowledge management as guidelines and best practices change over time, as well as governance.
“A key challenge with clinical decision support is knowledge management as guidelines and best practices change over time,40–42 as well as governance.43–45”
Quote in the paperSubstantial claim5 of 6 sources checked- cites [40] Ash, 2010, Identifying best practices for clinical decision support and knowledge management in the field, Stud Health Technol Inform
- cites [41] Sittig, 2008, Grand challenges in clinical decision support, J Biomed Inform source
Section 2.3.3 identifies formal knowledge-management processes as essential and says CDS content must accommodate ongoing upgrades and new knowledge. It also discusses standards for sponsorship and trust, but our copy does not explicitly identify governance as a separate challenge. (LLM, conf=0.82)
Experimental — accuracy not yet measured - cites [42] Sittig, 2010, The state of the art in clinical knowledge management: an inventory of tools and techniques, Int J Med Inform source
- cites [43] Wright, 2011, Governance for clinical decision support: case studies and recommended practices from leading institutions, J Am Med Inform Assoc source
- cites [44] Ash, 2012, Recommended practices for computerized clinical decision support and knowledge management in community settings: a qualitative study, BMC Med Inform Decis Mak source
- cites [45] Ash, 2012, Standard practices for computerized clinical decision support in community hospitals: a national survey, J Am Med Inform Assoc source
May combine two claims.
PremisecitationSuch feedback approaches can successfully identify individual alerts with optimization opportunities, as well as alerts with potentially suboptimal design or that are no longer functioning as designed.
“Such feedback approaches can successfully identify individual alerts with optimization opportunities, as well as alerts with potentially suboptimal design or that are no longer functioning as designed.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [52] Aaron, 2019, Cranky comments: detecting clinical decision support malfunctions through free-text override reasons, J Am Med Inform Assoc source
May combine two claims.
Findingown dataIn the second round, 20 participants selected 24 alert groups (55 total alerts) for Clickbusting, resulting in 29 alerts modified, 6 alerts turned off, and 22,201 weekly clicks busted (5.05%).
“In the second round, 20 participants selected 24 alert groups (55 total alerts) for Clickbusting, resulting in 29 alerts modified, 6 alerts turned off, and 22 201 weekly clicks busted (5.05%) (Table 1).”
Quote in the paperSubstantial claimMay combine two claims.
Findingnone givenThe Clickbusters initiative increased user engagement and involvement in CDS.
“In addition to the direct improvements in CDS, the initiative also increased user engagement and involvement in CDS.”
- Also stated in: § Conclusions · plain
“The program also involved more users in the process of evaluating and improving CDS and helped build a culture of continuous evaluation and improvement of clinical content in the electronic health record.”
Quote in the paperSubstantial claim- Rests on an assumption: Participation in the Clickbusters program indicates increased user engagement and involvement in CDS.
- Also stated in: § Conclusions · plain
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenSome HealthIT staff chose to participate in the Clickbusters program in addition to their regular job responsibilities.
“Additionally, some HealthIT staff chose to participate in the Clickbusters program, in addition to their regular job responsibilities.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe first round resulted in 13 alerts modified and 4 alerts turned off, with 49,026 weekly clicks busted (10.38%).
“This round resulted in 13 alerts modified and 4 alerts turned off, with 49 026 weekly clicks busted (10.38%).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataPrior to Clickbusters, VUMC had 419 BPA alerts in production, with 488,425 firings (42,982 interruptive) each week.
“Prior to beginning the Clickbusters initiative, VUMC had 419 BPA alerts in production, with 488 425 firings (42 982 interruptive) each week.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingnone givenClickbusters leadership devoted approximately 80 hours over 4 weeks to developing the curriculum and 40 hours over 2 weeks to retrieving alert data, ranking alerts, and creating the wiki pages.
“Clickbusters leadership (ABM, EMR, AW) devoted approximately 80 hours over 4 weeks to developing the curriculum and 40 hours over 2 weeks to retrieving alert data, ranking alerts, and creating the wiki pages.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe alerts were placed into 141 logical groups.
“The alerts were placed into 141 logical groups (eg, alerts related to suicide screening were grouped together).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataIn the first round, 8 participants selected 18 alert groups (29 total alerts) for Clickbusting.
“In the first round, 8 participants selected 18 alert groups (29 total alerts) for Clickbusting.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataAmong VUMC's BPA alerts prior to Clickbusters, 1.5% of total alerts and 9.1% of interruptive alerts were accepted.
“Among these, 1.5% of total alerts and 9.1% of interruptive alerts were accepted.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Findingown dataThese Physician Builders felt they needed a refresher.
“We learned that while they had all received some training from Epic on alert build, most had not used the related tools since their training, and they felt that they needed a refresher.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe Physician Builders who would be the first Clickbusters participants had all received some training from Epic on alert build.
“We learned that while they had all received some training from Epic on alert build, most had not used the related tools since their training, and they felt that they needed a refresher.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataMany Clickbusters participants (5 of the 24) were not certified Physician or Clinical Content Builders, but they were still able to complete reviews of the alerts and make suggestions for improvement based on information provided.
“Many of our Clickbusters participants (5 of the 24) were not certified Physician or Clinical Content Builders, but they were still able to complete reviews of the alerts and make suggestions for improvement based on information we provided”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe single alert turned off in the first round resulting in more than 40,000 fewer clicks was busted by a participant who was a primary care physician, but not a certified Physician Builder.
“the single alert turned off in the first round resulting in more than 40 000 fewer clicks was busted by a participant who was a primary care physician, but not a certified Physician Builder.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataThe Clickbusters results occurred in a short period and involved motivated users who had not previously participated in CDS review.
“While modest, these results occurred in a short period and involved motivated users that had not previously participated in CDS review.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
PremisecitationA frequently described approach in prior literature involves individuals or committees reviewing EHR utilization logs or dashboards, which are effective at identifying individual alerts with high override rates and alert trends across departments or provider type.
“One of the most frequently described approaches described in prior literature has involved individuals or committees reviewing EHR utilization logs or dashboards, as these are very effective at identifying individual alerts with high override rates, as well as alert trends across departments or provider type.27,28,34,38”
Quote in the paperSubstantial claim0 of 4 sources checked- cites [27] van der Sijs, 2008, Turning off frequently overridden drug alerts: limited opportunities for doing it safely, J Am Med Inform Assoc source
- cites [28] Seidling, 2014, What, if all alerts were specific - estimating the potential impact on drug interaction alert burden, Int J Med Inform source
- cites [34] McCoy, 2014, Clinical decision support alert appropriateness: a review and proposal for improvement, Ochsner J
- cites [38] Simpao, 2015, Optimization of drug-drug interaction alert rules in a pediatric hospital’s electronic health record system using a visual analytics dashboard, J Am Med Inform Assoc source
May combine two claims.
Interpretationown dataThe wide variation in participant time was due in part to the number of alerts selected and the number of Clickbusting steps completed; participants who selected more than one alert and made changes spent more time than those who selected a single alert and only reviewed it.
“Some reasons for this wide variation included the number of alerts selected for Clickbusting and the number of Clickbusting steps completed; participants who selected more than one alert and made changes to the alerts spent more time compared to those who only selected a single alert and only reviewed the alert.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningThe personnel effort for Clickbusters varies and can be shared across a larger group of individuals by engaging clinical builders and other participants beyond operational informatics teams.
“However, the personnel effort varies and can be shared across a larger group of individuals by engaging clinical builders and other participants beyond operational informatics teams.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationSome organizations have successfully implemented these tools and reduced the number of alerts and overrides.
“Although some organizations have successfully implemented these tools and reduced the number of alerts and overrides,38 other reports have found that many organizations are not following these recommendations, in part due to a lack of consensus for whether or not to turn off or modify, or how to modify, alerts.27,28,39”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [38] Simpao, 2015, Optimization of drug-drug interaction alert rules in a pediatric hospital’s electronic health record system using a visual analytics dashboard, J Am Med Inform Assoc source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationCDS alerts are included in all major commercial electronic health records (EHRs).
“CDS alerts are included in all major commercial electronic health records (EHRs); a large 2015 survey found that 95% of respondents had implemented drug–allergy, drug–drug, or drug–laboratory interaction alerts.8”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [8] Wright, 2009, Clinical decision support capabilities of commercially-available clinical information systems, J Am Med Inform Assoc source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationCommercial products exist to provide tools for evaluating CDS alerts.
“Commercial products also exist to provide tools for evaluating CDS alerts.35–37”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [35]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [36]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [37]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
InterpretationreasoningIn addition to direct improvements in CDS, Clickbusters increased user engagement and involvement in CDS.
“In addition to the direct improvements in CDS, the initiative also increased user engagement and involvement in CDS.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationA large 2015 survey found that 95% of respondents had implemented drug-allergy, drug-drug, or drug-laboratory interaction alerts.
“CDS alerts are included in all major commercial electronic health records (EHRs); a large 2015 survey found that 95% of respondents had implemented drug–allergy, drug–drug, or drug–laboratory interaction alerts.8”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [8] Wright, 2009, Clinical decision support capabilities of commercially-available clinical information systems, J Am Med Inform Assoc source
May combine two claims.
Study design and conductnone givenClickbusters participants followed standard processes for documenting changes and worked within existing governance frameworks.
“Clickbusters participants followed standard processes for documenting changes they made and worked within existing governance frameworks.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningCompared to other approaches, Clickbusters has moderate technical or build requirements and high personnel effort requirements.
“Compared to other approaches, Clickbusters has moderate technical or build requirements and high personnel effort requirements.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningChanges made by Clickbusters participants were generally well-designed and followed a standardized analysis process, so they could be reviewed efficiently and were generally approved quickly and without many required modifications.
“Because changes made by Clickbusters participants were generally well-designed and followed a standardized analysis process, they could be reviewed efficiently and were generally approved quickly and without many required modifications.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningThe Clickbusters program represents a novel approach to evaluating and optimizing CDS alerts.
“The Clickbusters program represents a novel approach to evaluating and optimizing CDS alerts.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenThe Clickbusters process could be readily replicated at other clinical sites and applied to other functions of the EHR, such as order sets, clinical documentation tools and information displays.
“The process could be readily replicated at other clinical sites and applied to other functions of the EHR, such as order sets, clinical documentation tools and information displays.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningIt may be possible to achieve some success with Clickbusters in settings without an active Physician Builder program, such as nonacademic medical centers, though the process may take more time and operational effort.
“It may still be possible to achieve some success with the Clickbusters program in settings where these groups are not present, such as nonacademic medical centers, though the process may take more time and operational effort.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationnone givenThe Clickbusters program brought more users into the process of evaluating and improving CDS and helped build a culture of continuous evaluation and improvement of clinical content in the EHR.
“The program also brought more users into the process of evaluating and improving CDS and helped build a culture of continuous evaluation and improvement of clinical content in the EHR.”
Quote in the paperSubstantial claim
The paper relies on this without support.
The existing Physician Builder program had not focused on optimizing existing CDS alerts.
“While they had developed documentation tools, order sets, reports, and CDS alerts, they did not focus on optimizing existing alerts.”
The paper relies on this without support.
The reduction in weekly alert firings observed after the Clickbusters program was attributable to the Clickbusters program rather than other changes.
“After 2 rounds, the Clickbusters program resulted in detailed, comprehensive reviews of 84 CDS alerts and reduced the number of weekly alert firings by more than 70 000 (15.43%).”
The paper relies on this without support.
Reducing alert firings and resulting clicks constitutes optimization of CDS alerts.
“At VUMC, the Clickbusters program was successful in optimizing CDS alerts by reducing alert firings and resulting clicks.”
The paper relies on this without support.
Participation in the Clickbusters program indicates increased user engagement and involvement in CDS.
“In addition to the direct improvements in CDS, the initiative also increased user engagement and involvement in CDS.”
The paper relies on this without support.
The reduction in firings among unmodified alerts (20%) reflects confounding changes that would have affected modified alerts to the same degree.
“we determined the reduction in firings for alerts not modified during Clickbusters after the round and multiplied the number of reduced clicks by that number (ie, unmodified alerts fired 20% less often after Clickbusters, so we determined the number of clicks actually busted to be 80% of the absolute difference in firings for modified alerts after Clickbusters).”
The paper relies on this without support.
Developing reports or dashboards for CDS alert review requires high effort, and reviewing their findings requires substantial personnel effort.
“However, efforts required to develop reports or dashboards can be high, and personnel effort required to review the findings are substantial.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous research has shown interruptive alert optimization by using strategies of eliciting end-user feedback, reducing alert volume, and reducing inappropriate firing of alerts, similar to the themes identified by participants in our study.
“Previous research has shown interruptive alert optimization by using strategies of eliciting end-user feedback, reducing alert volume, and reducing inappropriate firing of alerts, similar to the themes identified by participants in our study. 34 35 36 37”
From Qualitative Study of Inpatient Hospital Medicine Interruptive Alerts and Clinical Decision Support (2026) · cites this paper as [35]
Our findings assessing interruptive alert helpfulness and recommendations to change interruptive alert presentation, timing, and targeting were also like those found in previous studies.
“Our findings assessing interruptive alert helpfulness and recommendations to change interruptive alert presentation, timing, and targeting were also like those found in previous studies. 35 36 40”
From Qualitative Study of Inpatient Hospital Medicine Interruptive Alerts and Clinical Decision Support (2026) · cites this paper as [35]
Cites
Incidence of Adverse Drug Events and Potential Adverse Drug Events: Implications for Prevention
Bates, D. W. (1995) · JAMA · doi:10.1001/jama.1995.03530010043033 · Cited by 2,127 · see citing works
The “Meaningful Use” Regulation for Electronic Health Records
Blumenthal, D., & Tavenner, M. (2010) · New England Journal of Medicine · doi:10.1056/nejmp1006114 · Cited by 2,059 · see citing works
Effect of Clinical Decision-Support Systems: A Systematic Review
Bright, T. J., Wong, A., Dhurjati, R., Bristow, E., Bastian, L., Coeytaux, R. R., Samsa, G., Hasselblad, V., Williams, J. W., Musty, M. D., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-1-201207030-00450 · Cited by 1,200 · see citing works
Overriding of Drug Safety Alerts in Computerized Physician Order Entry
van der Sijs, H., Aarts, J., Vulto, A., & Berg, M. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1809 · Cited by 1,117 · see citing works
A primer on leading the improvement of systems: Fig 1
Berwick, D. M. (1996) · BMJ · doi:10.1136/bmj.312.7031.619 · Cited by 793 · see citing works
Medication-related clinical decision support in computerized provider order entry systems: a review
Kuperman, G. J., Bobb, A., Payne, T. H., Avery, A. J., Gandhi, T. K., Burns, G., Classen, D. C., & Bates, D. W. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2170 · Cited by 763 · see citing works
Middleton, B., Bloomrosen, M., Dente, M. A., Hashmat, B., Koppel, R., Overhage, J. M., Payne, T. H., Rosenbloom, S. T., Weaver, C., & Zhang, J. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001458 · Cited by 618 · see citing works
Grand challenges in clinical decision support
Sittig, D. F., Wright, A., Osheroff, J. A., Middleton, B., Teich, J. M., Ash, J. S., Campbell, E., & Bates, D. W. (2008) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2007.09.003 · Cited by 582 · see citing works
Self-citationImproving Outcomes with Clinical Decision Support: An Implementer's Guide, Second Edition
Osheroff, J. (2012) · doi:10.4324/9781498757461 · Cited by 517 · see citing works
Physicians' Decisions to Override Computerized Drug Alerts in Primary Care
Weingart, S. N., Toth, M., Sands, D. Z., Aronson, M. D., Davis, R. B., & Phillips, R. S. (2003) · Archives of Internal Medicine · doi:10.1001/archinte.163.21.2625 · Cited by 487 · see citing works
Improving Acceptance of Computerized Prescribing Alerts in Ambulatory Care
Shah, N. R., Seger, A. C., Seger, D. L., Fiskio, J. M., Kuperman, G. J., Blumenfeld, B., Recklet, E. G., Bates, D. W., & Gandhi, T. K. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1868 · Cited by 353 · see citing works
Overrides of Medication Alerts in Ambulatory Care
Isaac, T., Weissman, J. S., Davis, R. B., Massagli, M., Cyrulik, A., Sands, D. Z., & Weingart, S. N. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2008.551 · Cited by 309 · see citing works
Study Designs for PDSA Quality Improvement Research
Speroff, T., & O’Connor, G. T. (2004) · Quality Management in Health Care · doi:10.1097/00019514-200401000-00002 · Cited by 252 · see citing works
Return on Investment for a Computerized Physician Order Entry System
Kaushal, R., Jha, A. K., Franz, C., GLASER, J. P., Shetty, K. D., Jaggi, T., Blackford Middleton, A., Kuperman, G. J., Khorasani, R., Tanasijevic, M. J., et al. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1984 · Cited by 248 · see citing works
Phansalkar, S., van der Sijs, H., Tucker, A. D., Desai, A. A., Bell, D. S., Teich, J. M., Middleton, B., & Bates, D. W. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001089 · Cited by 239 · see citing works
A primer on PDSA: executing plan-do-study-act cycles in practice, not just in name
Leis, J. A., & Shojania, K. G. (2017) · BMJ Quality & Safety · doi:10.1136/bmjqs-2016-006245 · Cited by 235 · see citing works
Hsieh, T. C., Kuperman, G. J., Jaggi, T., Hojnowski-Diaz, P., Fiskio, J., Williams, D. H., Bates, D. W., & Gandhi, T. K. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1556 · Cited by 232 · see citing works
Strom, B. L., Schinnar, R., Aberra, F., Bilker, W., Hennessy, S., Leonard, C. E., & Pifer, E. (2010) · Archives of Internal Medicine · doi:10.1001/archinternmed.2010.324 · Cited by 217 · see citing works
Turning Off Frequently Overridden Drug Alerts: Limited Opportunities for Doing It Safely
van der Sijs, H., Aarts, J., van Gelder, T., Berg, M., & Vulto, A. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2311 · Cited by 175 · see citing works
Clinical decision support capabilities of commercially-available clinical information systems
Wright, A., Sittig, D. F., Ash, J. S., Sharma, S., Pang, J. E., & Middleton, B. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3111 · Cited by 156 · see citing works
Self-citationA review of analytics and clinical informatics in health care
Simpao, A. F., Ahumada, L. M., Gálvez, J. A., & Rehman, M. A. (2014) · Journal of Medical Systems · doi:10.1007/s10916-014-0045-x · Cited by 154 · see citing works
Wright, A. T., Sittig, D. F., Ash, J. S., Feblowitz, J. C., Meltzer, S. N., McMullen, C. K., Guappone, K. P., Carpenter, J., Richardson, J. E., Simonaitis, L., et al. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000113 · Cited by 147 · see citing works
Self-citationHigh-priority drug–drug interactions for use in electronic health records
Phansalkar, S., Desai, A. A., Bell, D., Yoshida, E., Doole, J., Czochanski, M., Middleton, B., & Bates, D. W. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000612 · Cited by 134 · see citing works
Analysis of clinical decision support system malfunctions: a case series and survey
Wright, A., Hickman, T.-T. T., McEvoy, D., Aaron, S., Ai, A., Andersen, J. M., Hussain, S., Ramoni, R., Fiskio, J., Sittig, D. F., et al. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw005 · Cited by 132 · see citing works
Self-citationDrug safety alert generation and overriding in a large Dutch university medical centre
van der Sijs, H., Mulder, A., van Gelder, T., Aarts, J., Berg, M., & Vulto, A. (2009) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.1800 · Cited by 119 · see citing works
(2014) · PubMed Central · Cited by 117 · see citing works
Self-citationGurwitz, J. H., Field, T. S., Rochon, P., Judge, J., Harrold, L. R., Bell, C. M., Lee, M., White, K., LaPrino, J., Erramuspe‐Mainard, J., et al. (2008) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.2008.02004.x · Cited by 106 · see citing works
Ash, J. S., Sittig, D. F., Guappone, K. P., Dykstra, R. H., Richardson, J., Wright, A., Carpenter, J., McMullen, C., Shapiro, M., Bunce, A., et al. (2012) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-12-6 · Cited by 103 · see citing works
Self-citationPractitioners’ views on computerized drug-drug interaction alerts in the VA system
Ko, Y., Abarca, J., Malone, D. C., Dare, D. C., Geraets, D., Houranieh, A., Jones, W. N., Nichol, W. P., Schepers, G. P., & Wilhardt, M. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2224 · Cited by 102 · see citing works
The state of the art in clinical knowledge management: an inventory of tools and techniques
Sittig, D. F., Wright, A., Simonaitis, L., Carpenter, J. D., Allen, G. O., Doebbeling, B. N., Sirajuddin, A. M., Ash, J. S., & Middleton, B. (2010) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2009.09.003 · Cited by 101 · see citing works
Self-citationSimpao, A. F., Ahumada, L. M., Desai, B. R., Bonafide, C. P., Gálvez, J. A., Rehman, M. A., Jawad, A. F., Palma, K. L., & Shelov, E. D. (2015) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002538 · Cited by 94 · see citing works
(2007) · PubMed · Cited by 93 · see citing works
Wright, A., Sittig, D. F., Ash, J. S., Bates, D. W., Feblowitz, J., Fraser, G., Maviglia, S. M., McMullen, C., Nichol, W. P., Pang, J. E., et al. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.002030 · Cited by 92 · see citing works
Self-citationWhat, if all alerts were specific - estimating the potential impact on drug interaction alert burden
Seidling, H. M., Klein, U., Schaier, M., Czock, D., Theile, D., Pruszydlo, M. G., Kaltschmidt, J., Mikus, G., & Haefeli, W. E. (2014) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2013.12.006 · Cited by 91 · see citing works
McEvoy, D. S., Sittig, D. F., Hickman, T.-T., Aaron, S., Ai, A., Amato, M., Bauer, D. W., Fraser, G. M., Harper, J., Kennemer, A., et al. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw114 · Cited by 75 · see citing works
Self-citationWaitman, L. R., Phillips, I. E., McCoy, A. B., Danciu, I., Halpenny, R. M., Nelsen, C. L., Johnson, D. C., Starmer, J. M., & Peterson, J. F. (2011) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(11)37041-9 · Cited by 73 · see citing works
Self-citationFrequency of inappropriate medical exceptions to quality measures
Persell, S. D., Dolan, N. C., Friesema, E. M., Thompson, J. A., Kaiser, D., & Baker, D. W. (2010) · Annals of Internal Medicine · doi:10.7326/0003-4819-152-4-201002160-00007 · Cited by 66 · see citing works
Physicians’ reasons for failing to comply with computerized preventive care guidelines
Litzelman, D. K., & Tierney, W. M. (1996) · Journal of General Internal Medicine · doi:10.1007/bf02599049 · Cited by 56 · see citing works
Cox, Z. L., Neal, E. B., Waitman, L. R., Peterson, N. B., Bhave, G., Siew, E. D., Danciu, I., Lewis, J. B., Peterson, J. F., & McCoy, A. B. (2012) · Applied Clinical Informatics · doi:10.4338/aci-2012-03-ra-0009 · Cited by 53 · see citing works
Self-citationStrom, B. L., Schinnar, R., Bilker, W., Hennessy, S., Leonard, C. E., & Pifer, E. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.000695 · Cited by 51 · see citing works
Tamblyn, R., Reidel, K., Huang, A., Taylor, L., Winslade, N., Bartlett, G., Grad, R., Jacques, A., Dawes, M., Larochelle, P., et al. (2010) · Medical Decision Making · doi:10.1177/0272989x09342752 · Cited by 44 · see citing works
Using statistical anomaly detection models to find clinical decision support malfunctions
Ray, S., McEvoy, D. S., Aaron, S., Hickman, T.-T., & Wright, A. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy041 · Cited by 44 · see citing works
Self-citationAsh, J. S., McCormack, J. L., Sittig, D. F., Wright, A., McMullen, C., & Bates, D. W. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000705 · Cited by 41 · see citing works
Self-citationCranky comments: detecting clinical decision support malfunctions through free-text override reasons
Aaron, S., McEvoy, D. S., Ray, S., Hickman, T.-T. T., & Wright, A. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy139 · Cited by 41 · see citing works
Self-citation(2010) · PubMed · Cited by 36 · see citing works
Self-citationWright, A. T., Ash, J. S., Aaron, S., Ai, A., Hickman, T.-T. T., Wiesen, J. F., Galanter, W., McCoy, A. B., Schreiber, R., Longhurst, C., et al. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.08.001 · Cited by 31 · see citing works
Self-citationSwiderski, S. M., Pedersen, C. A., Schneider, P. J., & Miller, A. S. (2007) · Journal of Patient Safety · doi:10.1097/01.jps.0000242991.86165.80 · Cited by 11 · see citing works
(2007) · PubMed · Cited by 9 · see citing works
(2008) · PubMed · Cited by 9 · see citing works
Turer, R. W., Arribas, M. A., Balgord, S., Brooks, S., Hopson, L. R., Bassin, B. S., & Medlin, R. P. (2020) · AEM Education and Training · doi:10.1002/aet2.10518 · Cited by 9 · see citing works
Self-citationIdentifying Best Practices for Clinical Decision Support and Knowledge Management in the Field
Ash Joan, S., Sittig Dean, F., Richard, D., Adam, W., Carmit, M., Joshua, R., & Blackford, M. (2010) · Studies in Health Technology and Informatics · doi:10.3233/978-1-60750-588-4-806
Flagged first, then by citations
Cited by
Sorted by citations
Using AI-generated suggestions from ChatGPT to optimize clinical decision support
Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad072 · Cited by 397 · see citing works
Self-citationAssessing the Value of ChatGPT for Clinical Decision Support Optimization
Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A. (2023) · doi:10.1101/2023.02.21.23286254 · Cited by 90 · see citing works
Self-citationDocumentation dynamics: note composition, burden, and physician efficiency
Apathy, N. C., Rotenstein, L., Bates, D. W., & Holmgren, A. J. (2023) · Health Services Research · doi:10.1111/1475-6773.14097 · Cited by 70 · see citing works
You, J. G., Hernandez-Boussard, T., Pfeffer, M. A., Landman, A., & Mishuris, R. G. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01506-4 · Cited by 59 · see citing works
Electronic health record usability, satisfaction, and burnout for family physicians.
Holmgren, A. J., Hendrix, N., Maisel, N., Everson, J., Bazemore, A., Rotenstein, L., Phillips, R. L., & Adler-Milstein, J. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.26956 · Cited by 56 · see citing works
Leveraging explainable artificial intelligence to optimize clinical decision support
Liu, S., McCoy, A. B., Peterson, J. F., Lasko, T. A., Sittig, D. F., Nelson, S. D., Andrews, J. C., Patterson, L., Cobb, C., Mulherin, D., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae019 · Cited by 44 · see citing works
Self-citationLiu, S., McCoy, A. B., Wright, A. P., Nelson, S. D., Huang, S. S., Ahmad, H. B., Carro, S., Franklin, J., Brogan, J., & Wright, A. T. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae041 · Cited by 35 · see citing works
Self-citationLevy, D. R., Withall, J. B., Mishuris, R. G., Tiase, V. L., Diamond, C., Douthit, B. J., Grabowska, M. E., Lee, R. Y., Moy, A. J., Sengstack, P. R., et al. (2024) · Applied Clinical Informatics · doi:10.1055/a-2385-1654 · Cited by 36 · see citing works
Self-citationFallon, A., Haralambides, K., Mazzillo, J., & Gleber, C. (2024) · Applied Clinical Informatics · doi:10.1055/a-2226-8144 · Cited by 22 · see citing works
The role of organizations in shaping physician use of electronic health records
Cross, D. A., Holmgren, A. J., & Apathy, N. C. (2023) · Health Services Research · doi:10.1111/1475-6773.14203 · Cited by 21 · see citing works
Reese, T. J., Liu, S., Steitz, B. D., McCoy, A. B., Russo, E. M., Koh, B., Ancker, J. S., & Wright, A. T. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac089 · Cited by 15 · see citing works
Self-citationClinical decision support systems in community pharmacies: a scoping review
Moon, J., Chladek, J. S., Wilson, P., & Chui, M. A. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad208 · Cited by 14 · see citing works
Rajamani, G., Diethelm, M., Gunderson, M. A., Talluri, V. S. M., Motz, P. A., Steinhaus, J. M., LaFlamme, A., Jarabek, B., Christiaansen, T., Blade, J. T., et al. (2023) · Applied Clinical Informatics · doi:10.1055/s-0043-1767684 · Cited by 7 · see citing works
A Multiyear Survey Evaluating Clinician Electronic Health Record Satisfaction
Garabedian, P. M., Rui, A., Volk, L. A., Neville, B. A., Lipsitz, S. R., Healey, M. J., & Bates, D. W. (2023) · Applied Clinical Informatics · doi:10.1055/s-0043-1770900 · Cited by 6 · see citing works
Ledger, T. S., Brooke-Cowden, K., & Coiera, E. W. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad193 · Cited by 6 · see citing works
Reimagining clinical AI: from clickstreams to clinical insights with EHR use metadata
Yan, C., Zhang, X., Kannampallil, T., Adler‐Milstein, J., & Chen, Y. (2025) · npj Health Systems · doi:10.1038/s44401-025-00040-5 · Cited by 5 · see citing works
Iscoe, M. S., Hooper, C., Levy, D. R., Buchanan, L., Dziura, J. D., Meeker, D., Taylor, R. A., D’Onofrio, G., Oladele, C., Sarpong, D. F., et al. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-098072 · Cited by 5 · see citing works
A Discount Approach to Reducing Nursing Alert Burden
Thompson, S., Kandaswamy, S., & Orenstein, E. W. (2024) · Applied Clinical Informatics · doi:10.1055/a-2345-6475 · Cited by 5 · see citing works
Advances in Clinical Decision Support Systems: Contributions from the 2022 Literature
Lehmann, C. U., Subbian, V., & Systems, S. E. F. T. I. Y. S. O. D. S. (2023) · Yearbook of Medical Informatics · doi:10.1055/s-0043-1768751 · Cited by 4 · see citing works
Resident clinical dashboards to support precision education in emergency medicine
Moser, J. S., Genes, N. G., Hekman, D. J., Krzyzaniak, S. M., Layng, T. A., Miller, D. T., Rider, A. C., Sagalowsky, S. T., Smith, M. E., & Schnapp, B. H. (2025) · AEM Education and Training · doi:10.1002/aet2.70020 · Cited by 3 · see citing works
Right Idea, Wrong Time: Focusing on Alert Timing for Effective Decision Support
Wilson, A. E., Bain, A. P., Webb, J. W., Lehmann, C. U., Moran, B., Shah, N., & O’Connell, E. J. (2025) · Applied Clinical Informatics · doi:10.1055/a-2605-4510 · Cited by 3 · see citing works
Blatt, M. I., Rupp, J. D., Lipton, M., Barrett, T. W., Boyd, J. S., & Ward, M. J. (2024) · Cureus · doi:10.7759/cureus.61675 · Cited by 2 · see citing works
Ravi, A., Weia, B., Sakumoto, M., Oates, A., & Liu, X. (2023) · ACI Open · doi:10.1055/s-0043-1775971 · Cited by 1 · see citing works
Sheehan, K. N., Cioci, A., Lucioni, T., & Hernandez, S. (2024) · Applied Clinical Informatics · doi:10.1055/s-0044-1786682 · Cited by 1 · see citing works
Vanderbilt Clinical Informatics Center Education Strategy: To Infinity and Beyond!
Russo, E. M., McCoy, A. B., Mize, D. L. E., Osterman, T. J., Nelson, S., Wanderer, J. P., & Wright, A. T. (2024) · Applied Clinical Informatics · doi:10.1055/a-2443-8318 · Cited by 1 · see citing works
Self-citationBlack, K. C., Snyder, N. A., Zhou, M., Zhu, Z., Uptegraft, C. C., Chintalapani, A., & Orwoll, B. (2024) · Applied Clinical Informatics · doi:10.1055/a-2250-6305 · Cited by 1 · see citing works
Hussain, S., Lehmann, H. P., Buresh, M. E., Niessen, T. M., Fingerhood, M., Ahmed, N., Cavallio, K., Maslen, A., & Knight, A. M. (2025) · Applied Clinical Informatics · doi:10.1055/a-2508-7086 · Cited by 1 · see citing works
“I worry we'll blow right by it:” Barriers to Uptake of the STRATIFY-CDS for Acute Heart Failure
Christensen, M. A., Reale, C., Anders, S. H., Coffman, T., Alaw, H., Mathe, J. L., Albert, D., Sachs, A. L., McCoy, A. B., Liu, D., et al. (2025) · Applied Clinical Informatics · doi:10.1055/a-2594-3770 · Cited by 1 · see citing works
Self-citationQualitative Study of Inpatient Hospital Medicine Interruptive Alerts and Clinical Decision Support
Khan, A., & Schnipper, J. (2026) · Applied Clinical Informatics · doi:10.1055/a-2955-9555
Ferro, D. F., Tobias, M., Carr, L. H., Wentz, P., Rodriguez, M., Pitts, C., Kane, E., & Shelov, E. (2026) · Applied Clinical Informatics · doi:10.1055/a-2793-0977
McCoy, A. B., Russo, E. M., & Wright, A. T. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad150
Self-citationMusser, R. C., Senior, R., Havrilesky, L. J., Buuck, J., Casarett, D., Ibrahim, S., & Davidson, B. A. (2024) · Applied Clinical Informatics · doi:10.1055/a-2247-9355
Sova, C., Poon, E. G., Musser, R. C., & Chowdhury, A. (2024) · Applied Clinical Informatics · doi:10.1055/s-0044-1787648
Sharma, S. (2024) · Journal of Informatics Education and Research · doi:10.52783/jier.v4i3.1672
Electronic Health Record Decision Aids for Prescribing—If You Build It, Will They Come?
Spann, A., & Reese, T. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.19044
Self-citationA Measurement Science Framework to Optimize CDS for Opioid Use Disorder Treatment in the ED
Iscoe, M. S., Hooper, C. D., Levy, D. R., Lutz, J., Paek, H., Rose, C. C., Kannampallil, T., Meeker, D., Dziura, J. D., & Melnick, E. R. (2025) · Applied Clinical Informatics · doi:10.1055/a-2595-0317
Christensen, M. A., Beebe, R., Chen, A., Reale, C., Faiman, J., Wrenn, J. O., Gradwohl, S. C., Slagle, J. M., Weitkamp, A. O., Nance, B., et al. (2026) · Applied Clinical Informatics · doi:10.1055/a-2808-9190
Designing Prescription Verification Systems That Know When the Pharmacist Must Take Over
Martínez, S., Gómez, C., & Navarro, L. (2025) · Archives of Pharmacy Practice · doi:10.51847/acwjvheaem
Denami, F., Ammendolia, A., Alessandro de, S. I. R. E., Marotta, N., Benedetto, G. L., Parrotta, E. I., Cuda, G., Gasparini, G., & Mercurio, M. (2026) · Bioengineering · doi:10.3390/bioengineering13080942
Rajput, S., & Mehta, R. B. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9109-0.ch008
Silva, J., Costa, P., & Beatriz, A. (2026) · Annals of Pharmacy Practice and Pharmacotherapy · doi:10.51847/5cptfj2hqs
- Authors
- McCoy, A. B., Russo, E. M., Johnson, K. B., Addison, B., Patel, N., Wanderer, J. P., Mize, D. E., Jackson, J. G., Reese, T. J., Littlejohn, S., Patterson, L., French, T., Preston, D., Rosenbury, A., Valdez, C., Nelson, S. D., Aher, C. V., Alrifai, M. W., Andrews, J., Cobb, C., Horst, S. N., Johnson, D. P., Knake, L. A., Lewis, A. A., Parks, L., Parr, S. K., Patel, P., Patterson, B. L., Smith, C. M., Suszter, K. D., Turer, R. W., Wilcox, L. J., Wright, A. P., & Wright, A.
- Affiliations
- Vanderbilt University Medical Center (US)
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
- Vanderbilt University (US)
- Union University (US)
- Corresponding author
- McCoy, Allison B
- ORCID
- McCoy, Allison B
- Russo, Elise M
- Johnson, Kevin B
- Patel, Neal
- Wanderer, Jonathan P
- Reese, Thomas J
- Nelson, Scott D
- Alrifai, Mhd Wael
- Andrews, Jennifer
- Horst, Sara N
- Johnson, David P
- Knake, Lindsey A
- Lewis, Adam A
- Parr, Sharidan K
- Patterson, Barron L
- Smith, Christine M
- Turer, Robert W
- Wilcox, Lyndy J
- Wright, Aileen P
- Wright, Adam
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2022-03-04
- Volume
- 29
- Issue
- 6
- Pages
- 1050-1059
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC9093034
- OpenAlex
- W4213054155
- Cited by
- 41
- References
- 51
- Citations by year
- 2026: 6 · 2025: 10 · 2024: 12 · 2023: 11 · 2022: 2
- FWCI
- 6.55
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- Vanderbilt University Medical Center
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- clinical decision support, alarm fatigue, clinician engagement, gamification, clinician burnout, patient safety, quality improvement, electronic health records
- MeSH terms
- Humans, Decision Support Systems, Clinical, Medical Order Entry Systems, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-02.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-02.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
