Qualitative Study of Inpatient Hospital Medicine Interruptive Alerts and Clinical Decision Support

Probably not a substantial claim: may not pass S3 (load-bearing).

Interviews were stopped when reviewers reached thematic sufficiency.

“Interviews were stopped when reviewers reached thematic sufficiency.”

• cites [28] K A LaDonna, 2021, Beyond the guise of saturation: rigor and qualitative interview data, J Grad Med Educ sourcePartly supported

  The sections “Shifting From Saturation to Sufficiency” and “Evaluating the Sufficiency of Qualitative Findings” discuss assessing whether interview data are sufficient. This editorial does not report that interviews were stopped when reviewers reached thematic sufficiency; it offers guidance for evaluating qualitative findings, not an account of that decision. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Effective governance of CDS has been demonstrated in other institutions, resulting in the transition of alert and reminder content to other more appropriate areas in the EHR, downgrading the severity of alerts, and assessing the effect of CDS on clinician behavior and patient care.

“Effective governance of CDS has been demonstrated in other institutions, resulting in the transition of alert and reminder content to other more appropriate areas in the EHR, downgrading the severity of alerts (often leading to them no longer firing), 38 39 and assessing the effect of CDS on clinician behavior and patient care. 40 41”

• cites [38] B A Van, 2021, Optimizing clinical decision support alerts in electronic medical records: a systematic review of reported strategies adopted by hospitals, J Am Med Inform Assoc sourcePartly supported

  Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites [39] A M Knight, 2019, The effect of eliminating intermediate severity drug-drug interaction alerts on overall medication alert burden and acceptance rate, Appl Clin Inform sourcePartly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites [40] A Wright, 2011, Governance for clinical decision support: case studies and recommended practices from leading institutions, J Am Med Inform Assoc sourcePartly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

At a large academic healthcare system, interruptive alerts fired 53,106 times per month in the inpatient setting, representing 73.6% of all alerts fired in the system and equivalent to approximately 71 hours of total provider effort per month.

“Another study showed that at a large academic healthcare system, interruptive alerts fired 53,106 times per month in the inpatient setting, representing 73.6% of all alerts fired in the system and was equivalent to approximately 71 hours of total provider effort per month.”

• cites [14] P Elias, 2019, Evaluating the impact of interruptive alerts within a health system: use, response time, and cumulative time burden, Appl Clin Inform sourcePartly supported

  The Results report that 73.6% of alerts occurred during inpatient encounters. Table 1 gives 53,103 inpatient alerts per month, close to the claimed 53,106, but reports 3,524 minutes of inpatient provider time per month—about 59 hours, not 71. (LLM, conf=0.90)

  Experimental — accuracy not yet measured

May combine two claims.

The inherent design of our site's EHR makes interruptive alerts one of the easiest and least expensive forms of CDS to implement, even though they are known to be disruptive and often ineffective.

“The inherent design of our site’s EHR makes interruptive alerts one of the easiest and least expensive forms of CDS to implement, even though they are known to be disruptive and often ineffective. 5 15 16”

• cites [5] H Edrees, 2020, High-priority drug-drug interaction clinical decision support overrides in a newly implemented commercial computerized provider order-entry system: override appropriateness and adverse drug events, J Am Med Inform Assoc sourcePartly supported

  The Introduction describes alert fatigue, and the Discussion reports that providers overrode nearly all high-priority drug–drug interaction alerts; these passages support the concern that alerts can be disruptive and ineffective. The paper does not say that interruptive alerts are among the easiest or least expensive forms of CDS to implement, or establish that claim about the citing authors’ EHR. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [15] A D Bryant, 2014, Drug interaction alert override rates in the meaningful use era: no evidence of progress, Appl Clin Inform sourcePartly supported

  The Introduction (page unavailable) says order-checking alerts interrupt workflow and are frequently overridden; the Discussion (page unavailable) questions whether drug-drug alerts provide value. The paper does not say that the citing site's EHR design makes interruptive alerts particularly easy or inexpensive to implement. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [16] H van der, 2006, Overriding of drug safety alerts in computerized physician order entry, J Am Med Inform Assoc sourcePartly supported

  The Results (page not available) describe unnecessary workflow interruptions and alert fatigue that can cause important alerts to be ignored, supporting the claim that alerts can be disruptive and ineffective. The paper does not assess the citing site's EHR or say that interruptive alerts are among the easiest or least expensive CDS tools to implement. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

The initial codes for qualitative domains and themes were created by both authors using conceptual frameworks from Osheroff's Five Rights of Clinical Decision Support and broader principles of human-centered design.

“The initial codes for qualitative domains and themes were created by both authors using conceptual frameworks from Osheroff’s Five Rights of Clinical Decision Support as well as broader principles of human-centered design.”

• cites [27] J A Osheroff, 2007, A roadmap for national action on clinical decision support, J Am Med Inform Assoc sourcePartly supported

  The Executive Summary (page not specified) describes CDS as delivering person-specific information to clinicians, staff, or patients at appropriate times, which overlaps with elements of the cited framework. Our copy does not name or set out Osheroff’s Five Rights. It also cannot establish how the citing authors created their qualitative codes; the broader human-centered design principles are not clearly attributed to this source. (LLM, conf=0.79)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Previous research by Paterno et al showed that by tiering drug-drug interaction alerts in the EHR, acceptance rates of severe and moderate drug-drug interaction alerts increased.

“Previous research by Paterno et al showed that by tiering drug–drug interaction alerts in the EHR, acceptance rates of severe and moderate drug–drug interaction alerts increased. 30”

• cites Paterno et alNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [30] M D Paterno, 2009, Tiering drug-drug interaction alerts by severity increases compliance rates, J Am Med Inform Assoc sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Critical care patients generate approximately 100 EHR alerts for end-users per day.

“One study showed that critical care patients generate approximately 100 EHR alerts for end-users per day.”

• cites [13] V Kizzier-Carnahan, 2019, Frequency of passive EHR alerts in the ICU: another form of alert fatigue?, J Patient Saf sourcePartly supported

  The Results (page not specified) report a mean of 64.1 passive EHR alerts generated per ICU patient per day, rising to 165.3 when repeated appearances during chart review are counted. The study supports substantial daily alert exposure, but neither measure is approximately 100 alerts, and its findings concern passive alerts rather than all EHR alerts. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

Interruptive alerts are one of the dominant forms of CDS and have the potential to distract from patient care and increase clinician's cognitive load.

“Interruptive alerts, or pop ups that require end-user action or acknowledgment, are one of the dominant forms of CDS and have the potential to distract from patient care and increase clinician’s cognitive load.”

• cites [5] H Edrees, 2020, High-priority drug-drug interaction clinical decision support overrides in a newly implemented commercial computerized provider order-entry system: override appropriateness and adverse drug events, J Am Med Inform Assoc sourcePartly supported

  The Introduction (page unavailable) says that too many CDS alerts can cause alert fatigue, leading providers to ignore or override them; the Discussion likewise calls for reducing alert burden. These passages support a potential burden on clinicians, but do not establish that interruptive pop-ups are a dominant form of CDS or specifically that they distract from patient care and increase cognitive load. (LLM, conf=0.79)

  Experimental — accuracy not yet measured
• cites [6] S N Shah, 2021, Renal medication-related clinical decision support (CDS) alerts and overrides in the inpatient setting following implementation of a commercial electronic health record: implications for designing more effective alerts, J Am Med Inform Assoc sourcePartly supported

  The Introduction and Discussion (page unavailable) describe alert fatigue from excessive or inappropriate CDS alerts and say clinicians may become desensitized to important alerts. They do not establish that interruptive pop-ups are a dominant form of CDS or specifically say they distract from patient care and increase cognitive load. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites [7] I Jankovic, 2020, Clinical decision support and implications for the clinician burnout crisis, Yearb Med Inform sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Bell et al demonstrated optimizing recipient targeting, timing, and alert design may minimize alert overrides in the inpatient cardiology setting.

“Bell et al demonstrated optimizing recipient targeting, timing, and alert design may minimize alert overrides in the inpatient cardiology setting.”

• cites [24] H Bell, 2019, Mixed methods study of medication-related decision support alerts experienced during electronic prescribing for inpatients at an English hospital, Eur J Hosp Pharm sourceSupported

May combine two claims.

Alerts fire so frequently that most are ignored, with prior research demonstrating rates of ignoring as high as 96%.

“Optimal alert volume is unknown, but alerts fire so frequently that most are ignored, with prior research demonstrating rates of ignoring as high as 96%, almost entirely defeating their purpose.”

• cites [5] H Edrees, 2020, High-priority drug-drug interaction clinical decision support overrides in a newly implemented commercial computerized provider order-entry system: override appropriateness and adverse drug events, J Am Med Inform Assoc sourceSupported
• cites [15] A D Bryant, 2014, Drug interaction alert override rates in the meaningful use era: no evidence of progress, Appl Clin Inform sourcePartly supported

  The Results report that most alerts were overridden: 93% overall and 95.1% of drug-drug alerts. But the paper does not report a prior rate of 96%, and its Discussion says the low number of alerts per physician and lack of correlation between alert quantity and override rate argue against frequent firing as the reason alerts were dismissed. (LLM, conf=0.87)

  Experimental — accuracy not yet measured
• cites [16] H van der, 2006, Overriding of drug safety alerts in computerized physician order entry, J Am Med Inform Assoc sourceSupported

May combine two claims.

Interruptive alerts are a commonly used form of CDS and previous research has shown negative unintended consequences for end-users.

“Although it is difficult to quantify interruptive alert burden, they are a commonly used form of CDS, and previous research has shown negative unintended consequences for end-users.”

• cites [1] D M Rind, 1994, Effect of computer-based alerts on the treatment and outcomes of hospitalized patients, Arch Intern Med sourceNot checked — no full text of the source
• cites [11] S N Weingart, 2009, An empirical model to estimate the potential impact of medication safety alerts on patient safety, health care utilization, and cost in ambulatory care, Arch Intern Med sourceNot checked — no full text of the source
• cites [12] E M Powers, 2018, Efficacy and unintended consequences of hard-stop alerts in electronic health record systems: a systematic review, J Am Med Inform Assoc sourceSupported

May combine two claims.

Alert creation and maintenance can be quite costly; one study found the cost of creating 94 new drug alerts to be $23,694 to 48,668, or approximately $252 to 518 per alert, with an additional annual maintenance cost of 10-20% of the creation cost.

“Finally, alert creation and maintenance can be quite costly. One study found the cost of creating 94 new drug alerts to be $23,694 to 48,668 (United States dollars), or approximately $252 to 518 per alert, with an additional annual maintenance cost of 10–20% of the creation cost.”

• cites [22] T S Field, 2008, Costs associated with developing and implementing a computerized clinical decision support system for medication dosing for patients with renal insufficiency in the long-term care setting, J Am Med Inform Assoc sourcePartly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured

May combine two claims.

The existing alert development team involves end-users potentially affected by the proposed CDS, which promotes interdisciplinary collaboration with the information systems team.

“The existing alert development team involves end-users potentially affected by the proposed CDS, which promotes interdisciplinary collaboration with the information systems team.”

Probably not a substantial claim: may not pass S2 (checkable).

Involving end-users in interruptive alert analysis is a recommendation for CDS improvement.

“Involve end-users in interruptive alert analysis”

Classified as probably not a substantial claim.

A finding not mentioned in previous qualitative CDS research was the contrast between the definition of a successful interruptive alert by end-users and the broader information systems administration.

“A finding not mentioned in previous qualitative CDS research was the contrast between the definition of a successful interruptive alert by end-users and the broader information systems administration.”

Probably not a substantial claim: may not pass S2 (checkable).

The sensitivity of an interruptive alert ideally should be proportional to the criticality of the alert.

“The sensitivity of an interruptive alert, i.e., the percentage of alert firing among all cases in which it should fire, ideally should be proportional to the criticality of the alert (Theme 1).”

Classified as probably not a substantial claim.

Changing interruptiveness of alerts via flags, changing timing, batching, and alternate forms of CDS is a recommendation for CDS improvement.

“Changing interruptiveness of alerts via flags, changing timing, batching, and alternate forms of CDS”

Classified as probably not a substantial claim. May combine two claims.

Participants recommended changing alert design by automating recommended actions as much as possible, integrating artificial intelligence into alerts, and changing the alert recipient.

“They also recommended changing alert design by automating the recommended actions as much as possible, integrating artificial intelligence into alerts, and changing the alert recipient (Theme 3).”

Probably not a substantial claim: may not pass S2 (checkable).

The interruptiveness of the alert should be relative to end-user's workflow priorities.

“The third aspect is the interruptiveness of the alert relative to end-user’s workflow priorities (Theme 3).”

May combine two claims.

Systems governance wanted accurate CDS as a means for a systemwide response to internal safety events and external regulations, with little appreciation for how false-positive alerts could negatively affect clinicians and patients.

“Overall, systems governance wanted accurate CDS as a means for a systemwide response to internal safety events and external regulations, with little appreciation for how false-positive alerts could negatively affect clinicians and patients.”

May combine two claims.

Silent mode is used to gather information about alert firing rate and accuracy, which helps with decisions about whether the alerts should be implemented.

“Silent mode is used to gather information about alert firing rate and accuracy, which helps with decisions about whether the alerts should be implemented.”

May combine two claims.

Responding clinicians expressed a loss of trust in interruptive alerts due to overwhelming interruptive alert volume and high rates of false positives, resulting in responding clinicians overriding most if not all interruptive alerts regardless of helpfulness.

“Responding clinicians expressed a loss of trust in interruptive alerts due to overwhelming interruptive alert volume and high rates of false positives, resulting in responding clinicians overriding most if not all interruptive alerts regardless of helpfulness.”

May combine two claims.

When patient care team roles are incompletely listed in the EHR, it is difficult to design role- and user-specific CDS, leading to interruptive alerts firing for inappropriate end-users.

“Additionally, when patient care team roles are incompletely listed in the EHR, it is difficult to design role- and user-specific CDS, leading to interruptive alerts firing for inappropriate end-users.”

Classified as probably not a substantial claim.

The broader administration defined alert success as alerts functioning as designed and helping the hospital comply with regulations or react to safety events.

“The broader administration defined alert success as alerts functioning as designed and helping the hospital comply with regulations or react to safety events.”

Classified as probably not a substantial claim.

Interruptive alert success is not defined by acceptance rates, end-user assessment of helpfulness, effect on clinician workflow, effect on clinician's cognitive load, unintended consequences, or patient outcomes.

“Interruptive alert success is not defined by acceptance rates, end-user assessment of helpfulness, effect on clinician workflow, effect on clinician’s cognitive load, unintended consequences, or patient outcomes.”

May combine two claims.

Sepsis alerts were almost universally viewed as unhelpful, likely in part due to their high rate of false positives.

“For example, sepsis alerts were almost universally viewed as unhelpful, likely in part due to their high rate of false positives.”

Probably not a substantial claim: may not pass S2 (checkable).

The alert's interruptiveness should be proportional to the criticality of the alert.

“The alert’s interruptiveness should also be proportional to the criticality of the alert (Theme 2), with interruptiveness defined as the degree to which workflow of the end-user in the EHR is interrupted.”

Classified as probably not a substantial claim. May combine two claims.

We identified many recommendations for improving CDS, originating from front-line inpatient clinicians, many of which might improve the experience of end-users without compromising (or possibly improving) patient care.

“We also identified many recommendations for improving CDS, originating from front-line inpatient clinicians, many of which might improve the experience of end-users without compromising (or possibly improving) patient care.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

A clinically critical event should result in an interruptive "hard stop" for the end-user, while less critical events should have less interruptive or noninterruptive alerts.

“For example, a clinically critical event should result in an interruptive “hard stop” for the end-user, while less critical events should have less interruptive or noninterruptive alerts.”

May combine two claims.

Responding clinicians at our site expressed mixed feelings about interruptive alerts, similar to findings of previous studies.

“Overall, responding clinicians at our site expressed mixed feelings about interruptive alerts, similar to findings of previous studies. 11”

• cites [11] S N Weingart, 2009, An empirical model to estimate the potential impact of medication safety alerts on patient safety, health care utilization, and cost in ambulatory care, Arch Intern Med sourceNot checked — no full text of the source

May combine two claims.

Responding clinicians felt clinically critical alerts, such as sepsis, should have high sensitivity (i.e., low rate of false negatives), even if it resulted in false positives, but felt less critical alerts should have lower sensitivity.

“For example, responding clinicians felt clinically critical alerts, such as sepsis, should have high sensitivity (i.e., low rate of false negatives), even if it resulted in false positives, but felt less critical alerts should have lower sensitivity.”

Classified as probably not a substantial claim. May combine two claims.

Participants recommended changing alert interruptiveness with noninterruptive flags, changing the timing of alert delivery or batching noncritical alerts, and using alternate forms of CDS when possible.

“Participants mentioned changing alert interruptiveness with noninterruptive flags, changing the timing of alert delivery or batching noncritical alerts, and using alternate forms of CDS when possible (Theme 1; Table 4 ).”

Classified as probably not a substantial claim.

Participants recommended changing alerts to be responsive to the medical record, user feedback, and system-wide evaluation.

“Participants also recommended changing alerts to be responsive to the medical record, user feedback, and system-wide evaluation (Theme 4).”

May combine two claims.

The current system promotes interdisciplinary discussion regarding alert creation but adds considerable time and complexity to the process.

“While the current system promotes interdisciplinary discussion regarding alert creation, it also adds considerable time and complexity to the process.”

May combine two claims.

We found three cross-cutting domains related to barriers to successful CDS and one domain related to facilitators of CDS success.

“We also found three cross-cutting domains related to barriers and one domain related to facilitators of CDS success ( Table 3 ), as well as several recommendations for CDS improvement ( Table 4 ).”

Classified as probably not a substantial claim.

We gathered several recommendations for CDS improvement from all stakeholder groups, including changing alert "interruptiveness," customizability, design, responsiveness, and involving end-users in alert analysis.

“We found one governance-level facilitator and gathered several recommendations for CDS improvement from all stakeholder groups, including changing alert “interruptiveness,” customizability, design, responsiveness, and involving end-users in alert analysis.”

May combine two claims.

Documentation, e.g., of abnormal vital signs, is often completed retrospectively, making it difficult to design CDS that responds to clinical situations in real time.

“Documentation, e.g., of abnormal vital signs, is often completed retrospectively, making it difficult to design CDS that responds to clinical situations in real time.”

May combine two claims.

End-users wanted high sensitivity interruptive alerts only in highly critical situations, while systems governance wanted high sensitivity interruptive alerts under all circumstances, resulting in over-alerting.

“While end-users wanted high sensitivity interruptive alerts only in highly critical situations, systems governance wanted high sensitivity interruptive alerts under all circumstances, resulting in over-alerting.”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). May combine two claims. The quote only partly states this claim.

Changing interruptive alert design by automating actions, integrating artificial intelligence, and changing alert recipient is a recommendation for CDS improvement.

“Changing interruptive alert design: automating actions, integrating artificial intelligence, changing alert recipient”

Probably not a substantial claim: may not pass S3 (load-bearing).

Lacking a group solely designated to optimizing CDS is a governance-level barrier to successful CDS.

“Lacking a group solely designated to optimizing CDS”

May combine two claims.

The implications of this study are that end-users wanted CDS that was relevant, targeted, less interruptive, and customizable.

“The implications of this study are that end-users wanted clinical decision support (CDS) that was relevant, targeted, less interruptive, and customizable.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Silent mode is a governance-level facilitator to successful CDS.

“Silent mode”

Classified as probably not a substantial claim. May combine two claims.

Participants recommended creating customizable CDS, including customizable parameters to trigger interruptive alert firing and options to defer or turn off alerts depending on type or criticality.

“Another recommendation was to create customizable CDS (Theme 2). This could include customizable parameters to trigger interruptive alert firing (for specific patients or specific users) and providing options to defer or turn off alerts depending on their type or criticality.”

Probably not a substantial claim: may not pass S2 (checkable).

Making interruptive alerts more responsive to medical record, user feedback, and system-wide evaluation is a recommendation for CDS improvement.

“Making interruptive alerts more responsive to medical record, user feedback, and system-wide evaluation”

May combine two claims.

We found three key domains for governance groups, clinical leads, and information systems members: factors that contribute to high volume and high sensitivity of interruptive alerts, differing definitions of interruptive alert success from end-users, and reluctance to add new interruptive alerts but limited changes or retirement of existing interruptive alerts.

“We found three domains across the governance, clinical lead, and information system groups about the causes of high-volume and high-sensitivity interruptive alerts, differing definitions of interruptive alert success from end-users, and reluctance to add new interruptive alerts but limited changes or retirement of existing interruptive alerts.”

May combine two claims.

Responding clinicians rarely found that sepsis interruptive alerts were accurate or provided new information.

“Responding clinicians rarely found that sepsis interruptive alerts were accurate or provided new information.”

Classified as probably not a substantial claim. May combine two claims.

Their biggest concern is that an interruptive alert would not fire when it should and felt the risks of under-alerting outweighed the risks of over-alerting.

“Their biggest concern is that an interruptive alert would not fire when it should and felt the risks of under-alerting outweighed the risks of over-alerting.”

Classified as probably not a substantial claim. May combine two claims.

Proposed improvements to the CDS governance system were to streamline the CDS request process and involve end-users in the process of analyzing existing alerts.

“Proposed improvements to the CDS governance system were to streamline the CDS request process and involve end-users in the process of analyzing existing alerts (Theme 5).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Responding clinicians mentioned that interruptive alert sensitivity and interruptiveness should be proportional to alert criticality.

“Responding clinicians mentioned that interruptive alert sensitivity and interruptiveness should be proportional to alert criticality.”

May combine two claims.

The EHR presents limitations in using noninterruptive CDS because of multiple ways to do the same task; there is no way to guarantee the end-user would encounter a specific workflow where noninterruptive CDS could be integrated.

“The EHR also presents limitations in using noninterruptive CDS because of multiple ways to do the same task; there is no way to guarantee the end-user would encounter a specific workflow where noninterruptive CDS could be integrated (Theme 2).”

May combine two claims.

End-users defined alert success as changing clinical management, improving patient outcomes, and having high acceptance rates.

“In contrast, end-users defined alert success as changing clinical management, improving patient outcomes, and having high acceptance rates.”

May combine two claims.

Most current alerts at our site were acquired as a bundle from external vendors and are not tiered by criticality.

“Also, unlike custom CDS created during the pre-vendor EHR era, most current alerts at our site were acquired as a bundle from external vendors and are not tiered by criticality.”

Classified as probably not a substantial claim.

Another unique contribution our study presents is the explicit concept of assessing CDS on three dimensions of proportionality: alert sensitivity proportional to criticality, alert interruptiveness proportional to criticality, and alert interruptiveness relative to workflow priorities.

“Another unique contribution our study presents is the explicit concept of assessing CDS on three dimensions of proportionality: alert sensitivity proportional to criticality, alert interruptiveness proportional to criticality, and alert interruptiveness relative to workflow priorities.”

May combine two claims.

End-users wanted alerts that were customizable, less disruptive, and lower in volume.

“End-users wanted alerts that were customizable, less disruptive, and lower in volume.”

Classified as probably not a substantial claim. May combine two claims.

The broader information systems administrative system defines successful interruptive alerts as those that function as designed and provide accurate information to end-users.

“Information systems (IS) participants and clinical leads acknowledged that the broader information systems administrative system defines successful interruptive alerts as those that function as designed, i.e., using their internal logic, and provide accurate information to end-users (Theme 1).”

May combine two claims.

Excess false-positive interruptive alerts result in a loss of trust in interruptive alerts and ultimately end-users overriding most interruptive alerts, regardless of their appropriateness.

“Excess false-positive interruptive alerts (Theme 1) result in a loss of trust in interruptive alerts and ultimately end-users overriding most interruptive alerts, regardless of their appropriateness.”

Probably not a substantial claim: may not pass S1 (asserted).

Making interruptive alerts customizable by user or patient, with option to defer or turn off alert, is a recommendation for CDS improvement.

“Making interruptive alerts customizable by user or patient; option to defer or turn off alert”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The interview responses of the end-users, mostly residents and APPs, may have been affected by being interviewed by an attending physician.

“Finally, the interview responses of the end-users, mostly residents and APPs, may have been affected by being interviewed by an attending physician.”

May combine two claims.

The most significant barriers to successful CDS at our site were limited ways of delivering CDS in the current build of the EHR, limited institutional resources to customize CDS, and fragmentation and limitations of governance.

“The most significant barriers to successful CDS at our site were limited ways of delivering CDS in the current build of the EHR, limited institutional resources to customize CDS, and fragmentation and limitations of governance.”

• AssumptionRests on an assumption: CDS teams may have limited technical capabilities in creating CDS.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Interruptive alerts are often used as a form of CDS because they are easy to implement compared with many other forms of CDS, especially in modern vendor-dominant EHRs commonly used in the inpatient setting.

“Interruptive alerts are often used as a form of CDS because they are easy to implement compared with many other forms of CDS, especially in modern vendor-dominant EHRs commonly used in the inpatient setting.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Optimizing interruptive CDS alerts would specifically involve reducing false-positive alerts, reducing alert interruptiveness when appropriate, and involving end-users in alert development, maintenance, and retirement.

“Optimizing interruptive CDS alerts, or “pop-ups,” would specifically involve reducing false-positive alerts, reducing alert interruptiveness when appropriate, and involving end-users in alert development, maintenance, and retirement.”

May combine two claims.

Customizing alerts at the institutional level would be ideal but is too resource intensive to be a practical and generalizable solution.

“Customizing alerts at the institutional level would be ideal but is unfortunately too resource intensive to be a practical and generalizable solution.”

• AssumptionRests on an assumption: CDS teams may have limited technical capabilities in creating CDS.

May combine two claims.

Differences in priorities of systems governance and end-users as well as EHR limitations contribute to the dysfunctionality of interruptive alerts.

“Differences in priorities of systems governance and end-users as well as EHR limitations contribute to the dysfunctionality of interruptive alerts.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Interruptive alerts can also provide real-time recommendations, which would make them more useful than those that do not, but they still carry the risk of alert fatigue.

“We should note that interruptive alerts can also provide real-time recommendations, which would make them more useful than those that do not, but they still carry the risk of alert fatigue.”

Classified as probably not a substantial claim. May combine two claims.

The two reviewers iteratively revised codes from 22 interviews, independently coded data and met to resolve discrepancies, and created and refined themes and modified interpretations in an iterative fashion.

“The two reviewers, Drs. Khan and Schnipper, iteratively revised codes from 22 interviews, independently coded data and met to resolve discrepancies, and created and refined themes and modified interpretations in an iterative fashion.”

Probably not a substantial claim: may not pass S2 (checkable).

This points to sepsis alerts being a category ripe for improvement, e.g., by incorporating additional clinical information and clinician actions into the logic.

“This points to sepsis alerts being a category ripe for improvement, e.g., by incorporating additional clinical information and clinician actions into the logic.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Restraints in governance group capacity may limit further optimization of CDS.

“Additionally, restraints in governance group capacity may limit further optimization of CDS.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Each group involved in CDS governance typically only sees a piece of the health care process and lacks some of the perspectives of others in the organization.

“Each group involved in CDS governance typically only sees a piece of the health care process and lacks some of the perspectives of others in the organization.”

May combine two claims.

Most hospitals have limited resources devoted to CDS development and maintenance, and fewer have been able to employ clinical informatics trained clinician-users to provide ongoing feedback.

“Most hospitals have limited resources devoted to CDS development and maintenance, and fewer have been able to employ clinical informatics trained clinician-users to provide ongoing feedback. 33”

• cites [33] D Bowman, 2011Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The Consolidated Criteria for REporting Qualitative (COREQ) research guidelines were followed.

“The Consolidated Criteria for REporting Qualitative (COREQ) research guidelines were followed.”

• cites [26] A Tong, 2007, Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups, Int J Qual Health Care sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

This included involving end-users who were trainees and advanced practice providers, which is especially important within academic teaching hospitals as they often bear the burden of entering medical orders in the EHR and receiving alerts.

“This included involving end-users who were trainees and advanced practice providers, which is especially important within academic teaching hospitals as they often bear the burden of entering medical orders in the EHR and receiving alerts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One unique strength of our study was involving stakeholders from every step in the process of CDS development, governance, and usage.

“One unique strength of our study was involving stakeholders from every step in the process of CDS development, governance, and usage.”

May combine two claims.

CDS broadly does not consistently employ Osheroff's Five Rights, which likely contributes to CDS falling short of its potential.

“Although Osheroff’s Five Rights of CDS is commonly acknowledged as the backbone of effective CDS, CDS broadly does not consistently employ the five rights, which likely contributes to CDS falling short of its potential.”

• cites [23] J Osheroff, 2012 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Transcripts were not returned to participants for review, and participants did not provide feedback on study findings.

“Transcripts were not returned to participants for review, and participants did not provide feedback on study findings.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Although we attempted to get a broad, representative group of participants, we were limited by who responded to requests to participate in the study and those respondents may have been more likely to respond because they were either very satisfied or dissatisfied with interruptive alerts.

“Although we attempted to get a broad, representative group of participants, we were limited by who responded to requests to participate in the study and those respondents may have been more likely to respond because they were either very satisfied or dissatisfied with interruptive alerts.”

May combine two claims.

While interruptive alerts were initially developed as a tool to enhance patient care, they have resulted in clinician alert fatigue and workflow disruption due to their high volume and false-positive rate.

“While interruptive alerts were initially developed as a tool to enhance patient care, they have resulted in clinician alert fatigue and workflow disruption due to their high volume and false-positive rate.”