Electronic health record activity changes around new decision support implementation: monitoring using audit logs and topic modeling

Classified as probably not a substantial claim.

Prior studies found that providers often perceived or experienced increased workload such as extended duration of patient consultation, interrupted workflow, and alert overload associated with using CDS tools.

“Prior studies found that providers often perceived or experienced increased workload such as extended duration of patient consultation, interrupted workflow, and alert overload associated with using CDS tools.10,11”

• cites [10] Fletcher, 2023, Workload and workflow implications associated with the use of electronic clinical decision support tools used by health professionals in general practice: a scoping review, BMC Prim Care sourceSupported
• cites [11] Olakotan, 2021, The appropriateness of clinical decision support systems alerts in supporting clinical workflows: a systematic review, Health Informatics J sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

Increased workload and misalignment with clinical workflows are among the most common barriers to CDS adoption in clinical settings.

“Increased workload and misalignment with clinical workflows are among the most common barriers to CDS adoption in clinical settings.12,13”

• cites [12] Abell, 2023, Identifying barriers and facilitators to successful implementation of computerized clinical decision support systems in hospitals: a NASSS framework-informed scoping review, Implement Sci sourcePartly supported

  The Results identify fit with workflows as the most commonly reported determinant of hospital CDSS implementation (19 of 44 studies). They also report that perceived time and effort and loss of productivity can deter use, but do not establish increased workload as one of the most common barriers. The review concerns hospital settings, rather than clinical settings generally; page numbers were not supplied. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [13] Meunier, 2023, Barriers and facilitators to the use of clinical decision support systems in primary care: a mixed-methods systematic review, Ann Fam Med sourcePartly supported

  The Framework Synthesis reports increased workload for 33 of 45 CDSSs, and the barriers table lists disruption of usual workflow for 25. These findings support both barriers in primary care, but the review does not establish that they are among the most common barriers across clinical settings generally. Page numbers were not available in our copy. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim. May combine two claims.

There is currently no consensus on how to define the idle time threshold, and accurately identifying active EHR use remains an open research question.

“There is currently no consensus on how to define this threshold, and accurately identifying active EHR use remains an open research question.33”

• cites [33] Rule, 2022, Using event logs to observe interactions with electronic health records: an updated scoping review shows increasing use of vendor-derived measures, J Am Med Inform Assoc sourceSupported

Probably not a substantial claim: may not pass S2 (checkable).

Expert review is important for assessing quality of topics for real-world applications.

“Expert review is important for assessing quality of topics for real-world applications.58,59”

• cites [58] Chang, 2009Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [59] Meaney, 2023, Quality indices for topic model selection and evaluation: a literature review and case study, BMC Med Inform Decis Mak sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

The 5-minute idle time threshold has been used in several studies analyzing raw log data.

“Notably, this threshold has been used in several studies analyzing raw log data.24,25,31,46,47”

• cites [24] Ouyang, 2016, Internal medicine resident computer usage: an electronic audit of an inpatient service, JAMA Intern Med sourceSupported
• cites [25] Wang, 2019, Characterizing electronic health record usage patterns of inpatient medicine residents using event log data, PLoS One sourceSupported
• cites [31] Lou, 2022, Temporal associations between EHR-derived workload, burnout, and errors: a prospective cohort study, J Gen Intern Med sourceSupported
• cites [46] Lou, 2023, Characterizing the macrostructure of electronic health record work using raw audit logs: an unsupervised action embeddings approach, J Am Med Inform Assoc sourceNot checked — no full text of the source
• cites [47] Jones, 2020, Learning tasks of pediatric providers from electronic health record audit logs, AMIA Annu Symp ProcNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Studies using raw audit logs fit into three broad categories: manual mapping, identification of networks, application of automatic methods.

“Studies using raw audit logs fit into three broad categories:”

• cites [37] Amroze, 2019, Use of electronic health record access and audit logs to identify physician actions following noninterruptive alert opening: descriptive study, JMIR Med Inform sourcePartly supported

  The Methods and Results describe manually identifying and interpreting EHR audit-log entries with physician–data manager collaboration, which fits the claim’s manual-mapping category. The paper does not classify studies using raw audit logs into three broad categories or establish the other two categories. No page numbers are available in our copy. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [72] Arndt, 2017, Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations, Ann Fam Med sourceCould not verify

  The Methods (p. 2) describe assigning EHR log events to 15 task categories, not classifying studies that use raw audit logs. Searches for the proposed three categories and their named methods found no passage in our copy that addresses that classification. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [76] Cohen, 2019, Variation in physicians' electronic health record documentation and potential patient harm from that variation, J Gen Intern Med sourceCould not verify

  The Quantitative section describes this study’s use of EHR clickstream logs, and the Analytic Approach describes its own analysis. Neither section classifies studies using raw audit logs into three categories. Searches for the proposed categories and for a three-category classification found no such passage in our copy; page numbers were not available. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [77] Ancker, 2014, How is the electronic health record being used? Use of EHR data to assess physician-level variability in technology use, J Am Med Inform Assoc sourceCould not verify

  The Metric development section describes five categories of EHR-use metrics, not three categories of studies using raw audit logs. The Introduction and Discussion discuss ways to assess EHR use, but the searched passages do not classify audit-log studies as manual mapping, network identification, and automatic methods (searched: audit logs, manual mapping, networks, automatic methods). (LLM, conf=0.72)

  Experimental — accuracy not yet measured
• cites [78] Cutrona, 2017, Primary care providers' opening of time-sensitive alerts sent to commercial electronic health record InBaskets, J Gen Intern Med sourceCould not verify

  The Discussion describes uses of EHR audit logs, and Dependent Variables explains how this study used them to track alert opening. Neither section classifies audit-log studies into manual mapping, network identification, and automatic methods; searches for those terms and the proposed three-category classification found no supporting passage in our copy. (LLM, conf=0.72)

  Experimental — accuracy not yet measured
• cites [79] Gray, 2011, Using digital crumbs from an electronic health record to identify, study and improve health care teams, AMIA Annu Symp ProcNot checked — no full text of the source
• cites [80] Hripcsak, 2011, Use of electronic clinical documentation: time spent and team interactions, J Am Med Inform Assoc sourcePartly supported

  The Methods section (page unavailable) describes a study using detailed electronic health record usage logs and social network analysis, so this paper exemplifies a study in the network category. The Introduction lists three potential ways audit logs could advance informatics, but it does not classify studies into the three claimed categories of manual mapping, network identification, and automatic methods. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [81] Chen, 2017, Identifying collaborative care teams through electronic medical record utilization patterns, J Am Med Inform Assoc sourcePartly supported

  The Research Design and Methods and Conclusions sections describe an automated method for identifying collaborative networks from EMR utilization logs, so this paper is an example of one category. They do not classify audit-log studies into the three stated categories or discuss manual mapping as a category. The passages provided have no page numbers. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites [82] Durojaiye, 2019, Evaluation of multidisciplinary collaboration in pediatric trauma care using EHR data, J Am Med Inform Assoc sourceCould not verify

  The Materials and Methods sections describe constructing networks from EHR event logs and classifying those networks, but do not classify studies using raw audit logs into three categories. Searches for the proposed categories and for manual mapping found no passage stating that taxonomy; page numbers were not available in our copy. (LLM, conf=0.70)

  Experimental — accuracy not yet measured
• cites [83] Zhu, 2019, Measuring electronic communication networks in virtual care teams using electronic health records access-log data, Int J Med Inform sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Alert burden can lead to high rates of alert overrides and physician burnout (both physical and cognitive).

“Alert burden can lead to high rates of alert overrides and physician burnout (both physical and cognitive).9,16,64–66”

• cites [9] Chen, 2023, Monitoring the implementation of tobacco cessation support tools: using novel electronic health record activity metrics, JMIR Med Inform sourcePartly supported

  The Discussion section “Balancing Alert Efficacy With Burden” says that an overabundance of interruptive alerts may lead to frequent overrides and that providers experience alert-related fatigue. It does not say that alert burden causes physician burnout, or distinguish physical from cognitive burnout. No page number is available in our copy. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [16] Gregory, 2017, Electronic health record alert-related workload as a predictor of burnout in primary care providers, Appl Clin Inform sourcePartly supported

  Section 4.1 (page not available in our copy) reports that perceived alert burden predicted physical fatigue and cognitive weariness among primary care providers; measured time spent managing alerts did not. The paper's limitations note that its cross-sectional design cannot establish the direction of the effect. The passages on alert workload do not support the sentence's additional claim about high rates of alert overrides. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [64] Genco, 2016, Clinically inconsequential alerts: the characteristics of opioid drug alerts and their utility in preventing adverse drug events in the emergency department, Ann Emerg Med sourcePartly supported

  The Introduction (Background; page unavailable) says frequent, minimally significant warnings cause providers to repeatedly override alerts and reports high override rates. It calls this “alert fatigue,” but does not say alert burden causes physician burnout, physical or cognitive. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [65] Wong, 2019, Evaluation of harm associated with high dose-range clinical decision support overrides in the intensive care unit, Drug Saf sourcePartly supported

  The Results report that 93.0% of high dose-range alerts were overridden in the studied ICUs, and the Discussion recommends turning off unnecessary warnings. The paper does not report physician burnout, whether physical or cognitive, or establish that alert burden caused the overrides. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [66] McGreevey, 2020, Reducing alert burden in electronic health records: state of the art recommendations from four health systems, Appl Clin Inform sourcePartly supported

  The Background and Significance section reports high alert-override rates and says clinicians are more likely to ignore alerts as the number they encounter increases. It also reports an association between perceived alert burden and burnout among primary care physicians; the Clinical Relevance Statement identifies burnout as a potential consequence of over-alerting. Neither passage distinguishes physical from cognitive burnout, and the reported association concerns perceived rather than measured alert burden. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

Increased workload (either perceived or experienced) associated with CDS has been identified as a major barrier to CDS adoption.

“In addition, increased workload (either perceived or experienced) associated with CDS has been identified as a major barrier to CDS adoption.12,13”

• cites [12] Abell, 2023, Identifying barriers and facilitators to successful implementation of computerized clinical decision support systems in hospitals: a NASSS framework-informed scoping review, Implement Sci sourceSupported
• cites [13] Meunier, 2023, Barriers and facilitators to the use of clinical decision support systems in primary care: a mixed-methods systematic review, Ann Fam Med sourceSupported

Classified as probably not a substantial claim. May combine two claims.

EHR information overload poses risk to patient safety and excessive alerting is among the primary contributors to this overload.

“EHR information overload poses risk to patient safety and excessive alerting is among the primary contributors to this overload.84”

• cites [84] Nijor, 2022, Patient safety issues from information overload in electronic medical records, J Patient Saf sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

EHR-based CDS tools can streamline workflows by automatically presenting relevant information, thus reducing the need for providers to manually search through extensive records.

“EHR-based CDS tools can streamline workflows by automatically presenting relevant information, thus reducing the need for providers to manually search through extensive records.10,13”

• cites [10] Fletcher, 2023, Workload and workflow implications associated with the use of electronic clinical decision support tools used by health professionals in general practice: a scoping review, BMC Prim Care sourcePartly supported

  The Introduction says eCDS tools can draw directly on electronic medical records, and the Results (Decrease in consultation duration) report that some helped clinicians complete tasks more quickly. Our copy does not establish that automatically presenting relevant information reduces providers’ need to search through extensive records manually; that specific mechanism is unsupported. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites [13] Meunier, 2023, Barriers and facilitators to the use of clinical decision support systems in primary care: a mixed-methods systematic review, Ann Fam Med sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Prior studies found that CDS alerts, if addressed properly, were able to improve adherence to clinical guidelines.

“Prior studies found that CDS alerts, if addressed properly, were able to improve prescribing behaviors and medication safety4,63 and adherence to clinical guidelines.6,8”

• cites [6] Guiriguet, 2016, Alerts in electronic medical records to promote a colorectal cancer screening programme: a cluster randomised controlled trial in primary care, Br J Gen Pract sourcePartly supported

  The Per-protocol analysis (page unavailable) found significantly higher colorectal cancer screening uptake among eligible patients attending primary care after adjustment. The Implications for research and practice section (page unavailable) says improving professionals’ responses to alerts could increase their benefits, but overall uptake did not increase significantly. The study measured screening uptake, not adherence to clinical guidelines generally or the effect of alerts being ‘addressed properly.’ (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [8] Chak, 2018, Electronic medical alerts increase screening for chronic hepatitis B: a randomized, double-blind, controlled trial, Cancer Epidemiol Biomarkers Prev sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Prior studies found that CDS alerts, if addressed properly, were able to improve prescribing behaviors and medication safety.

“Prior studies found that CDS alerts, if addressed properly, were able to improve prescribing behaviors and medication safety4,63 and adherence to clinical guidelines.6,8”

• cites [4] Page, 2017, A systematic review of the effectiveness of interruptive medication prescribing alerts in hospital CPOE systems to change prescriber behavior and improve patient safety, Int J Med Inform sourceNot checked — no full text of the source
• cites [63] Taheri Moghadam, 2021, The effects of clinical decision support system for prescribing medication on patient outcomes and physician practice performance: a systematic review and meta-analysis, BMC Med Inform Decis Mak sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Within 4 hours before check-in, the CDS-related activity prevalence difference was 0.008 (95% CI 0.009 to 0.025; P=.336) post- versus pre-implementation.

“CDS-related | 0.120 [0.111-0.129] | 0.128 [0.114-0.143] | 0.008 [−0.009 to 0.025] | .336 | 0.070”

• AssumptionRests on an assumption: The 4-hour before-check-in window was likely to cover most of the physician’s EHR activities related to preparing for the patient visit.

Classified as probably not a substantial claim.

The 14 EHR-use activities were categorized into 4 groups: CDS-focused activity, CDS-related activity, EHR modification, and review of patient data.

“We further categorized the 14 activities into 4 groups: CDS-focused activity, CDS-related activity, EHR modification, and review of patient data (Table 1).”

The quote only partly states this claim.

Activity-level analysis showed that reductions mainly occurred in performing administrative activities for patient visit and billing after check-in.

“Activity-level analysis showed that reductions mainly occurred in reviewing patient summary data (both before and after patient check-in) and performing administrative activities for patient visit and billing after check-in.”

The quote only partly states this claim.

Activity-level analysis showed that reductions mainly occurred in reviewing patient summary data (both before and after patient check-in).

“Activity-level analysis showed that reductions mainly occurred in reviewing patient summary data (both before and after patient check-in) and performing administrative activities for patient visit and billing after check-in.”

Probably not a substantial claim: may not pass S2 (checkable).

The approach offers a scalable, data-driven framework for evaluating the real-world impact of EHR-embedded CDS tools.

“Our approach offers a scalable, data-driven framework for evaluating the real-world impact of EHR-embedded CDS tools.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The approach could be adapted to explore the impact of other healthcare quality improvement strategies using EHR-integrated CDS interventions.

“Built on a generic machine learning framework, this approach could be adapted to explore impact of other healthcare quality improvement strategies using EHR-integrated CDS interventions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

CD3 and CD4 are indirectly related to the firing of the tobacco-support alert.

“We assigned CD3 and CD4 to the CDS-related group because the representative activities in these topics (eg, load a note template, viewed report with patient data) could be indirectly related to the firing of the tobacco-support alert.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The optimal topic number was chosen by testing a range from 5 to 25 and selecting the number that achieved maximum within-topic semantic coherence and cross-topic exclusivity.

“When developing the model, we determined the optimal value for n by testing a range of topic numbers (from 5 to 25) and chose the number that achieved the maximum within-topic semantic coherence56 and cross-topic exclusivity.57”

• cites [56] Mimno, 2011Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [57] Bischof, 2012Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The CDS monitoring approach is generic.

“Our CDS monitoring approach is generic.”

Probably not a substantial claim: may not pass S3 (load-bearing).

CD2 focuses on cases where CDS alerts were bypassed or postponed by providers.

“CD2 focuses on cases where CDS alerts were bypassed or postponed by providers.”

Classified as probably not a substantial claim.

The covariates included in the regression analysis were encounter type, provider type, clinic type, implementation period, and time window length.

“The covariates included into the regression analysis included encounter type, provider type, clinic type, implementation period, and time window length.”

• AssumptionRests on an assumption: The time window length serves as a proxy for complexity of the patient’s condition.

Classified as probably not a substantial claim.

Adjusted mean prevalence of each EHR activity or activity group pre- and post-CDS-implementation was estimated using the inverse-probability-weighted regression adjustment (IPWRA) method, adjusting for clinic and patient characteristics that were unbalanced between periods or deemed to have potential impact on EHR activities.

“Adjusted mean prevalence (expressed as proportions out of 1.0) of each EHR activity or activity group pre- and post-CDS-implementation was estimated by using the inverse-probability-weighted regression adjustment (IPWRA) method, adjusting for clinic and patient characteristics that are unbalanced between pre- and post-CDS-implementation periods or those deemed to have potential impact on EHR act”

Classified as probably not a substantial claim.

Four domain experts (3 physicians and 1 EHR specialist) reviewed the topics and assigned EHR activity labels to topics.

“In our study, 4 domain experts (3 physicians and 1 EHR specialist) reviewed the topics and assigned EHR activity labels to topics.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One sensitivity analysis was conducted to validate the findings from the secondary analysis, using a fine-grained matching method (PSM within strata defined by clinic type, encounter type, and provider type) to obtain pre- and post-CDS encounters balanced in these characteristics and repeating the analysis for the 4-hour before-check-in window.

“Furthermore, we conducted one sensitivity analysis to validate the findings from the secondary analysis.”

• Also stated in: § Monitoring changes in providers’ EHR-use activities · plain

  “Specifically, we used a fine-grained matching method (ie, conducting PSM within strata defined by clinic type, encounter type, and provider type) to obtain pre- and post-CDS encounters balanced in these characteristics and repeated the analysis for the 4-hour before-check-in window on this new dataset.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The study was approved by the Wake Forest University School of Medicine Institutional Review Board (IRB00066841) and used de-identified EHR data.

“The study was approved by the Wake Forest University School of Medicine Institutional Review Board (IRB00066841) and used de-identified EHR data.”

Classified as probably not a substantial claim. May combine two claims.

Clinical practice guidelines recommend routine screening for tobacco use and referral to evidence-based tobacco control interventions for all cancer patients.

“Clinical practice guidelines recommend, for all cancer patients, routine screening for tobacco use and referral to evidence-based tobacco control interventions,39,40”

• cites [39] Shields, 2016, Smoking cessation, version 1.2016, NCCN clinical practice guidelines in oncology, J Natl Compr Canc Netw sourceNot checked — no full text of the source
• cites [40] Hanna, 2013, Tobacco cessation and control a decade later: American Society of Clinical Oncology policy statement update, J Clin Oncol sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

The labeled topics were grouped through expert review, and prevalence of activities and time on activities were analyzed at both the group level and the activity level.

“We then grouped the labeled topics (ie, EHR activities) through expert review and analyzed prevalence of activities and time on activities at both the group level and the activity level.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Two sensitivity analyses were conducted to validate the findings from the primary analysis.

“Additionally, we conducted 2 sensitivity analyses to validate the findings from our primary analysis.”

Classified as probably not a substantial claim.

A topic review protocol was developed and refined through pilot testing to ensure reliability of topic label assignment.

“To ensure the reliability of topic label assignment, we developed a topic review protocol (Appendix S5) to facilitate this process and refined the protocol through a pilot testing (Appendix S6).”

Probably not a substantial claim: may not pass S2 (checkable).

Topics emerge from latent structures within the data, making their labeling inherently subjective and dependent on contextual interpretation and domain expertise rather than absolute correctness.

“Instead, topics emerge from latent structures within the data, making their labeling inherently subjective and dependent on contextual interpretation and domain expertise rather than absolute correctness.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The number of topics was automatically determined based on within-topic coherence and across-topic divergence.

“The number of topics was automatically determined based on within-topic coherence and across-topic divergence”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In the second sensitivity analysis, data from one medical oncology clinic and office visits were used, and pre- and post-CDS encounters were matched at the individual provider level, to reduce the potential influence of data heterogeneity.

“In the second analysis, we reduced the potential influence of data heterogeneity by focusing on data from one medical oncology clinic and office visits and matching the pre- and post-CDS encounters at individual provider level.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The effectiveness of topic modeling, such as the coherence of the topics learned, is influenced by the heterogeneity of data used to train the model.

“However, like other unsupervised learning techniques, the effectiveness of topic modeling (eg, the coherence of the topics learned) is influenced by the heterogeneity of data used to train the model.”

Classified as probably not a substantial claim.

Topic modeling does not assume or rely on predefined labels for the topics it identifies.

“Like other unsupervised learning methods, topic modeling does not assume or rely on predefined labels for the topics it identifies.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

An iterative procedure was used to synthesize feedback from the 4 domain experts and resolve difficult cases through group meetings and email discussions.

“We used an iterative procedure to synthesize feedback from the 4 domain experts (see Appendix S6) and resolved difficult cases through group meetings and email discussions.”

Classified as probably not a substantial claim. May combine two claims.

The average length of a patient visit in this population was expected to be less than 30 minutes based on the literature and supported by insights from physician researchers on the team.

“Third, the average length of a patient visit in this population was expected to be less than 30 minutes based on the literature43 and supported by insights from physician researchers on our team.”

• cites [43] Guy, 2012, Visit duration for outpatient physician office visits among patients with cancer, J Oncol Pract sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

The identified topics were assigned clinically relevant EHR activity labels by 4 domain experts.

“the identified topics were assigned clinically relevant EHR activity labels by 4 domain experts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the first sensitivity analysis, a fine-grained matching method (PSM within strata defined by clinic type and encounter type) was used to obtain pre- and post-CDS encounters more balanced in encounter types.

“In the first analysis, we used a fine-grained matching method (ie, conducting PSM within strata defined by clinic type and encounter type) to obtain pre- and post-CDS encounters more balanced in encounter types.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

To our knowledge, this is the first study that develops automated CDS monitoring approach by leveraging EHR audit logs.

“To our knowledge, this is the first study that develops automated CDS monitoring approach by leveraging EHR audit logs.”

• AssumptionRests on an assumption: The relative change in the prevalence of CDS-focused and CDS-related topics can reflect change in EHR activities associated with the tobacco control CDS tool.

Probably not a substantial claim: may not pass S3 (load-bearing).

It is possible that certain patient visits lasted longer than 1 hour.

“It is possible that certain patient visits lasted longer than 1 hour.”

Classified as probably not a substantial claim.

Prevalence of EHR activity groups and individual EHR activities pre- and post-CDS-implementation was estimated by inverse-probability-weighted regression adjustment method, adjusting for encounter type, provider type, clinic type, implementation period, and actual time window length.

“Prevalence (expressed as proportions out of 1.0) of EHR activity groups and individual EHR activities pre- and post-CDS-implementation was estimated by inverse-probability-weighted regression adjustment method, adjusting for encounter type, provider type, clinic type, implementation period, and actual time window length.”

May combine two claims.

Methods for evaluating CDS impact on EHR activities remain limited, and existing approaches are hard to scale up for large implementation programs.

“Methods for evaluating CDS impact on EHR activities remain limited, and existing approaches (eg, direct observation, qualitative interview, and survey) are hard to scale up for large implementation programs.”

• cites [67] Hanauer, 2013, Computerized prescriber order entry implementation in a physician assistant-managed hematology and oncology inpatient service: effects on workflow and task switching, J Oncol Pract sourceNot checked — no full text of the source
• cites [68] Birmingham, 2011, FluAlert: a qualitative evaluation of providers' desired characteristics and concerns regarding computerized influenza vaccination alerts, Prev Med sourceNot checked — no full text of the source
• cites [69] Ayatollahi, 2015, Physicians' and nurses' opinions about the impact of a computerized provider order entry system on their workflow, Perspect Health Inf ManagNot checked — no full text of the source
• cites [70] Baysari, 2020, The efficiency-thoroughness trade-off after implementation of electronic medication management: a qualitative study in paediatric oncology, Int J Qual Health Care sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

The approach falls into the third category (application of automatic methods).

“Our approach falls into the third category.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

CDS tools may also indirectly impact clinical activities outside the EHR.

“and may also indirectly impact clinical activities outside the EHR.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Unlike simpler EHR-derived metrics, applying a topic model to raw log data is more computationally intensive.

“For example, unlike simpler EHR-derived metrics, such as those from the alert activity table that track whether a provider responded to or postponed a CDS alert, applying a topic model to raw log data is more computationally intensive.”

May combine two claims.

The topic modeling approach does not depend on hard boundaries between clusters or groups; rather, it identifies soft clusters as surrogates of higher-level EHR activities.

“In contrast, the topic modeling approach does not depend on hard boundaries between clusters or groups; rather, it identifies soft clusters (ie, topics that have different distributions over the micro-level events or actions) as surrogates of higher-level EHR activities.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The development of automated monitoring approaches, like the one presented in this study, can help identify workflow bottlenecks early, enabling timely adjustments to help providers maintain a balance between responding to CDS prompts and delivering holistic, patient-centered care.

“The development of automated monitoring approaches, like the one presented in this study, can help identify workflow bottlenecks early, enabling timely adjustments to help providers maintain a balance between responding to CDS prompts and delivering holistic, patient-centered care.”

• AssumptionRests on an assumption: The relative change in the prevalence of CDS-focused and CDS-related topics can reflect change in EHR activities associated with the tobacco control CDS tool.

Classified as probably not a substantial claim. May combine two claims.

Multiple comparisons (N = 18 regression models in the main analysis) were corrected by the Bonferroni correction, and a P value lower than .0027 (=.05/18) was regarded as significant.

“Multiple comparisons (N = 18 regression models in the main analysis) were corrected by the Bonferroni correction, and a P value lower than .0027 (=.05/18) was regarded as significant.”

Probably not a substantial claim: may not pass S2 (checkable).

Because the topic model can automatically infer the prevalences of EHR activities in individual EHR sessions, it provides a powerful tool to study relationships between EHR activities in different clinical contexts.

“Because the topic model can automatically infer the prevalences of EHR activities (ie, topics) in individual EHR sessions, it provides a powerful tool to study relationships between EHR activities in different clinical contexts.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

By combining signal detection with qualitative insights, this approach can support continuous improvement in provider workflows and enhance the alignment of CDS tools with clinical practice and patient care.

“By combining signal detection with qualitative insights, this approach can support continuous improvement in provider workflows and enhance the alignment of CDS tools with clinical practice and patient care.”

• AssumptionRests on an assumption: The relative change in the prevalence of CDS-focused and CDS-related topics can reflect change in EHR activities associated with the tobacco control CDS tool.

Classified as probably not a substantial claim.

The approach builds on a generic machine learning framework.

“Fourth, our approach builds on a generic machine learning framework.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Interpreting the output of a topic model requires broader context, such as comparisons across providers or over different time periods for the same provider, to yield meaningful insights.

“Moreover, interpreting the output of a topic model requires broader context, such as comparisons across providers or over different time periods for the same provider, to yield meaningful insights.”

May combine two claims.

Prior studies used embedding techniques or clustering methods, while the study used a latent-variable machine learning approach (structural topic modeling).

“While prior studies used embedding techniques or clustering methods, we used a latent-variable machine learning approach (structural topic modeling).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The results from the two sensitivity analyses showed compatible results.

“The results from the two sensitivity analyses (Tables A9-1 to A9-6 in Appendix S9) showed compatible results.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

This study may inform evaluation of CDS-based strategies and future efforts integrating machine learning into implementation research.

“This study may inform evaluation of CDS-based strategies and future efforts integrating machine learning into implementation research.”

• AssumptionRests on an assumption: The relative change in the prevalence of CDS-focused and CDS-related topics can reflect change in EHR activities associated with the tobacco control CDS tool.

May combine two claims.

The approach automatically identified EHR activities at a deeper level of granularity through the use of raw EHR logs and provided contextual clues surrounding providers' EHR use.

“In contrast, our approach automatically identified EHR activities at a deeper level of granularity through the use of raw EHR logs and also provided contextual clues surrounding providers’ EHR use.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Our CDS monitoring approach is adaptable to other settings-for example, primary care clinics that utilize CDS tools to support vaccination initiatives and the management of chronic diseases.

“Second, our CDS monitoring approach is adaptable to other settings—for example, primary care clinics that utilize CDS tools to support vaccination initiatives and the management of chronic diseases.”

• AssumptionRests on an assumption: The relative change in the prevalence of CDS-focused and CDS-related topics can reflect change in EHR activities associated with the tobacco control CDS tool.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The study advances research on use of EHR audit logs and expands available methodologic approaches.

“Our study advances research on use of EHR audit logs and expands available methodologic approaches.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Implementing such applications will require addressing additional technical challenges and operational considerations.

“However, implementing such applications will require addressing additional technical challenges and operational considerations.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The topic model can learn from new data easily.

“The topic model can learn from new data easily.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Mapping between topics and EHR activities requires manual efforts and domain expertise.

“However, mapping between topics and EHR activities requires manual efforts and domain expertise.”

May combine two claims.

Because EHR audit logs do not capture clinical activities outside EHR, our approach is useful for studying changes in EHR activities due to CDS implementation but less useful for studying the reasons for these changes.

“Because EHR audit logs do not capture clinical activities outside EHR (eg, phone, pager, face-to-face interactions), our approach is useful for studying changes in EHR activities due to CDS implementation but less useful for studying the reasons for these changes.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Technically, a trained topic model can be applied to individual EHR log files, making real-time monitoring feasible.

“Technically, a trained topic model can be applied to individual EHR log files, making real-time monitoring feasible.”

Classified as probably not a substantial claim.

It is possible that unobserved factors may exist and confound the regression analysis.

“It is therefore possible that unobserved factors may exist and confound the regression analysis.”

Classified as probably not a substantial claim. May combine two claims.

We used a pre-post quasi-experimental design and adjusted for observed patient and clinic characteristics unbalanced between pre- and post-CDS-implementation periods or deemed to have differential impact on EHR-use activities.

“We used a pre-post quasi-experimental design and adjusted for observed patient and clinic characteristics unbalanced between pre- and post-CDS-implementation periods or deemed to have differential impact on EHR-use activities.”

May combine two claims.

Methods built on raw audit logs are more generalizable and can be adapted to different EHR systems compared to evaluation methods using vendor-provided EHR-use metrics.

“Compared to evaluation methods using vendor-provided EHR-use metrics, methods built on raw audit logs are more generalizable and can be adapted to different EHR systems.”

Probably not a substantial claim: may not pass S2 (checkable).

Our novel approach uses machine learning and EHR audit logs to advance methods for implementation monitoring.

“Our novel approach uses machine learning and EHR audit logs to advance methods for implementation monitoring.”

• AssumptionRests on an assumption: The relative change in the prevalence of CDS-focused and CDS-related topics can reflect change in EHR activities associated with the tobacco control CDS tool.

May combine two claims.

This method only approximates the actual idle time and will introduce noise to data analysis.

“Although this method is commonly used in EHR audit log research, it only approximates the actual idle time and will introduce noise to data analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing).

We assessed the robustness of our findings through sensitivity analyses.

“In this study, we assessed the feasibility of using this approach to detect change signals and validated our findings based on our understanding of this CDS tool’s workflow as well as through sensitivity analyses.”

• Also stated in: § Limitations and future work · plain

  “Finally, constrained by the study scope, we did not compare the time-on-activity measure with actual time measurements, although we assessed the robustness of our findings through sensitivity analyses.”

May combine two claims.

Metrics built on audit log data can reduce reporting bias and may minimize Hawthorne effects.

“Metrics built on this data can reduce reporting bias and may minimize Hawthorne effects (ie, the change in participants’ engagement with an intervention that occurs when they are aware of attention from observers).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The results from the sensitivity analysis showed compatible results.

“The results from the sensitivity analysis (Tables A9-7 to A9-9 in Appendix S9) showed compatible results.”

Classified as probably not a substantial claim.

Adapting the approach to different EHR systems or CDS tools requires retraining the topic model on domain-specific log data and relabeling the topics.

“Adapting our approach to different EHR systems or CDS tools requires retraining the topic model on domain-specific log data and relabeling the topics.”

May combine two claims.

We expect the impact of this noise to be negligible because we used the relative time change for monitoring the impact of CDS and used the same method for all time windows.

“However, we expect the impact of this noise to be negligible because we used the relative time change for monitoring the impact of CDS and used the same method for all time windows.”

Probably not a substantial claim: may not pass S3 (load-bearing).

We did not compare the time-on-activity measure with actual time measurements.

“Finally, constrained by the study scope, we did not compare the time-on-activity measure with actual time measurements, although we assessed the robustness of our findings through sensitivity analyses.”

Classified as probably not a substantial claim. May combine two claims.

The approach is agnostic to specific study designs and can be applied to any EHR-embedded CDS tools.

“It is agnostic to specific study designs and can be applied to any EHR-embedded CDS tools.”