CoChat
Sign Up
Cited by 92Health Policy Implementation ScienceHealth Sciences Research and EducationClinical practice guidelines implementation

A systematic review of experimentally tested implementation strategies across health and human service settings: evidence from 2010-2022

Open access
Ashcraft, Laura Ellen, Goodrich, David E., Hero, Joachim, Phares, Angela, Bachrach, Rachel L., Quinn, Deirdre A., Qureshi, Nabeel, Ernecoff, Natalie C. +4 more·2024·Implementation Science
DOI 10.1186/s13012-024-01369-5PMC PMC11194895
Open this paper in CoChat — free account

Abstract

Background: Studies of implementation strategies range in rigor, design, and evaluated outcomes, presenting interpretation challenges for practitioners and researchers. This systematic review aimed to describe the body of research evidence testing implementation strategies across diverse settings and domains, using the Expert Recommendations for Implementing Change (ERIC) taxonomy to classify strategies and the Reach Effectiveness Adoption Implementation and Maintenance (RE-AIM) framework to classify outcomes. Methods: We conducted a systematic review of studies examining implementation strategies from 2010-2022 and registered with PROSPERO (CRD42021235592). We searched databases using terms “implementation strategy”, “intervention”, “bundle”, “support”, and their variants. We also solicited study recommendations from implementation science experts and mined existing systematic reviews. We included studies that quantitatively assessed the impact of at least one implementation strategy to improve health or health care using an outcome that could be mapped to the five evaluation dimensions of RE-AIM. Only studies meeting prespecified methodologic standards were included. We described the characteristics of studies and frequency of implementation strategy use across study arms. We also examined common strategy pairings and cooccurrence with significant outcomes. Findings: Our search resulted in 16,605 studies; 129 met inclusion criteria. Studies tested an average of 6.73 strategies (0-20 range). The most assessed outcomes were Effectiveness (n=82; 64%) and Implementation (n=73; 56%). The implementation strategies most frequently occurring in the experimental arm were Distribute Educational Materials (n=99), Conduct Educational Meetings (n=96), Audit and Provide Feedback (n=76), and External Facilitation (n=59). These strategies were often used in combination. Nineteen implementation strategies were frequently tested and associated with significantly improved outcomes. However, many strategies were not tested sufficiently to draw conclusions. Conclusion: This review of 129 methodologically rigorous studies built upon prior implementation science data syntheses to identify implementation strategies that had been experimentally tested and summarized their impact on outcomes across diverse outcomes and clinical settings. We present recommendations for improving future similar efforts.

AI snapshot

Written from the paper’s full text.

TL;DR

This systematic review synthesized 129 experimentally tested studies (143 publications) from 2010–2022, mapping implementation strategies to ERIC and outcomes to RE-AIM. It found educational strategies were ubiquitous, 19 strategies were frequently tested and associated with positive outcomes, but many ERIC strategies lacked sufficient experimental evidence.

Key findings

  • Of 129 included studies, the most assessed outcomes were Effectiveness (n=82; 64%) and Implementation (n=73; 56%); 98 studies (76%) reported at least one significantly positive outcome.
  • Experimental arms averaged 8.33 strategies (range 1–21), tested strategies averaged 6.73 (range 0–20), and control arms averaged 1.64 strategies (range 0–20). The most common experimental-arm strategies were Distribute Educational Materials (n=99), Conduct Educational Meetings (n=96), Audit and Provide Feedback (n=76), and External Facilitation (n=59).
  • Nineteen implementation strategies fell in the upper-right quadrant—tested in more than the median 8.5 studies and associated with a significant RE-AIM outcome in more than 75% of studies—including Conduct Educational Meetings, Distribute Educational Materials, External Facilitation, and Audit and Provide Feedback.
  • Ten ERIC strategies were not used in any reviewed studies: Alter Patient/Consumer Fees, Change Liability Laws, Change Service Sites, Develop Disincentives, Develop Resource Sharing Agreements, Identify Early Adopters, Make Billing Easier, Start a Dissemination Organization, Use Capitated Payments, and Use Data Experts.

Methods

  • Systematic review registered with PROSPERO (CRD42021235592) and reported using PRISMA-P; searched CINAHL, PubMed, and Web of Science, plus expert recommendations and prior systematic reviews, for English-language experimental studies published January 1, 2010–September 20, 2022 in health or human service settings with a quantitative outcome mappable to RE-AIM and a strategy classifiable by ERIC.
  • Inclusion required a concurrent control group, experimental design, and a rigor score of 7 or 8 from two independent reviewers; data were abstracted in pairs, strategies were coded using a modified ERIC taxonomy (70 named strategies), and outcomes were coded into RE-AIM domains.
  • Analyses used descriptive statistics for strategy frequency and pairings, and a quadrant plot based on median number of studies (8.5) and median percent of studies with at least one positive RE-AIM outcome (75%).

Limitations

  • Stated: only experimental studies were included, omitting observational investigations; the study period centered on the creation of *Implementation Science* rather than ERIC standardization, so retroactive mapping may have missed studies; indexing and search terms may have missed relevant studies (e.g., policies, financial strategies, community health promotion, electronic medical record reminders); and ERIC is not universally applied and can be unclear or missing key domains.
  • Stated: the authors could not assess study bias due to non-standard reporting and heterogeneity of designs, measurement, and analytic approaches, and could not gather primary data to evaluate effect sizes.
  • Evident from the text: most studies used multifaceted bundles, making it difficult to attribute effects to individual strategies; many strategies were not tested sufficiently to draw conclusions; and implementation strategy fidelity, costs, and health disparities were not assessed.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13012-024-01369-5

Cites

116 of 116 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 61 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Educational strategies were employed ubiquitously.

    “Educational strategies were employed ubiquitously; nineteen other commonly used implementation strategies, including External Facilitation and Audit and Provide Feedback, were associated with positive outcomes in these experimental trials.”
    Quote in the paperNot a substantial claim
  • Findingown data

    In about half the studies (53%; n=69), the Control Arms were active controls that included at least one strategy.

    “In about half the studies (53%; n=69), the Control Arms were “active controls” that included at least one strategy”
    Quote in the paperNot a substantial claim
  • Findingown data

    Of 129 included studies, 103 (79%) were conducted in a healthcare setting.

    “Of 129 included studies (Table 1; see also Additional File 5 for Summary of Included Studies), 103 (79%) were conducted in a healthcare setting.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Most studies (n=98, 76%) reported at least one significantly positive outcome.

    “Most studies (n = 98, 76%) reported at least one significantly positive outcome.”
    Quote in the paperSubstantial claim
  • Findingown data

    The implementation strategies most frequently occurring in the experimental arm were Distribute Educational Materials (n=99), Conduct Educational Meetings (n=96), Audit and Provide Feedback (n=76), and External Facilitation (n=59).

    “The implementation strategies most frequently occurring in the experimental arm were Distribute Educational Materials (n=99), Conduct Educational Meetings (n=96), Audit and Provide Feedback (n=76), and External Facilitation (n=59).”
    Quote in the paperNot a substantial claim
  • May combine two claims. 46
  • Classified as probably not a substantial claim. 46

  • May combine two claims.

    Findingown data

    Control Arms averaged approximately two strategies (range 0-20) and Experimental Arms averaged eight strategies (range 1-21).

    “The number of strategies varied widely across studies, with Control Arms averaging approximately two strategies (Range = 0-20, including studies with no strategy in the comparison group) and Experimental Arms averaging eight strategies (Range = 1-21).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The review identified 14,646 articles through the initial literature search, 17 through expert recommendation, and 1,942 through reviewing prior systematic reviews.

    “We identified 14,646 articles through the initial literature search, 17 articles through expert recommendation (three of which were not included in the initial search), and 1,942 articles through reviewing prior systematic reviews (Fig. 1).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Adoption and Implementation outcomes showed positive change in three-quarters of studies (n=78), while Reach (n=18; 58%), Effectiveness (n=44; 54%), and Maintenance (n=23; 58%) outcomes evidenced positive change in approximately half of studies.

    “Adoption and Implementation outcomes showed positive change in three-quarters of studies (n=78), while Reach (n=18; 58%), Effectiveness (n=44; 54%), and Maintenance (n=23; 58%) outcomes evidenced positive change in approximately half of studies.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The search identified 16,605 studies, of which 129 met inclusion criteria.

    “Our search resulted in 16,605 studies; 129 met inclusion criteria.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Non-US studies (n=73) included more sites and target population on average, with an overall median of 32 sites and 1,531 patients assessed in each study.

    “Non-US studies (n=73) included more sites and target population on average, with an overall median of 32 sites and 1,531 patients assessed in each study.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Studies included a median of 29 sites and 1,419 target population.

    “Studies included a median of 29 sites and 1,419 target population (e.g., patients or students).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Six hundred seventeen articles were reviewed for study rigor, with 385 excluded for reasons related to study design and rigor, and 86 removed for other reasons.

    “Six hundred seventeen articles were reviewed for study rigor with 385 excluded for reasons related to study design and rigor, and 86 removed for other reasons (e.g., not a research article).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    After removing duplicates, 9,399 articles were included in the initial abstract screening.

    “After removing duplicates, 9,399 articles were included in the initial abstract screening.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Control arms included an average of 1.64 implementation strategies, with up to 20 strategies reported.

    “with an average of 1.64 (and up to 20) strategies reported in control arms”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The full text of 1,426 articles was screened in pairs.

    “We screened the full text of 1,426 articles in pairs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Nineteen implementation strategies were frequently tested and associated with significantly improved outcomes.

    “Nineteen implementation strategies were frequently tested and associated with significantly improved outcomes.”
    • Also stated in: § Supplementary Information · plain
      “Educational strategies were employed ubiquitously; nineteen other commonly used implementation strategies, including External Facilitation and Audit and Provide Feedback, were associated with positive outcomes in these experimental trials.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The most evaluated RE-AIM outcomes were Effectiveness (n=82, 64%) and Implementation (n=73, 56%), followed by Maintenance (n=40, 31%), Adoption (n=33, 26%), and Reach (n=31, 24%).

    “The most assessed outcomes were Effectiveness (n=82; 64%) and Implementation (n=73; 56%).”
    • Also stated in: § Descriptive results · plain
      “Organized by RE-AIM, the most evaluated outcomes were Effectiveness (n = 82, 64%) and Implementation (n = 73, 56%); followed by Maintenance (n=40; 31%), Adoption (n=33; 26%), and Reach (n=31; 24%).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The final number of studies abstracted was 129, representing 143 publications.

    “The final number of studies abstracted was 129 representing 143 publications.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Among three additional expert-recommended articles, one met inclusion criteria and was added to the analysis.

    “Among the three additional expert-recommended articles, one met inclusion criteria and was added to the analysis.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Implementation and Effectiveness outcomes were assessed most frequently, with Implementation being the most commonly reported significantly positive outcome.

    “Implementation and Effectiveness outcomes were assessed most frequently, with Implementation being the mostly commonly reported significantly positive outcome.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Implementation-phase strategies with strong evidence included coaching and support (External and Internal Facilitation) and involving additional key partners (Intervene with Patients to Enhance Uptake and Adherence).

    “Strategies that target the implementation phase include those that provide coaching and support (External and Internal Facilitation), involve additional key partners (Intervene with Patients to Enhance Uptake and Adherence)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Other commonly paired strategies included Develop a Formal Implementation Blueprint, Promote Adaptability, Conduct Ongoing Training, Purposefully Reexamine the Implementation, and Develop and Implement Tools for Quality Monitoring.

    “Other commonly paired strategies included Develop a Formal Implementation Blueprint, Promote Adaptability, Conduct Ongoing Training, Purposefully Reexamine the Implementation, and Develop and Implement Tools for Quality Monitoring.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The five most used strategies in the Experimental Arm included Audit and Provide Feedback (n=76), External Facilitation (n=59), Tailor Strategies (n=43), Assess for Readiness and Identify Barriers and Facilitators (n=43), and Organize Implementation Teams (n=42).

    “The five most used strategies in the Experimental Arm included Audit and Provide Feedback (n=76), External Facilitation (n=59), Tailor Strategies (n=43), Assess for Readiness and Identify Barriers and Facilitators (n=43), and Organize Implementation Teams (n=42).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    We identified 19 implementation strategies that were tested in at least 8 studies (more than the median) and were associated with positive results at least 75% of the time.

    “We identified 19 implementation strategies that were tested in at least 8 studies (more than the median) and were associated with positive results at least 75% of the time.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: At least one significantly positive RE-AIM outcome is an adequate indicator of a strategy's supportive evidence.
    • Rests on an assumption: The median split (8.5 studies and 75% positive outcomes) appropriately distinguishes stronger from weaker evidence.

  • May combine two claims.

    Findingown data

    Of the 19 strategies with stronger evidence, Conduct Educational Meetings, Distribute Educational Materials, External Facilitation, and Audit and Provide Feedback continued to occur frequently, appearing in 59-99 studies.

    “Of those 19 implementation strategies, Conduct Educational Meetings, Distribute Educational Materials, External Facilitation, and Audit and Provide Feedback continued to occur frequently, appearing in 59-99 studies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The most common strategies in the Experimental Arm were Distribute Educational Materials (n=99), Conduct Educational Meetings (n=96), Audit and Provide Feedback (n=76), and External Facilitation (n=59).

    “The most common strategies in the Experimental Arm were Distribute Educational Materials (n=99), Conduct Educational Meetings (n=96), Audit and Provide Feedback (n=76), and External Facilitation (n=59).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The top 10% of tested strategies included Conduct Educational Meetings (n=68), Audit and Provide Feedback (n=63), External Facilitation (n=54), Distribute Educational Materials (n=49), Tailor Strategies (n=41), Assess for Readiness and Identify Barriers and Facilitators (n=38), and Organize Clinician Implementation Team Meetings (n=37).

    “Overall, the top 10% of tested strategies included Conduct Educational Meetings (n=68), Audit and Provide Feedback (n=63), External Facilitation (n=54), Distribute Educational Materials (n=49), Tailor Strategies (n=41), Assess for Readiness and Identify Barriers and Facilitators (n=38) and Organize Clinician Implementation Team Meetings (n=37).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Findingnone given

    Our experience with this review seemed to suggest that ERIC's use was increasing over time.

    “However, our experience with this review seemed to suggest that ERIC’s use was increasing over time.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Reach and Adoption of the intervention were reported for the Experimental Arm but not for the Control Arm, prohibiting statistical comparisons of strategies on the relative impact of the EBP between study arms.

    “Similarly, Reach and Adoption of the intervention would be reported for the Experimental Arm but not for the Control Arm, prohibiting statistical comparisons of strategies on the relative impact of the EBP between study arms.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    We identified commonly occurring implementation strategies, frequently used bundles, and the strategies with the highest degree of supportive evidence, while simultaneously identifying gaps in the literature.

    “We identified commonly occurring implementation strategies, frequently used bundles, and the strategies with the highest degree of supportive evidence, while simultaneously identifying gaps in the literature.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The systematic review was developed and reported using the PRISMA-P model.

    “We used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA-P) model [29–31] to develop and report on the methods for this systematic review (Additional File 1).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [29] Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4:1. source

      Not checked in the free run.

    • cites [30] Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;349:g7647. source

      Not checked in the free run.

    • cites [31] Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ [Internet]. 2021;372. Available from: https://www.bmj.com/content/372/bmj.n71 source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    Studies were included only if they had a concurrent control group, an experimental design, and received a rigor score of 7 or 8 from two independent reviewers.

    “Articles were included if they 1) included a concurrent control group, 2) had an experimental design, and 3) received a score of 7 or 8 from two independent reviewers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The systematic review protocol was registered with PROSPERO (CRD42021235592).

    “We conducted a systematic review of studies examining implementation strategies from 2010-2022 and registered with PROSPERO (CRD42021235592).”
    • Also stated in: § Registration · plain
      “The protocol was registered with PROSPERO (PROSPERO 2021 CRD42021235592).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Title and abstract screening was conducted by two reviewers, with conflicts adjudicated by a senior team member.

    “Each study’s title and abstract were read by two reviewers, who dichotomously scored studies on each of the six eligibility criteria described above as yes=1 or no=0, resulting in a score ranging from 1 to 6.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Full text screening was conducted in pairs, with conflicts adjudicated by a senior team member.

    “During the full text screening process, we reviewed, in pairs, each article that had progressed through abstract screening. Conflicts between reviewers were adjudicated by a senior member of the team for a final inclusion decision (MJC, SSR, DEG).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study was considered non-human subjects research by the RAND institutional review board.

    “This study was considered to be non-human subjects research by the RAND institutional review board.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation

    Implementation strategies were classified using standard definitions based on minor modifications to the ERIC taxonomy, resulting in 70 named strategies.

    “Implementation strategies were classified using standard definitions (MJC, SSR, DEG), based on minor modifications to the ERIC taxonomy [2–4]. Modifications resulted in 70 named strategies and were made to decrease redundancy and improve clarity.”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [2] Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21. source

      Not checked in the free run.

    • cites [3] Waltz TJ, Powell BJ, Chinman MJ, Smith JL, Matthieu MM, Proctor EK, et al. Expert recommendations for implementing change (ERIC): protocol for a mixed methods study. Implement Sci IS. 2014;9:39. source

      Not checked in the free run.

    • cites [4] Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, et al. A Compilation of Strategies for Implementing Clinical Innovations in Health and Mental Health. Med Care Res Rev. 2012;69:123–57. source

      Not checked in the free run.

    • cites [37] Yakovchenko V, Chinman MJ, Lamorte C, Powell BJ, Waltz TJ, Merante M, et al. Refining Expert Recommendations for Implementing Change (ERIC) strategy surveys using cognitive interviews with frontline providers. Implement Sci Commun. 2023;4:1–14. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Data from the full text review were abstracted in pairs, with conflicts resolved by senior team members using a standard Qualtrics abstraction form.

    “Data from the full text review were abstracted in pairs, with conflicts resolved by senior team members (DEG, MJC) using a standard Qualtrics abstraction form.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The ERIC taxonomy has been widely adopted and further refined.

    “The ERIC taxonomy has been widely adopted and further refined [6–13].”
    Quote in the paperNot a substantial claim0 of 8 sources checked
    • cites [6] Perry CK, Damschroder LJ, Hemler JR, Woodson TT, Ono SS, Cohen DJ. Specifying and comparing implementation strategies across seven large implementation interventions: a practical application of theory. Implement Sci. 2019;14(1):32. source

      Not checked in the free run.

    • cites [7] Community Preventive Services Task Force. Community Preventive Services Task Force: All Active Findings June 2023 [Internet]. 2023 [cited 2023 Aug 7]. Available from: https://www.thecommunityguide.org/media/pdf/CPSTF-All-Findings-508.pdf

      Not checked in the free run.

    • cites [8] Solberg LI, Kuzel A, Parchman ML, Shelley DR, Dickinson WP, Walunas TL, et al. A Taxonomy for External Support for Practice Transformation. J Am Board Fam Med JABFM. 2021;34:32–9. source

      Not checked in the free run.

    • cites [9] Leeman J, Birken SA, Powell BJ, Rohweder C, Shea CM. Beyond “implementation strategies”: classifying the full range of strategies used in implementation science and practice. Implement Sci. 2017;12:1–9. source

      Not checked in the free run.

    • cites [10] Leeman J, Calancie L, Hartman MA, Escoffery CT, Herrmann AK, Tague LE, et al. What strategies are used to build practitioners’ capacity to implement community-based interventions and are they effective?: a systematic review. Implement Sci. 2015;10:1–15. source

      Not checked in the free run.

    • cites [11] Nathan N, Shelton RC, Laur CV, Hailemariam M, Hall A. Editorial: Sustaining the implementation of evidence-based interventions in clinical and community settings. Front Health Serv. 2023;3:1176023. source

      Not checked in the free run.

    • cites [12] Balis LE, Houghtaling B, Harden SM. Using implementation strategies in community settings: an introduction to the Expert Recommendations for Implementing Change (ERIC) compilation and future directions. Transl Behav Med. 2022;12:965–78. source

      Not checked in the free run.

    • cites [13] Nathan N, Powell BJ, Shelton RC, Laur CV, Wolfenden L, Hailemariam M, et al. Do the Expert Recommendations for Implementing Change (ERIC) strategies adequately address sustainment? Front Health Serv. 2022;2:905909. source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductcitation

    The review did not assess implementation strategy fidelity or effects on health disparities.

    “We did not assess implementation strategy fidelity or effects on health disparities as these are recently adopted reporting standards [27, 28] and not yet widely implemented in current publications.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [27] Shelton RC, Chambers DA, Glasgow RE. An Extension of RE-AIM to Enhance Sustainability: Addressing Dynamic Context and Promoting Health Equity Over Time. Front Public Health. 2020;8:134. source

      Not checked in the free run.

    • cites [28] Holtrop JS, Estabrooks PA, Gaglio B, Harden SM, Kessler RS, King DK, et al. Understanding and applying the RE-AIM framework: Clarifications and resources. J Clin Transl Sci. 2021;5:e126. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationnone given

    This systematic review provides a transdisciplinary evaluation of implementation strategies across population, practice setting, and evidence-based interventions using a standardized taxonomy of strategies and outcomes.

    “This systematic review provides a transdisciplinary evaluation of implementation strategies across population, practice setting, and evidence-based interventions using a standardized taxonomy of strategies and outcomes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    The search timeframe 2010-2022 was selected to coincide with the advent of Implementation Science and when the term 'implementation strategy' became conventionally used.

    “This timeframe was selected to coincide with the advent of Implementation Science and when the term “implementation strategy” became conventionally used [2, 4, 33].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [2] Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21. source

      Not checked in the free run.

    • cites [4] Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, et al. A Compilation of Strategies for Implementing Clinical Innovations in Health and Mental Health. Med Care Res Rev. 2012;69:123–57. source

      Not checked in the free run.

    • cites [33] Eccles MP, Mittman BS. Welcome to Implementation Science. Implement Sci. 2006;1:1. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The ERIC taxonomy comprises 73 implementation strategies organized into nine clusters.

    “In 2015, the Expert Recommendations for Implementing Change (ERIC) study organized a panel of implementation scientists to compile a standardized set of implementation strategy terms and definitions [2–4].”
    • Also stated in: § Background · plain
      “These 73 strategies were then organized into nine “clusters” [5].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [2] Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21. source

      Not checked in the free run.

    • cites [3] Waltz TJ, Powell BJ, Chinman MJ, Smith JL, Matthieu MM, Proctor EK, et al. Expert recommendations for implementing change (ERIC): protocol for a mixed methods study. Implement Sci IS. 2014;9:39. source

      Not checked in the free run.

    • cites [4] Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, et al. A Compilation of Strategies for Implementing Clinical Innovations in Health and Mental Health. Med Care Res Rev. 2012;69:123–57. source

      Not checked in the free run.

    • cites [5] Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10:109. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    Outcomes were coded into RE-AIM outcome domains following best practices recommended by framework experts.

    “Outcomes were then coded into RE-AIM outcome domains following best practices as recommended by framework experts [26–28].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [26] Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89:1322–7. source

      Not checked in the free run.

    • cites [27] Shelton RC, Chambers DA, Glasgow RE. An Extension of RE-AIM to Enhance Sustainability: Addressing Dynamic Context and Promoting Health Equity Over Time. Front Public Health. 2020;8:134. source

      Not checked in the free run.

    • cites [28] Holtrop JS, Estabrooks PA, Gaglio B, Harden SM, Kessler RS, King DK, et al. Understanding and applying the RE-AIM framework: Clarifications and resources. J Clin Transl Sci. 2021;5:e126. source

      Not checked in the free run.

  • May combine two claims.

    Interpretationnone given

    This review of 129 methodologically rigorous studies built upon prior implementation science data syntheses to identify implementation strategies that had been experimentally tested and summarized their impact on outcomes across diverse outcomes and clinical settings.

    “This review of 129 methodologically rigorous studies built upon prior implementation science data syntheses to identify implementation strategies that had been experimentally tested and summarized their impact on outcomes across diverse outcomes and clinical settings.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The absence of the ten strategies may have been the result of inclusion criteria requiring an experimental design.

    “Likewise, the absence of these ten strategies may have been the result of our inclusion criteria, which required an experimental design.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    Considering that most studies employed multiple or multifaceted strategies, there seems to be a benefit of using a targeted bundle of strategies that are purposefully aligned with site/clinic/population norms, rather than simply adding more strategies.

    “Considering that most studies employed multiple or multifaceted strategies, it seems that there is a benefit of using a targeted bundle of strategies that are purposefully aligns with site/clinic/population norms, rather than simply adding more strategies [66].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [66] Harvey G, Kitson A. Translating evidence into healthcare policy and practice: Single versus multi-faceted implementation strategies – is there a simple answer to a complex question? Int J Health Policy Manag. 2015;4:123–6. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The need to improve real-world implementation may outweigh the theoretical need to identify individual strategy effectiveness.

    “Therefore, the need to improve real-world implementation may outweigh the theoretical need to identify individual strategy effectiveness.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The complexity of health and human service settings, the imperative to improve public health outcomes, and engagement with community partners often require the use of multiple strategies simultaneously.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Education strategies are generally considered to be necessary but not sufficient for successful implementation.

    “education strategies are generally considered to be “necessary but not sufficient” for successful implementation [43, 44]”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [43] Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess Winch Engl. 2004;8:iii–iv 1-72.

      Not checked in the free run.

    • cites [44] Beidas RS, Kendall PC. Training Therapists in Evidence-Based Practice: A Critical Review of Studies From a Systems-Contextual Perspective. Clin Psychol Publ Div Clin Psychol Am Psychol Assoc. 2010;17:1–30.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    To truly assess the impacts of strategies on outcomes, it may be necessary to track fidelity to implementation strategies (not just the EBPs they support).

    “In addition, to truly assess the impacts of strategies on outcomes, it may be necessary to track fidelity to implementation strategies (not just the EBPs they support).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Rogal and colleagues documented that VA hospitals implementing a new evidence-based hepatitis C treatment chose more than 20 strategies, when multiple years of data linking strategies to outcomes showed that 1-3 specific strategies would have yielded the same outcome.

    “Rogal and colleagues documented that VA hospitals implementing a new evidence-based hepatitis C treatment chose >20 strategies, when multiple years of data linking strategies to outcomes showed that 1-3 specific strategies would have yielded the same outcome [39].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [39] Gold HT, McDermott C, Hoomans T, Wagner TH. Cost data in implementation science: categories and approaches to costing. Implement Sci. 2022;17:11. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    While we cannot easily disaggregate the effects of multifaceted strategies, doing so may not yield meaningful results.

    “While we cannot easily disaggregate the effects of multifaceted strategies, doing so may not yield meaningful results.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Because strategies often synergize, disaggregated results could either underestimate the true impact of individual strategies or conversely, actually undermine their effectiveness.

    “Because strategies often synergize, disaggregated results could either underestimate the true impact of individual strategies or conversely, actually undermine their effectiveness (i.e., when their effectiveness comes from their combination with other strategies).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Implementation strategies often synergize.

  • May combine two claims.

    Interpretationown data

    The upper right quadrant included 19 strategies that occurred in more than the median number of studies (8.5) and had more than the median percent of studies with a significant RE-AIM outcome (75%); those 19 strategies were viewed as having stronger evidence.

    “The upper right quadrant included strategies that occurred in more than the median number of studies (8.5) and had more than the median percent of studies with a significant RE-AIM outcome (75%); thus those 19 strategies were viewed as having stronger evidence.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: At least one significantly positive RE-AIM outcome is an adequate indicator of a strategy's supportive evidence.
    • Rests on an assumption: The median split (8.5 studies and 75% positive outcomes) appropriately distinguishes stronger from weaker evidence.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Reasons for the absence of these strategies may include challenges with testing certain strategies experimentally, limitations in search terms, and the relative paucity of implementation strategy trials compared to clinical trials.

    “Reasons for the absence of these strategies in our review may include challenges with testing certain strategies experimentally (e.g., changing liability laws), limitations in our search terms, and the relative paucity of implementation strategy trials compared to clinical trials.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Other studies have found that more is not always better.

    “others have found that “more is not always better” [63–65]”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [63] Rogal SS, Yakovchenko V, Waltz TJ, Powell BJ, Kirchner JE, Proctor EK, et al. The association between implementation strategy use and the uptake of hepatitis C treatment in a national sample. Implement Sci. 2017;12:1–13. source

      Not checked in the free run.

    • cites [64] Smith SN, Almirall D, Prenovost K, Liebrecht C, Kyle J, Eisenberg D, et al. Change in patient outcomes after augmenting a low-level implementation strategy in community practices that are slow to adopt a collaborative chronic care model: a cluster randomized implementation trial. Med Care. 2019;57:503. source

      Not checked in the free run.

    • cites [65] Rogal SS, Yakovchenko V, Waltz TJ, Powell BJ, Gonzalez R, Park A, et al. Longitudinal assessment of the association between implementation strategy use and the uptake of hepatitis C treatment: Year 2. Implement Sci. 2019;14:1–12. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    We were not able to gather primary data to evaluate effect sizes across studies in order to systematically evaluate bias.

    “Further, we were not able to gather primary data to evaluate effect sizes across studies in order to systematically evaluate bias, which would be fruitful for future study.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    Only experimental studies were included, omitting many informative observational investigations that cover the range of implementation strategies.

    “For one, we only included experimental studies, omitting many informative observational investigations that cover the range of implementation strategies.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    We asked experts to inform us about any studies that they would include and cross-checked their lists with what was identified through our search terms, which yielded very few additional studies.

    “To manage this, we asked experts to inform us about any studies that they would include and cross-checked their lists with what was identified through our search terms, which yielded very few additional studies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given

    The study period was centered on the creation of the journal Implementation Science and not on the standardization and operationalization of implementation strategies in the publication of the ERIC taxonomy (which came later).

    “Second, our study period was centered on the creation of the journal Implementation Science and not on the standardization and operationalization of implementation strategies in the publication of the ERIC taxonomy (which came later).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    These findings support prior recommendations for improved empirical rigor in implementation studies.

    “These findings support prior recommendations for improved empirical rigor in implementation studies [46, 95].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [46] Powell BJ, Fernandez ME, Williams NJ, Aarons GA, Beidas RS, Lewis CC, et al. Enhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda. Front Public Health [Internet]. 2019 [cited 2021 Mar 31];7. Available from: https://www.frontiersin.org/articles/https://doi.org/10.3389/fpubh.2019.00003/full source

      Not checked in the free run.

    • cites [95] Chambers DA, Emmons KM. Navigating the field of implementation science towards maturity: challenges and opportunities. Implement Sci. 2024;19:26, s13012-024-01352–0. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The search terms may not have captured tests of policies, financial strategies, community health promotion initiatives, or electronic medical record reminders, due to differences in terminology used in corresponding subfields of research.

    “For example, the search terms may not have captured tests of policies, financial strategies, community health promotion initiatives, or electronic medical record reminders, due to differences in terminology used in corresponding subfields of research (e.g., health economics, business, health information technology, and health policy).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    This study reaffirmed the need to apply not only a standard naming convention (e.g., ERIC) but also a standard reporting for implementation strategies.

    “This study reaffirmed the need to apply not only a standard naming convention (e.g., ERIC) but also a standard reporting of for implementation strategies.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Some have commented that ERIC strategies can be unclear and are missing key domains.

    “Also, some have commented that ERIC strategies can be unclear and are missing key domains.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    We included standard coding using the ERIC taxonomy, which was a strength.

    “We included standard coding using the ERIC taxonomy, which was a strength, but future work should consider including the additional strategies that have been recommended to augment ERIC”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The broad terms for implementation strategies may have led to inadvertent omissions of studies of specific strategies.

    “As such, the broad terms for implementation strategies (e.g., “evidence-based interventions”[7] or “behavior change techniques” [107]) may have led to inadvertent omissions of studies of specific strategies.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [7] Community Preventive Services Task Force. Community Preventive Services Task Force: All Active Findings June 2023 [Internet]. 2023 [cited 2023 Aug 7]. Available from: https://www.thecommunityguide.org/media/pdf/CPSTF-All-Findings-508.pdf

      Not checked in the free run.

    • cites [107] Michie S, Atkins L, West R. The behaviour change wheel. Guide Des Interv 1st Ed G B Silverback Publ. 2014;1003:1010.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    We addressed this particular concern by reviewing other systematic reviews of implementation strategies and soliciting recommendations from global implementation science experts.

    “We addressed this particular concern by reviewing other systematic reviews of implementation strategies and soliciting recommendations from global implementation science experts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    There were many studies evaluating numerous outcomes, risking data dredging.

    “Moreover, there were many studies evaluating numerous outcomes, risking data dredging.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    Indexing approaches used by academic databases may have missed relevant studies.

    “Additionally, indexing approaches used by academic databases may have missed relevant studies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Strategy listings like ERIC are only useful when they are widely adopted and used in conjunction with guidelines for specifying and reporting strategies.

    “Another potential limitation comes from the ERIC taxonomy itself—i.e., strategy listings like ERIC are only useful when they are widely adopted and used in conjunction with guidelines for specifying and reporting strategies [1] in protocol and outcome papers.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1] Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    We addressed this limitation by being cautious in our reporting of findings, specifically in identifying effective implementation strategies.

    “We addressed this limitation by being cautious in our reporting of findings, specifically in identifying “effective” implementation strategies.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    It was challenging in the present review to disentangle the impact of the intervention(s) (i.e., the EBP) versus the impact of the implementation strategy(ies) for each RE-AIM dimension.

    “It was challenging in the present review to disentangle the impact of the intervention(s) (i.e., the EBP) versus the impact of the implementation strategy(ies) for each RE-AIM dimension.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning

    Although the ERIC paper has been widely cited (over three thousand times, accessed about 186 thousand times), it is still not universally applied, making tracking the impact of specific strategies more difficult.

    “Although the ERIC paper has been widely cited (over three thousand times, accessed about 186 thousand times), it is still not universally applied, making tracking the impact of specific strategies more difficult.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    We found the utility of the ERIC taxonomy to outweigh any of the taxonomy's current limitations.

    “In sum, we found the utility of the ERIC taxonomy to outweigh any of the taxonomy’s current limitations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Understanding the effectiveness of discrete strategies deployed alone or in combination requires reliable and low burden tracking methods to collect information about strategy use and outcomes.

    “Understanding the effectiveness of discrete strategies deployed alone or in combination requires reliable and low burden tracking methods to collect information about strategy use and outcomes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    This could have resulted in over- or underestimating the results of our synthesis.

    “This could have resulted in over- or underestimating the results of our synthesis.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    As with all reviews, the search terms influenced our findings.

    “As with all reviews, the search terms influenced our findings.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    To address this limitation, strategies were retroactively mapped to ERIC, but it is possible that some studies were missed.

    “To address this limitation, we retroactively mapped strategies to ERIC, but it is possible that some studies were missed.”
    Quote in the paperNot a substantial claim
73 of 73

  • The paper relies on this without support.

    Implementation strategies are intended to have broad applicability.

    “Given that these strategies are intended to have broad applicability, there remains a need to understand how well implementation strategies work across EBPs and settings and the extent to which implementation knowledge is generalizable.”

  • The paper relies on this without support.

    At least one significantly positive RE-AIM outcome is an adequate indicator of a strategy's supportive evidence.

    “We classified the strength of evidence for each strategy by evaluating both the number of studies in which each strategy appeared in the Experimental Arm and the percentage of times there was at least one significantly positive RE-AIM outcome.”

  • The paper relies on this without support.

    The median split (8.5 studies and 75% positive outcomes) appropriately distinguishes stronger from weaker evidence.

    “Due to the non-normal distribution of both factors, we used the median (rather than the mean) to create four quadrants.”

  • The paper relies on this without support.

    Implementation strategies often synergize.

    “Because strategies often synergize, disaggregated results could either underestimate the true impact of individual strategies or conversely, actually undermine their effectiveness”

  • The paper relies on this without support.

    The complexity of health and human service settings, the imperative to improve public health outcomes, and engagement with community partners often require the use of multiple strategies simultaneously.

    “The complexity of health and human service settings, imperative to improve public health outcomes, and engagement with community partners often requires the use of multiple strategies simultaneously.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A team-based approach is associated with improved outcomes.

    “A team-based approach is associated with improved outcomes 20”

    From Structured Development of Learning and Assessment Tasks to Prevent Generative AI Misuse and Enhance AI Literacy in the Faculty in Physiotherapy Education (2025) · cites this paper as [20]

    The Results (Tested strategies; page unavailable) identify clinician implementation team meetings as a commonly tested strategy. The review also assesses positive outcomes for implementation strategies, but the passages do not establish that a team-based approach specifically was associated with improved outcomes. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • A team-based approach is better suited to understand and address end-user needs, offering diverse perspectives.

    “and is better suited to understand and address end-user needs, offering diverse perspectives.20,21”

    From Structured Development of Learning and Assessment Tasks to Prevent Generative AI Misuse and Enhance AI Literacy in the Faculty in Physiotherapy Education (2025) · cites this paper as [20]

    The Results (Descriptive results; page unavailable) list clinician implementation team meetings and stakeholder-engagement strategies, but do not assess whether teams are better suited to understand end-user needs or offer diverse perspectives. Searches for “end-user needs,” “diverse perspectives,” and team-based approaches found no passage making that claim. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Overall ICU-setting percentage is incorrect

    Minor

    Five of the 129 studies is 3.88%, which rounds to 4% rather than the reported 5%.

    Reported 5 · recomputed 3.875969

  • U.S. "Other" setting percentage is incorrect

    Minor

    The table gives 56 U.S. studies; 15 of 56 is 26.79%, not 58%.

    Reported 58 · recomputed 26.785714

  • Control-arm educational-material counts disagree

    Minor

    The prose reports 52 control arms using Distribute Educational Materials, whereas Table 2 reports 51.

  • Blueprint strategy appears more often in experimental arms than overall

    Minor

    The table counts this strategy in 42 experimental arms but only 41 studies overall. An overall count covering any study arm cannot be smaller than its experimental-arm count.

  • Paired-screening percentage does not match the abstract count

    Minor

    The reported 4,075 abstracts are 43.36% of the 9,399 abstracts screened, not 48%.

    Reported 48 · recomputed 43.355676

  • Control-arm educational-meeting counts disagree

    Minor

    The prose reports 30 control arms using Conduct Educational Meetings, whereas Table 2 reports 29.

  • Positive-outcome percentage footnote gives the wrong denominator

    Minor

    The footnote says the positive-outcome percentages use all 129 studies as their denominator. For example, the reported 58% for Reach is 18 of the 31 studies assessing Reach, not 18 of 129.

  • U.S. primary-care percentage is incorrect

    Minor

    The table gives 56 U.S. studies; 21 of 56 is 37.5%, rather than 36%.

    Reported 36 · recomputed 37.5

8 of 8

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Dopp AR, Kerns SEU, Panattoni L, Ringel JS, Eisenberg D, Powell BJ, et al. Translating economic evaluations into financing strategies for implementing evidence-based practices. Implement Sci IS. 2021;16:66. · DOI · Paper page
  2. Michie S, Atkins L, West R. The behaviour change wheel. Guide Des Interv 1st Ed G B Silverback Publ. 2014;1003:1010.

Cited by

Sorted by citations

Related

Authors
Ashcraft, L. E., Goodrich, D. E., Hero, J., Phares, A., Bachrach, R. L., Quinn, D. A., Qureshi, N., Ernecoff, N. C., Lederer, L. G., Scheunemann, L. P., Rogal, S. S., & Chinman, M. J.
Affiliations
  • Philadelphia VA Medical Center (US)
  • University of Pittsburgh (US)
  • RAND Corporation (US)
  • VA Pittsburgh Healthcare System (US)
  • University of Michigan (US)
Corresponding author
Ashcraft, Laura Ellen
ORCID
Venue
Implementation Science
Publisher
Springer Science and Business Media LLC
Published
2024-06-24
Volume
19
Issue
1
Pages
43-43
ISSN
1748-5908 (ISSN-L)
DOI
10.1186/s13012-024-01369-5
PMC
PMC11194895
OpenAlex
W4399975434
Semantic Scholar
f2cff0e46f60639de8787c31296aa5197bea7f3a
Cited by
89
References
116
Citations by year
2026: 44 · 2025: 38 · 2024: 7
FWCI
8.00
Type
journal article
Language
en
Update notices
correction (2024-07-24)
Open access
Licence
Locations
  • Implementation Science (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • University of Pittsburgh (K23-DA048182)
  • Health Services Research and Development (PEC 19-207, CDA 20-057, CDA 20-224)
  • Agency for Healthcare Research and Quality (K08HS027210)
  • Patient-Centered Outcomes Research Institute (AOSEPP2 Task Order 12)
  • VA Center for Health Equity Research and Promotion (CIN 13-405)
Topics
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
Keywords
implementation strategies, implementation research, healthcare implementation, RE-AIM framework, audit and feedback, educational meetings, external facilitation, teaching materials, implementation outcomes
MeSH terms
Implementation Science, Delivery of Health Care, Humans

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review