CoChat
Sign Up
Cited by 490Health Policy Implementation ScienceHealth Sciences Research and EducationClinical practice guidelines implementation

Ten recommendations for using implementation frameworks in research and practice

Open access
Moullin, Joanna C., Dickson, Kelsey S., Stadnick, Nicole A., Albers, Bianca, Nilsen, Per, Broder-Fingert, Sarabeth, Mukasa, Barbara, Aarons, Gregory A.·2020·Implementation Science Communications
DOI 10.1186/s43058-020-00023-7PMC PMC7427911
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Recent reviews of the use and application of implementation frameworks in implementation efforts highlight the limited use of frameworks, despite the value in doing so. As such, this article aims to provide recommendations to enhance the application of implementation frameworks, for implementation researchers, intermediaries, and practitioners. Discussion: Ideally, an implementation framework, or multiple frameworks should be used prior to and throughout an implementation effort. This includes both in implementation science research studies and in real-world implementation projects. To guide this application, outlined are ten recommendations for using implementation frameworks across the implementation process. The recommendations have been written in the rough chronological order of an implementation effort; however, we understand these may vary depending on the project or context: (1) select a suitable framework(s), (2) establish and maintain community stakeholder engagement and partnerships, (3) define issue and develop research or evaluation questions and hypotheses, (4) develop an implementation mechanistic process model or logic model, (5) select research and evaluation methods (6) determine implementation factors/determinants, (7) select and tailor, or develop, implementation strategy(s), (8) specify implementation outcomes and evaluate implementation, (9) use a framework(s) at micro level to conduct and tailor implementation, and (10) write the proposal and report. Ideally, a framework(s) would be applied to each of the recommendations. For this article, we begin by discussing each recommendation within the context of frameworks broadly, followed by specific examples using the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Summary: The use of conceptual and theoretical frameworks provides a foundation from which generalizable implementation knowledge can be advanced. On the contrary, superficial use of frameworks hinders being able to use, learn from, and work sequentially to progress the field. Following the provided ten recommendations, we hope to assist researchers, intermediaries, and practitioners to improve the use of implementation science frameworks.

AI snapshot

Written from the paper’s full text.

TL;DR

This article provides ten recommendations for selecting and applying implementation frameworks throughout implementation research and practice, illustrated with examples using the EPIS framework. It argues that comprehensive framework use supports generalizable implementation knowledge, while superficial use hinders learning and progress in the field.

Key findings

  • The paper outlines ten recommendations in rough chronological order: (1) select a suitable framework(s), (2) establish and maintain community stakeholder engagement and partnerships, (3) define issue and develop research or evaluation questions and hypotheses, (4) develop an implementation mechanistic process model or logic model, (5) select research and evaluation methods, (6) determine implementation factors/determinants, (7) select and tailor, or develop, implementation strategy(s), (8) specify implementation outcomes and evaluate implementation, (9) use a framework(s) at micro level to conduct and tailor implementation, and (10) write the proposal and report.
  • Framework selection should consider four criteria: the purpose of the framework, the level(s) included, the degree of inclusion and depth of analysis or operationalization of implementation concepts, and the framework’s orientation (setting and type of intervention).
  • The article recommends that frameworks be used prior to and throughout an implementation effort; it states that suboptimal use can result in wasted resources, erroneous conclusions, specification errors in implementation methods and data analyses, and attenuated reviews of funding applications.
  • An international four-member stakeholder panel—including at least one stakeholder from each target audience (researchers, evidence-based practice, intermediaries, and practitioners)—reviewed the recommendations and tested/trialed two tools, which were refined according to their suggestions.

Methods

  • Design: A discussion/recommendations article, not an empirical study; recommendations were developed from a recent systematic review and the collective experience of the co-authors applying the EPIS framework in national and international implementation efforts.
  • Data or sample: No primary dataset; an international four-member stakeholder panel reviewed the recommendations and tested/trialed the tools. EPIS application examples are drawn from cited studies, including JJ-TRIALS, LOCI, ATTAIN, and a school nursing trial to reduce LGBTQ adolescent suicide.
  • Main analysis: Narrative synthesis with applied examples; the article provides two tools—the Implementation Framework Application Worksheet and the Implementation Framework Utilization Checklist (Additional files 1 and 2).

Limitations

  • The paper states that the sequence of recommendations is not prescriptive; recommendations one to three in particular may vary or occur concurrently, and applying all recommendations may be unrealistic under real-world resource constraints, requiring strategic selection of frameworks.
  • It states that depending on the stage in the implementation process, it may not be necessary to apply all recommendations; those early in the exploration phase may not be able to produce process or logic models or articulate mechanisms yet.
  • Not stated in the paper: there is no formal limitations section. Evident from the text: the recommendations and tools were not empirically tested beyond stakeholder review/trial, and the tools themselves are in additional files not included in the provided text.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s43058-020-00023-7

Cites

81 of 81 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 60 cited sources checked · 0 supported · 0 flagged

  • Premisecitation

    Frameworks provide explicit guidance and justification for proposed hypotheses to be tested that strengthen proposals, projects, trials, and products, both research and practice based.

    “frameworks provide explicit guidance and justification for proposed hypotheses to be tested that strengthen proposals, projects, trials, and products, both research and practice based [2, 4].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [2] Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53. source

      Not checked in the free run.

    • cites [4] Proctor EK, Powell BJ, Baumann AA, Hamilton AM, Santens RL. Writing implementation research grant proposals: ten key ingredients. Implement Sci. 2012;7:96. source

      Not checked in the free run.

  • Interpretationreasoning

    Poor use of frameworks can slow or misguide the progress of implementation science as a field.

    “the poor use of frameworks goes beyond proposals and projects and can slow or misguide the progress of implementation science as a field.”
    Quote in the paperSubstantial claim
  • Premisecitation

    Recent reviews and analyses of implementation science studies show limited or suboptimal use of implementation frameworks.

    “Recent reviews of the use and application of implementation frameworks in implementation efforts highlight the limited use of frameworks, despite the value in doing so.”
    • Also stated in: § Background · plain
      “Recent analyses of implementation science studies show suboptimal use of implementation frameworks [1, 8].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [1] Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implement Sci. 2019;14:1. source

      Not checked in the free run.

    • cites [8] Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139. source

      Not checked in the free run.

  • Premisecitation

    Suboptimal use of frameworks can impact the viability and success of implementation efforts.

    “Suboptimal use of frameworks can impact the viability and success of implementation efforts [4].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [4] Proctor EK, Powell BJ, Baumann AA, Hamilton AM, Santens RL. Writing implementation research grant proposals: ten key ingredients. Implement Sci. 2012;7:96. source

      Not checked in the free run.

  • Premisecitation

    Use of frameworks to explicitly guide hypothesis formation is the minority, even within implementation efforts using theory to guide other aspects of the research process.

    “Despite its explanatory power, use of frameworks to explicitly guide hypothesis formation are the minority, even within implementation efforts using theory to guide other aspects of the research process [38–40].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [38] Birken SA, Powell BJ, Shea CM, Haines ER, Alexis Kirk M, Leeman J, Rohweder C, Damschroder L, Presseau J. Criteria for selecting implementation science theories and frameworks: results from an international survey. Implement Sci. 2017;12:124. source

      Not checked in the free run.

    • cites [39] Davies P, Walker AE, Grimshaw JM. A systematic review of the use of theory in the design of guideline dissemination and implementation strategies and interpretation of the results of rigorous evaluations. Implement Sci. 2010;5:14. source

      Not checked in the free run.

    • cites [40] Johnson AM, Moore JE, Chambers DA, Rup J, Dinyarian C, Straus SE. How do researchers conceptualize and plan for the sustainability of their NIH R01 implementation projects? Implement Sci. 2019;14:50. source

      Not checked in the free run.

  • Classified as probably not a substantial claim. 37
  • May combine two claims. 13
  • The quote only partly states this claim. 1

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    For intermediaries and practitioners, implementation frameworks can provide shared language to familiarize stakeholders with implementation and function as practical tools for planning, executing, and evaluating real-world implementation efforts.

    “For intermediaries and practitioners, they can provide shared language to familiarize stakeholders with implementation and function as practical tools for planning, executing, and evaluating real-world implementation efforts.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [1] Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implement Sci. 2019;14:1. source

      Not checked in the free run.

    • cites [2] Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    There is great value in effectively using implementation frameworks, models, and theories.

    “There is great value in effectively using implementation frameworks, models, and theories [1, 2].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [1] Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implement Sci. 2019;14:1. source

      Not checked in the free run.

    • cites [2] Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53. source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Implementation frameworks can provide a structure for describing or guiding the process of translating effective interventions and research evidence into practice, analyzing what influences implementation outcomes, and evaluating implementation efforts.

    “Implementation frameworks can provide a structure for the following: (1) describing and/or guiding the process of translating effective interventions and research evidence into practice (process frameworks), (2) analyzing what influences implementation outcomes (determinant frameworks), and (3) evaluating implementation efforts (outcome frameworks) [2].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [2] Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Suboptimal use of frameworks can result in wasted resources, erroneous conclusions, specification errors in implementation methods and data analyses, and attenuated reviews of funding applications.

    “This can result in wasted resources, erroneous conclusions, specification errors in implementation methods and data analyses, and attenuated reviews of funding applications [5].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5] Pedhazur EJ. Multiple regression in behavioral research: explanation and prediction. 2nd ed. Fort Worth, TX: Harcourt Brace; 1982.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Use of frameworks to inform the articulation of an implementation need and the development of practice-related or research questions and hypotheses has the potential to optimize implementation efforts and outcomes.

    “Use of frameworks to inform the articulation of an implementation need (i.e., a research-practice gap) and the development of practice-related or research questions and hypotheses has the potential to optimize implementation efforts and outcomes [2].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [2] Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    The applied examples are based on a recent systematic review and the co-authors' collective experience applying the EPIS framework in national and international implementation efforts.

    “These applied examples are based on the results of a recent systematic review [1], and the collective experience of the co-authors applying the EPIS framework in national and international implementation efforts.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1] Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implement Sci. 2019;14:1. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Involving stakeholders in defining local health system needs and selecting EBP(s) and/or implementation strategies enhances their collaboration and ownership of the implementation effort.

    “having them involved in defining the local health system needs and selecting EBP(s) and/or implementation strategies in the EPIS implementation phase, as these are important to enhance their collaboration and ownership of the implementation effort [26].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [26] Miller WL, Rubinstein EB, Howard J, Crabtree BF. Shifting implementation science theory to empower primary care practices. Ann Fam Med. 2019;17:250–6. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Engaging stakeholders across multiple levels of an implementation ecosystem is recommended best practice for implementation researchers.

    “Engaging stakeholders across multiple levels of an implementation ecosystem (e.g., policy/legislative, funders, community, organizational, provider, client/patient) is recommended best practice for implementation researchers [30] and as indicated in the multi-level nature of the majority of implementation frameworks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30] Brookman-Frazee L, Stahmer A, Stadnick N, Chlebowski C, Herschell A, Garland AF. Characterizing the use of research-community partnerships in studies of evidence-based interventions in children’s community services. Adm Policy Ment Hlth. 2016;43:93–104. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Being specific about the problem to be addressed entails being precise about the behaviors you are trying to change and whose behavior is involved.

    “Being specific about the problem to be addressed entails being precise about the behaviors you are trying to change and whose behavior is involved [36].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [36] Michie S, Atkins L, West R. The behaviour change wheel: a guide to designing interventions. Great Britain: Silverback Publishing; 2014.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Frameworks facilitate the framing and formulation of implementation questions, including those related to needs assessment.

    “frameworks facilitate the framing and formulation of implementation questions, including those related to needs assessment”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    In real-world practice, suboptimal use of implementation frameworks can lead stakeholders to misjudge their implementation context or develop inappropriate implementation strategies.

    “In real-world practice, the suboptimal use of implementation frameworks can lead stakeholders to misjudge their implementation context or develop inappropriate implementation strategies.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Study design and conductreasoning

    An international four-member stakeholder panel reviewed the recommendations and tools, and the tools were refined according to their suggestions.

    “To ensure that the recommendations are clear, practical, and comprehensive, we invited an international stakeholder panel who come from different perspectives (e.g., researcher, NGO administrator, intermediary, provider/physician) to review the recommendations and consider their utility applied to their implementation efforts.”
    • Also stated in: § Recommendations for using implementation framework(s) · plain
      “The tools were refined according to their suggestions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    There is a need for explicit guidance on the use of implementation frameworks generally.

    “there is a need for explicit guidance on the use of frameworks generally.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Implementation frameworks generally encourage stakeholder engagement prior to funding and for it to continue during implementation effort justification and as part of future implementation iterations and adaptations.

    “Implementation frameworks generally encourage stakeholder engagement prior to funding, and for it to continue during implementation effort justification and as part of future implementation iterations and adaptations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Poor use of frameworks can slow the translation of research evidence into practice and thereby limit public health impact.

    “poor use of frameworks can slow the translation of research evidence into practice, and thereby limit public health impact.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    When used in research, implementation frameworks can guide the design and conduct of studies, inform the theoretical and empirical thinking of research teams, and aid interpretation of findings.

    “When used in research, they can guide the design and conduct of studies, inform the theoretical and empirical thinking of research teams, and aid interpretation of findings.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [1] Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implement Sci. 2019;14:1. source

      Not checked in the free run.

    • cites [2] Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Stakeholder engagement is an integral component of implementation.

    “Stakeholder engagement is an integral component of implementation [25, 26].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [25] Drahota A, Meza R, Brikho G, Naaf M, Estabillo J, Spurgeon E, Vejnoska S, Dufek E, Stahmer AC, Aarons GA. Community-academic partnerships: a systematic review of the state of the literature and recommendations for future research. Milbank Q. 2016;94:163–214.. source

      Not checked in the free run.

    • cites [26] Miller WL, Rubinstein EB, Howard J, Crabtree BF. Shifting implementation science theory to empower primary care practices. Ann Fam Med. 2019;17:250–6. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationcitation

    Comprehensive framework application can help address identified priorities for enhancing the impact of implementation strategies.

    “Additionally, comprehensive framework(s) application can help address identified priorities (e.g., methods for tailoring strategies, specifying, and testing mechanisms) for enhancing the impact of implementation strategies [63].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [63] Powell BJ, Fernandez ME, Williams NJ, Aarons GA, Beidas RS, Lewis CC, McHugh SM, Weiner BJ. Enhancing the impact of implementation strategies in healthcare: a research agenda. Front Public Health. 2019;7:3. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    EPIS was used to code qualitative data and select quantitative survey measures in the LOCI study.

    “In addition, to the developed implementation model and implementation strategy, EPIS was used to code qualitative data and select quantitative survey measures.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [55] Aarons GA, Ehrhart MG, Moullin JC, Torres EM, Green AE. Testing the leadership and organizational change for implementation (LOCI) intervention in substance abuse treatment: a cluster randomized trial study protocol. Implement Sci. 2017;12:29. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Mechanism measurement and evaluation is critical to increase the rigor and relevance of implementation science.

    “Mechanism measurement and evaluation is critical to increase the rigor and relevance of implementation science [46].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [46] Geng E, Peiris D, Kruk ME. Implementation science: relevance in the real world without sacrificing rigor. PLOS Med. 2017;14:e1002288. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Each of the four matching approaches is highly participatory but varies in strengths and weaknesses of application.

    “Each approach is highly participatory but varies in strengths and weaknesses of application.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [62] Powell BJ, Beidas RS, Lewis CC, Aarons GA, McMillen JC, Proctor EK, Mandell DS. Methods to improve the selection and tailoring of implementation strategies. J Behav Health Serv Res. 2017;44:177–94. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The DAP uses an iterative, data-informed approach to facilitate implementation across each phase of EPIS.

    “The DAP uses an iterative, data-informed approach to facilitate implementation across each phase of EPIS.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [69] Aarons GA, Green AE, Palinkas LA, Self-Brown S, Whitaker DJ, Lutzker JR, Silovsky JF, Hecht DB, Chaffin MJ. Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implement Sci. 2012;7:32. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Proctor et al. outlined that implementation strategies should be named, defined, and specified based on seven components: actor, action, action targets, temporality, dose, outcomes, and theory/justification.

    “Furthermore, as outlined by Proctor et al. [8], implementation strategies should be named, defined, and specified, based on seven components enabling their measurement and replication: actor, action, action targets, temporality (when), dose (duration and how often), outcomes, and theory/justification.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [8] Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A critical and core component of the DAP is the creation of an Implementation Resource Team, a multiple stakeholder collaborative designed to support implementation, data interpretation, and explicitly address adaptations during the implementation process.

    “A critical and core component of the DAP is the creation of an Implementation Resource Team that is a multiple stakeholder collaborative designed to support implementation, data interpretation, and explicitly address adaptations during the implementation process.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [69] Aarons GA, Green AE, Palinkas LA, Self-Brown S, Whitaker DJ, Lutzker JR, Silovsky JF, Hecht DB, Chaffin MJ. Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implement Sci. 2012;7:32. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The PARIHS framework can be used to identify critical evidentiary and contextual elements that may impact EBP implementation.

    “For example, the PARIHS framework [67] can be used to identify critical evidentiary (e.g., patient experience, information from the local setting) and contextual (e.g., leadership, receptive context) elements that may impact EBP implementation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [67] Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. BMJ Qual Saf. 1998;7:149–58. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In that study, EPIS and DAP were used to inform constructs and measurement of implementation outcomes in the sustainment phase.

    “(4) inform the constructs and measurement of the implementation outcomes in the sustainment phase.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [68] Willging CE, Green AE, Ramos MM. Implementing school nursing strategies to reduce LGBTQ adolescent suicide: a randomized cluster trial study protocol. Implement Sci. 2016;11:145. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    The BCW methodology guides users to conduct a thorough behavioral diagnosis using the COM-B and to then prioritize which behaviors to address.

    “For example, the BCW methodology guides users to conduct a thorough behavioral diagnosis using the COM-B and to then prioritize which behaviors to address.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In that study, EPIS and DAP were used to inform constructs measured in the multi-level needs assessment during the exploration phase.

    “Within this study, the EPIS framework and the DAP were used to (1) inform the constructs measured in the multi-level needs assessment during the exploration phase”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [68] Willging CE, Green AE, Ramos MM. Implementing school nursing strategies to reduce LGBTQ adolescent suicide: a randomized cluster trial study protocol. Implement Sci. 2016;11:145. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Assessing implementation determinants using a theoretical framework helps specify the target of the implementation effort.

    “Another important reason for assessing implementation determinants using a theoretical framework is to specify the target of the implementation effort.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Implementation strategies vary in purpose and complexity, ranging from discrete strategies to multifaceted branded strategies that integrate at least two discrete strategies.

    “Implementation strategies vary in purpose and complexity, ranging from discrete strategies [56] such as audit and feedback [57] to multifaceted, and often branded, strategies that integrate at least two discrete strategies”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [56] Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, Glass JE, York JL. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2012;69:123–57. source

      Not checked in the free run.

    • cites [57] Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O’Brien MA, Johansen M, Grimshaw J, Oxman AD: Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews 2012. source

      Not checked in the free run.

    • cites [37] Aarons GA, Ehrhart MG, Farahnak LR, Hurlburt MS. Leadership and organizational change for implementation (LOCI): a randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implement Sci. 2015;10:11. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Frameworks can help identify and/or expand the implementation determinants or aspects assumed to impact the implementation process at different levels and across multiple phases for consideration or measurement.

    “frameworks can help to identify and/or expand the implementation determinants or aspects assumed to impact the implementation process at different levels and across multiple phases for consideration or measurement.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    The distinct aims and purposes of implementation efforts require distinct evaluation designs such as mixed-methods, hybrid effectiveness-implementation, and quality improvement approaches.

    “The distinct aims and purposes of implementation efforts require distinct evaluation designs such as mixed-methods, hybrid effectiveness-implementation, and quality improvement approaches including formative evaluations or Plan-Do-Study-Act cycles [48].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [48] Landsverk J, Brown CH, Chamberlain P, Palinkas L, Ogihara M, Czaja S, Goldhaber-Fiebert JD, Rolls Reutz J, McCue Horwitz S. Design and analysis in dissemination and implementation research. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and Implementation Research in Health: Translating Science to Practice. New York, NY: Oxford University Press; 2012.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Stetler et al. provide four progressive yet integrated stages of formative evaluation.

    “Stetler et al. [71] provide four progressive yet integrated stages of formative evaluation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [71] Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, Kimmel B, Sharp ND, Smith JL. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006;21:S1–8. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Powell and colleagues outlined four primary methods for matching implementation strategies to barriers: conjoint analysis, intervention mapping, concept mapping, group model building.

    “Powell and colleagues have outlined four primary methods for matching implementation strategies to barriers (conjoint analysis, intervention mapping, concept mapping, group model building) [62].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [62] Powell BJ, Beidas RS, Lewis CC, Aarons GA, McMillen JC, Proctor EK, Mandell DS. Methods to improve the selection and tailoring of implementation strategies. J Behav Health Serv Res. 2017;44:177–94. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    Prospective framework application may reveal different determinants and aid selection of particular influencing factors to target during subsequent implementation studies.

    “Prospective framework application may also reveal different determinants and aid selection of particular influencing factors to target during subsequent implementation studies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In the LOCI study, implementation leadership, organizational climate and culture, implementation climate, and psychological safety climate were selected as determinants to test their influence on the fidelity of the EBP being implemented.

    “In this study, implementation leadership from first-level team leaders/managers, organizational climate and culture, implementation climate, and psychological safety climate were selected as determinants to test their influence on the fidelity of the EBP being implemented.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [55] Aarons GA, Ehrhart MG, Moullin JC, Torres EM, Green AE. Testing the leadership and organizational change for implementation (LOCI) intervention in substance abuse treatment: a cluster randomized trial study protocol. Implement Sci. 2017;12:29. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Implementation frameworks can inform program logics that describe anticipated relationships between inputs, activities, outputs, and implementation and client outcomes, thereby supporting explicit formulation of key assumptions and outlining of crucial project details.

    “Within research and practice projects, implementation frameworks can inform the program logics that describe the anticipated relationships between inputs, activities, outputs, and implementation and client outcomes, thereby supporting the explicit formulation of key assumptions and outlining of crucial project details.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Existing standards and guidelines such as STaRI exist to improve transparent and accurate reporting of implementation studies.

    “Consistent with these recommendations are existing standards and guidelines to improve transparent and accurate reporting of implementation studies such as the Standards for Reporting Implementation Studies (STaRI; Pinnock et al. [85]).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [85] Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, Rycroft-Malone J, Meissner P, Murray E, Patel A, Sheikh A. Standards for reporting implementation studies (StaRI) statement. bmj. 2017;356:i6795. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Proctor and colleagues provide a taxonomy for classifying and reporting types of implementation research outcomes that includes guidance on level of analysis and measurement, theoretical basis, and maps salience of outcome onto phases of implementation.

    “Again, the work of Proctor and colleagues [22] provides a useful taxonomy for classifying and reporting types of implementation research outcomes that also includes guidance regarding level of analysis and measurement, theoretical basis, and maps the salience of outcome onto the phases of implementation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] Proctor EK, Silmere H, Raghavan R, Hovmand P, Aarons GA, Bunger A, Griffey R, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Hlth. 2011;38:65–76. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In that study, EPIS and DAP were used to support identification of stakeholders and activities involved in the Implementation Resource Team developed in the preparation phase.

    “(2) support the identification of the stakeholders and activities involved in the Implementation Resource Team that was developed in the preparation phase”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [68] Willging CE, Green AE, Ramos MM. Implementing school nursing strategies to reduce LGBTQ adolescent suicide: a randomized cluster trial study protocol. Implement Sci. 2016;11:145. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Frameworks can help operationalize constructs of importance to an evaluation and the identification of suitable measures.

    “They can also help to operationalize constructs of importance to an evaluation and the identification of suitable measures.”
    Quote in the paperSubstantial claim
41 of 41

  • The paper relies on this without support.

    Following the ten recommendations will improve the use of implementation science frameworks by researchers, intermediaries, and practitioners.

    “Following the provided ten recommendations, we hope to assist researchers, intermediaries, and practitioners to improve the use of implementation science frameworks.”

  • The paper relies on this without support.

    Implementation frameworks are necessary for selecting, tailoring, or developing implementation strategies.

    “Implementation frameworks are necessary for selecting, tailoring, or developing implementation strategies.”

  • The paper relies on this without support.

    Implementation is a dynamic, context-specific process.

    “Implementation is a dynamic, context-specific process.”

  • The paper relies on this without support.

    Documenting the use of implementation frameworks is vital for an implementation project to be replicable and analyzable.

    “As part of this documentation, detailing the use of an implementation framework(s) is vital for the implementation project to be replicable and analyzable.”

  • The paper relies on this without support.

    Superficial use of implementation frameworks hinders researchers' and practitioners' learning and efforts to sequentially progress the field.

    “Superficial use of frameworks hinders researchers’ and practitioners’ learning and efforts to sequentially progress the field.”
5 of 5

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Determinants are assigned the wrong recommendation number

    Minor

    The text directs readers to recommendation 5 for identifying barriers and facilitators, but the numbered list assigns implementation determinants to recommendation 6; recommendation 5 concerns research and evaluation methods.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. World Health Organization. Changing mindsets: strategy on health policy and systems research. Geneva, Switzerland: World Health Organization; 2012.
  2. Lewis C, Boyd M, Beidas R, Lyon A, Chambers D, Aarons G, Mittman B: A research agenda for mechanistic dissemination and implementation research. In Conference on the Science of Dissemination and Implementation; Bethesda, MD. 2015.
  3. Suhrheinrich J, et al. Exploring inner-context factors associated with implementation outcomes in a randomized trial of classroom pivotal response teaching. Under Review.

Cited by

Sorted by citations

Related

Authors
Moullin, J. C., Dickson, K. S., Stadnick, N. A., Albers, B., Nilsen, P., Broder-Fingert, S., Mukasa, B., & Aarons, G. A.
Affiliations
  • Curtin University (AU)
  • San Diego State University (US)
  • University of California San Diego (US)
  • The University of Melbourne (AU)
  • Mildmay Uganda (UG)
Corresponding author
Moullin, Joanna C.
ORCID
Venue
Implementation Science Communications
Publisher
Springer Science and Business Media LLC
Published
2020-04-30
Volume
1
Issue
1
Pages
42-42
ISSN
2662-2211 (ISSN-L)
DOI
10.1186/s43058-020-00023-7
PMC
PMC7427911
OpenAlex
W3025092189
Semantic Scholar
6f1f2fb71b6148c7b3f5e557b4c4f1c1286402bd
Cited by
488
References
81
Citations by year
2026: 67 · 2025: 115 · 2024: 91 · 2023: 101 · 2022: 61 · 2021: 45 · 2020: 7 · 2019: 1
FWCI
55.86
Type
journal article
Language
en
Open access
Licence
Locations
  • Implementation Science Communications (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • eScholarship (California Digital Library) (submittedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
  • eSpace (Curtin University) (submittedVersion, cc-by-nc-nd)
Funders
  • National Institute of Mental Health (R03MH117493-01, K23MH115100, K23MH110602, K23MH109673)
  • National Institute on Drug Abuse (R01DA038466)
Topics
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
Keywords
implementation framework, implementation science, EPIS framework, implementation process, stakeholder engagement, logic model, implementation strategies, implementation outcomes, implementation factors

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review