Ten recommendations for using implementation frameworks in research and practice

In the paper

“When used in research, they can guide the design and conduct of studies, inform the theoretical and empirical thinking of research teams, and aid interpretation of findings.”

What the sources say

• Supported

  The Background says implementation frameworks help select and structure research questions, methods, strategies, measures, and results. The Discussion recommends using them throughout the research process, from developing questions to presenting findings, and describes their role in testing and advancing knowledge. (LLM, conf=0.90)

  cites [1] Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implement Sci. 2019;14:1. source
• Partly supported

  The Discussion says implementation frameworks can identify factors influencing outcomes and provide a structure for evaluating intervention studies. That supports their use in planning evaluations and interpreting findings, but the passages do not specifically say they guide the conduct of studies or inform research teams’ theoretical and empirical thinking. (LLM, conf=0.78)

  cites [2] Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53. sourceExperimental — accuracy not yet measured

In the paper

“A critical and core component of the DAP is the creation of an Implementation Resource Team that is a multiple stakeholder collaborative designed to support implementation, data interpretation, and explicitly address adaptations during the implementation process.”

What the sources say

• Supported

  Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  cites [69] Aarons GA, Green AE, Palinkas LA, Self-Brown S, Whitaker DJ, Lutzker JR, Silovsky JF, Hecht DB, Chaffin MJ. Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implement Sci. 2012;7:32. source

In the paper

“For intermediaries and practitioners, they can provide shared language to familiarize stakeholders with implementation and function as practical tools for planning, executing, and evaluating real-world implementation efforts.”

What the sources say

• Not checked yetcites [1] Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implement Sci. 2019;14:1. source
• Not checked yetcites [2] Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53. source

In the paper

“There is great value in effectively using implementation frameworks, models, and theories [1, 2].”

What the sources say

• Not checked yetcites [1] Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implement Sci. 2019;14:1. source
• Not checked yetcites [2] Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53. source

In the paper

“Implementation frameworks can provide a structure for the following: (1) describing and/or guiding the process of translating effective interventions and research evidence into practice (process frameworks), (2) analyzing what influences implementation outcomes (determinant frameworks), and (3) evaluating implementation efforts (outcome frameworks) [2].”

What the sources say

• Not checked yetcites [2] Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53. source

In the paper

“This can result in wasted resources, erroneous conclusions, specification errors in implementation methods and data analyses, and attenuated reviews of funding applications [5].”

What the sources say

• Not checked yetcites [5] Pedhazur EJ. Multiple regression in behavioral research: explanation and prediction. 2nd ed. Fort Worth, TX: Harcourt Brace; 1982.

In the paper

“Use of frameworks to inform the articulation of an implementation need (i.e., a research-practice gap) and the development of practice-related or research questions and hypotheses has the potential to optimize implementation efforts and outcomes [2].”

What the sources say

• Not checked yetcites [2] Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53. source

In the paper

“These applied examples are based on the results of a recent systematic review [1], and the collective experience of the co-authors applying the EPIS framework in national and international implementation efforts.”

What the sources say

• Not checked yetcites [1] Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implement Sci. 2019;14:1. source

In the paper

“having them involved in defining the local health system needs and selecting EBP(s) and/or implementation strategies in the EPIS implementation phase, as these are important to enhance their collaboration and ownership of the implementation effort [26].”

What the sources say

• Not checked yetcites [26] Miller WL, Rubinstein EB, Howard J, Crabtree BF. Shifting implementation science theory to empower primary care practices. Ann Fam Med. 2019;17:250–6. source

In the paper

“Engaging stakeholders across multiple levels of an implementation ecosystem (e.g., policy/legislative, funders, community, organizational, provider, client/patient) is recommended best practice for implementation researchers [30] and as indicated in the multi-level nature of the majority of implementation frameworks.”

What the sources say

• Not checked yetcites [30] Brookman-Frazee L, Stahmer A, Stadnick N, Chlebowski C, Herschell A, Garland AF. Characterizing the use of research-community partnerships in studies of evidence-based interventions in children’s community services. Adm Policy Ment Hlth. 2016;43:93–104. source

In the paper

“Being specific about the problem to be addressed entails being precise about the behaviors you are trying to change and whose behavior is involved [36].”

What the sources say

• Not checked yetcites [36] Michie S, Atkins L, West R. The behaviour change wheel: a guide to designing interventions. Great Britain: Silverback Publishing; 2014.

In the paper

“frameworks facilitate the framing and formulation of implementation questions, including those related to needs assessment”

In the paper

“In real-world practice, the suboptimal use of implementation frameworks can lead stakeholders to misjudge their implementation context or develop inappropriate implementation strategies.”

In the paper

“To ensure that the recommendations are clear, practical, and comprehensive, we invited an international stakeholder panel who come from different perspectives (e.g., researcher, NGO administrator, intermediary, provider/physician) to review the recommendations and consider their utility applied to their implementation efforts.”

• Also stated in: § Recommendations for using implementation framework(s) · plain

  “The tools were refined according to their suggestions.”

In the paper

“there is a need for explicit guidance on the use of frameworks generally.”

In the paper

“Implementation frameworks generally encourage stakeholder engagement prior to funding, and for it to continue during implementation effort justification and as part of future implementation iterations and adaptations.”

In the paper

“poor use of frameworks can slow the translation of research evidence into practice, and thereby limit public health impact.”

In the paper

“Stakeholder engagement is an integral component of implementation [25, 26].”

What the sources say

• Not checked yetcites [25] Drahota A, Meza R, Brikho G, Naaf M, Estabillo J, Spurgeon E, Vejnoska S, Dufek E, Stahmer AC, Aarons GA. Community-academic partnerships: a systematic review of the state of the literature and recommendations for future research. Milbank Q. 2016;94:163–214.. source
• Not checked yetcites [26] Miller WL, Rubinstein EB, Howard J, Crabtree BF. Shifting implementation science theory to empower primary care practices. Ann Fam Med. 2019;17:250–6. source

In the paper

“Additionally, comprehensive framework(s) application can help address identified priorities (e.g., methods for tailoring strategies, specifying, and testing mechanisms) for enhancing the impact of implementation strategies [63].”

What the sources say

• Not checked yetcites [63] Powell BJ, Fernandez ME, Williams NJ, Aarons GA, Beidas RS, Lewis CC, McHugh SM, Weiner BJ. Enhancing the impact of implementation strategies in healthcare: a research agenda. Front Public Health. 2019;7:3. source

In the paper

“In addition, to the developed implementation model and implementation strategy, EPIS was used to code qualitative data and select quantitative survey measures.”

What the sources say

• Not checked yetcites [55] Aarons GA, Ehrhart MG, Moullin JC, Torres EM, Green AE. Testing the leadership and organizational change for implementation (LOCI) intervention in substance abuse treatment: a cluster randomized trial study protocol. Implement Sci. 2017;12:29. source

In the paper

“Mechanism measurement and evaluation is critical to increase the rigor and relevance of implementation science [46].”

What the sources say

• Not checked yetcites [46] Geng E, Peiris D, Kruk ME. Implementation science: relevance in the real world without sacrificing rigor. PLOS Med. 2017;14:e1002288. source

In the paper

“Each approach is highly participatory but varies in strengths and weaknesses of application.”

What the sources say

• Not checked yetcites [62] Powell BJ, Beidas RS, Lewis CC, Aarons GA, McMillen JC, Proctor EK, Mandell DS. Methods to improve the selection and tailoring of implementation strategies. J Behav Health Serv Res. 2017;44:177–94. source

In the paper

“The DAP uses an iterative, data-informed approach to facilitate implementation across each phase of EPIS.”

What the sources say

• Not checked yetcites [69] Aarons GA, Green AE, Palinkas LA, Self-Brown S, Whitaker DJ, Lutzker JR, Silovsky JF, Hecht DB, Chaffin MJ. Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implement Sci. 2012;7:32. source

In the paper

“Furthermore, as outlined by Proctor et al. [8], implementation strategies should be named, defined, and specified, based on seven components enabling their measurement and replication: actor, action, action targets, temporality (when), dose (duration and how often), outcomes, and theory/justification.”

What the sources say

• Not checked yetcites [8] Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139. source

In the paper

“For example, the PARIHS framework [67] can be used to identify critical evidentiary (e.g., patient experience, information from the local setting) and contextual (e.g., leadership, receptive context) elements that may impact EBP implementation.”

What the sources say

• Not checked yetcites [67] Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. BMJ Qual Saf. 1998;7:149–58. source

In the paper

“(4) inform the constructs and measurement of the implementation outcomes in the sustainment phase.”

What the sources say

• Not checked yetcites [68] Willging CE, Green AE, Ramos MM. Implementing school nursing strategies to reduce LGBTQ adolescent suicide: a randomized cluster trial study protocol. Implement Sci. 2016;11:145. source

In the paper

“For example, the BCW methodology guides users to conduct a thorough behavioral diagnosis using the COM-B and to then prioritize which behaviors to address.”

In the paper

“Within this study, the EPIS framework and the DAP were used to (1) inform the constructs measured in the multi-level needs assessment during the exploration phase”

What the sources say

• Not checked yetcites [68] Willging CE, Green AE, Ramos MM. Implementing school nursing strategies to reduce LGBTQ adolescent suicide: a randomized cluster trial study protocol. Implement Sci. 2016;11:145. source

In the paper

“Another important reason for assessing implementation determinants using a theoretical framework is to specify the target of the implementation effort.”

In the paper

“Implementation strategies vary in purpose and complexity, ranging from discrete strategies [56] such as audit and feedback [57] to multifaceted, and often branded, strategies that integrate at least two discrete strategies”

What the sources say

• Not checked yetcites [56] Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, Glass JE, York JL. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2012;69:123–57. source
• Not checked yetcites [57] Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O’Brien MA, Johansen M, Grimshaw J, Oxman AD: Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews 2012. source
• Not checked yetcites [37] Aarons GA, Ehrhart MG, Farahnak LR, Hurlburt MS. Leadership and organizational change for implementation (LOCI): a randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implement Sci. 2015;10:11. source

In the paper

“frameworks can help to identify and/or expand the implementation determinants or aspects assumed to impact the implementation process at different levels and across multiple phases for consideration or measurement.”

In the paper

“The distinct aims and purposes of implementation efforts require distinct evaluation designs such as mixed-methods, hybrid effectiveness-implementation, and quality improvement approaches including formative evaluations or Plan-Do-Study-Act cycles [48].”

What the sources say

• Not checked yetcites [48] Landsverk J, Brown CH, Chamberlain P, Palinkas L, Ogihara M, Czaja S, Goldhaber-Fiebert JD, Rolls Reutz J, McCue Horwitz S. Design and analysis in dissemination and implementation research. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and Implementation Research in Health: Translating Science to Practice. New York, NY: Oxford University Press; 2012.

In the paper

“Stetler et al. [71] provide four progressive yet integrated stages of formative evaluation.”

What the sources say

• Not checked yetcites [71] Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, Kimmel B, Sharp ND, Smith JL. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006;21:S1–8. source

In the paper

“Powell and colleagues have outlined four primary methods for matching implementation strategies to barriers (conjoint analysis, intervention mapping, concept mapping, group model building) [62].”

What the sources say

• Not checked yetcites [62] Powell BJ, Beidas RS, Lewis CC, Aarons GA, McMillen JC, Proctor EK, Mandell DS. Methods to improve the selection and tailoring of implementation strategies. J Behav Health Serv Res. 2017;44:177–94. source

In the paper

“Prospective framework application may also reveal different determinants and aid selection of particular influencing factors to target during subsequent implementation studies.”

In the paper

“In this study, implementation leadership from first-level team leaders/managers, organizational climate and culture, implementation climate, and psychological safety climate were selected as determinants to test their influence on the fidelity of the EBP being implemented.”

What the sources say

• Not checked yetcites [55] Aarons GA, Ehrhart MG, Moullin JC, Torres EM, Green AE. Testing the leadership and organizational change for implementation (LOCI) intervention in substance abuse treatment: a cluster randomized trial study protocol. Implement Sci. 2017;12:29. source

In the paper

“Within research and practice projects, implementation frameworks can inform the program logics that describe the anticipated relationships between inputs, activities, outputs, and implementation and client outcomes, thereby supporting the explicit formulation of key assumptions and outlining of crucial project details.”

In the paper

“Consistent with these recommendations are existing standards and guidelines to improve transparent and accurate reporting of implementation studies such as the Standards for Reporting Implementation Studies (STaRI; Pinnock et al. [85]).”

What the sources say

• Not checked yetcites [85] Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, Rycroft-Malone J, Meissner P, Murray E, Patel A, Sheikh A. Standards for reporting implementation studies (StaRI) statement. bmj. 2017;356:i6795. source

In the paper

“Again, the work of Proctor and colleagues [22] provides a useful taxonomy for classifying and reporting types of implementation research outcomes that also includes guidance regarding level of analysis and measurement, theoretical basis, and maps the salience of outcome onto the phases of implementation.”

What the sources say

• Not checked yetcites [22] Proctor EK, Silmere H, Raghavan R, Hovmand P, Aarons GA, Bunger A, Griffey R, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Hlth. 2011;38:65–76. source

In the paper

“(2) support the identification of the stakeholders and activities involved in the Implementation Resource Team that was developed in the preparation phase”

What the sources say

• Not checked yetcites [68] Willging CE, Green AE, Ramos MM. Implementing school nursing strategies to reduce LGBTQ adolescent suicide: a randomized cluster trial study protocol. Implement Sci. 2016;11:145. source

In the paper

“They can also help to operationalize constructs of importance to an evaluation and the identification of suitable measures.”