CoChat
Sign Up
Cited by 405Health Policy Implementation ScienceHealth Sciences Research and EducationClinical practice guidelines implementation

“Scaling-out” evidence-based interventions to new populations or new health care delivery systems

Open access
Aarons, Gregory A., Sklar, Marisa, Mustanski, Brian, Benbow, Nanette, Brown, C. Hendricks·2017·Implementation Science
DOI 10.1186/s13012-017-0640-6PMC PMC5588712
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Implementing treatments and interventions with demonstrated effectiveness is critical for improving patient health outcomes at a reduced cost. When an evidence-based intervention (EBI) is implemented with fidelity in a setting that is very similar to the setting wherein it was previously found to be effective, it is reasonable to anticipate similar benefits of that EBI. However, one goal of implementation science is to expand the use of EBIs as broadly as is feasible and appropriate in order to foster the greatest public health impact. When implementing an EBI in a novel setting, or targeting novel populations, one must consider whether there is sufficient justification that the EBI would have similar benefits to those found in earlier trials. DISCUSSION: In this paper, we introduce a new concept for implementation called "scaling-out" when EBIs are adapted either to new populations or new delivery systems, or both. Using existing external validity theories and multilevel mediation modeling, we provide a logical framework for determining what new empirical evidence is required for an intervention to retain its evidence-based standard in this new context. The motivating questions are whether scale-out can reasonably be expected to produce population-level effectiveness as found in previous studies, and what additional empirical evaluations would be necessary to test for this short of an entirely new effectiveness trial. We present evaluation options for assessing whether scaling-out results in the ultimate health outcome of interest. CONCLUSION: In scaling to health or service delivery systems or population/community contexts that are different from the setting where the EBI was originally tested, there are situations where a shorter timeframe of translation is possible. We argue that implementation of an EBI in a moderately different setting or with a different population can sometimes "borrow strength" from evidence of impact in a prior effectiveness trial. The collection of additional empirical data is deemed necessary by the nature and degree of adaptations to the EBI and the context. Our argument in this paper is conceptual, and we propose formal empirical tests of mediational equivalence in a follow-up paper.

AI snapshot

Written from the paper’s full text.

TL;DR

This conceptual paper introduces “scaling-out” as the deliberate implementation, testing, improvement, and sustainment of evidence-based interventions (EBIs) in new populations and/or new delivery systems. It argues that under some conditions, scale-out can “borrow strength” from prior effectiveness trials if core elements, mechanisms, and mediational pathways are retained or empirically tested.

Key findings

  • The paper defines three scale-out types: Type I (population fixed, different delivery system), Type II (delivery system fixed, different population), and Type III (different population and delivery system); it focuses mainly on Types I and II.
  • It proposes that EBIs in moderately different settings or populations can sometimes borrow strength from prior effectiveness evidence, with additional empirical data needed depending on the nature and degree of adaptations; this requires retaining core elements, preserving mechanisms of action, ensuring sufficient organizational/system support, and testing “mediational equivalence.”
  • It outlines four levels of evidence for scale-out evaluations, from level 0 (minimal or no new empirical evidence) to level 3 (full randomized hybrid trial), mapped to RE-AIM domains; Table 3 gives typical evaluation levels by whether population and delivery system are same or different.
  • Examples include PrEP, where adherence reduced HIV risk by close to 90% in trials, but CDC guidelines did not recommend PrEP for at-risk adolescents due to lack of efficacy/safety evidence; adolescents aged 13–19 accounted for 4% of new HIV diagnoses in 2015. SafeCare was scaled out in one US county with population 3.2 million using an interagency collaborative team.

Methods

  • Conceptual/theoretical paper; no empirical sample, data collection, or primary analysis.
  • Uses external validity theories (Cook’s proximal similarity and heterogeneous irrelevancies), multilevel mediation modeling, RE-AIM, and implementation frameworks including EPIS, the Dynamic Sustainability Framework, and CFIR.
  • Develops a logical framework, tables of scale-out types, four evidence levels, and a sequential mediation schematic; formal statistical tests of mediational equivalence are deferred to a follow-up paper.

Limitations

  • The paper states its argument is conceptual and that formal empirical tests of mediational equivalence are proposed in a follow-up paper.
  • It states that details remain unanswered regarding how best to assess changes in complex implementation systems, and that rigorous non-inferiority statistical methods require additional development.
  • Evident from the text: the proposed framework and evidence levels are not empirically validated here; Type III scale-out is only mentioned and not developed; examples are illustrative rather than evaluated.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13012-017-0640-6

Cites

89 of 89 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 62 cited sources checked · 0 supported · 0 flagged

  • Premisereasoning

    PrEP delivery requires a higher level of engagement of healthcare providers than has typically been engaged for HIV primary prevention.

    “However, PrEP delivery requires a higher level of engagement of healthcare providers than has typically been engaged for HIV primary prevention; in fact, it more closely mimics the types of care an individual living with HIV received than the less medicalized services more typically provided for HIV prevention.”
    Quote in the paperSubstantial claim
  • Interpretationcitation

    If we can legitimately borrow strength from previous studies and a modest amount of empirical evidence, this could accelerate and expand benefit to populations that have experienced health disparities that might never be included in a rigorous randomized effectiveness trial.

    “Indeed, if we can legitimately borrow strength from previous studies and a modest amount of empirical evidence, this could accelerate and expand benefit to populations that have experienced health disparities that might never be included in a rigorous randomized effectiveness trial [6].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] Brown CH, Beardslee W. Realizing population-level improvements for all children’s cognitive, affective, and behavioral health. Am J Prev Med. 2016;51:S101–5. source

      Not checked in the free run.

  • Interpretationcitation

    It would be reasonable to expect beneficial outcomes from Familias Unidas when an intervention increases the mean on parent-child communication to the level found in previous studies.

    “Therefore, it would be reasonable to expect beneficial outcomes from this intervention when an intervention increases the mean on parent-child communication to the level found in previous studies.”
    Quote in the paperSubstantial claim
  • Interpretationreasoning

    We have no justification to anticipate health impact in scale-out if we are unwilling to embrace the three fundamental premises (core elements retained, mechanism same, sufficient organizational support) either based on strength of generalizability of existing evidence or new data.

    “Logically, we have no justification to anticipate health impact in scale-out if we are unwilling to embrace these three fundamental premises either based on the strength of generalizability of existing evidence or new data.”
    Quote in the paperNot a substantial claim
  • Premisecitation

    Concern regarding the effectiveness of an intervention when adapted has been addressed for a number of specific EBIs, including child maltreatment, substance abuse, child anxiety, HIV, school-based social competence, psychological treatments, and health risk prevention programs.

    “Concern regarding the effectiveness of an intervention when adapted has been addressed for a number of specific EBIs, including child maltreatment interventions [50], substance abuse treatment [51], child anxiety interventions [52], HIV treatment [53], school-based social competence interventions [54], psychological treatments for a variety of disorders [55, 56], and health risk prevention program”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [50] Aarons GA, Palinkas LA. Implementation of evidence-based practice in child welfare: service provider perspectives. Adm Policy Ment Health Ment Health Serv Res. 2007;34:411–9. source

      Not checked in the free run.

    • cites [51] Backer TE. The failure of success: challenges of disseminating effective substance abuse prevention programs. J Community Psychol. 2000;28:363–73. source

      Not checked in the free run.

    • cites [52] Kendall PC, Gosch E, Furr JM, Sood E. Flexibility within fidelity. J Am Acad Child Adolesc Psychiatry. 2008;47:987–93. source

      Not checked in the free run.

  • Classified as probably not a substantial claim. 33
  • May combine two claims. 15

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The logic behind borrowing strength from previous studies requires justification that the delivery system retains critical components of the implementation strategy.

    “We note that the logic behind borrowing strength from previous studies requires that we justify that key elements of the intervention still exist and are delivered with fidelity, that the delivery system retains critical components of the implementation strategy, and that broader ecological systems are still supportive of the delivery and sustainment of this intervention.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    In delivering PrEP to prevent HIV incidence, adherence to the medication would be the most important intermediate outcome.

    “In delivering PrEP to prevent HIV incidence, adherence to the medication would be the most important intermediate outcome.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    A more limited evaluation for scaling-out would typically collect a small amount of-or even no-health outcome data, but a substantial amount of implementation process and output data.

    “Typically, a more limited evaluation would collect a small amount of—or even no—health outcome data, but a substantial amount of implementation process and output data, what others have called implementation outcomes [10].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10] Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health Ment Health Serv Res. 2011;38:65–76. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    When an evidence-based intervention is implemented with fidelity in a setting identical or very similar to where it was previously tested and found effective, similar benefits can reasonably be anticipated.

    “When an EBI is implemented with fidelity in a setting that is identical to or very similar to where it was previously tested and found to be effective, it is reasonable to anticipate that the EBI would provide similar benefits to those found earlier.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Staffing, training, and costs required to deliver PrEP in STD clinics are complex and need to be addressed for successful implementation.

    “Staffing, training, and costs required to deliver PrEP in this setting are complex and need to be addressed for successful implementation in STD clinics [31].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [31] Weiss G. Implementing PrEP in STD clinics: findings from a 2015 assessment of local health department engagement in PrEP implementation. In: 2016 National STD Prevention Conference. Atlanta: CDC; 2016.

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Given that state laws and regulations vary in terms of parental consent requirements for medical services, PrEP implementation in adolescents requires different strategies and research that can inform effective delivery.

    “Given that state laws and regulations vary in terms of parental consent requirements for medical services, PrEP implementation in this new population requires different strategies and research that can inform effective delivery [35–37].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [35] Culp L, Caucci L. State adolescent consent laws and implications for HIV pre-exposure prophylaxis. Am J Prev Med. 2013;44:S119–24. source

      Not checked in the free run.

    • cites [36] Mustanski B, Fisher CB. HIV rates are increasing in gay/bisexual teens: IRB barriers to research must be resolved to bend the curve. Am J Prev Med. 2016;51:249–52. source

      Not checked in the free run.

    • cites [37] Hosek S, Celum C, Wilson CM, Kapogiannis B, Delany-Moretlwe S, Bekker L-G. Preventing HIV among adolescents with oral PrEP: observations and challenges in the United States and South Africa. J Int AIDS Soc. 2016;19(7(Suppl 6)):21107. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Cook's principle of causal explanation requires investigation into the causal mediating mechanisms that underlie a relationship of interest, explaining how and why an effect occurs.

    “Cook’s principle of causal explanation [5] requires investigation into the causal mediating mechanisms that underlie a relationship of interest, explaining how and why an effect occurs.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5] Cook TD. Meta-analysis: its potential for causal description and causal explanation within program evaluation. In: Social prevention and the social sciences: theoretical controversies, research problems, and evaluation strategies. Berlin: Walter de Gruyter; 1991. p. 245–85.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    With a lack of empirical data in the new setting, the burden of proof necessarily rests on similarity with previous work.

    “With a lack of empirical data in the new setting, the burden of proof necessarily rests on similarity with previous work.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Three key interconnected implementation/sustainment components for successful scaling-out are alignment of the surrounding ecological context, alignment of the health or service delivery system and organizations where it is delivered, and integrity of the EBI itself as adaptations occur.

    “we focus on three key interconnected implementation/sustainment components: (1) the alignment of the surrounding ecological context including characteristics of the target population as well as broader cultural and contextual factors (e.g., policies, funding), (2) the alignment of the health or service delivery system and organizations where it”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [47] Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health Ment Health Serv Res. 2011;38:4–23. source

      Not checked in the free run.

    • cites [40] Chambers DA, Glasgow R, Stange K. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117. source

      Not checked in the free run.

    • cites [48] Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:1. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The logic behind borrowing strength from previous studies requires justification that broader ecological systems are still supportive of delivery and sustainment.

    “We note that the logic behind borrowing strength from previous studies requires that we justify that key elements of the intervention still exist and are delivered with fidelity, that the delivery system retains critical components of the implementation strategy, and that broader ecological systems are still supportive of the delivery and sustainment of this intervention.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    The adaptation-fidelity tension is a critical component of scaling-out and is addressed head on in more dynamic models of implementation process.

    “This “adaptation-fidelity” tension is a critical component of scaling-out in that it is addressed head on as proposed in more dynamic models of implementation process [60–62].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [60] Stirman SW, Gutner CA, Langdon K, Graham JR. Bridging the gap between research and practice in mental health service settings: An overview of developments in implementation theory and research. Behav Ther. 2016;47(6):920–36. source

      Not checked in the free run.

    • cites [61] Kemp L. Adaption and fidelity: a recipe analogy for achieving both in population scale implementation. Prev Sci. 2016;17:429–38. source

      Not checked in the free run.

    • cites [62] Chambers DA, Norton WE. The adaptome: advancing the science of intervention adaptation. Am J Prev Med. 2016;51:S124–31. source

      Not checked in the free run.

    • Rests on an assumption: Overly strict intervention fidelity may be at odds with effective implementation of EBIs in real-world practice settings outside highly controlled efficacy trials.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Level 0 depends almost exclusively on the assumption that the new implementation will follow in the same footsteps as the previous intervention did when it produced findings of health impact.

    “Level 0 depends almost exclusively on the assumption that the new implementation will follow in the same footsteps as the previous intervention did when it produced findings of health impact.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The new implementation will follow in the same footsteps as the previous intervention did when it produced findings of health impact.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    If testing a new scale-out requires the full empirical evaluation required for establishing an EBI, this would be exceptionally costly, time consuming, and would delay implementation, especially to populations underrepresented in scientific trials or in settings where delivery could reasonably produce benefit.

    “If testing of this new scale-out requires the full empirical evaluation that would be required for establishing an EBI, this would be exceptionally costly, time consuming, and would delay implementation, especially to populations underrepresented in scientific trials or in settings where its delivery could reasonably produce benefit.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    An actual design may involve different levels of assessment across the columns.

    “We note that an actual design may involve different levels of assessment across the columns.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The logic behind borrowing strength from previous studies requires justification that key elements of the intervention still exist and are delivered with fidelity.

    “We note that the logic behind borrowing strength from previous studies requires that we justify that key elements of the intervention still exist and are delivered with fidelity, that the delivery system retains critical components of the implementation strategy, and that broader ecological systems are still supportive of the delivery and sustainment of this intervention.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Evidence of the effectiveness of scaling-out can be supported if we establish that mediational pathways have equivalent strength as they did in the original trials.

    “We describe four levels of evidence and recommend that evidence of the effectiveness of scaling-out can be supported if we establish that mediational pathways have equivalent strength as they did in the original trials.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    An expectation of beneficial impact when scaling-up relies upon Cook's principle of proximal similarity because a nearly identical intervention is delivered in the same way to a similar population.

    “An expectation of beneficial impact when scaling-up relies upon Cook’s principle of proximal similarity [5] because a nearly identical intervention is delivered in the same way to a similar population.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5] Cook TD. Meta-analysis: its potential for causal description and causal explanation within program evaluation. In: Social prevention and the social sciences: theoretical controversies, research problems, and evaluation strategies. Berlin: Walter de Gruyter; 1991. p. 245–85.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    There must be sufficient organizational or system support to deliver the intervention as intended to sufficient numbers of the target population.

    “Third, there must be sufficient organizational or system support to deliver the intervention as intended to sufficient numbers of the target population.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    For all types of scaling-out, it may be possible to use the identical EBI, but more often the intervention needs to be adapted to fit the new population and/or new delivery system.

    “For all types of scaling-out, it may be possible to use the identical EBI (e.g., delivered using the same manualized version and/or dosage). However, we more often need to adapt the intervention to fit the new population and/or new delivery system.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    In all variants of scaling-out, there is more concern about impact on effectiveness and health outcomes, as there is more uncertainty whether the empirically supported causal association between intervention and outcome found in previous studies will hold when adapted and tested under yet-unstudied conditions.

    “In all variants of scaling-out, there is more concern about the impact on effectiveness and health outcomes, as there is more uncertainty whether the empirically supported causal association between intervention and outcome found in previous studies will hold when adapted and tested under yet-unstudied conditions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    Even as the EBI is adapted to new settings or populations, it still must retain its core elements.

    “First, even as the EBI is adapted to new settings or populations, it still must retain its core elements [12].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12] Mohr DC, Schueller SM, Riley WT, Brown CH, Cuijpers P, Duan N, Kwasny MJ, Stiles-Shields C, Cheung K. Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. J Med Internet Res. 2015;17:e166. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Assuming the conceptual theory of mediation holds in a scaled-out adaptation, relying on Cook's principles of heterogeneous irrelevancies and proximal similarity, we could assess impact on the mediating variable using a level 1 design for assessing a proxy outcome.

    “Assuming the conceptual theory of mediation holds in a scaled-out adaptation, relying on Cook’s principles of heterogeneous irrelevancies and proximal similarity, then we could assess impact on the mediating variable using a level 1 design for assessing a proxy outcome.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The conceptual theory of mediation holds in a scaled-out adaptation.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    We can infer that positive effects would be expected to translate into corresponding benefit on the distal outcome as well.

    “We can also infer that positive effects would be expected to translate into corresponding benefit on the distal outcome as well.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    The more similar the delivery system, broader contextual setting, intervention, and population, the more we should be able to rely on previous evidence, and the less new empirical evidence should be needed, to anticipate a successful scaling-out.

    “Intuitively one would expect that the more similar the delivery system, the broader the contextual setting, the intervention, and the population, the more we should be able to rely on previous evidence, and the less new empirical evidence should be needed, to anticipate a successful scaling-out.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Discussions of adaptation to fit an EBI for a particular cultural group most commonly pertain to cultural adaptations.

    “Discussions of this issue most commonly pertain to adaptations to fit an EBI for a particular cultural group [58].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [58] Castro FG, Barrera M Jr, Martinez CR Jr. The cultural adaptation of prevention interventions: resolving tensions between fidelity and fit. Prev Sci. 2004;5:41–5. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Adaptations to accommodate an EBI are likely to be needed at the service system or organization levels, and often to the EBI itself.

    “However, adaptations to accommodate an EBI are likely to be needed at the service system or organization levels, and often to the EBI itself [47, 60].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [47] Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health Ment Health Serv Res. 2011;38:4–23. source

      Not checked in the free run.

    • cites [60] Stirman SW, Gutner CA, Langdon K, Graham JR. Bridging the gap between research and practice in mental health service settings: An overview of developments in implementation theory and research. Behav Ther. 2016;47(6):920–36. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Recent methodologic developments have addressed challenging issues in causality for single and multiple randomized trials, evolving interventions, natural experiments, and multilevel as well as multidimensional situations.

    “Recent methodologic developments have addressed challenging issues in causality for single [77] and multiple randomized trials [25], evolving interventions [12], natural experiments [78, 79], and multilevel [78] as well as multidimensional situations [80].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [77] VanderWeele TJ. Mediation analysis: a practitioner's guide. Annu Rev Public Health. 2016;37:17–32. source

      Not checked in the free run.

    • cites [25] Huang S, MacKinnon DP, Perrino T, Gallo CG, Cruden G, Brown CH. A statistical method for synthesizing mediation analyses using the product of coefficient approach across multiple trials. Stat Methods Appl. 2016;25:565–79. source

      Not checked in the free run.

    • cites [12] Mohr DC, Schueller SM, Riley WT, Brown CH, Cuijpers P, Duan N, Kwasny MJ, Stiles-Shields C, Cheung K. Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. J Med Internet Res. 2015;17:e166. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    For type III scaling-out (both population and service delivery different), logic requires more empirical examination that mediational pathways remain in the new context compared to types I or II, potentially using proxy measures for mediational relationships and generally requiring some measures of distal health outcome.

    “For type III scaling-out where both population and service delivery are different (bottom right), logic would require more empirical examination that the mediational pathways remain in this new context compared to types I or II scaling-out. Potentially, proxy measures could be used to assess these mediational relationships, and generally some measures of the distal health outcome would be required”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    Major implementation strategy adaptations often occur, and are sometimes required to support implementation in delivery systems with widely different system and/or organizational cultures and climates, readiness, and resources, and to populations with widely different histories, norms and values.

    “Major implementation strategy adaptations often occur, and indeed are sometimes required to support implementation in delivery systems with widely different system and/or organizational cultures and climates, readiness, and resources [81], to populations having widely different histories, norms and values.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [81] Czaja SJ, Valente TW, Nair SN, Villamar J, Brown CH. Characterizing implementation strategies using a systems engineering survey and interview tool: a comparison across 10 prevention programs for drug abuse and HIV sexual risk behaviors. Implement Sci. 2016;11:70. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    At the heart of the scale-out approach is borrowing strength through the use of mediational modeling.

    “At the heart of the scale-out approach is borrowing strength through the use of mediational modeling.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given

    It seems natural to require much more evidence when both population and delivery system change; doing so would demand new empirical evidence beyond that for either delivery system or population fixed scaling-out.

    “It seems natural to require much more evidence when both population and delivery system change; doing so would demand new empirical evidence beyond that for either delivery system or population fixed scaling-out.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    By testing underlying mechanisms or mediators of effects, the efficiency of testing EBIs for new populations or new service systems can be streamlined through greater understanding and use of prior data to borrow strength from previous effectiveness studies.

    “We take a mechanistic approach in suggesting that by testing underlying mechanisms or mediators of effects, the efficiency of testing EBIs for new populations or new service systems can be streamlined through a greater understanding of and use of prior data to borrow strength from previous effectiveness studies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Scaling-out necessitates a better understanding of how to facilitate delivery of EBIs with appropriate content adherence and competence in delivery.

    “Scaling-out, while allowing for appropriate system, organization, and intervention adaptation, necessitates a better understanding of how to facilitate delivery of EBIs with appropriate content adherence and competence in delivery”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    There are multiple implementation strategies that, in specific circumstances, can effectively address different barriers to implementation, as well as the same barrier.

    “That is, there are multiple implementation strategies that, in specific circumstances can effectively address different barriers to implementation, as well as the same barrier (e.g., financing) [83–85].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [83] Eaton J, McCay L, Semrau M, Chatterjee S, Baingana F, Araya R, Ntulo C, Thornicroft G, Saxena S. Scale-up of services for mental health in low-income and middle-income countries. Lancet. 2011;378:1592–603. source

      Not checked in the free run.

    • cites [84] Paina L, Peters DH. Understanding pathways for scaling-up health services through the lens of complex adaptive systems. Health Policy Plan. 2012;27:365–73. source

      Not checked in the free run.

    • cites [85] Mangham LJ, Hanson K. Scaling-up in international health: what are the key issues? Health Policy Plan. 2010;25:85–96. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Under Cook's principle of proximal similarity, generalization is justified when all relevant properties of the causal association, such as prototypical and necessary components of an EBI, are adequately represented in the new context.

    “Under this principle, generalization is justified when all relevant properties of the causal association, such as prototypical and necessary components of an EBI, are adequately represented in the new context.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5] Cook TD. Meta-analysis: its potential for causal description and causal explanation within program evaluation. In: Social prevention and the social sciences: theoretical controversies, research problems, and evaluation strategies. Berlin: Walter de Gruyter; 1991. p. 245–85.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Scaling-out provides an opportunity to use a strategic approach to improve the efficiency of moving an EBI from one setting to another and/or from one population to another.

    “Scaling-out provides an opportunity to use a strategic approach to improve the efficiency of moving an EBI from one setting to another and/or from one population to another.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Mediational analysis has an extensive history but has had increased interest in implementation science.

    “Mediational analysis has an extensive history [74, 75] but has had increased interest in implementation science [76].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [74] MacKinnon DP. Introduction to statistical mediation analysis. Mahwah, NJ: Lawrence Erlbaum Associates; 2008.

      Not checked in the free run.

    • cites [75] MacKinnon DP, Kisbu-Sakarya Y, Gottschall AC. 16 developments in mediation analysis. The Oxford Handbook of Quantitative Methods in Psychology: Vol 2: Statistical Analysis. 2013;2:338.

      Not checked in the free run.

    • cites [76] Anselmi L, Binyaruka P, Borghi J. Understanding causal pathways within health systems policy evaluation through mediation analysis: an application to payment for performance (P4P) in Tanzania. Implement Sci. 2017;12:10. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Similar scaling-out approaches are possible when implementing other types of interventions including evidence-based decision support systems such as Communities that Care.

    “Similar scaling-out approaches are possible when implementing other types of interventions including evidence-based decision support systems, such as Communities that Care [68–72], which helps communities decide which among a menu of EBIs are most appropriate to implement, rather than a single intervention.”
    Quote in the paperNot a substantial claim0 of 5 sources checked
    • cites [68] Hawkins JD, Oesterle S, Brown EC, Abbott RD, Catalano RF. Youth problem behaviors 8 years after implementing the communities that care prevention system: a community-randomized trial. JAMA Pediatr. 2014;168:122–9. source

      Not checked in the free run.

    • cites [69] Hawkins JD, Catalano Jr RF. Communities that care: action for drug abuse prevention. San Francisco: Jossey-Bass; 1992.

      Not checked in the free run.

    • cites [70] Fagan AA, Hanson K, Hawkins JD, Arthur MW. Translational research in action: implementation of the communities that care prevention system in 12 communities. J Community Psychol. 2009;37:809–29. source

      Not checked in the free run.

    • cites [71] Brown EC, Hawkins JD, Arthur MW, Briney JS, Fagan AA. Prevention service system transformation using communities that care. J Community Psychol. 2011;39:183–201. source

      Not checked in the free run.

    • cites [72] Brown E, Hawkins JD, Rhew I, Shapiro V, Abbott R, Oesterle S, Arthur M, Briney J, Catalano R. Prevention system mediation of communities that care effects on youth outcomes. Prev Sci. 2014;15:623–32. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    For type II scaling-out (delivery system fixed, different population), reach should typically be assessed with high accuracy, but implementation fidelity, adoption, and sustainment could be assessed with proxy measures.

    “For type II delivery system fixed, different population scaling-out (lower left), we would typically want to assess reach with high accuracy due to the focus on a new population, but may make the case that assessing implementation fidelity, adoption, and sustainment could be done with proxy measures.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    For type I scaling-out (population fixed, different delivery system), mediational relationships are expected to hold, reach could be measured with a proxy, but stronger evidence is needed around implementation fidelity, adoption, and sustainment.

    “For type I population fixed, different delivery system scaling-out (upper right), the mediational relationships are again expected to hold, reach could be measured with a proxy, but because the delivery system changes we would need stronger evidence around implementation fidelity, adoption, and sustainment.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The less system/organization adaptation needed, the more readily an EBI can be assimilated.

    “The less system/organization adaptation needed, the more readily an EBI can be assimilated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    More rapid implementation is especially important to the delivery of effective interventions to minorities and other populations experiencing health or health service disparities that would otherwise not benefit from the extensive research required to demonstrate effectiveness.

    “To close this discussion of a framework for asserting that a scaling-out is expected to share the original version’s impact on health outcomes by relying on previous studies and new empirical data, we note that more rapid implementation is especially important to the delivery of effective interventions to minorities and other populations experiencing health or health service disparities that would”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [90] Brown CH, Mohr DC, Gallo CG, Mader C, Palinkas L, Wingood G, Prado G, Kellam SG, Pantin H, Poduska J, et al. A computational future for preventing HIV in minority communities: how advanced technology can improve implementation of effective programs. J Acquir Immune Defic Syndr. 2013;63:S72–84. source

      Not checked in the free run.

    • cites [91] Perrino T, Beardslee W, Bernal G, Brincks A, Cruden G, Howe G, Murry V, Pantin H, Prado G, Sandler I. Toward scientific equity for the prevention of depression and depressive symptoms in vulnerable youth. Prev Sci. 2015;16:642–51. source

      Not checked in the free run.

42 of 42

  • The paper relies on this without support.

    Adolescents aged 13–19 who engage in risky sexual behavior and/or injection drugs are at increased risk of HIV infection and accounted for 4% of new HIV diagnoses in 2015.

    “However, adolescents between the ages of 13–19 who engage in risky sexual behavior and/or injection drugs are also at increased risk of HIV infection, and accounted for 4% of new HIV diagnoses in 2015.”

  • The authors state this as an assumption.

    The conceptual theory of mediation holds in a scaled-out adaptation.

    “Assuming the conceptual theory of mediation holds in a scaled-out adaptation, relying on Cook’s principles of heterogeneous irrelevancies and proximal similarity, then we could assess impact on the mediating variable using a level 1 design for assessing a proxy outcome.”

  • The authors state this as an assumption.

    The new implementation will follow in the same footsteps as the previous intervention did when it produced findings of health impact.

    “Level 0 depends almost exclusively on the assumption that the new implementation will follow in the same footsteps as the previous intervention did when it produced findings of health impact.”

  • The paper relies on this without support.

    Overly strict intervention fidelity may be at odds with effective implementation of EBIs in real-world practice settings outside highly controlled efficacy trials.

    “However, there is mounting evidence that overly strict intervention fidelity may be at odds with effective implementation of EBIs in real-world practice settings (i.e., outside of highly controlled efficacy trials), thus, raising concern about the balance between delivering EBIs with fidelity and making adaptations believed to be necessary for usual care contexts.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A hybrid type 2 design is ideal when studying interventions that already have evidence of effectiveness in other settings or populations, but not in either the context or population in the current trial, resulting in uncertainty that there would be a similar clinical benefit.

    “A hybrid type 2 design is ideal when studying interventions that already have evidence of effectiveness in other settings or populations, but not in either the context or population in the current trial resulting in uncertainty that there would be a similar clinical benefit (Aarons et al., 2017).”

    From An introduction to effectiveness-implementation hybrid designs (2019) · cites this paper as Aarons et al., 2017

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • During scale-up or scale-out, other changes may be necessary, particularly for contexts where the intervention is likely to reach populations that may differ from the population that received the intervention during implementation.

    “During scale-up or scale-out, other changes may be necessary, particularly for contexts where the intervention is likely to reach populations that may differ from the population that received the intervention during implementation [43].”

    From The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions (2019) · cites this paper as [43]

Cites

Flagged first, then by citations

2 of the 91 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

  1. Institute of Medicine. National Research Council: preventing mental, emotional, and behavioral disorders among young people: progress and possibilities. Washington, DC: The National Academies Press; 2009.
  2. U.S. Public Health Service. In: Services USDoHaH, editor. Preexposure prophylaxis for the prevention of HIV infection in the United States - 2014: a clinical practice guideline. Atlanta: Centers for Disease Control and Prevention; 2014.
  3. MacKinnon DP. Introduction to statistical mediation analysis. Mahwah, NJ: Lawrence Erlbaum Associates; 2008.

Cited by

Sorted by citations

Related

Authors
Aarons, G. A., Sklar, M., Mustanski, B., Benbow, N., & Brown, C. H.
Affiliations
  • University of California San Diego (US)
  • Brown University (US)
  • Northwestern University (US)
Corresponding author
Aarons, Gregory A.
ORCID
Venue
Implementation Science
Publisher
Springer Science and Business Media LLC
Published
2017-09-06
Volume
12
Issue
1
Pages
111-111
ISSN
1748-5908 (ISSN-L)
DOI
10.1186/s13012-017-0640-6
PMC
PMC5588712
OpenAlex
W2751779860
Semantic Scholar
fa16633491dbb36e730405e32620ef57ee0196f9
Cited by
399
References
91
Citations by year
2026: 25 · 2025: 52 · 2024: 41 · 2023: 41 · 2022: 56 · 2021: 66 · 2020: 51 · 2019: 51 · 2018: 16
FWCI
5.69
Type
journal article
Language
en
Open access
Licence
Locations
  • Implementation Science (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • eScholarship (California Digital Library) (submittedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
Funders
  • National Institute on Drug Abuse (P30DA027828, R01DA038466)
  • National Institute of Mental Health (R01MH072961, R01MH092950)
  • Agency for Healthcare Research and Quality (F32HS024192)
Topics
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
Keywords
horizontal scaling, evidence-based interventions, generalizability, intervention adaptation, health care delivery systems, multilevel mediation, implementation fidelity, intervention generalizability
MeSH terms
Delivery of Health Care, Health Plan Implementation, Humans, Evidence-Based Medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review