CoChat
Sign Up
Cited by 627Health Policy Implementation ScienceEvaluation and Performance AssessmentMeta-analysis and systematic reviews

Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation

Open access
Gottfredson, Denise C., Cook, Thomas D., Gardner, Frances E. M., Gorman-Smith, Deborah, Howe, George W., Sandler, Irwin N., Zafft, Kathryn M.·2015·Prevention Science
DOI 10.1007/s11121-015-0555-xPMC PMC4579256
Open this paper in CoChat — free accountOpen PDF

Abstract

A decade ago, the Society of Prevention Research (SPR) endorsed a set of standards for evidence related to research on prevention interventions. These standards (Flay et al., Prevention Science 6:151-175, 2005) were intended in part to increase consistency in reviews of prevention research that often generated disparate lists of effective interventions due to the application of different standards for what was considered to be necessary to demonstrate effectiveness. In 2013, SPR's Board of Directors decided that the field has progressed sufficiently to warrant a review and, if necessary, publication of "the next generation" of standards of evidence. The Board convened a committee to review and update the standards. This article reports on the results of this committee's deliberations, summarizing changes made to the earlier standards and explaining the rationale for each change. The SPR Board of Directors endorses "The Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation."

AI snapshot

Written from the paper’s full text.

TL;DR

This article reports the Society for Prevention Research’s 2015 revision of its standards of evidence for prevention interventions, updating the 2005 Flay et al. criteria for efficacy, effectiveness, and scale-up research. The committee added or elevated standards on theory testing, implementation fidelity, cost reporting, independent replication/investigator allegiance, and rigorous evaluation of scale-up outcomes.

Key findings

  • Revised standards are cumulative across efficacy, effectiveness, and scale-up; an efficacy statement should specify “Intervention X is efficacious for producing Y outcomes for Z population at time T in setting S,” and there must be at least one long-term follow-up beyond the intervention, with 6 months described as a reasonable minimum.
  • New or elevated standards include testing the intervention’s causal theory/mediators, measuring implementation quality and fidelity, reporting cost information (cost-effectiveness required at the effectiveness stage), and requiring evidence that effects are not biased by investigator allegiance.
  • The scale-up section adds standards for readiness, materials, training/technical assistance, fidelity monitoring, recruitment/reach, and rigorous evaluation of scale-up outcomes; only EBIs meeting all effectiveness criteria should be made available for scaling up.
  • The paper cites replication concerns: 92% of findings from original research teams replicated versus 65% from independent teams; developer/creator trials had effect sizes more than twice those from other trials; and interventions that monitored implementation obtained effect sizes two to three times larger than those that reported no monitoring.

Methods

  • Consensus/standards-development paper: in 2013 the SPR Board convened a committee chaired by Denise Gottfredson to review and update the 2005 standards; the article reports the committee’s deliberations and rationale for changes.
  • No empirical sample or primary data are analyzed; the paper synthesizes prior research and examples, including within-study comparisons of designs and reviews of implementation research, to justify the standards.
  • Standards are organized around Cook and Campbell validity types and cumulative stages, with Tables 1–3 summarizing required and desirable standards for efficacy, effectiveness, and scaling up.

Limitations

  • The paper states the standards are forward-looking and that it would be “premature and inappropriate” for many to be translated into current requirements; they are not intended as a checklist for a single work.
  • The paper notes lack of consensus on key issues, no accepted standard for cost assessment, little rigorous research on scale-up outcomes, and community readiness scales lacking tested predictive validity.
  • Evident from the text: this is a normative standards document, not an empirical test of whether adopting the standards improves prevention outcomes; no data are presented on that question.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s11121-015-0555-x

Cites

110 of 110 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 121 cited sources checked · 0 supported · 0 flagged

  • Findingcitation

    The past decade has witnessed a disturbingly high rate of failures to replicate when independent evaluation teams conduct studies of prevention interventions and practices that had met all criteria for effectiveness.

    “On the other hand, the past decade has also witnessed a disturbingly high rate of failures to replicate when independent evaluation teams conduct studies of prevention interventions and practices that had met all criteria for effectiveness.6”
    Quote in the paperSubstantial claim
  • Findingcitation

    When implementation quality is monitored, implementation quality improves.

    “When implementation quality is monitored, implementation quality improves.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Durlak and DuPre 2008 source

      Not checked in the free run.

  • Premisecitation

    The earlier SPR standards were intended in part to increase consistency in reviews of prevention research that often generated disparate lists of effective interventions due to the application of different standards for what was considered necessary to demonstrate effectiveness.

    “These standards (Flay et al., Prevention Science 6:151–175, 2005) were intended in part to increase consistency in reviews of prevention research that often generated disparate lists of effective interventions due to the application of different standards for what was considered to be necessary to demonstrate effectiveness.”
    • Also stated in: § Introduction · plain
      “These standards were intended in part to increase consistency in reviews of prevention research that often generated disparate lists of effective interventions due to the application of different standards for what was considered to be necessary to demonstrate effectiveness.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Flay et al. 2005 source

      Not checked in the free run.

  • Interpretationreasoning

    Only by studying outcomes of scale-up efforts will we learn how interventions work in these new circumstances.

    “Only by studying outcomes of scale-up efforts will we learn how interventions work in these new circumstances.”
    Quote in the paperNot a substantial claim
  • Premisecitation

    Regression discontinuity design studies produced comparable causal estimates to randomized studies at points around the cutoff point.

    “The regression discontinuity design studies produced comparable causal estimates to the randomized studies at points around the cutoff point.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Cook et al. 2008 source

      Not checked in the free run.

  • May combine two claims. 29
  • Classified as probably not a substantial claim. 61

  • May combine two claims.

    Findingnone given

    The standard of independent evaluation is seldom met in Prevention Science, even though such studies are clearly feasible.

    “This standard is seldom met in Prevention Science, even though such studies are clearly feasible.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The earlier standards recommended that monitoring and evaluation tools suitable for use by practitioners be made available to adopting organizations so that they could demonstrate to their constituencies that their prevention dollars were well spent.

    “The earlier standards instead recommended that monitoring and evaluation tools suitable for use by practitioners be made available to adopting organizations so that they could demonstrate to their constituencies that their prevention dollars were well spent.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Flay et al. 2005 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Evaluations of the effects of interventions that are being widely implemented in a community but have not previously undergone randomized efficacy trials have great promise for identifying interventions with significant public health impact.

    “For example, evaluations of the effects of interventions that are being widely implemented in a community but have not previously undergone randomized efficacy trials have great promise for identifying interventions with significant public health impact.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Often, effectiveness and scale-up research generate new questions that are later addressed in new efficacy or effectiveness studies.

    “Often, effectiveness and scale-up research generate new questions that are later addressed in new efficacy or effectiveness studies.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    Much valuable prevention research does not follow the preventive intervention research cycle at all.

    “We also recognize that much valuable prevention research does not follow the preventive intervention research cycle at all.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    A recent report from an SPR task force on type 2 translational research (Spoth et al. 2013) addresses the challenges to translating EBIs into broader usage and sets an agenda for needed research to advance this new science of translation.

    “A recent report from an SPR task force on type 2 translational research (Spoth et al. 2013) addresses the challenges to translating EBIs into broader usage and sets an agenda for needed research to advance this new science of translation.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Spoth et al. 2013

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    The SPR task force recognizes that successful translation of EBIs into broader usage will require a more elaborated and less linear progression of research than is implied by the progression from efficacy to effectiveness to dissemination in the prior SPR standards.

    “This task force recognizes that successful translation of EBIs into broader usage will require a more elaborated and less linear progression of research than is implied by the progression from efficacy to effectiveness to dissemination in the prior SPR standards.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Spoth et al. 2013

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Much can be learned about the factors influencing adoption, implementation, and sustainability of practices by conducting research on interventions that have not been developed according to the sequence implied in the preventive intervention research cycle.

    “Likewise, much can be learned about the factors influencing adoption, implementation, and sustainability of practices by conducting research on interventions that have not been developed according to the sequence implied in the preventive intervention research cycle.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    In the past 10 years, as it became clear that preventive interventions do indeed produce desired outcomes at least under optimal conditions, the emphasis in Prevention Science has shifted more toward understanding how these EBIs can be implemented on a broader scale to produce larger impacts on entire populations.

    “In the past 10 years, as it became clear that preventive interventions do indeed produce desired outcomes at least under optimal conditions, the emphasis in Prevention Science has shifted more toward understanding how these EBIs can be implemented on a broader scale to produce larger impacts on entire populations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Because outcome research results are specific to the intervention actually tested, the samples (or populations), the point in time and settings from which they were drawn, and the outcomes measured, it is essential that conclusions from the research be clear regarding the intervention, population(s), time, and settings, and the outcomes for which efficacy is claimed.

    “Because outcome research results are specific to the intervention actually tested, the samples (or populations), the point in time and settings from which they were drawn, and the outcomes measured, it is essential that conclusions from the research be clear regarding the intervention, population(s), time, and settings, and the outcomes for which efficacy is claimed.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Flay et al. (2005) defined efficacy trials as studies of programs or policies delivered under optimal conditions, and effectiveness trials as studies conducted in real-world conditions.

    “Flay et al. (2005) defined efficacy trials as studies of programs or policies delivered under optimal conditions, and effectiveness trials as studies conducted in real-world conditions.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Flay et al. 2005 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Chen (1990) and MacKinnon (2008) discuss the two components of the theoretical mechanism: the "action theory" corresponds to how the treatment will affect mediators, and the "conceptual theory" focuses on how the mediators are related to the outcome variables.

    “Chen (1990) and MacKinnon (2008) discuss the two components of this theoretical mechanism: The “action theory” corresponds to how the treatment will affect mediators, and the “conceptual theory” focuses on how the mediators are related to the outcome variables.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Chen (1990)

      Not checked in the free run.

    • cites MacKinnon (2008)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    While some preventive intervention research continues to be developed along a continuum, the progression along this continuum often involves numerous critical feedback loops to inform needed research at earlier phases of the research cycle.

    “We recognize that while some preventive intervention research continues to be developed along a continuum, the progression along this continuum often involves numerous critical feedback loops to inform needed research at earlier phases of the research cycle.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The traditional preventive intervention research cycle framework was appropriate to describe the prevention field 10 years ago, when Prevention Science was dominated by studies of risk and protective factors and evaluations of newly developed interventions.

    “This framework was appropriate to describe the prevention field 10 years ago, when Prevention Science was dominated by studies of risk and protective factors and evaluations of newly developed interventions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Interventions may become more or less effective relative to the usual state that would be present in the absence of the EBI.

    “Interventions may become more or less effective relative to the usual state that would be present in the absence of the EBI.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Flay et al. (2005) argued that even interventions that have been demonstrated to be effective in the real world might not be ready for broad dissemination.

    “Further, Flay et al. (2005) argued that even interventions that have been demonstrated to be effective in the real world might not be ready for broad dissemination.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Flay et al. 2005 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Standards of evidence are expected to change over time as methods develop and as Prevention Science and practice advance.

    “Standards of evidence are expected to change over time as methods develop and as Prevention Science and practice advance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Flay et al. (2005) stated that effectiveness trials require an additional burden of proof above and beyond that required for efficacy trials because it is necessary to demonstrate that the intervention was in fact delivered under real-world conditions and that the outcomes are generalizable to the population targeted by the intervention.

    “Flay et al. (2005) stated that effectiveness trials require an additional burden of proof above and beyond that required for efficacy trials because, for example, it is necessary to demonstrate that the intervention was in fact delivered under real-world conditions and that the outcomes are generalizable to the population targeted by the intervention.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Flay et al. 2005 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    The SPR task force suggests that research on factors that are likely to influence the success of scale-up efforts should be incorporated throughout each phase of the development and testing of EBIs.

    “Specifically, it suggests that research on factors that are likely to influence the success of scale-up efforts should be incorporated throughout each phase of the development and testing of EBIs.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Spoth et al. 2013

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Flay et al. (2005) did not consider standards for research on broad dissemination efforts, but rather assumed that once an intervention had cleared all hurdles related to efficacy and effectiveness, further rigorous research on its effects was not necessary as long as fidelity to the proven model was demonstrated.

    “They did not consider standards for research on broad dissemination efforts, but rather assumed that once an intervention had cleared all hurdles related to efficacy and effectiveness, further rigorous research on its effects was not necessary as long as fidelity to the proven model was demonstrated.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Flay et al. 2005 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Flay et al. (2005) document was organized around the traditional preventive intervention research cycle (Mrazek and Haggerty 1994), which depicted a mostly linear progression for prevention research beginning with basic research on the nature of the problem to be addressed, progressing through the development of interventions, pilot testing, efficacy trials, effectiveness trial, and finally to

    “The Flay et al. (2005) document was therefore organized around the traditional preventive intervention research cycle (Mrazek and Haggerty 1994), which depicted a mostly linear progression (although feedback across stages was anticipated) for prevention research beginning with basic research on the nature of the problem to be addressed, progressing through the development of interventions, pilot”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Flay et al. 2005 source

      Not checked in the free run.

    • cites Mrazek and Haggerty 1994

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Research on the outcomes of scale-up efforts is necessary because neither prevention interventions nor alternatives to these interventions are static.

    “This research is necessary because neither prevention interventions nor alternatives to these interventions are static.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationnone given

    A clear and complete description of the intervention is necessary to guide practice, provide a basis for sound measurement of its implementation, and for replication.

    “A clear and complete description of the intervention is necessary to guide practice, provide a basis for sound measurement of its implementation, and for replication.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The SPR task force recommends a more collaborative, practice-oriented framework for developing EBIs than is implied in the traditional preventive intervention research cycle.

    “It also recommends a more collaborative, practice-oriented framework for developing EBIs than is implied in the traditional preventive intervention research cycle.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Spoth et al. 2013

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Flay et al. (2005) provided additional criteria that must be met in order to justify a decision to broadly disseminate.

    “They therefore provided additional criteria that must be met in order to justify a decision to broadly disseminate.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Flay et al. 2005 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Spoth et al. (2013) call for an increase in research on the outcomes of scale-up efforts.

    “Spoth et al. (2013) also call for an increase in research on the outcomes of scale-up efforts.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Spoth et al. 2013

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Important work conducted by economists, public policy analysts, and evaluation researchers often generates conclusive answers to questions about what works under what conditions, while focusing less on describing the intervention, testing the theoretical model underlying the intervention, or creating conditions conducive to subsequent scaling up.

    “In addition, important work conducted by economists, public policy analysts, and evaluation researchers often generates conclusive answers to questions about what works under what conditions, while focusing less on describing the intervention, testing the theoretical model underlying the intervention, or creating conditions conducive to subsequent scaling up.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Dismantling designs that randomly assign subjects to different core components are seldom feasible when the subjects are schools or communities.

    “Of course, such designs are seldom feasible when the subjects are schools or communities.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The level of implementation of the intervention is meaningful only in comparison to what is present in the comparison condition.

    “The level of implementation of the intervention is meaningful only in comparison to what is present in the comparison condition.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    Early trials of an intervention can help to provide information (e.g., about costs and variability in the quality of implementation) that may not be critical in the early stages of development but that will become critical later.

    “For example, we identified numerous ways in which early trials of an intervention can help to provide information (e.g., about costs and variability in the quality of implementation) that may not be critical in the early stages of development but that will become critical later.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Random assignment is the strongest possible design for generating statistically unbiased estimates of intervention effects.

    “Although random assignment is the strongest possible design for generating statistically unbiased estimates of intervention effects”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    It will take many years before researchers begin to build these elements into their research proposals and before funding agencies adapt their priorities to encourage inclusion of these elements in funded research.

    “Clearly, it will take many years before researchers begin to build these elements into their research proposals and before funding agencies adapt their priorities to encourage inclusion of these elements in funded research.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Making these theories explicit should help the developer to identify the features of the intervention that are most central to the action theory.

    “Making these theories explicit should help the developer to identify the features of the intervention that are most central to the action theory.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The timing of intervention can be randomly assigned to allow a short-term comparison of outcomes between the early and later intervention groups.

    “Also, the timing of intervention can be randomly assigned to allow a short-term comparison of outcomes between the early and later intervention groups.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Sound research on the risks of smoking and the effectiveness of specific strategies for reducing smoking were necessary building blocks for the tobacco control movement to occur, but not sufficient to produce change without a range of additional efforts aimed at translating the research into broad change.

    “Sound research on the risks of smoking and the effectiveness of specific strategies for reducing smoking were necessary building blocks for this movement to occur, but not sufficient to produce change without a range of additional efforts aimed at translating the research into broad change.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: High-quality research on prevention interventions will lead to beneficial change.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Demonstrated public health impact is essential at a later stage of program development, but it should not be ignored at the efficacy stage.

    “Demonstrated public health impact is essential at a later stage of program development, but it should not be ignored at the efficacy stage.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    The positive effects of an intervention may diminish rapidly or slowly, or broaden and increase over time.

    “The positive effects of an intervention may diminish rapidly or slowly, or broaden and increase over time.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    There are now a total of seven studies comparing regression discontinuity and experimental estimates at the regression discontinuity cutoff score and six comparing experimental and interrupted time series or comparison time series designs with a nontreatment comparison series.

    “There are now a total of seven studies comparing regression discontinuity and experimental estimates at the regression discontinuity cutoff score and there are six comparing experimental and interrupted time series or comparison time series designs with a nontreatment comparison series.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Perceived obstacles to random assignment are often not as difficult to overcome as initially anticipated.

    “perceived obstacles to random assignment are often not as difficult to overcome as initially anticipated”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    In so doing, the underlying assumptions about hypothesized similarities in the causal structure as well as anticipated limitations to application across populations, settings, times, and outcomes will be documented.

    “In so doing, the underlying assumptions about hypothesized similarities in the causal structure as well as anticipated limitations to application across populations, settings, times, and outcomes will be documented.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    During effectiveness trials, developer control over staff selection and supervision and sometimes training and quality control is loosened.

    “During effectiveness trials, developer control over staff selection and supervision and (sometimes) training and quality control is loosened.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Even small violations of the assumption of statistical independence can have very large impacts on the standard error of the effect size estimate and can greatly inflate the type I error rate.

    “Even small violations of this assumption can have very large impacts on the standard error of the effect size estimate (Kenny and Judd 1986; Murray 1998), which in turn can greatly inflate the type I error rate (e.g., Scariano and Davenport 1987).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Kenny and Judd 1986 source

      Not checked in the free run.

    • cites Murray 1998

      Not checked in the free run.

    • cites Scariano and Davenport 1987 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Differential measurement attrition can occur even when the rates of attrition are comparable across groups.

    “Note that differential measurement attrition can occur even when the rates of attrition are comparable across groups.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    In early stages of testing, the developer's team generally controls intervention materials, training, staff selection and supervision, feedback on implementation, data collection, and data analysis.

    “In early stages of testing, the developer’s team generally controls intervention materials, training, staff selection and supervision, feedback on implementation, data collection, and data analysis.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    Efficacy trials often have small sample sizes and are thus incapable of demonstrating effectiveness across different population subgroups.

    “Efficacy trials also often have small sample sizes and are thus incapable of demonstrating effectiveness across different population subgroups.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    When intervention and comparison sites were from different states or different cities within a state, the groups were not equivalent and differences could not be adjusted away.

    “Conversely, when intervention and comparison sites were from different states or even different cities within a state, the groups were not at all equivalent and differences could not be adjusted away.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    Propensity score matching is theoretically preferable because it is nonparametric and requires demonstrated overlap between treatment and comparison groups on observed variables.

    “though the latter is theoretically preferable because it is nonparametric and requires demonstrated overlap between the treatment and comparison groups on observed variables.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    More leverage for reducing selection bias comes from preintervention theoretical analysis of the selection process, selecting local comparison groups that maximize group overlap, and using a heterogeneous and extensive collection of covariates including pretest waves of the main outcome.

    “More leverage for reducing selection bias comes from (a) preintervention theoretical analysis of the selection process into treatment—or even direct observation of this process—so as to know which covariates to choose, (b) selecting local comparison groups that maximize group overlap before any covariate choice, and (c) using a heterogeneous and extensive collection of covariates that, at a min”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Cook et al. 2009 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Appropriate application of imputation methods requires assumptions about the pattern of missingness, and these assumptions are often not justified in practice.

    “Appropriate application of these imputation methods requires assumptions about the pattern of missingness, however, and these assumptions are often not justified in practice.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    Non-randomized matched control designs should be used only when initial group differences are minimized, the selection process is fully known and modeled, or covariates lead to no detectable pretest differences.

    “They should be used only under the following conditions:(a) Initial group differences are minimized, especially by identifying intact comparison groups that are local to the treatment group and demonstrably heavily overlap with it on at least pretest measures of the outcome (Cook et al. 2008 ); (b) the process by which treatment subjects select into the intervention group (or are selected into it)”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Cook et al. 2008 source

      Not checked in the free run.

    • cites Diaz and Handa 2006 source

      Not checked in the free run.

    • cites Shadish et al. 2008 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationnone given

    The comparison time series design, but not the single group interrupted time series design, can provide a useful alternative to randomized designs.

    “We conclude that the comparison time series design, but not the single group interrupted time series design, can provide a useful alternative to randomized designs.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    There is a continuum of developer control over intervention implementation and research.

    “There is a continuum of developer control over intervention implementation and research.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Most efficacy studies are limited in terms of generalizability because the intervention tested is one in which the developer has exerted substantial control, the population is often carefully selected to be amenable to the intervention, and the setting is likely to be one that is conveniently accessed by the researcher.

    “Most efficacy studies are limited in terms of generalizability because the intervention tested is one in which the developer has exerted substantial control, the population is often carefully selected to be amenable to the intervention, and the setting is likely to be one that is conveniently accessed by the researcher.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    The conditions of intact group matching and complete knowledge of the selection process are rare.

    “These conditions—intact group matching and complete knowledge of the selection process—are rare.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Reporting of interventions tested and the methods used to evaluate them is often suboptimal, even in best journals, and this often makes it difficult to judge the quality of the evidence from potentially important prevention trials.

    “Unfortunately, reporting of interventions tested and the methods used to evaluate them is often suboptimal, even in our best journals (Grant et al. 2013), and this often makes it difficult to judge the quality of the evidence from potentially important prevention trials.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Grant et al. 2013 source

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    Probability samples, at least of individuals, are rarely used in Prevention Science because tests of interventions generally require that the subjects be located in a limited geographical area.

    “However, probability samples (at least of individuals) are rarely used in Prevention Science because tests of interventions generally require that the subjects be located in a limited geographical area.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    Even when executed faithfully, randomized trials often suffer from differential attrition across study groups, which often reduces group equivalence and renders the study results ambiguous.

    “But even when executed faithfully, randomized trials often suffer from differential attrition across study groups, which often reduces group equivalence and renders the study results ambiguous.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Broad reforms are often made in response to highly publicized, emotionally laden incidents, and these incidents may result in numerous reforms at roughly the same time, making it impossible to isolate the effects of only one using time series analysis.

    “Often broad reforms are made in response to highly publicized, often emotionally laden, incidents. These incidents may result in numerous reforms that fall into place at roughly the same time, making it impossible to isolate the effects of only one of them using time series analysis.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Effect sizes from trials conducted by program developers or creators were more than twice the size of effect sizes from trials conducted by others.

    “They reported that effect sizes from trials conducted by program developers/creators were more than twice the size of effect sizes from trials conducted by others.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Petrosino and Soydan (2005) source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The practical significance of an outcome can be expressed in terms of the combination of strength of effect and size of population affected.

    “As a general rule, the practical significance of the outcome can be expressed in terms of the combination of strength of effect and size of population affected.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Metrics of practical significance are likely to differ depending on the nature of the evidence-based intervention.

    “However, metrics of practical significance are likely to differ depending on the nature of the EBI.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Reinvention or adaptation is essential to the success of dissemination and implementation efforts and is likely to lead to at least equal intervention effects as shown in the original efficacy or effectiveness trial.

    “Rabin and Brownson (2012) identify reinvention or adaptation as essential to the success of dissemination and implementation efforts and suggest that such adaptation is likely to lead to “at least equal intervention effects as shown in the original efficacy or effectiveness trial” (p. 35).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Rabin and Brownson 2012 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationcitation

    Fidelity and adaptability can coexist in an intervention.

    “Durlak and Dupre (2008) note the apparent contradiction between the importance of fidelity and adaptability in the research on implementation quantity and suggest that the apparent contradiction is resolved by recognizing that both qualities can coexist in an intervention.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Durlak and Dupre 2008 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Careful attention to recruitment can improve the reach of the intervention.

    “but careful attention to recruitment can improve the reach of the intervention.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Services to which clients would be exposed in the absence of an EBI change over time.

    “Similarly, the services to which clients would be exposed in the absence of an EBI change over time.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    A pattern of findings in which the average effect size across all tests of an outcome construct is positive and any outcome construct in the undesired direction can be attributed to chance supports a claim of effectiveness.

    “Such a pattern of findings supports a claim of effectiveness.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Interventions often change over time, presumably to meet evolving client needs.

    “Interventions often change over time, presumably to meet evolving client needs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    High-quality implementation of larger scale interventions such as passage of new laws or changes to broad policies often requires some type of technical assistance to those charged with implementing the changes.

    “While much of the research on the provision of technical assistance concerns interventions involving direct services, it is likely that high-quality implementation of larger scale interventions such as passage of new laws or changes to broad policies often requires some type of technical assistance to those charged with implementing the changes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    The characteristics related to high-quality implementation of larger scale EBIs such as laws or broad policies are not as well documented as those for smaller scale interventions.

    “The characteristics related to high-quality implementation of larger scale EBIs such as laws or broad policies are not as well documented but are likely to involve some of these same activities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Sustained use of EBIs can be facilitated by a community partnership or coalition that continually reviews and addresses problems as they arise.

    “However, we note that sustained use of EBIs can be facilitated by a community partnership or coalition, as discussed above, that continually reviews and addresses problems as they arise.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Initial training is usually required to provide knowledge about the intervention, but training by itself seldom improves the quality of implementation.

    “Research on training and technical assistance suggests that although initial training is usually required to provide knowledge about the intervention, training by itself seldom improves the quality of implementation (Fixsen et al. 2005).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Fixsen et al. 2005

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Failure to reach those segments of population most at risk for experiencing problems and traditionally underserved communities is especially problematic for prevention efforts.

    “Failure to reach those segments of population most at risk for experiencing problems (e.g., inner city neighborhoods, schools with weak leadership) and traditionally underserved communities (e.g., rural communities) is especially problematic for prevention efforts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Services with overlapping goals are frequently developed and implemented.

    “Services with overlapping goals are frequently developed and implemented.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The observed pattern of more positive findings from research conducted by original research teams is equally consistent with a fidelity explanation.

    “Of course, the observed pattern of more positive findings from research conducted by original research teams is equally consistent with a fidelity explanation: Program developers are likely to understand their interventions better and be more attuned to variations in implementation quality that might diminish intervention effectiveness.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    A national evaluation of Head Start indicated that effects of the program were small and faded out more rapidly than anticipated.

    “This point was recently made in response to evidence from a national evaluation of Head Start, a major early childhood program, indicating that effects of the program were small and faded out more rapidly than anticipated (Puma et al. 2012).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Puma et al. 2012

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Continued assessment of outcomes is needed after scale-up in order to monitor changes in effect sizes over time and reevaluate prevention options.

    “and that continued assessment of outcomes is needed after scale-up in order to monitor changes in effect sizes over time and reevaluate prevention options.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    The quality of the Head Start program has improved since 1980, and the quality of the environments to which youth would be exposed in the absence of the program has also improved.

    “While the quality of the Head Start program has improved since that time, so has the quality of the environments to which youth would be exposed in the absence of the program.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Ludwig and Phillips 2008 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Curran et al. (2013) discuss models for blending design components of effectiveness and implementation research that may be particularly useful in studying scale-up efforts.

    “Curran et al. (2013) also discuss models for blending design components of effectiveness and implementation research that may be particularly useful in the studying of scale-up efforts.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Curran et al. (2013) source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    It is premature to draw conclusions about which mechanisms underlie the phenomenon of changing effect sizes over time.

    “It is premature to draw conclusions about which mechanisms underlie the phenomenon of changing effect sizes over time, and they are likely to vary by the intervention in question as well.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Premisecitation

    Alternative early childhood programs developed and evaluated since the initiation of Head Start are far more cost effective than Head Start.

    “Ludwig and Phillips (2008) commented that alternative early childhood programs have been developed and evaluated since the initiation of Head Start that are far more cost effective than Head Start.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ludwig and Phillips 2008 source

      Not checked in the free run.

80 of 80

  • The authors state this as an assumption.

    High-quality research on prevention interventions will lead to beneficial change.

    “An assumption on which these standards are based is that high-quality research on such interventions will lead to beneficial change.”

  • The paper relies on this without support.

    The earlier standards did not provide sufficient guidance on standards for replication.

    “It noted that important research has been conducted in the past decade that might alter design recommendations for efficacy studies, that the earlier standards did not provide sufficient guidance on standards for replication, and that the field had made important strides in understanding prerequisites for scaling up of effective interventions that should now be incorporated into the SPR standards.”

  • The paper relies on this without support.

    The field had made important strides in understanding prerequisites for scaling up of effective interventions.

    “It noted that important research has been conducted in the past decade that might alter design recommendations for efficacy studies, that the earlier standards did not provide sufficient guidance on standards for replication, and that the field had made important strides in understanding prerequisites for scaling up of effective interventions that should now be incorporated into the SPR standards.”

  • The paper relies on this without support.

    Important research has been conducted in the past decade that might alter design recommendations for efficacy studies.

    “It noted that important research has been conducted in the past decade that might alter design recommendations for efficacy studies, that the earlier standards did not provide sufficient guidance on standards for replication, and that the field had made important strides in understanding prerequisites for scaling up of effective interventions that should now be incorporated into the SPR standards.”

  • The paper relies on this without support.

    There is currently no accepted standard to guide cost assessment, and considerable variability exists in what elements are included.

    “There is currently no accepted standard to guide cost assessment, and considerable variability exists in what elements are included.”

  • The paper relies on this without support.

    Costs incurred during efficacy and effectiveness trials are likely to include significant costs related to conducting the research that are difficult to separate from the costs likely to be incurred by communities later adopting the intervention.

    “Costs incurred during efficacy and effectiveness trials are likely to include significant costs related to conducting the research that are difficult to separate from the costs likely to be incurred by communities later adopting the intervention.”

  • The paper relies on this without support.

    Costs are likely to change over time as programs evolve.

    “Further, costs are likely to change over time as programs evolve.”

  • The paper relies on this without support.

    Researchers do not usually estimate cost-effectiveness in efficacy trials.

    “Researchers do not usually estimate cost-effectiveness in efficacy trials.”

  • The paper relies on this without support.

    Cost-effectiveness information will influence decisions to adopt the intervention at a later stage.

    “This information will influence decisions to adopt the intervention at a later stage and so should be collected during earlier stages if possible.”

  • The paper relies on this without support.

    Selection effects are the most serious and prevalent source of bias in prevention research.

    “Although there are many sources of bias in the estimation of causal effects, selection effects are the most serious and prevalent in prevention research.”

  • The paper relies on this without support.

    Correct randomization procedures are not always implemented or sometimes break down in practice.

    “Because correct randomization procedures are not always implemented or sometimes break down in practice”

  • The authors state this as an assumption.

    The comparison time series designs produce unbiased estimates of treatment effects assumes that there are few studies with conflicting results sitting in file drawers.

    “this assumes, of course, that there are few studies with conflicting results sitting in “file drawers””

  • The authors state this as an assumption.

    Missing data are missing at random, which means that there is no discernible pattern to the missingness once measured variables are controlled.

    “The required assumption is that missing data are “missing at random,” which means that there is no discernible pattern to the missingness once measured variables are controlled.”

  • The paper relies on this without support.

    There is no consensus on the best way to handle multiple comparisons in prevention research.

    “There is no consensus on the best way to handle this issue in prevention research.”

  • The paper relies on this without support.

    The prevention research enterprise is no closer to reaching the high standard for replication to rule out chance findings than it was 10 years ago.

    “The prevention research enterprise appears no closer to reaching this high standard for replication to rule out chance findings, and funding agencies are no more likely today to fund replications simply to verify the results of an earlier study than they were 10 years ago.”

  • The paper relies on this without support.

    Funding agencies are no more likely today to fund replications simply to verify the results of an earlier study than they were 10 years ago.

    “funding agencies are no more likely today to fund replications simply to verify the results of an earlier study than they were 10 years ago.”

  • The paper relies on this without support.

    When replication studies are conducted, they are much more likely to be for testing variations in the intervention or generalizing results than for ruling out chance findings.

    “When replication studies are conducted, they are much more likely to be for the purpose of testing variations in the intervention or of generalizing results to different settings or populations than for ruling out chance findings.”

  • The paper relies on this without support.

    Incomplete and inaccurate reporting results in underuse of the research.

    “Incomplete and inaccurate reporting results in underuse of the research.”

  • The paper relies on this without support.

    Incomplete reporting is a problem in other disciplines as well.

    “Incomplete reporting is a problem in other disciplines as well.”

  • The paper relies on this without support.

    The goal of making independent evaluations of preventive interventions routine is too distant at this time to justify establishing a requirement that these evaluations be completed prior to scaling up.

    “The goal of making independent evaluations of preventive interventions routine is arguably too distant at this time to justify establishing a requirement that these evaluations be completed prior to scaling up.”

  • The paper relies on this without support.

    Cost-benefit information would likely be a powerful tool at the adoption and sustaining stages of scale-up.

    “Although this information on longer term benefits of prevention interventions is seldom available, such information would likely to be a powerful tool at the adoption and sustaining stages of scale-up.”

  • The paper relies on this without support.

    Agreed-upon structures and mechanisms to facilitate the sharing of individual-level datasets for combination and data synthesis do not currently exist.

    “Although models are available (Perrino et al. 2013), agreed-upon structures and mechanisms to facilitate the sharing of individual-level datasets for combination and data synthesis do not currently exist.”

  • The paper relies on this without support.

    An intervention that is ready to be scaled up in terms of research support and materials, training, and technical assistance will not be adopted, implemented, or sustained in the absence of local capacity to adopt, implement, and sustain its use.

    “An intervention may be ready to be scaled up in terms of the research supporting its effectiveness and (if appropriate) the materials, training, and technical assistance available to communities, but it will not be adopted, implemented, or sustained in the absence of local capacity to adopt, implement, and sustain its use.”

  • The paper relies on this without support.

    No rigorous research has tested the validity of available community readiness scales for predicting improved implementation quality or sustainability of an intervention.

    “Although good psychometric properties have been reported for these scales, we know of no rigorous research that has tested the validity of the scales for predicting improved implementation quality or sustainability of an intervention.”

  • The paper relies on this without support.

    At present, very little research is available to guide the use of data from community readiness assessments in practice.

    “At present, very little research is available to guide the use of data from such assessments in practice.”

  • The paper relies on this without support.

    Standards do not currently exist to guide decisions about which cost elements should be included in cost accounting.

    “As noted earlier, standards do not currently exist to guide decisions about which cost elements should be included in cost accounting.”

  • The paper relies on this without support.

    Intent-to-treat analyses often neglect important variation in the effect of an intervention across individual characteristics, context, and time.

    “For example, intent-to-treat analyses often neglect important variation in the effect of an intervention across individual characteristics, context, and time.”

  • The paper relies on this without support.

    Conditions encountered in a scale-up effort are likely to be less well suited for research than conditions encountered in efficacy and even effectiveness trials.

    “Whatever design is used, it is likely that conditions encountered in a scale-up effort will be less well suited for research than conditions encountered in efficacy and even effectiveness trials.”

  • The paper relies on this without support.

    Some effective interventions require further research and development to boost their small to negligible effects.

    “Some of our “effective” interventions require further research and development to boost their small to negligible effects.”

  • The paper relies on this without support.

    Some effective interventions require updated research to demonstrate continued effectiveness.

    “Others require updated research to demonstrate continued effectiveness.”

  • The paper relies on this without support.

    Many policy or legislative changes fail to change behavior.

    “Despite the potential of larger scale interventions to reach large proportions of the population, history shows that many policy or legislative changes fail to change behavior.”

  • The authors state this as an assumption.

    Investing in high-quality research will produce higher quality evidence on which to base sound policies and practices.

    “The underlying assumption in providing these standards is that investing in high-quality research will produce higher quality evidence on which to base sound policies and practices.”

  • The paper relies on this without support.

    More informed decisions will enhance well-being more in the long run than will best guesses based on existing evidence.

    “More informed decisions will enhance well-being more in the long run than will best guesses based on existing evidence.”
33 of 33

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Type 1 translational prevention science requires an intervention to have an action theory linking it to biological mediators and a conceptual theory linking mediators to outcomes.

    “A core foundation of type 1 translational prevention science is that the intervention must have an “action theory” that corresponds to how the intervention will affect the underlying biologically based mechanisms of change, or mediators, as well as a conceptual theory that describes how the mediators will predict the hypothesized outcome variables [10].”

    From A translational neuroscience perspective on mindfulness meditation as a prevention strategy (2016) · cites this paper as [10]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Multiple-outcome guidance points to the pretest-differences standard

    Minor

    The text directs readers to Standard 6.d. for an omnibus test correcting for alpha inflation, but Standard 6.d. addresses pretest differences. Standard 6.e. addresses multiple outcomes.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. Boruch, R. F. (Ed). (2005). Place randomized trials: Special issue. Annals of the American Academy of Political and Social Sciences, 599, whole issue.
  2. Greene, W. H. (1993). Econometric analysis. New York: MacMillan.
  3. Schochet, P. Z. (2007). Guidelines for multiple testing in experimental evaluations of educational interventions. Princeton: Mathematica Policy Research, Inc.
  4. Trochim, W. (2000). The research methods knowledge base (2nd ed.). Cincinnati: Atomic Dog Publishing.
  5. Wagenaar, A. C., & Komro, K. A. (2013). Natural experiments: Research design elements for optimal causal inference without randomization. In A. C. Wagenaar & S. C. Burris (Eds.), Public health law research: Theory and methods (pp. 307–324). Somerset: Wiley.
  6. Winokur Early, K., Hand, G., Blankenship, J., & Chapman, S. (2012). Redirection continues to save money and reduce recidivism. Tallahassee: Justice Research Center.

Cited by

Sorted by citations

Related

Authors
Gottfredson, D. C., Cook, T. D., Gardner, F. E. M., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M.
Affiliations
  • University of Maryland, College Park (US)
  • Northwestern University (US)
  • University of Oxford (GB)
  • George Washington University (US)
  • Arizona State University (US)
Corresponding author
Gottfredson, Denise C.
ORCID
Venue
Prevention Science
Publisher
Springer Science and Business Media LLC
Published
2015-04-07
Volume
16
Issue
7
Pages
893-926
ISSN
  • 1389-4986 (ISSN-L)
  • 1573-6695
DOI
10.1007/s11121-015-0555-x
PMC
PMC4579256
OpenAlex
W2029591572
Cited by
627
References
110
Citations by year
2026: 34 · 2025: 52 · 2024: 44 · 2023: 51 · 2022: 62 · 2021: 76 · 2020: 69 · 2019: 63 · 2018: 60 · 2017: 60 · 2016: 43 · 2015: 13
FWCI
107.02
Type
journal article
Language
en
Open access
Licence
Locations
  • Prevention Science (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • U.S. Department of Education
  • Substance Abuse and Mental Health Services Administration
  • Office of Justice Programs
Topics
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords
evidence standards, prevention interventions, efficacy, effectiveness, evidence-based practice
MeSH terms
Preventive Health Services, Research, Efficiency, Organizational, Evidence-Based Medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review