CoChat
Sign Up
Cited by 519Health Policy Implementation ScienceEvaluation and Performance AssessmentHealth Sciences Research and Education

An Overview of Research and Evaluation Designs for Dissemination and Implementation

Open access
Brown, C. Hendricks, Curran, Geoffrey, Palinkas, Lawrence A., Aarons, Gregory A., Wells, Kenneth B., Jones, Loretta, Collins, Linda M., Duan, Naihua +10 more·2017·Annual Review of Public Health
DOI 10.1146/annurev-publhealth-031816-044215PMC PMC5384265
Open this paper in CoChat — free account

Abstract

The wide variety of dissemination and implementation designs now being used to evaluate and improve health systems and outcomes warrants review of the scope, features, and limitations of these designs. This article is one product of a design workgroup that was formed in 2013 by the National Institutes of Health to address dissemination and implementation research, and whose members represented diverse methodologic backgrounds, content focus areas, and health sectors. These experts integrated their collective knowledge on dissemination and implementation designs with searches of published evaluations strategies. This article emphasizes randomized and nonrandomized designs for the traditional translational research continuum or pipeline, which builds on existing efficacy and effectiveness trials to examine how one or more evidence-based clinical/prevention interventions are adopted, scaled up, and sustained in community or service delivery systems. We also mention other designs, including hybrid designs that combine effectiveness and implementation research, quality improvement designs for local knowledge, and designs that use simulation modeling.

Integrity signals

Registry

Crossref · 10.1146/annurev-publhealth-031816-044215

Cites

114 of 114 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 68 cited sources checked · 0 supported · 0 flagged

  • Premisecitation

    A review of randomized trials at system, practitioner, and patient levels for lipid management using health information technology found better management in system-level and individual-patient implementations.

    “A recent review of system, practitioner, and individual patient-level randomized trials to evaluate level-specific lipid management approaches that all used a health information technology component concluded that systems and individual patient implementations showed better management (4).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [4] source

      Not checked in the free run.

  • Premisecitation

    Gaps in understanding D&I have been well documented.

    “Limitations in our understanding of dissemination and implementation have been well-documented (1, 2, 86).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [1] source

      Not checked in the free run.

    • cites [2] source

      Not checked in the free run.

    • cites [86] source

      Not checked in the free run.

  • Interpretationnone given

    Efficacy trials can only answer whether a clinical/preventive intervention could work under rigorous conditions.

    “Recognizing that efficacy trials can answer only questions of whether a clinical/preventive intervention could work under rigorous conditions”
    Quote in the paperSubstantial claim
  • Premisecitation

    Qualitative methods can be used alone or in mixed-methods designs to precede, confirm, complement, or extend quantitative effectiveness evaluation.

    “While many of these designs rely on quantitative analysis, qualitative methods can also be used by themselves or in mixed-methods designs that combine qualitative and quantitative methods, to precede, confirm, complement or extend quantitative evaluation of effectiveness (83).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [83] source

      Not checked in the free run.

  • Interpretationcitation

    Smaller units such as wards, teams, clinicians, or patients can sometimes be randomized within each organization, using site as a blocking factor.

    “Instead of larger health organizational sites being randomized, smaller units can sometimes be randomized within each organization. Randomization could occur at the level of the ward, team, clinician, or even at the level of patients or subjects within each site, again assessing health care service utilization as the primary outcome. Such a design uses site as a blocking factor”
    Quote in the paperNot a substantial claim
  • May combine two claims. 23
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 21

  • May combine two claims. The quote only partly states this claim.

    Interpretationnone given

    In head-to-head randomized implementation trials, each strategy is manualized and delivered with comparable fidelity.

    “Both implementation strategies are manualized and carried out with equivalent attention to fidelity.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    A system that helps choose evidence-based programs functions as an implementation strategy and can itself be evaluated with a sound design.

    “A decision support system to select evidence-based programs is, in fact, an implementation strategy, and such a support system can also be tested for impact using a well-crafted design.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Incomplete factorial designs drop at least one arm from a full factorial.

    “With incomplete factorial designs, one or more arms of a complete factorial are excluded from the study.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    A large family of efficacy and effectiveness designs now exists, including designs that randomize across persons, places, and time, and nonrandomized designs such as pre-post comparisons, regression discontinuity, time series, and multiple baseline comparisons.

    “There is now a large family of designs for testing efficacy or effectiveness that randomize across persons, place, and time (or combinations of these) (15), as well as designs that do not use randomization, such as pre-post comparisons, regression discontinuity (106), time series and multiple baseline comparisons (5).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [15] source

      Not checked in the free run.

    • cites [106] source

      Not checked in the free run.

    • cites [5] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    In the preparation phase of MOST, components are selected and pilot tested with a clear optimization criterion.

    “The first phase, preparation, involves selection and pilot testing components with a clear optimization criterion (e.g., most effective components subject to a maximum cost).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    A second head-to-head design assigns different implementation strategies aimed at separate outcomes, so no site gets both; each site can then act as both intervention and control because the outcome targets concern different patient groups.

    “Another type of head-to-head trial design involves two implementation strategies that target different outcomes, with no site receiving both. This design allows each site to serve as both an active intervention and a control because the two implementation targets focus on different patient populations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A head-to-head randomized implementation trial compares two active, qualitatively distinct strategies to see which better implements a clinical/preventive intervention.

    “A head-to-head randomized implementation trial is a comparative effectiveness implementation trial that tests which of two active, qualitatively different implementation strategies is more successful in implementing a clinical/preventive intervention.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [11] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A comprehensive implementation strategy often requires components involving system leadership, clinic level, clinician level, and processes such as planning, educating, financing, restructuring, managing quality, and/or policy.

    “For example, a comprehensive implementation strategy often requires components that involve system leadership, the clinic level, the clinician level, as well as key processes: planning, educating, financing, restructuring, managing quality, and/or policy (90).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [90] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    D&I research explicitly attends to external validity, whereas most randomized efficacy and effectiveness trials mainly emphasize internal validity.

    “Dissemination and implementation research pays explicit (although not exclusive) attention to external validity, in contrast to the main emphasis on internal validity in most randomized efficacy and effectiveness trials (20, 48, 52).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [20] source

      Not checked in the free run.

    • cites [48] source

      Not checked in the free run.

    • cites [52] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationnone given

    D&I is a separate research stage whose design repertoire is not as mature as that for efficacy and effectiveness studies.

    “Dissemination and implementation research represents a distinct stage, and the designs for this newer field of research are currently less well-established than those for efficacy and effectivness.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Trials of efficacy or effectiveness almost always randomize, whereas implementation studies frequently face a choice between a randomized design that offers strong internal validity but is hard to mount and an observational study with weak experimental control but possible value.

    “Efficacy or effectiveness trials nearly always use random assignment, while implementation research often requires tradeoffs between a randomized trial design that can have high internal validity but is difficult to mount, and an observational intervention study that has little experimental control but still may provide valuable information (71).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [71] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Poor understanding of the full range of D&I designs has hindered the development of D&I science and practice.

    “Understanding the full range of these designs has impeded the development of dissemination and implementation science and practice.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Effectiveness trials determine whether a clinical/preventive intervention does work in a realistic context.

    “These less-controlled conditions allow an effectiveness trial to determine if a clinical/preventive intervention does work in a realistic context.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    The traditional pipeline does not imply that research always moves in one direction; the sequence of intervention studies is often cyclical.

    “This traditional pipeline does not imply that research always continues to move in one direction as the sequential progression of intervention studies is often cyclical (14).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    In the traditional translation pipeline, the main emphasis is on producing generalizable knowledge.

    “In the traditional translation pipeline, most of the emphasis is on producing generalizable knowledge.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Implementation factorial designs examine two or more implementation strategies simultaneously.

    “Factorial designs for implementation investigate the combination of two or more implementation strategies at a time.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    The Community Youth Development Study reported that the Communities that Care system increased adoption of evidence-based programs and reduced youth substance use and violence.

    “This study found that the CTC decision support system led to greater adoption of evidence-based programs and prevented youth substance use and violence.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [19] source

      Not checked in the free run.

    • cites [56] Hawkins JD, 2002

      Not checked in the free run.

    • cites [73] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Across diverse fields, reasonable consensus has developed about when particular designs should be used and what sample size and design protocols are needed to maximize internal validity.

    “Within this growing family of randomized, nonrandomized, and mixed method designs, reasonable consensus has grown across diverse fields about when certain designs should be used and what sample size requirements and design protocols are necessary to maximize internal validity (42, 87).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [42] source

      Not checked in the free run.

    • cites [87] Piantadosi S, 1997

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    In randomized new-versus-usual-practice trials, sites are allocated at random to either the new active implementation or the usual practice condition; group-level allocation makes this a cluster-randomized design.

    “Here the sites (practices, communities, organizations) are assigned randomly to the new active implementation or usual practice condition. As random assignment is at the group rather than individual level, this forms a cluster-randomized design (75, 95).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [75] Murray DM, 1998

      Not checked in the free run.

    • cites [95] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    In the evaluation phase of MOST, a standard randomized implementation trial compares the optimized intervention against an appropriate comparison condition.

    “In the third phase, evaluation, a standard randomized implementation trial is conducted comparing the optimized intervention against an appropriate comparison condition.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Kilbourne and colleagues (2014) used SMART to test an adaptive form of REP.

    “In a subsequent study, Kilbourne et al. (2014) are using SMART to examine an adaptive version of REP (64).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [64] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The SMART implementation design belongs to the factorial experiment family and uses multiple randomizations, allowing a site's implementation process to be altered when it is not working.

    “The Sequential Multiple Assignment Randomized Implementation Trial (SMART) implementation design is a special case of the factorial experiment (32, 67, 78) involving multistage randomizations where the site-level implementation process can be modified if unsuccessful.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [32] source

      Not checked in the free run.

    • cites [67] source

      Not checked in the free run.

    • cites [78] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Adaptive implementation can allocate available resources optimally and can switch tactics when a strategy is failing.

    “Such an adaptive approach to enhancing implementation can make the best allocation of available resources (31) and change its approach if a strategy is failing.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [31] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    In the simplest roll-out design, each time interval provides a between-unit comparison between units assigned to the new strategy and those still in usual practice, and a within-unit comparison when units switch at randomly assigned times.

    “Thus, at each time interval there is a between-units comparison of those units that are assigned to the new implementation strategy and those that remain as a usual practice setting. In addition, a within-unit comparison can be made as units change from a usual practice setting to a new implementation strategy at some randomly assigned time.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Communities and organizations often accept roll-out designs over traditional designs when receiving the new strategy is seen as advantageous or when withholding it for the whole study is unethical.

    “Communities and organizations often are willing to accept this type of roll-out design over a traditional design when there are obvious or perceived advantages in receiving the new implementation strategy, or when it is unethical to withhold a new implementation strategy throughout the study (17).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Adaptations may be unlimited when interventions are complex and target populations and settings are highly diverse.

    “Such adaptations are potentially infinite if the interventions are complex and the intended populations and settings highly varied.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A smoking cessation implementation trial examined whether direct patient reimbursement for medication and/or physician group training affected medication use, with patients nested in physicians and all four combinations tested.

    “An example of this design is a trial for smoking cessation implementation, testing whether direct patient reimbursement for medication and/or physician group training influence the use of these medications. Patients are nested within physicians, and all four combinations are tested (107).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [107] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Randomization can often make inferences stronger.

    “Many of the designs we described involved randomization, which can often strengthen inferences”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    A pairwise enrollment roll-out design could also be used in implementation trials and would remove the need to list all sites at the start.

    “This type of pairwise enrollment roll-out design shown in Figure 4(c) could also work for implementation trials, thereby eliminating the need to enumerate all sites at the beginning of the study.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    D&I research usually needs multilevel data with many group-level and individual units because statistical power is driven mainly by the number of highest-level units.

    “Dissemination and implementation research, because it often tests system-level strategies, generally requires multilevel data with sizeable numbers of group level (e.g., practices or organizations) as well as individuals, as the statistical power of such designs is most strongly influenced by the number units at the highest level (75, 77).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [75] Murray DM, 1998

      Not checked in the free run.

    • cites [77] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Factorial designs can be used to evaluate multiple implementation components.

    “A wide range of factorial designs can also be used to evaluate multiple implementation components.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    Other roll-out designs could be used, but the authors know of none yet used in implementation.

    “In addition, other types of roll-out designs could be used, but to our knowledge have not yet been used in implementation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Randomization is sometimes impossible, unnecessary, or inadvisable.

    “but there are many situations where randomization is not possible or required, nor even advisable.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Blocking and matching in the design and covariate adjustment at the randomization unit can raise statistical power.

    “Some general approaches to increasing statistical power include blocking and matching within the design itself and analytical adjustment with covariates at the unit of randomization to reduce imbalance.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The translational pipeline is useful for implementing a predetermined clinical or preventive intervention or one from a set of evidence-based programs.

    “This pipeline is useful in implementing a predetermined clinical/preventive intervention, or ones from a suite of evidence-based programs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    In that original HIV design, two communities were randomized to receive the intervention in year one or year two, with assessments in both communities in year one to evaluate impact.

    “In their original design (62), a pair of communities were randomized to receive the intervention in the first or second year; assessments were made for both communities in the first year to evaluate impact.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [62] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    D&I studies involve major organizational and community changes, so community leaders, organizational leaders, and policy makers have more at stake than evaluators.

    “More than efficacy and effectiveness studies, dissemination and implementation studies involve major changes in organizations and communities, and as such, community leaders, organizational leaders, and policy makers have far more at stake than the evaluators.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Even the most attractive scientific design will not occur without endorsement and agreement from the communities and organizations involved.

    “The most attractive scientific design on paper will not happen without the endorsement and agreement of the communities and organizations where this takes place.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Design choice is complex and depends on research questions, existing knowledge, whether generalizable or local knowledge is sought, stakeholder values and resources, and available opportunities.

    “The choice of design is a complex process, with major input regarding the research questions; the state of existing knowledge; the intention to obtain generalizable or local knowledge; the community, organizational, and funder values, expectations, and resources; and the available opportunities to conduct such research (71).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [71] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    In a post-test-only design, effects of a new implementation can be confounded with policy changes and other external factors, while a randomized new-versus-IAU comparison retains some protection.

    “the effects of examining a new implementation in a post-test only design can be confounded with changes in multiple policies and other external factors, whereas a randomized new versus IAU maintains some protection.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    Implementation strategies are systemic rather than linear, and no fully developed causal inference method comparable to single-person randomized trials addresses this cyclic nature.

    “In addition, implementation strategies are inherently systemic rather than linear, and to date there is no fully-developed causal inference approach comparable to that of single-person randomized trials that addresses this cyclic nature of implementation.”
    Quote in the paperSubstantial claim
41 of 41

  • The authors state this as an assumption.

    Wide-scale use of evidence-based clinical/preventive interventions generally requires targeted information dissemination and often a concerted, deliberate implementation strategy to move along the diffusion, dissemination, and implementation continuum.

    “A tacit assumption of this pipeline is that wide-scale use of evidence-based clinical/preventive interventions generally requires a targeted information dissemination and often a concerted, deliberate strategy for implementation to move to this end of the diffusion, dissemination, and implementation continuum (53, 81, 94).”

  • The authors state this as an assumption.

    For a clinical/preventive intervention to have population-level impact, it must not only be effective but also reach a large portion of the population, be delivered with fidelity, and be maintained.

    “A second assumption is that for a clinical/preventive intervention to have population-level impact, it must not only be an effective program, it must reach a large portion of the population, be delivered with fidelity, and maintained (50).”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Other aspects of reporting that may be important for improving understanding of modification and its impact, such as when in the implementation process the modification was made, were also not included in the prior frameworks.

    “Further, we recognized that other aspects of reporting that may be important for improving understanding of modification and its impact, such as when in the implementation process the modification was made, were also not included [28–31].”

    From The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions (2019) · cites this paper as [30]

    The Introduction (page unavailable) describes adaptations during preparation and implementation, and the Between-Site Designs section describes changing an implementation strategy if it fails. Neither passage addresses whether prior frameworks included reporting when a modification was made. Searches for modification timing and reporting found no passage that settles that claim. (LLM, conf=0.69)

    Experimental — accuracy not yet measured

  • Several methodological issues have hampered the sustainability literature and the broader development of the field.

    “Several methodological issues have hampered the sustainability literature and the broader development of the field (16, 116, 118, 135).”

    From The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care (2018) · cites this paper as [16]

  • There was no evidence that economic methods have as yet been integrated into more advanced evaluative designs within the fields of improvement and implementation design, particularly 'hybrid' designs.

    “There was no evidence that economic methods have as yet been integrated into more advanced evaluative designs within the fields of improvement and implementation design, particularly ‘hybrid’ designs”

    From Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review (2019) · cites this paper as [39]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Cheung citation year differs from its reference entry

    Minor

    The text dates the Cheung et al. work to 2015, while the corresponding reference entry dates it to 2014.

  • Cellulitis example cites an unrelated HIV-testing study

    Minor

    The cellulitis guideline comparison cites reference 8, but reference 8 is a study of rapid HIV testing in a dental clinic.

2 of 2

Cites

Flagged first, then by citations

1 of the 115 references couldn’t be matched to a known paper.

Research works printed but not linked: 5

  1. Fisher RAS, 1935, The Design of Experiments
  2. Hawkins JD, 2002, Investing in Your Community's Youth: An Introduction to the Communities that Care System
  3. Med. Res. Counc. (MRC) Health Serv. Public Health Res. Board, 2000, A Framework for the Development and Evaluation of RCTs for Complex Interventions to Improve Health
  4. Neumann MS, 2000, AIDS Educ. Prev.: Off. Publ. Int. Soc. AIDS Educ., 12, 35
  5. Weiss CH, 2014, Phys. Rev. X, 4, 041008

Cited by

Sorted by citations

Related

Authors
Brown, C. H., Curran, G., Palinkas, L. A., Aarons, G. A., Wells, K. B., Jones, L., Collins, L. M., Duan, N., Mittman, B. S., Wallace, A., Tabak, R. G., Ducharme, L., Chambers, D. A., Neta, G., Wiley, T., Landsverk, J., Cheung, K., & Cruden, G.
Affiliations
  • Northwestern University (US)
  • University of Arkansas for Medical Sciences (US)
  • University of Southern California (US)
  • University of California San Diego (US)
  • University of California, Los Angeles (US)
  • Healthy African American Families II (US)
  • Pennsylvania State University (US)
  • Columbia University Irving Medical Center (US)
  • VA Greater Los Angeles Healthcare System (US)
  • University of Iowa (US)
  • Washington University in St. Louis (US)
  • National Institutes of Health (US)
  • Oregon Social Learning Center (US)
  • Columbia University (US)
ORCID
Venue
Annual Review of Public Health
Publisher
Annual Reviews
Published
2017-03-20
Volume
38
Issue
1
Pages
1-22
ISSN
  • 0163-7525 (ISSN-L)
  • 1545-2093
DOI
10.1146/annurev-publhealth-031816-044215
PMC
PMC5384265
OpenAlex
W2335826478
Semantic Scholar
0da4bddebc1df0e0d3bc34393a513dcdd5717a1c
Cited by
509
References
115
Citations by year
2026: 28 · 2025: 44 · 2024: 66 · 2023: 59 · 2022: 63 · 2021: 71 · 2020: 74 · 2019: 60 · 2018: 30 · 2017: 10 · 2016: 4
FWCI
8.42
Type
journal article
Language
en
Open access
Licence
Locations
  • Annual Review of Public Health (publishedVersion, cc-by-sa)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (acceptedVersion)
  • PubMed Central (acceptedVersion)
Funders
  • National Institute of Mental Health (R01MH076158, R01MH072961)
  • National Institute on Drug Abuse (P30DA027828, P50DA035763)
  • National Institute of Diabetes and Digestive and Kidney Diseases (P30DK092950)
Topics
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
  • Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
Keywords
dissemination research, implementation research, research design, randomized experiments, hybrid design, computer simulation, evidence-based interventions, scale-up, sustainability, health care system
MeSH terms
Clinical Protocols, Humans, Outcome and Process Assessment, Health Care, Research Design, Randomized Controlled Trials as Topic, Evidence-Based Medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review