The DOI matches its Crossref record; 64 of the 67 references it cites were found in a registry or an index.
Experimental and quasi-experimental designs in implementation research
Open accessAbstract
Implementation science is focused on maximizing the adoption, appropriate use, and sustainability of effective clinical practices in real world clinical settings. Many implementation science questions can be feasibly answered by fully experimental designs, typically in the form of randomized controlled trials (RCTs). Implementation-focused RCTs, however, usually differ from traditional efficacy- or effectiveness-oriented RCTs on key parameters. Other implementation science questions are more suited to quasi-experimental designs, which are intended to estimate the effect of an intervention in the absence of randomization. These designs include pre-post designs with a non-equivalent control group, interrupted time series (ITS), and stepped wedges, the last of which require all participants to receive the intervention, but in a staggered fashion. In this article we review the use of experimental designs in implementation science, including recent methodological advances for implementation studies. We also review the use of quasi-experimental designs in implementation science, and discuss the strengths and weaknesses of these approaches. This article is therefore meant to be a practical guide for researchers who are interested in selecting the most appropriate study design to answer relevant implementation science questions, and thereby increase the rate at which effective clinical practices are adopted, spread, and sustained.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quasiexperiments are appropriate for practical situations and ethical compliance for studies involving vulnerable participants such as older adults, 17 and allow scholars to conduct rigorous studies under certain limitations and non-control conditions. 18”
From Mixed reality technology for older adults: Evaluating the impact of a novel virtual humanoid coach in a community-based physical exercise program in the Philippines (2024) · cites this paper as [18]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · cited 1×
“Second, as a corollary of this point, implementation RCTs typically feature different outcome measures than efficacy or effectiveness RCTs, with an emphasis on the extent to which a health intervention was successfully implemented rather than an evaluation of the health effects of that intervention (Proctor et al., 20…”
BackgroundEvaluating the public health impact of health promotion interventions: the RE-AIM framework.
Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999) · American Journal of Public Health · doi:10.2105/ajph.89.9.1322 · Cited by 7,217 · cited 1×
“A variety of evaluation-oriented implementation frameworks may guide the choices of such measures (e.g., RE-AIM; Gaglio et al., 2013; Glasgow et al., 1999).”
BackgroundFisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
Curran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012) · Medical Care · doi:10.1097/mlr.0b013e3182408812 · Cited by 4,058 · cited 1×
“Hybrid implementation-effectiveness studies attend to both effectiveness and implementation outcomes (Curran et al., 2012); these designs are also covered in more detail elsewhere in this issue (Landes, this issue).”
BackgroundSegmented regression analysis of interrupted time series studies in medication use research
Wagner, A. K., Soumerai, S. B., Zhang, F., & Ross-Degnan, D. (2002) · Journal of Clinical Pharmacy and Therapeutics · doi:10.1046/j.1365-2710.2002.00430.x · Cited by 3,279 · cited 2×
“More formally, ITS evaluations focus on identifying whether there is discontinuity in the trend (change in slope or level) after the intervention relative to before the intervention, using segmented regression to model pre- and post-intervention trends (Gebski et al., 2012; Penfold and Zhang, 2013; Taljaard et al., 20…”
BackgroundInterrupted time series regression for the evaluation of public health interventions: a tutorial
Lopez Bernal, J., Cummins, S., & Gasparrini, A. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyw098 · Cited by 3,049 · cited 4×
“Relative to an RCT, ITS designs can also allow for a more comprehensive assessment of the longitudinal effects of an intervention (positive or negative), as effects can be traced over all included time points (Bernal et al., 2017; Penfold and Zhang, 2013).”
BackgroundInfluentialImproving Primary Care for Patients With Chronic Illness
Bodenheimer, T. (2002) · JAMA · doi:10.1001/jama.288.14.1775 · Cited by 2,961
Organizing Care for Patients with Chronic Illness
Wagner, E. H., Austin, B. T., & Korff, M. V. (1996) · The Milbank Quarterly · doi:10.2307/3350391 · Cited by 2,877 · cited 1×
“, 2014) to implement care practices consistent with the collaborative chronic care model (CCM; Bodenheimer et al., 2002a, 2002b; Wagner et al., 1996) in nine outpatient mental health teams in VA medical centers.”
MethodTheory of Statistical Estimation
Fisher, R. A. (1925) · Mathematical Proceedings of the Cambridge Philosophical Society · doi:10.1017/s0305004100009580 · Cited by 2,684
Improving Primary Care for Patients With Chronic Illness: The Chronic Care Model, Part 2
Bodenheimer, T., Wagner, E. H., & Grumbach, K. (2002) · JAMA · doi:10.1001/jama.288.15.1909 · Cited by 2,413
Design and analysis of stepped wedge cluster randomized trials.
Hussey, M. A., & Hughes, J. (2007) · Contemporary Clinical Trials · doi:10.1016/j.cct.2006.05.007 · Cited by 1,297 · cited 1×
“In a stepped wedge, all participants receive the intervention, but are assigned to the timing of the intervention in a staggered fashion (Betran et al., 2018; Brown and Lilford, 2006; Hussey and Hughes, 2007), ty-”
BackgroundMethods for Evaluating Changes in Health Care Policy: The Difference-in-Differences Approach
Dimick, J. B., & Ryan, A. M. (2014) · JAMA · doi:10.1001/jama.2014.16153 · Cited by 1,193 · cited 1×
“, Morgan, 2018) can statistically mitigate internal validity concerns, although this approach may be of limited utility when comparing secular trends between different study cohorts (Dimick and Ryan, 2014).”
BackgroundUse of Interrupted Time Series Analysis in Evaluating Health Care Quality Improvements
Penfold, R. B., & Zhang, F. (2013) · Academic Pediatrics · doi:10.1016/j.acap.2013.08.002 · Cited by 1,150 · cited 3×
“, 2003): a common recommendation is at least eight time points both pre- and post-intervention (Penfold and Zhang, 2013).”
BackgroundCollins, L. M., Murphy, S. A., & Strecher, V. (2007) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2007.01.022 · Cited by 1,082
The RE-AIM framework: a systematic review of use over time
Gaglio, B., Shoup, J. A., & Glasgow, R. E. (2013) · American Journal of Public Health · doi:10.2105/ajph.2013.301299 · Cited by 1,029 · cited 1×
“A variety of evaluation-oriented implementation frameworks may guide the choices of such measures (e.g., RE-AIM; Gaglio et al., 2013; Glasgow et al., 1999).”
BackgroundThe stepped wedge trial design: a systematic review
Brown, C. A., & Lilford, R. J. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-54 · Cited by 907
Adaptive designs in clinical trials: why use them, and how to run and report them
Pallmann, P., Bedding, A. W., Choodari-Oskooei, B., Dimairo, M., Flight, L., Hampson, L. V., Holmes, J., Mander, A. P., Odondi, L., Sydes, M. R., et al. (2018) · BMC Medicine · doi:10.1186/s12916-018-1017-7 · Cited by 829
Kilbourne, A. M., Neumann, M. S., Pincus, H. A., Bauer, M. S., & Stall, R. (2007) · Implementation Science · doi:10.1186/1748-5908-2-42 · Cited by 653
A Conceptual Framework for Adaptive Preventive Interventions
Collins, L. M., Murphy, S. A., & Bierman, K. L. (2004) · Prevention Science · doi:10.1023/b:prev.0000037641.26017.00 · Cited by 608
Research designs for studies evaluating the effectiveness of change and improvement strategies
Eccles, M. (2003) · Quality and Safety in Health Care · doi:10.1136/qhc.12.1.47 · Cited by 586 · cited 4×
“widespread implementation of policy mandates, quality improvement initiatives, or dissemination campaigns (Eccles et al., 2003).”
BackgroundInfluentialA "SMART" Design for Building Individualized Treatment Sequences
Lei, H., Nahum-Shani, I., Lynch, K., Oslin, D., & Murphy, S. A. (2012) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032511-143152 · Cited by 557 · cited 1×
“Researchers interested in optimizing sequences of implementation strategies that adapt to ongoing needs over time may be interested in a variant of factorial designs known as the sequential, multiple-assignment randomized trial (SMART; Almirall et al., 2012; Collins et al., 2014b; Kilbourne et al., 2014b; Lei et al.,…”
BackgroundAdaptive Designs for Clinical Trials
Bhatt, D. L., & Mehta, C. R. (2016) · New England Journal of Medicine · doi:10.1056/nejmra1510061 · Cited by 515 · cited 1×
“These designs are becoming increasingly popular in clinical treatment (Bhatt and Mehta, 2016) but could also hold promise for implementation scientists, especially as interest grows in rapid-cycle testing of implementation strategies or efforts.”
BackgroundA strategy for optimizing and evaluating behavioral interventions
Collins, L. M., Murphy, S. A., Nair, V. N., & Strecher, V. J. (2005) · Annals of Behavioral Medicine · doi:10.1207/s15324796abm3001_8 · Cited by 492
Collins, L. M., Dziak, J. J., & Li, R. (2009) · Psychological Methods · doi:10.1037/a0015826 · Cited by 431
Stetler, C. B., Legro, M. W., Rycroft‐Malone, J., Bowman, C., Curran, G. M., Guihan, M., Hagedorn, H., Piñeros, S. L., & Wallace, C. M. (2006) · Implementation Science · doi:10.1186/1748-5908-1-23 · Cited by 429 · cited 2×
“For sites that failed to implement Life Goals under REP, two additional implementation strategies were considered as augmentations to REP: External Facilitation (EF; Kilbourne et al., 2014b; Stetler et al., 2006), consisting of phone-based mentoring in strategic skills from a study team member; and Internal Facilitati…”
BackgroundSelecting and Improving Quasi-Experimental Designs in Effectiveness and Implementation Research
Handley, M. A., Lyles, C. R., McCulloch, C., & Cattamanchi, A. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014128 · Cited by 428
Single-Subject Experimental Design for Evidence-Based Practice
Byiers, B. J., Reichle, J. E., & Symons, F. J. (2012) · American Journal of Speech-Language Pathology · doi:10.1044/1058-0360(2012/11-0036 · Cited by 339
Collins, L. M., Dziak, J. J., Kugler, K. C., & Trail, J. B. (2014) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2014.06.021 · Cited by 335
The Existence and Construction of Balanced Incomplete Block Designs
Hanani, H. (1961) · doi:10.1214/aoms/1177705047 · Cited by 326 · cited 1×
“Stepped wedge designs have their analytic roots in balanced incomplete block designs, in which all pairs of treatments occur an equal number of times within each block (Hanani, 1961).”
BackgroundCollaborative care for bipolar disorder, Part II: impact on clinical outcome, function, and costs.
Bauer, M. S., McBride, L., Williford, W. O., Glick, H., Kinosian, B., Altshuler, L., Beresford, T., Kilbourne, A. M., & Sajatovic, M. (2006) · Psychiatric Services · doi:10.1176/ps.2006.57.7.937 · Cited by 312
Effect of regression to the mean on decision making in health care
Morton, V., & Torgerson, D. J. (2003) · BMJ · doi:10.1136/bmj.326.7398.1083 · Cited by 305
Experimental design and primary data analysis methods for comparing adaptive interventions.
Nahum-Shani, I., Qian, M., Almirall, D., Pelham, W. E., Gnagy, B., Fabiano, G. A., Waxmonsky, J. G., Yu, J., & Murphy, S. A. (2012) · Psychological Methods · doi:10.1037/a0029372 · Cited by 298 · cited 1×
“Researchers interested in optimizing sequences of implementation strategies that adapt to ongoing needs over time may be interested in a variant of factorial designs known as the sequential, multiple-assignment randomized trial (SMART; Almirall et al., 2012; Collins et al., 2014b; Kilbourne et al., 2014b; Lei et al.,…”
BackgroundLong-term Effectiveness and Cost of a Systematic Care Program for Bipolar Disorder
Simon, G. E., Ludman, E. J., Bauer, M. S., Unützer, J., & Operskalski, B. H. (2006) · Archives of General Psychiatry · doi:10.1001/archpsyc.63.5.500 · Cited by 263 · cited 1×
“struct patients in self-management, and improving clinical information systems and care management across physical and mental health providers (Bauer et al., 2006; Kilbourne et al., 2012a, 2008; Simon et al., 2006).”
BackgroundOutcomes of a Partnered Facilitation Strategy to Implement Primary Care–Mental Health
Kirchner, J. E., Ritchie, M. J., Pitcock, J. A., Parker, L. E., Curran, G. M., & Fortney, J. C. (2014) · Journal of General Internal Medicine · doi:10.1007/s11606-014-3027-2 · Cited by 256 · cited 3×
“For sites that failed to implement Life Goals under REP, two additional implementation strategies were considered as augmentations to REP: External Facilitation (EF; Kilbourne et al., 2014b; Stetler et al., 2006), consisting of phone-based mentoring in strategic skills from a study team member; and Internal Facilitati…”
BackgroundMethodTaljaard, M., McKenzie, J. E., Ramsay, C. R., & Grimshaw, J. M. (2014) · Implementation Science · doi:10.1186/1748-5908-9-77 · Cited by 237 · cited 2×
“More formally, ITS evaluations focus on identifying whether there is discontinuity in the trend (change in slope or level) after the intervention relative to before the intervention, using segmented regression to model pre- and post-intervention trends (Gebski et al., 2012; Penfold and Zhang, 2013; Taljaard et al., 20…”
BackgroundAlmirall, D., Compton, S. N., Gunlicks‐Stoessel, M., Duan, N., & Murphy, S. A. (2012) · Statistics in Medicine · doi:10.1002/sim.4512 · Cited by 223 · cited 1×
“Researchers interested in optimizing sequences of implementation strategies that adapt to ongoing needs over time may be interested in a variant of factorial designs known as the sequential, multiple-assignment randomized trial (SMART; Almirall et al., 2012; Collins et al., 2014b; Kilbourne et al., 2014b; Lei et al.,…”
BackgroundCollins, L. M., Nahum-Shani, I. B., & Almirall, D. (2014) · Clinical Trials · doi:10.1177/1740774514536795 · Cited by 200
Reducing bias using propensity score matching
Morgan, C. J. (2018) · Journal of Nuclear Cardiology · doi:10.1007/s12350-017-1012-y · Cited by 193
Kilbourne, A. M., Post, E. P., Nossek, A., Drill, L., Cooley, S., & Bauer, M. S. (2008) · Psychiatric Services · doi:10.1176/ps.2008.59.7.760 · Cited by 142
Kilbourne, A. M., Almirall, D., Eisenberg, D., Waxmonsky, J., Goodrich, D. E., Fortney, J. C., Kirchner, J. E., Solberg, L. I., Main, D., Bauer, M. S., et al. (2014) · Implementation Science · doi:10.1186/s13012-014-0132-x · Cited by 130
Fretheim, A., Zhang, F., Ross-Degnan, D., Oxman, A., Cheyne, H., Foy, R., Goodacre, S., Herrin, J., Kerse, N., McKinlay, R. J., et al. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.10.003 · Cited by 119
Fretheim, A., Soumerai, S. B., Zhang, F., Oxman, A. D., & Ross‐Degnan, D. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.03.016 · Cited by 98
Davey, C., Hargreaves, J., Thompson, J. A., Copas, A. J., Beard, E., Lewis, J. J., & Fielding, K. (2015) · Trials · doi:10.1186/s13063-015-0838-3 · Cited by 98
Pellegrini, C. A., Hoffman, S. A., Collins, L. M., & Spring, B. J. (2014) · Contemporary Clinical Trials · doi:10.1016/j.cct.2014.05.007 · Cited by 88 · cited 1×
“, sites or providers) to different combinations of implementation strategies, and can be used to evaluate the effectiveness of each strategy individually to inform an optimal combination (e.g., Coulton et al., 2009; Pellegrini et al., 2014; Wyrick et al., 2014).”
MethodModelling interrupted time series to evaluate prevention and control of infection in healthcare
Gebski, V., Ellingson, K., Edwards, J. R., Jernigan, J. A., & Kleinbaum, D. G. (2012) · Epidemiology and Infection · doi:10.1017/s0950268812000179 · Cited by 87 · cited 1×
“More formally, ITS evaluations focus on identifying whether there is discontinuity in the trend (change in slope or level) after the intervention relative to before the intervention, using segmented regression to model pre- and post-intervention trends (Gebski et al., 2012; Penfold and Zhang, 2013; Taljaard et al., 20…”
Pridemore, W., & Snowden, A. J. (2009) · American Journal of Public Health · doi:10.2105/ajph.2008.146183 · Cited by 80 · cited 2×
“A number of recent implementation studies have used ITS designs, including an evaluation of implementation of a comprehensive smoke-free policy in a large UK mental health organization to reduce physical assaults (Robson et al., 2017); the impact of a national policy limiting alcohol availability on suicide mortality…”
BackgroundBauer, M. S., Miller, C. J., Kim, B., Lew, R., Stolzmann, K. L., Sullivan, J. L., Riendeau, R. P., Pitcock, J. A., Williamson, A. K., Connolly, S. L., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.0230 · Cited by 79
Self-citationRobson, D., Spaducci, G., McNeill, A. D., Stewart, D., Craig, T., Yates, M., & Szatkowski, L. C. (2017) · The Lancet Psychiatry · doi:10.1016/s2215-0366(17)30209-2 · Cited by 65 · cited 1×
“studies have used ITS designs, including an evaluation of implementation of a comprehensive smoke-free policy in a large UK mental health organization to reduce physical assaults (Robson et al., 2017); the impact of a national policy limiting alcohol availability on suicide mortality in Slovenia (Pridemore and Snowden…”
MethodPartnering with health system operations leadership to develop a controlled implementation trial
Bauer, M. S., Miller, C., Kim, B., Lew, R., Weaver, K. R., Coldwell, C. M., Henderson, K., Holmes, S. K., Seibert, M. N., Stolzmann, K. L., et al. (2015) · Implementation Science · doi:10.1186/s13012-016-0385-7 · Cited by 62
Self-citationLife Goals Collaborative Care for Patients With Bipolar Disorder and Cardiovascular Disease Risk
Kilbourne, A. M., Goodrich, D. E., Lai, Z., Clogston, J., Waxmonsky, J. A., & Bauer, M. S. (2012) · Psychiatric Services · doi:10.1176/appi.ps.201100528 · Cited by 61
Wyrick, D. L., Rulison, K. L., Fearnow-Kenney, M., Milroy, J. J., & Collins, L. M. (2014) · Translational Behavioral Medicine · doi:10.1007/s13142-013-0247-7 · Cited by 57 · cited 1×
“, sites or providers) to different combinations of implementation strategies, and can be used to evaluate the effectiveness of each strategy individually to inform an optimal combination (e.g., Coulton et al., 2009; Pellegrini et al., 2014; Wyrick et al., 2014).”
MethodCheung, Y. K., Chakraborty, B., & Davidson, K. W. (2015) · Biometrics · doi:10.1111/biom.12258 · Cited by 55 · cited 1×
“Adaptive designs could potentially be incorporated into both SMART designs and stepped wedge studies, as well as traditional RCTs to further advance implementation science (Cheung et al., 2015).”
MethodCoulton, S., Perryman, K. A., Bland, M., Cassidy, P., Crawford, M., Deluca, P., Drummond, C., Gilvarry, E. A., Godfrey, C., Heather, N., et al. (2009) · BMC Health Services Research · doi:10.1186/1472-6963-9-114 · Cited by 52
Kilbourne, A. M., Almirall, D., Goodrich, D. E., Lai, Z., Abraham, K. M., Nord, K. M., & Bowersox, N. W. (2014) · Implementation Science · doi:10.1186/s13012-014-0163-3 · Cited by 50
Kilbourne, A. M., Neumann, M. S., Waxmonsky, J. A., Bauer, M. S., Kim, H. M., Pincus, H. A., & Thomas, M. R. (2012) · Psychiatric Services · doi:10.1176/appi.ps.201200032 · Cited by 40
Betrán, A. P., Bergel, E., Griffin, S., Melo, A., Nguyen, M. H., Carbonell, A., Mondlane, S., Merialdi, M., Temmerman, M., Gülmezoglu, A. M., et al. (2018) · The Lancet Global Health · doi:10.1016/s2214-109x(17)30421-7 · Cited by 40 · cited 1×
“In a stepped wedge, all participants receive the intervention, but are assigned to the timing of the intervention in a staggered fashion (Betran et al., 2018; Brown and Lilford, 2006; Hussey and Hughes, 2007), ty-”
BackgroundImmediate versus sustained effects: interrupted time series analysis of a tailored intervention
Hanbury, A., Farley, K., Thompson, C. A., Wilson, P. M., Chambers, D., & Holmes, H. (2013) · Implementation Science · doi:10.1186/1748-5908-8-130 · Cited by 34 · cited 1×
“, 2017); the impact of a national policy limiting alcohol availability on suicide mortality in Slovenia (Pridemore and Snowden, 2009); and the effect of delivery of a tailored intervention for primary care providers to increase psychological referrals for women with mild to moderate postnatal depression (Hanbury et al…”
BackgroundNeCamp, T., Kilbourne, A. M., & Almirall, D. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280217708654 · Cited by 33 · cited 10×
“SMARTs are multistage randomized trials in which some or all participants are randomized more than once, often based on ongoing information (e.g., treatment response).”
BackgroundMethodInfluentialKilbourne, A. M., Goodrich, D. E., Nord, K. M., Poppelen, C. V., Kyle, J., Bauer, M. S., Waxmonsky, J. A., Lai, Z., Kim, H. M., Eisenberg, D., et al. (2014) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0598-5 · Cited by 33
Kilbourne, A., Bramlet, M., Barbaresso, M. M., Nord, K. M., Goodrich, D. E., Lai, Z.-S., Post, E., Almirall, D., Verchinina, L., Duffy, S., et al. (2014) · Contemporary Clinical Trials · doi:10.1016/j.cct.2014.07.007 · Cited by 28
Pape, U., Millett, C., Lee, J., Car, J., & Majeed, A. (2013) · Journal of the Royal Society of Medicine · doi:10.1258/jrsm.2012.110319 · Cited by 24 · cited 1×
“Similarly, this design is particularly sensitive to secular trends that may differentially affect the treatment and control groups (Cousins et al., 2014; Pape et al., 2013), as well as regression to the mean confounding study results (Morton and Torgerson, 2003).”
BackgroundSchildcrout, J., Schisterman, E., Mercaldo, N., Rathouz, P., & Heagerty, P. (2018) · Epidemiology · doi:10.1097/ede.0000000000000764 · Cited by 19
A method to reduce imbalance for site-level randomized stepped wedge implementation trial designs
Lew, R., Miller, C. J., Kim, B., Wu, H.-S., Stolzmann, K. L., & Bauer, M. (2019) · Implementation Science · doi:10.1186/s13012-019-0893-3 · Cited by 18
Self-citationCousins, K., CONNOR, J. L., & Kypri, K. (2014) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2014.07.015 · Cited by 11
Research works printed but not linked: 1
- Neumann, 2000, Replicating effective programs: HIV/AIDS prevention technology transfer, AIDS Educ. Prev., 12, 35
Cited by
Sorted by citations
Designing and undertaking randomised implementation trials: guide for researchers
Wolfenden, L., Foy, R., Presseau, J., Grimshaw, J., Ivers, N. M., Powell, B. J., Taljaard, M., Wiggers, J., Sutherland, R., Nathan, N., et al. (2021) · BMJ · doi:10.1136/bmj.m3721 · Cited by 287
Reid, R. J., Wodchis, W. P., Kuluski, K., Lee-Foon, N. K., Lavis, J. N., Rosella, L. C., & Desveaux, L. (2024) · SSM - Health Systems · doi:10.1016/j.ssmhs.2024.100010 · Cited by 109
Designs and methods for implementation research: Advancing the mission of the CTSA program
Hwang, S., Birken, S. A., Melvin, C. L., Rohweder, C. L., & Smith, J. D. (2020) · Journal of Clinical and Translational Science · doi:10.1017/cts.2020.16 · Cited by 99
Thomas, D. R., Lin, J., Gatz, E., Gurung, A., Gupta, S., Norberg, K., Fancsali, S. E., Aleven, V. A. W. M. M., Branstetter, L., Brunskill, E., et al. (2024) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3636555.3636896 · Cited by 91
“Randomized controlled trials (RCTs), are universally accepted as the ideal methodology for assessing intervention efficacy and impact [23].”
BackgroundMethodInfluentialA Meta-analysis of University STEM Summer Bridge Program Effectiveness
Bradford, B. C., Beier, M. E., & Oswald, F. L. (2021) · CBE—Life Sciences Education · doi:10.1187/cbe.20-03-0046 · Cited by 73
Metadata
- Authors
- Miller, C. J., Smith, S. N., & Pugatch, M.
- Affiliations
- United States Department of Veterans Affairs (US)
- University of Michigan (US)
- Corresponding author
- Miller, Christopher J.
- Venue
- Psychiatry Research
- Publisher
- Elsevier BV
- Published
- 2020-01
- Volume
- 283
- Pages
- 112452
- ISSN
- 0165-1781 (ISSN-L)
- PMC
- PMC6923620
- OpenAlex
- W2951715721
- Cited by
- 373
- References
- 67
- Citations by year
- 2026: 63 · 2025: 89 · 2024: 67 · 2023: 63 · 2022: 56 · 2021: 20 · 2020: 14
- FWCI
- 4.18
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Psychiatry Research (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health
- U.S. Department of Veterans Affairs
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- implementation research, implementation science, experimental design, quasi-experimental design, randomized controlled trials, interrupted time series, stepped wedge design, pre-post design, research design selection, clinical adoption, intervention sustainability
- MeSH terms
- Implementation Science, Humans, Research Design, Randomized Controlled Trials as Topic, Control Groups, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
