Experimental and quasi-experimental designs in implementation research

Miller, Christopher J.; Smith, Shawna N.; Pugatch, Marianne

doi:10.1016/j.psychres.2019.06.027

Cited by 373Health Policy Implementation ScienceStatistical Methods in Clinical TrialsAdvanced Causal Inference Techniques

Experimental and quasi-experimental designs in implementation research

Open access

Miller, Christopher J., Smith, Shawna N., Pugatch, Marianne·2020·Psychiatry Research

DOI 10.1016/j.psychres.2019.06.027PMC PMC6923620

Open this paper in CoChat — free account Open PDF

Abstract

Implementation science is focused on maximizing the adoption, appropriate use, and sustainability of effective clinical practices in real world clinical settings. Many implementation science questions can be feasibly answered by fully experimental designs, typically in the form of randomized controlled trials (RCTs). Implementation-focused RCTs, however, usually differ from traditional efficacy- or effectiveness-oriented RCTs on key parameters. Other implementation science questions are more suited to quasi-experimental designs, which are intended to estimate the effect of an intervention in the absence of randomization. These designs include pre-post designs with a non-equivalent control group, interrupted time series (ITS), and stepped wedges, the last of which require all participants to receive the intervention, but in a staggered fashion. In this article we review the use of experimental designs in implementation science, including recent methodological advances for implementation studies. We also review the use of quasi-experimental designs in implementation science, and discuss the strengths and weaknesses of these approaches. This article is therefore meant to be a practical guide for researchers who are interested in selecting the most appropriate study design to answer relevant implementation science questions, and thereby increase the rate at which effective clinical practices are adopted, spread, and sustained.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1016/j.psychres.2019.06.027

Cites

64 of 64 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Quasi-experiments are appropriate for practical situations and ethical compliance for studies involving vulnerable participants such as older adults, and allow scholars to conduct rigorous studies under certain limitations and non-control conditions.
“Quasiexperiments are appropriate for practical situations and ethical compliance for studies involving vulnerable participants such as older adults, 17 and allow scholars to conduct rigorous studies under certain limitations and non-control conditions. 18”
From Mixed reality technology for older adults: Evaluating the impact of a novel virtual humanoid coach in a community-based physical exercise program in the Philippines (2024) · cites this paper as [18]
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · cited 1× · see citing works
“Second, as a corollary of this point, implementation RCTs typically feature different outcome measures than efficacy or effectiveness RCTs, with an emphasis on the extent to which a health intervention was successfully implemented rather than an evaluation of the health effects of that intervention (Proctor et al., 20…”
Background
Evaluating the public health impact of health promotion interventions: the RE-AIM framework.
Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999) · American Journal of Public Health · doi:10.2105/ajph.89.9.1322 · Cited by 7,217 · cited 1× · see citing works
“A variety of evaluation-oriented implementation frameworks may guide the choices of such measures (e.g., RE-AIM; Gaglio et al., 2013; Glasgow et al., 1999).”
Background
Design of Experiments
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980 · see citing works
Effectiveness-implementation Hybrid Designs: Combining Elements of Clinical Effectiveness and Implementation Research to Enhance Public Health Impact
Curran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012) · Medical Care · doi:10.1097/mlr.0b013e3182408812 · Cited by 4,058 · cited 1× · see citing works
“Hybrid implementation-effectiveness studies attend to both effectiveness and implementation outcomes (Curran et al., 2012); these designs are also covered in more detail elsewhere in this issue (Landes, this issue).”
Background
Segmented regression analysis of interrupted time series studies in medication use research
Wagner, A. K., Soumerai, S. B., Zhang, F., & Ross-Degnan, D. (2002) · Journal of Clinical Pharmacy and Therapeutics · doi:10.1046/j.1365-2710.2002.00430.x · Cited by 3,279 · cited 2× · see citing works
“More formally, ITS evaluations focus on identifying whether there is discontinuity in the trend (change in slope or level) after the intervention relative to before the intervention, using segmented regression to model pre- and post-intervention trends (Gebski et al., 2012; Penfold and Zhang, 2013; Taljaard et al., 20…”
Background
Interrupted time series regression for the evaluation of public health interventions: a tutorial
Lopez Bernal, J., Cummins, S., & Gasparrini, A. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyw098 · Cited by 3,049 · cited 4× · see citing works
“Relative to an RCT, ITS designs can also allow for a more comprehensive assessment of the longitudinal effects of an intervention (positive or negative), as effects can be traced over all included time points (Bernal et al., 2017; Penfold and Zhang, 2013).”
BackgroundInfluential
Improving Primary Care for Patients With Chronic Illness
Bodenheimer, T. (2002) · JAMA · doi:10.1001/jama.288.14.1775 · Cited by 2,961 · see citing works
Organizing Care for Patients with Chronic Illness
Wagner, E. H., Austin, B. T., & Korff, M. V. (1996) · The Milbank Quarterly · doi:10.2307/3350391 · Cited by 2,877 · cited 1× · see citing works
“, 2014) to implement care practices consistent with the collaborative chronic care model (CCM; Bodenheimer et al., 2002a, 2002b; Wagner et al., 1996) in nine outpatient mental health teams in VA medical centers.”
Method
Theory of Statistical Estimation
Fisher, R. A. (1925) · Mathematical Proceedings of the Cambridge Philosophical Society · doi:10.1017/s0305004100009580 · Cited by 2,684 · see citing works
Improving Primary Care for Patients With Chronic Illness: The Chronic Care Model, Part 2
Bodenheimer, T., Wagner, E. H., & Grumbach, K. (2002) · JAMA · doi:10.1001/jama.288.15.1909 · Cited by 2,413 · see citing works
Design and analysis of stepped wedge cluster randomized trials.
Hussey, M. A., & Hughes, J. (2007) · Contemporary Clinical Trials · doi:10.1016/j.cct.2006.05.007 · Cited by 1,297 · cited 1× · see citing works
“In a stepped wedge, all participants receive the intervention, but are assigned to the timing of the intervention in a staggered fashion (Betran et al., 2018; Brown and Lilford, 2006; Hussey and Hughes, 2007), ty-”
Background
Methods for Evaluating Changes in Health Care Policy: The Difference-in-Differences Approach
Dimick, J. B., & Ryan, A. M. (2014) · JAMA · doi:10.1001/jama.2014.16153 · Cited by 1,193 · cited 1× · see citing works
“, Morgan, 2018) can statistically mitigate internal validity concerns, although this approach may be of limited utility when comparing secular trends between different study cohorts (Dimick and Ryan, 2014).”
Background
Use of Interrupted Time Series Analysis in Evaluating Health Care Quality Improvements
Penfold, R. B., & Zhang, F. (2013) · Academic Pediatrics · doi:10.1016/j.acap.2013.08.002 · Cited by 1,150 · cited 3× · see citing works
“, 2003): a common recommendation is at least eight time points both pre- and post-intervention (Penfold and Zhang, 2013).”
Background
The Multiphase Optimization Strategy (MOST) and the Sequential Multiple Assignment Randomized Trial (SMART)
Collins, L. M., Murphy, S. A., & Strecher, V. (2007) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2007.01.022 · Cited by 1,082 · see citing works
The RE-AIM framework: a systematic review of use over time
Gaglio, B., Shoup, J. A., & Glasgow, R. E. (2013) · American Journal of Public Health · doi:10.2105/ajph.2013.301299 · Cited by 1,029 · cited 1× · see citing works
“A variety of evaluation-oriented implementation frameworks may guide the choices of such measures (e.g., RE-AIM; Gaglio et al., 2013; Glasgow et al., 1999).”
Background
The stepped wedge trial design: a systematic review
Brown, C. A., & Lilford, R. J. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-54 · Cited by 907 · see citing works
Adaptive designs in clinical trials: why use them, and how to run and report them
Pallmann, P., Bedding, A. W., Choodari-Oskooei, B., Dimairo, M., Flight, L., Hampson, L. V., Holmes, J., Mander, A. P., Odondi, L., Sydes, M. R., et al. (2018) · BMC Medicine · doi:10.1186/s12916-018-1017-7 · Cited by 829 · see citing works
Implementing evidence-based interventions in health care: application of the replicating effective programs framework
Kilbourne, A. M., Neumann, M. S., Pincus, H. A., Bauer, M. S., & Stall, R. (2007) · Implementation Science · doi:10.1186/1748-5908-2-42 · Cited by 653 · see citing works
A Conceptual Framework for Adaptive Preventive Interventions
Collins, L. M., Murphy, S. A., & Bierman, K. L. (2004) · Prevention Science · doi:10.1023/b:prev.0000037641.26017.00 · Cited by 608 · see citing works
Research designs for studies evaluating the effectiveness of change and improvement strategies
Eccles, M. (2003) · Quality and Safety in Health Care · doi:10.1136/qhc.12.1.47 · Cited by 586 · cited 4× · see citing works
“widespread implementation of policy mandates, quality improvement initiatives, or dissemination campaigns (Eccles et al., 2003).”
BackgroundInfluential
A "SMART" Design for Building Individualized Treatment Sequences
Lei, H., Nahum-Shani, I., Lynch, K., Oslin, D., & Murphy, S. A. (2012) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032511-143152 · Cited by 557 · cited 1× · see citing works
“Researchers interested in optimizing sequences of implementation strategies that adapt to ongoing needs over time may be interested in a variant of factorial designs known as the sequential, multiple-assignment randomized trial (SMART; Almirall et al., 2012; Collins et al., 2014b; Kilbourne et al., 2014b; Lei et al.,…”
Background
Adaptive Designs for Clinical Trials
Bhatt, D. L., & Mehta, C. R. (2016) · New England Journal of Medicine · doi:10.1056/nejmra1510061 · Cited by 506 · cited 1× · see citing works
“These designs are becoming increasingly popular in clinical treatment (Bhatt and Mehta, 2016) but could also hold promise for implementation scientists, especially as interest grows in rapid-cycle testing of implementation strategies or efforts.”
Background
A strategy for optimizing and evaluating behavioral interventions
Collins, L. M., Murphy, S. A., Nair, V. N., & Strecher, V. J. (2005) · Annals of Behavioral Medicine · doi:10.1207/s15324796abm3001_8 · Cited by 492 · see citing works
Design of experiments with multiple independent variables: A resource management perspective on complete and reduced factorial designs.
Collins, L. M., Dziak, J. J., & Li, R. (2009) · Psychological Methods · doi:10.1037/a0015826 · Cited by 431 · see citing works
Selecting and Improving Quasi-Experimental Designs in Effectiveness and Implementation Research
Handley, M. A., Lyles, C. R., McCulloch, C., & Cattamanchi, A. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014128 · Cited by 428 · see citing works
Role of "external facilitation" in implementation of research findings: a qualitative evaluation of facilitation experiences in the Veterans Health Administration
Stetler, C., Legro, M., Rycroft-Malone, J., Bowman, C., Curran, G., Guihan, M., Hagedorn, H., Piñeros, S. L., & Wallace, C. (2006) · Implementation science : IS · doi:10.1186/1748-5908-1-23 · Cited by 392 · cited 2× · see citing works
“For sites that failed to implement Life Goals under REP, two additional implementation strategies were considered as augmentations to REP: External Facilitation (EF; Kilbourne et al., 2014b; Stetler et al., 2006), consisting of phone-based mentoring in strategic skills from a study team member; and Internal Facilitati…”
Background
Single-Subject Experimental Design for Evidence-Based Practice
Byiers, B. J., Reichle, J. E., & Symons, F. J. (2012) · American Journal of Speech-Language Pathology · doi:10.1044/1058-0360(2012/11-0036 · Cited by 339 · see citing works
Factorial Experiments
Collins, L. M., Dziak, J. J., Kugler, K. C., & Trail, J. B. (2014) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2014.06.021 · Cited by 335 · see citing works
The Existence and Construction of Balanced Incomplete Block Designs
Hanani, H. (1961) · doi:10.1214/aoms/1177705047 · Cited by 326 · cited 1×
“Stepped wedge designs have their analytic roots in balanced incomplete block designs, in which all pairs of treatments occur an equal number of times within each block (Hanani, 1961).”
Background
Collaborative care for bipolar disorder, Part II: impact on clinical outcome, function, and costs.
Bauer, M. S., McBride, L., Williford, W. O., Glick, H., Kinosian, B., Altshuler, L., Beresford, T., Kilbourne, A. M., & Sajatovic, M. (2006) · Psychiatric Services · doi:10.1176/ps.2006.57.7.937 · Cited by 312 · see citing works
Effect of regression to the mean on decision making in health care
Morton, V., & Torgerson, D. J. (2003) · BMJ · doi:10.1136/bmj.326.7398.1083 · Cited by 305 · see citing works
Experimental design and primary data analysis methods for comparing adaptive interventions.
Nahum-Shani, I., Qian, M., Almirall, D., Pelham, W. E., Gnagy, B., Fabiano, G. A., Waxmonsky, J. G., Yu, J., & Murphy, S. A. (2012) · Psychological Methods · doi:10.1037/a0029372 · Cited by 298 · cited 1× · see citing works
“Researchers interested in optimizing sequences of implementation strategies that adapt to ongoing needs over time may be interested in a variant of factorial designs known as the sequential, multiple-assignment randomized trial (SMART; Almirall et al., 2012; Collins et al., 2014b; Kilbourne et al., 2014b; Lei et al.,…”
Background
Long-term Effectiveness and Cost of a Systematic Care Program for Bipolar Disorder
Simon, G. E., Ludman, E. J., Bauer, M. S., Unützer, J., & Operskalski, B. H. (2006) · Archives of General Psychiatry · doi:10.1001/archpsyc.63.5.500 · Cited by 264 · cited 1× · see citing works
“struct patients in self-management, and improving clinical information systems and care management across physical and mental health providers (Bauer et al., 2006; Kilbourne et al., 2012a, 2008; Simon et al., 2006).”
Background
Outcomes of a Partnered Facilitation Strategy to Implement Primary Care–Mental Health
Kirchner, J. E., Ritchie, M. J., Pitcock, J. A., Parker, L. E., Curran, G. M., & Fortney, J. C. (2014) · Journal of General Internal Medicine · doi:10.1007/s11606-014-3027-2 · Cited by 256 · cited 3× · see citing works
“For sites that failed to implement Life Goals under REP, two additional implementation strategies were considered as augmentations to REP: External Facilitation (EF; Kilbourne et al., 2014b; Stetler et al., 2006), consisting of phone-based mentoring in strategic skills from a study team member; and Internal Facilitati…”
BackgroundMethod
The use of segmented regression in analysing interrupted time series studies: an example in pre-hospital ambulance care
Taljaard, M., McKenzie, J. E., Ramsay, C. R., & Grimshaw, J. M. (2014) · Implementation Science · doi:10.1186/1748-5908-9-77 · Cited by 237 · cited 2× · see citing works
“More formally, ITS evaluations focus on identifying whether there is discontinuity in the trend (change in slope or level) after the intervention relative to before the intervention, using segmented regression to model pre- and post-intervention trends (Gebski et al., 2012; Penfold and Zhang, 2013; Taljaard et al., 20…”
Background
Designing a pilot sequential multiple assignment randomized trial for developing an adaptive treatment strategy
Almirall, D., Compton, S. N., Gunlicks‐Stoessel, M., Duan, N., & Murphy, S. A. (2012) · Statistics in Medicine · doi:10.1002/sim.4512 · Cited by 223 · cited 1× · see citing works
“Researchers interested in optimizing sequences of implementation strategies that adapt to ongoing needs over time may be interested in a variant of factorial designs known as the sequential, multiple-assignment randomized trial (SMART; Almirall et al., 2012; Collins et al., 2014b; Kilbourne et al., 2014b; Lei et al.,…”
Background
Optimization of behavioral dynamic treatment regimens based on the sequential, multiple assignment, randomized trial (SMART)
Collins, L. M., Nahum-Shani, I. B., & Almirall, D. (2014) · Clinical Trials · doi:10.1177/1740774514536795 · Cited by 194 · see citing works
Reducing bias using propensity score matching
Morgan, C. J. (2018) · Journal of Nuclear Cardiology · doi:10.1007/s12350-017-1012-y · Cited by 193 · see citing works
Improving Medical and Psychiatric Outcomes Among Individuals With Bipolar Disorder: A Randomized Controlled Trial
Kilbourne, A. M., Post, E. P., Nossek, A., Drill, L., Cooley, S., & Bauer, M. S. (2008) · Psychiatric Services · doi:10.1176/ps.2008.59.7.760 · Cited by 142 · see citing works
Protocol: Adaptive Implementation of Effective Programs Trial (ADEPT): cluster randomized SMART trial comparing a standard versus enhanced implementation strategy to improve outcomes of a mood disorders program
Kilbourne, A. M., Almirall, D., Eisenberg, D., Waxmonsky, J., Goodrich, D. E., Fortney, J. C., Kirchner, J. E., Solberg, L. I., Main, D., Bauer, M. S., et al. (2014) · Implementation Science · doi:10.1186/s13012-014-0132-x · Cited by 130 · see citing works
A reanalysis of cluster randomized trials showed interrupted time-series studies were valuable in health system evaluation
Fretheim, A., Zhang, F., Ross‐Degnan, D., Oxman, A. D., Cheyne, H., Foy, R., Goodacre, S. W., Herrin, J. S., Kerse, N. M., McKinlay, R. J., et al. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.10.003 · Cited by 126 · see citing works
Interrupted time-series analysis yielded an effect estimate concordant with the cluster-randomized controlled trial result
Fretheim, A., Soumerai, S. B., Zhang, F., Oxman, A. D., & Ross‐Degnan, D. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.03.016 · Cited by 98 · see citing works
Analysis and reporting of stepped wedge randomised controlled trials: synthesis and critical appraisal of published studies, 2010 to 2014
Davey, C., Hargreaves, J., Thompson, J. A., Copas, A. J., Beard, E., Lewis, J. J., & Fielding, K. (2015) · Trials · doi:10.1186/s13063-015-0838-3 · Cited by 93 · see citing works
Modelling interrupted time series to evaluate prevention and control of infection in healthcare
Gebski, V., Ellingson, K., Edwards, J. R., Jernigan, J. A., & Kleinbaum, D. G. (2012) · Epidemiology and Infection · doi:10.1017/s0950268812000179 · Cited by 85 · cited 1× · see citing works
“More formally, ITS evaluations focus on identifying whether there is discontinuity in the trend (change in slope or level) after the intervention relative to before the intervention, using segmented regression to model pre- and post-intervention trends (Gebski et al., 2012; Penfold and Zhang, 2013; Taljaard et al., 20…”
Optimization of remotely delivered intensive lifestyle treatment for obesity using the Multiphase Optimization Strategy: Opt-IN study protocol
Pellegrini, C. A., Hoffman, S. A., Collins, L. M., & Spring, B. J. (2014) · Contemporary Clinical Trials · doi:10.1016/j.cct.2014.05.007 · Cited by 83 · cited 1× · see citing works
“, sites or providers) to different combinations of implementation strategies, and can be used to evaluate the effectiveness of each strategy individually to inform an optimal combination (e.g., Coulton et al., 2009; Pellegrini et al., 2014; Wyrick et al., 2014).”
Method
Reduction in suicide mortality following a new national alcohol policy in Slovenia: an interrupted time-series analysis.
Pridemore, W., & Snowden, A. J. (2009) · American Journal of Public Health · doi:10.2105/ajph.2008.146183 · Cited by 80 · cited 2× · see citing works
“A number of recent implementation studies have used ITS designs, including an evaluation of implementation of a comprehensive smoke-free policy in a large UK mental health organization to reduce physical assaults (Robson et al., 2017); the impact of a national policy limiting alcohol availability on suicide mortality…”
Background
Effectiveness of Implementing a Collaborative Chronic Care Model for Clinician Teams on Patient Outcomes and Health Status in Mental Health
Bauer, M. S., Miller, C. J., Kim, B., Lew, R., Stolzmann, K. L., Sullivan, J. L., Riendeau, R. P., Pitcock, J. A., Williamson, A. K., Connolly, S. L., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.0230 · Cited by 76 · see citing works
Self-citation
Effect of implementation of a smoke-free policy on physical violence in a psychiatric inpatient setting: an interrupted time series analysis
Robson, D., Spaducci, G., McNeill, A. D., Stewart, D., Craig, T., Yates, M., & Szatkowski, L. C. (2017) · The Lancet Psychiatry · doi:10.1016/s2215-0366(17)30209-2 · Cited by 65 · cited 1× · see citing works
“studies have used ITS designs, including an evaluation of implementation of a comprehensive smoke-free policy in a large UK mental health organization to reduce physical assaults (Robson et al., 2017); the impact of a national policy limiting alcohol availability on suicide mortality in Slovenia (Pridemore and Snowden…”
Method
Partnering with health system operations leadership to develop a controlled implementation trial
Bauer, M. S., Miller, C., Kim, B., Lew, R., Weaver, K. R., Coldwell, C. M., Henderson, K., Holmes, S. K., Seibert, M. N., Stolzmann, K. L., et al. (2015) · Implementation Science · doi:10.1186/s13012-016-0385-7 · Cited by 62 · see citing works
Self-citation
Life Goals Collaborative Care for Patients With Bipolar Disorder and Cardiovascular Disease Risk
Kilbourne, A. M., Goodrich, D. E., Lai, Z., Clogston, J., Waxmonsky, J. A., & Bauer, M. S. (2012) · Psychiatric Services · doi:10.1176/appi.ps.201100528 · Cited by 61 · see citing works
Moving beyond the treatment package approach to developing behavioral interventions: addressing questions that arose during an application of the Multiphase Optimization Strategy (MOST)
Wyrick, D. L., Rulison, K. L., Fearnow-Kenney, M., Milroy, J. J., & Collins, L. M. (2014) · Translational Behavioral Medicine · doi:10.1007/s13142-013-0247-7 · Cited by 57 · cited 1× · see citing works
“, sites or providers) to different combinations of implementation strategies, and can be used to evaluate the effectiveness of each strategy individually to inform an optimal combination (e.g., Coulton et al., 2009; Pellegrini et al., 2014; Wyrick et al., 2014).”
Method
Sequential Multiple Assignment Randomized Trial (SMART) with Adaptive Randomization for Quality Improvement in Depression Treatment Program
Cheung, Y. K., Chakraborty, B., & Davidson, K. W. (2015) · Biometrics · doi:10.1111/biom.12258 · Cited by 55 · cited 1× · see citing works
“Adaptive designs could potentially be incorporated into both SMART designs and stepped wedge studies, as well as traditional RCTs to further advance implementation science (Cheung et al., 2015).”
Method
Screening and brief interventions for hazardous alcohol use in accident and emergency departments: a randomised controlled trial protocol
Coulton, S., Perryman, K. A., Bland, M., Cassidy, P., Crawford, M., Deluca, P., Drummond, C., Gilvarry, E. A., Godfrey, C., Heather, N., et al. (2009) · BMC Health Services Research · doi:10.1186/1472-6963-9-114 · Cited by 52 · see citing works
Enhancing outreach for persons with serious mental illness: 12-month results from a cluster randomized trial of an adaptive implementation strategy
Kilbourne, A. M., Almirall, D., Goodrich, D. E., Lai, Z., Abraham, K. M., Nord, K. M., & Bowersox, N. W. (2014) · Implementation Science · doi:10.1186/s13012-014-0163-3 · Cited by 50 · see citing works
Public-Academic Partnerships: Evidence-Based Implementation: The Role of Sustained Community-Based Practice and Research Partnerships
Kilbourne, A. M., Neumann, M. S., Waxmonsky, J. A., Bauer, M. S., Kim, H. M., Pincus, H. A., & Thomas, M. R. (2012) · Psychiatric Services · doi:10.1176/appi.ps.201200032 · Cited by 40 · see citing works
Provision of medical supply kits to improve quality of antenatal care in Mozambique: a stepped-wedge cluster randomised trial
Betrán, A. P., Bergel, E., Griffin, S., Melo, A., Nguyen, M. H., Carbonell, A., Mondlane, S., Merialdi, M., Temmerman, M., Gülmezoglu, A. M., et al. (2018) · The Lancet Global Health · doi:10.1016/s2214-109x(17)30421-7 · Cited by 40 · cited 1× · see citing works
“In a stepped wedge, all participants receive the intervention, but are assigned to the timing of the intervention in a staggered fashion (Betran et al., 2018; Brown and Lilford, 2006; Hussey and Hughes, 2007), ty-”
Background
Comparing cluster-level dynamic treatment regimens using sequential, multiple assignment, randomized trials: Regression estimation and sample size considerations
NeCamp, T., Kilbourne, A. M., & Almirall, D. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280217708654 · Cited by 33 · cited 10× · see citing works
“SMARTs are multistage randomized trials in which some or all participants are randomized more than once, often based on ongoing information (e.g., treatment response).”
BackgroundMethodInfluential
Long-Term Clinical Outcomes from a Randomized Controlled Trial of Two Implementation Strategies to Promote Collaborative Care Attendance in Community Practices
Kilbourne, A. M., Goodrich, D. E., Nord, K. M., Poppelen, C. V., Kyle, J., Bauer, M. S., Waxmonsky, J. A., Lai, Z., Kim, H. M., Eisenberg, D., et al. (2014) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0598-5 · Cited by 33 · see citing works
Immediate versus sustained effects: interrupted time series analysis of a tailored intervention
Hanbury, A., Farley, K., Thompson, C. A., Wilson, P. M., Chambers, D., & Holmes, H. (2013) · Implementation Science · doi:10.1186/1748-5908-8-130 · Cited by 31 · cited 1× · see citing works
“, 2017); the impact of a national policy limiting alcohol availability on suicide mortality in Slovenia (Pridemore and Snowden, 2009); and the effect of delivery of a tailored intervention for primary care providers to increase psychological referrals for women with mild to moderate postnatal depression (Hanbury et al…”
Background
SMI Life Goals: Description of a Randomized Trial of a Collaborative Care Model to Improve Outcomes for Persons with Serious Mental Illness
Kilbourne, A., Bramlet, M., Barbaresso, M. M., Nord, K. M., Goodrich, D. E., Lai, Z.-S., Post, E., Almirall, D., Verchinina, L., Duffy, S., et al. (2014) · Contemporary Clinical Trials · doi:10.1016/j.cct.2014.07.007 · Cited by 28 · see citing works
Disentangling secular trends and policy impacts in health studies: use of interrupted time series analysis
Pape, U., Millett, C., Lee, J., Car, J., & Majeed, A. (2013) · Journal of the Royal Society of Medicine · doi:10.1258/jrsm.2012.110319 · Cited by 24 · cited 1× · see citing works
“Similarly, this design is particularly sensitive to secular trends that may differentially affect the treatment and control groups (Cousins et al., 2014; Pape et al., 2013), as well as regression to the mean confounding study results (Morton and Torgerson, 2003).”
Background
Extending the case–control design to longitudinal data: stratified sampling based on repeated binary outcomes
Schildcrout, J., Schisterman, E., Mercaldo, N., Rathouz, P., & Heagerty, P. (2018) · Epidemiology · doi:10.1097/ede.0000000000000764 · Cited by 19
A method to reduce imbalance for site-level randomized stepped wedge implementation trial designs
Lew, R., Miller, C. J., Kim, B., Wu, H.-S., Stolzmann, K. L., & Bauer, M. (2019) · Implementation Science · doi:10.1186/s13012-019-0893-3 · Cited by 18 · see citing works
Self-citation
Effects of the Campus Watch intervention on alcohol consumption and related harm in a university population
Cousins, K., CONNOR, J. L., & Kypri, K. (2014) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2014.07.015 · Cited by 11 · see citing works

Flagged first, then by citations

3 of the 67 references couldn’t be matched to a known paper.

Research works printed but not linked: 1

Neumann MS, Sogolow ED, 2000 Replicating effective programs: HIV/AIDS prevention technology transfer. AIDS Educ Prev 12 (5 Suppl), 35– 48. 11063068

Sorted by citations

Designing and undertaking randomised implementation trials: guide for researchers
Wolfenden, L., Foy, R., Presseau, J., Grimshaw, J., Ivers, N. M., Powell, B. J., Taljaard, M., Wiggers, J., Sutherland, R., Nathan, N., et al. (2021) · BMJ · doi:10.1136/bmj.m3721 · Cited by 279 · see citing works
Actioning the Learning Health System: An applied framework for integrating research into health systems
Reid, R. J., Wodchis, W. P., Kuluski, K., Lee-Foon, N. K., Lavis, J. N., Rosella, L. C., & Desveaux, L. (2024) · SSM - Health Systems · doi:10.1016/j.ssmhs.2024.100010 · Cited by 109 · see citing works
Designs and methods for implementation research: Advancing the mission of the CTSA program
Hwang, S., Birken, S. A., Melvin, C. L., Rohweder, C. L., & Smith, J. D. (2020) · Journal of Clinical and Translational Science · doi:10.1017/cts.2020.16 · Cited by 97 · see citing works
Improving Student Learning with Hybrid Human-AI Tutoring: A Three-Study Quasi-Experimental Investigation
Thomas, D. R., Lin, J., Gatz, E., Gurung, A., Gupta, S., Norberg, K. A., Fancsali, S. E., Aleven, V., Branstetter, L. G., Brunskill, E., et al. (2023) · International Conference on Learning Analytics and Knowledge · doi:10.1145/3636555.3636896 · Cited by 73 · see citing works
“Randomized controlled trials (RCTs), are universally accepted as the ideal methodology for assessing intervention efficacy and impact [23].”
BackgroundMethodInfluential
A Meta-analysis of University STEM Summer Bridge Program Effectiveness
Bradford, B. C., Beier, M. E., & Oswald, F. L. (2021) · CBE—Life Sciences Education · doi:10.1187/cbe.20-03-0046 · Cited by 73 · see citing works
Structural interventions that affect racial inequities and their impact on population health outcomes: a systematic review
Clark, E., Cranston, E., Polin, T., Ndumbe-Eyoh, S., Macdonald, D., Betker, C., & Dobbins, M. (2022) · BMC Public Health · doi:10.1186/s12889-022-14603-w · Cited by 64 · see citing works
Game-Based Learning in Higher Education: The Pedagogical Effect of Genially Games in English as a Foreign Language Instruction
Cabrera-Solano, P. (2022) · International Journal of Educational Methodology · doi:10.12973/ijem.8.4.719 · Cited by 71 · see citing works
“This type of design involves the application of pre and post-tests, and allows the researchers to conduct rigorous studies (Miller et al., 2020).”
Background
Evidence-based practice implementation in healthcare in China: a living scoping review
Zhao, J., Bai, W., Zhang, Q., Su, Y., Wang, J., Xiao-ning, D., Zhou, Y., Kong, C., Qing, Y., Gong, S., et al. (2022) · The Lancet Regional Health - Western Pacific · doi:10.1016/j.lanwpc.2021.100355 · Cited by 63 · see citing works
A serial mediation model of the relationship between entrepreneurial education, orientation, motivation and intentions
Otache, I., Edopkolor, J. E., & Kadiri, U. (2022) · The International Journal of Management Education · doi:10.1016/j.ijme.2022.100645 · Cited by 56 · see citing works
Effectiveness of gamified intelligent tutoring in physical education through the lens of self-determination theory
Hsia, L., Lin, Y., Lin, C.-H., & Hwang, G. (2024) · Computers & Education · doi:10.1016/j.compedu.2024.105212 · Cited by 52 · see citing works
CRISPR‐driven diagnostics: Molecular mechanisms, clinical efficacy and translational challenges
Wang, Z., Wang, Q., Zhang, J., Li, B., Li, Y., Chen, Z., Guo, D., & Feng, S. (2025) · Clinical and Translational Medicine · doi:10.1002/ctm2.70482 · Cited by 44 · see citing works
“132,133”
A quasi-experimental analysis on the causal effects of COVID-19 on urban park visits: The role of park features and the surrounding built environment
Zhang, W., & Li, J. (2023) · Urban forestry & urban greening · doi:10.1016/j.ufug.2023.127898 · Cited by 49 · see citing works
“Another advantage of ITS design is that it involves a before-and-after comparison within a single group instead of a comparison with a control group, which is appropriate for research on COVID-19, because no control group was available since the outbreak occurred on a worldwide scale (Bernal et al., 2018b; Miller et a…”
Background
TikTok for developing learning motivation and oral proficiency in MICE learners
Gao, S.-Y., Tsai, Y.-Y., Huang, J.-H., Ma, Y.-X., & Wu, T.-L. (2023) · Journal of Hospitality Leisure Sport & Tourism Education · doi:10.1016/j.jhlste.2022.100415 · Cited by 52 · see citing works
Bridging the “Know-Do” Gaps in Five Non-Communicable Diseases Using a Common Framework Driven by Implementation Science
Donohue, J. F., Elborn, J. S., Lansberg, P. J., Javed, A., Tesfaye, S. H., Rugo, H. S., Devi, R., Jithoo, N., Huang, P.-H., Subramaniam, K., et al. (2023) · Journal of Healthcare Leadership · doi:10.2147/jhl.s394088 · Cited by 55 · see citing works
“86 ● Refinement of clinical practice guidelines based on RCTs taking into consideration emerging data from real-world clinical practice.”
A typology of quantitative approaches to discovery
Dolničar, S., Zinn, A. K., & Demeter, C. (2023) · Annals of Tourism Research · doi:10.1016/j.annals.2023.103704 · Cited by 41 · see citing works
Quantifying the impact of the Grain-for-Green Program on ecosystem service scarcity value in Qinghai, China
Hu, Y., Zhang, S., Shi, Y., & Guo, L. (2023) · Scientific Reports · doi:10.1038/s41598-023-29937-7 · Cited by 39 · see citing works
“(7) Q Sup s,t1 = − ∑ k∈KESV Naive s,k,t1 − ∑ k∈KESV Naive s,k,t0”
Systematic review of interventions to enhance preceptors’ role in undergraduate health student clinical learning
Griffiths, M., Creedy, D. K., Carter, A. G., & Donnellan-Fernandez, R. E. (2022) · Nurse Education in Practice · doi:10.1016/j.nepr.2022.103349 · Cited by 38 · see citing works
Advancing climate change health adaptation through implementation science
Neta, G. I., Pan, W., Ebi, K. L., Buss, D. F., Castranio, T., Lowe, R., Ryan, S. J., Ibarra, A. M. S., Hapairai, L. K., Sehgal, M., et al. (2022) · The Lancet Planetary Health · doi:10.1016/s2542-5196(22)00199-1 · Cited by 53 · see citing works
Evaluating technology integration in education: a framework for professional development
Siyam, Y., Siyam, N., Hussain, M., & Alqaryouti, O. (2025) · Discover Education · doi:10.1007/s44217-025-00448-z · Cited by 34 · see citing works
A systematic review of digital health technologies for the care of older adults during COVID-19 pandemic
Zou, C., Harvard, A., Qian, J., & Fox, B. I. (2023) · DIGITAL HEALTH · doi:10.1177/20552076231191050 · Cited by 34 · see citing works
“RCTs, quasi-experimental studies, and qualitative studies) yield different results, with RCTs generally considered more robust than quasi-experimental studies.(78) Also, the inclusion of diverse populations could lead to various experiences of DHTs.”
Result
Pilot Quasi-Experimental Research on the Effectiveness of the Woebot AI Chatbot for Reducing Mild Depression Symptoms among Athletes
Qi, Y. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2301256 · Cited by 36 · see citing works
“However, they can still reveal the relationships and impact of the studied factors, providing valuable information and support (Miller et al., 2020).”
The association between internet use and depression among older adults in China: The mediating role of social networks
Wang, C., Zhu, Y., Ma, J., & Chu, J. (2023) · Digital Health · doi:10.1177/20552076231207587 · Cited by 33 · see citing works
“48,49 Other approaches also have their limitations, such as FE can only eliminate time-ﬁ xed confounding terms 50 and PSM that rely exclusively on the hypothesis that “ observables are ignored.”
BackgroundResult
Implementation Science Methodologies for Complementary and Integrative Health Research
Clark, D. M., Edwards, E., Murray, P., & Langevin, H. M. (2021) · The Journal of Alternative and Complementary Medicine · doi:10.1089/acm.2020.0446 · Cited by 23 · see citing works
“Cluster randomized designs, in which sites or clinics rather than individual patients are the units of randomization, are common.(40,41) When combinations of implementation strategies are being assessed, factorial or fractional-factorial designs that randomize sites or providers to different combinations of strategies…”
BackgroundMethodResultInfluential
The impact of Information and Communication Technology (ICT) on learning outcomes in early childhood and primary education: a meta-analysis of moderating factors
Zuo, R., Li, W., & Zhang, X. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1540169 · Cited by 17 · see citing works
“In experimental designs, participants were randomly assigned to either the treatment group or the control group, whereas in the quasi-experimental studies, participants were not randomly assigned (Miller et al., 2020).”
Investigation of interventions to reduce nurses’ medication errors in adult intensive care units: A systematic review
Mohanna, Z., Kusljic, S., & Jarden, R. (2021) · Australian Critical Care · doi:10.1016/j.aucc.2021.05.012 · Cited by 23 · see citing works
Effectiveness of Fluid and Caffeine Modifications on Symptoms in Adults With Overactive Bladder: A Systematic Review
Park, J., Lee, H., Kim, Y., Norton, C., Woodward, S., & Lee, S. (2023) · International Neurourology Journal · doi:10.5213/inj.2346014.007 · Cited by 20 · see citing works
“A QES is an experimental study without randomization or a control group [23].”
Method
Methodological Aspects in Randomized Clinical Trials of Nutritional Interventions
Martínez‐López, E., Pérez-Guerrero, E. E., Torres-Carrillo, N. M., López-Quintero, A., Betancourt‐Núñez, A., & Gutiérrez-Hurtado, I. A. (2022) · Nutrients · doi:10.3390/nu14122365 · Cited by 22 · see citing works
“A RCT, regardless of the type of intervention, is essentially a research design that must meet two requirements: the manipulation of a variable (the independent variable), and the random allocation of the intervention between study groups [6].”
Background
Interventions to reduce loneliness among Chinese older adults: A network meta‐analysis of randomized controlled trials and quasi‐experimental studies
Li, J., Zhou, X., & Wang, Q. (2022) · Applied Psychology Health and Well-Being · doi:10.1111/aphw.12375 · Cited by 24 · see citing works
Strength-based capacity-building interventions to promote adolescents’ mental health: A systematic review and meta-analysis
Fu, C., Chien, W. T., Zhang, Y., & Lam, S. K. K. (2025) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-025-02741-6 · Cited by 20 · see citing works
“…15 [62, 66, 70–72, 74, 76, 78, 81, 89, 91, 96, 98, 102, 104] used an quasi-experimental design, which refers to non-randomized pre–post-tests with a control group [108], two [88, 106] used quasi-c-RCTs, which refers to the absence of randomization, and one [105] used a Solomon four-group design.”
Building a Life Course Intervention Research Framework.
Russ, S. A., Hotez, E., Berghaus, M., Hoover, C. G., Verbiest, S., Schor, E., & Halfon, N. (2022) · Pediatrics · doi:10.1542/peds.2021-053509e · Cited by 19 · see citing works
Implementation of unassisted and community-based HIV Self-Testing (HIVST) during the COVID-19 pandemic among Men-who-have-sex-with-Men (MSM) and Transgender Women (TGW): A demonstration study in Metro Manila, Philippines
Rosadiño, J. D. T., Pagtakhan, R. G., Briñes, M. T., Dinglasan, J. L. G., Cruz, D. P., Corciega, J. O. L., Pagtakhan, A. B., Regencia, Z. J. G., & Baja, E. S. (2021) · medRxiv · doi:10.1371/journal.pone.0282644 · Cited by 18 · see citing works
“This community-based, volunteer-run organization offers HIV and other sexually-health-related services in Metro Manila, Philippines.”
Influential
Impact of Higher Order Thinking Skills (HOTS) Module Based on the Cognitive Apprenticeship Model (CAM) on Student’s Performance
Ibrahim, N. N., Ayub, A., & Yunus, A. S. (2020) · International Journal of Learning, Teaching and Educational Research · doi:10.26803/ijlter.19.7.14 · Cited by 18 · see citing works
Effect of Digital Safety Interventions on Parental Practices in Safeguarding Children’s Digital Activities: Systematic Review and Meta-Analysis
Zgambo, M., Anyango, E., Arabiat, D. H., Ngune, I., Mörelius, E., Zhang, M., & Whitehead, L. (2025) · JMIR Pediatrics and Parenting · doi:10.2196/70745 · Cited by 15 · see citing works
“The larger effect observed in the quasi-experiment could be because of the lack of randomization in these studies [59].”
How has the COVID-19 pandemic affected the utilisation of online consultation and face-to-face medical treatment? An interrupted time-series study in Beijing, China
Zhang, S., & Ma, C. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-062272 · Cited by 19 · see citing works
Designing clinically useful psychopharmacological trials: challenges and ways forward
Chevance, A., Ravaud, P., Cornelius, V. R., Mayo‐Wilson, E., & Furukawa, T. A. (2022) · The Lancet Psychiatry · doi:10.1016/s2215-0366(22)00041-4 · Cited by 17 · see citing works
Model Pembelajaran Sains Teknologi Masyarakat (STM) DAN Penilaian Autentik terhadap Hasil Belajar Siswa
Jayadiningrat, M. G., Widiana, I., Ariani, N. W. T., & Widiani, N. (2022) · Jurnal Pedagogi dan Pembelajaran · doi:10.23887/jp2.v5i3.50268 · Cited by 14
The ‘Tasty School’ model is feasible for food education in primary schools
Laitinen, A. L., Antikainen, A., Mikkonen, S. J., Kähkönen, K., Talvia, S. S., Varjonen, S., Paavola, S., Karhunen, L. J., & Tilles‐Tirkkonen, T. (2022) · Journal of Human Nutrition and Dietetics · doi:10.1111/jhn.13071 · Cited by 20 · see citing works
Enhancing Students’ Science Literacy Skills; Implications for Scientific Approach in Elementary School
Kartimi, K., & Winarso, W. (2021) · Al Ibtida: Jurnal Pendidikan Guru MI · doi:10.24235/al.ibtida.snj.v8i2.9175 · Cited by 14
“However, it did not fully function to control for external variables that affect the execution of the experiment (Miller, Smith & Pugatch, 2020).”
The Effects of an Inductive Reasoning Learning Strategy Assisted by the GeoGebra Software on Students’ Motivation for the Functional Graph II Topic
Abdullah, A. H., Misrom, N. S. B., Kohar, U. A., Hamzah, M. H., Ashari, Z. M., Ali, D. F. B., Samah, N., Tahir, L., & Rahman, S. N. S. A. (2020) · IEEE Access · doi:10.1109/access.2020.3014202 · Cited by 14 · see citing works
“It appears that adequate time is essential to ensure that a given treatment in an experimental group is effective [76].”
Background
Implementation of project-based learning for improve students' critical thinking skills in creative product and entrepreneurship subjects
Febrianti, R., Yufrizal, A., Putra, R. P., & Phongdala, P. (2023) · Jurnal Pendidikan Teknologi Kejuruan · doi:10.24036/jptk.v6i4.34523 · Cited by 13 · see citing works
“The design used in this study is to use an equivalent control group design (Miller et al., 2020).”
Method
Implementation of food education in school environments improves pupils’ eating patterns and social participation in school dining
Laitinen, A. L., Antikainen, A., Mikkonen, S. J., Kähkönen, K., Talvia, S. S., Varjonen, S., Paavola, S., Karhunen, L. J., & Tilles‐Tirkkonen, T. (2022) · Public Health Nutrition · doi:10.1017/s1368980022002154 · Cited by 15 · see citing works
“Setting and participants The present study had a quasi-experimental design(30) with fifteen intervention schools, which were supported to implement food education based on a tailored, teacherdelivered Tasty School model and ten control schools that”
Method
Implementation Science to Improve Quality of Neurological Care
Samanta, D., & Landes, S. J. (2021) · Pediatric Neurology · doi:10.1016/j.pediatrneurol.2021.05.009 · Cited by 16 · see citing works
Organic food research: Key contributors, research hotspots, and emerging trends
Azizan, A., Awal, A., Zain, W. Z. W. M., Hairoman, N., Endrini, S., & Hassan, F. (2025) · Applied Food Research · doi:10.1016/j.afres.2025.101109 · Cited by 10 · see citing works
A pragmatic evaluation of a public health knowledge broker mentoring education program: a convergent mixed methods study
Clark, E., Dhaliwal, B., Ciliska, D., Neil-Sztramko, S., Steinberg, M., & Dobbins, M. (2021) · Implementation Science Communications · doi:10.1186/s43058-022-00267-5 · Cited by 12 · see citing works
“In future iterations, a time series or stepped wedge design, where organizations are observed prior to intervention, may help further determine the program’s specific effects [53].”
Background
A quasi-experimental analysis of capabilities and limitations of generative AI in academic content evaluation in social sciences
Zhu, Y., Lu, Y., Xie, H. X., Ye, J., & Chen, M. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104365 · Cited by 11 · see citing works
THE EFFECTS OF SOCIO-SCIENTIFIC INQUIRY BASED LEARNING ON STUDENTS' PROBLEM-SOLVING SKILLS
Nugroho, A. A., Sajidan, S., Suranto, S., & Masykuri, M. (2025) · Journal of Baltic Science Education · doi:10.33225/jbse/25.24.149 · Cited by 11 · see citing works
The involvement of specialists in primary healthcare teams for managing diabetes: a systematic review and meta-analysis
Li, J., Xu, Z., Zhou, H. C., Li, Z., & Yuan, B. (2025) · BMC Family Practice · doi:10.1186/s12875-025-02743-y · Cited by 14 · see citing works
“This may be attributed to the well-controlled design of RCTs, which could get a more reliable estimation of effects [52].”
Measuring the effects of nurse-led frailty intervention on community-dwelling older people in Ethiopia: a quasi-experimental study
Kasa, A. S., Traynor, V., & Drury, P. (2024) · BMC Geriatrics · doi:10.1186/s12877-024-04909-2 · Cited by 11 · see citing works
“The study was conducted using a pre-, post-, and follow-up single group quasi-experimental design [36–38].”
Method
Leveraging Single-Case Experimental Designs to Promote Personalized Psychological Treatment: Step-by-Step Implementation Protocol with Stakeholder Involvement of an Outpatient Clinic for Personalized Psychotherapy
Scholten, S., Schemer, L., Herzog, P., Haas, J. W., Heider, J., Winter, D., Reis, D., & Glombiewski, J. A. (2024) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-024-01363-5 · Cited by 18 · see citing works
“Therefore, pre-and post-implementation need to be clearly distinguished (Miller et al., 2020).”
The Effectiveness of School-Based Programs on Aggressive Behaviors among Children and Adolescents: A Systematic Review and Meta-Analysis
Shen, L., Jiang, S., & Tan, S. (2024) · Research on Social Work Practice · doi:10.1177/10497315241227147 · Cited by 10 · see citing works
“The RCT design, often dubbed the “ gold standard ” for interventions, entails random allocation of a representative population sample into experimental and control groups, followed by intervention application (Miller et al., 2020).”
Group Segmentation as a Strategy for Implementing the Intervention Programme in Emotional Education for Infant and Primary Teachers
Carbonero-Martín, M., Arteaga-Cedeño, W. L., Martín-Antón, L., & Molinero-González, P. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192315702 · Cited by 11 · see citing works
“We adopted an experimental pre-test and post-test research design with a control group [40].”
Method
Effectiveness of the interventions against workplace violence suffered by health and support professionals: A meta-analysis
Okubo, C. V. C., Martins, J., da Silva, M. M. T., Galdino, M. J., Haddad, M., Cardelli, A. A. M., & Silveira, R. C. D. C. P. (2022) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.5923.3639 · Cited by 11 · see citing works
“In addition to that, effectiveness can also be affected by the follow-up time of the studies, as research has found that the outcomes can improve with increased study follow-up times(22).”
BackgroundInfluential
The effect of transactional analysis training on emotional intelligence in health professions students
Seow, H. Y., Wu, M. H. L., Mohan, M., Mamat, N. H., Kutzsche, H. E., & Pau, A. K. H. (2022) · BMC Medical Education · doi:10.1186/s12909-022-03455-y · Cited by 14 · see citing works
Impact of virtual reality training on nurses’ preparedness and self-efficacy in emergencies and disasters: A quasi-experimental study
Alsaqer, K., & Alhmoud, S. H. (2025) · International Emergency Nursing · doi:10.1016/j.ienj.2025.101662 · Cited by 13 · see citing works
Effects of nurse-led interventions on enhancing patient-related outcomes in colorectal cancer management throughout the cancer care continuum: A systematic review and meta-analysis
Han, Y., Han, Y., Huang, W., Liu, Y., Wang, Z., Zhao, W. K., & Zhang, W. (2025) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2025.105100 · Cited by 9 · see citing works
Harnessing Implementation Science in Clinical Psychology: Past, Present, and Future
Beidas, R. S., Boyd, M. R., Casline, E. P., Scott, K., Patel, Z. S., Mills, C., Mustanski, B. S., Schriger, S. H., Williams, F. S., Waller, C., et al. (2025) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-081423-021727 · Cited by 12 · see citing works
“The Implementation Research Logic Model (IRLM) (Figure 1a) is a blueprint for integrating multiple FTMs that specifies the relationships between core IS components (Smith et al. 2020).”
The Effect of Integrated Simulation Experiential Learning Disaster Nursing for Enhancing Learning Outcomes Among Undergraduate Nursing Students: A Quasi-Experimental Study
Emaliyawati, E., Ibrahim, K., Trisyani, Y., & Songwathana, P. (2025) · Advances in Medical Education and Practice · doi:10.2147/amep.s489163 · Cited by 10 · see citing works
“20 This research is part of the first author’s dissertation and is a trial of the learning model at stage III.”
The benefits of single-subject research designs and multi-methodological approaches for neuroscience research
Becker, A. M. (2023) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2023.1190412 · Cited by 9 · see citing works
THE EFFECTS OF SCENARIOS ON DECISION-MAKING QUALITY IN EARLY DESIGN – AN EMPIRICAL STUDY
Parolin, G., McAloone, T., & Pigosso, D. (2023) · Proceedings of the Design Society · doi:10.1017/pds.2023.338 · Cited by 10
An expert patient program to improve the empowerment and quality of life of people with multiple sclerosis: protocol for a multicenter pre-post intervention study
Robles-Sanchez, M.-A., Bujan, P. A., Bosch‐Farré, C., Coll, C., Arévalo, M. J., Anglada, E., Menéndez, R., Montalbán, X., Sastre‐Garriga, J., Ramió‐Torrentà, L., et al. (2023) · Frontiers in Neurology · doi:10.3389/fneur.2023.1172640 · Cited by 11 · see citing works
“The common goal of this kind of study is to find whether a health intervention affects the participants and their clinical variables and health-related outcomes (13).”
Background
Effects of implementation strategies aimed at improving high-value verification methods of nasogastric tube placement: A systematic review
Li, J., Sun, X., & Wu, X. (2022) · Frontiers in Nutrition · doi:10.3389/fnut.2022.1009666 · Cited by 10 · see citing works
The effect of clinical pharmacists’ intervention in adverse drug reaction reporting: a retrospective analysis with a 9-year interrupted time series
Lan, T., Wang, H., Li, X., Yin, H., Shao, D., Jiang, Y.-Y., & Yu, Q. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08320-8 · Cited by 10
Evaluating digital mathematical games in improving the basic mathematical skills of university students
Go, M. B., Golbin, R. A., Velos, S. P., Dayupay, J. P., Dionaldo, W., Cababat, F. G., Bongo, M. F., Troussas, C., & Ocampo, L. A. (2022) · International Journal of Mathematical Education in Science and Technology · doi:10.1080/0020739x.2022.2089604 · Cited by 23 · see citing works
“…unique to implementation science, can satisfactorily serve their purpose, especially in cases when experimental designs would become inappropriate (Miller et al., 2020) due to, for instance, in the case of this work, the difference of programmes, topics, and assigned instructors for each course.”
Background
The impact of divorce education programs on co‐parental conflict: Review of programs with parent follow‐up reports
Guyette, E. J., & Harris, S. M. (2024) · Family Court Review · doi:10.1111/fcre.12797 · Cited by 9 · see citing works
Telerehabilitation Intervention in Transitional Care for People with COVID-19: Pre-Post Study with a Non-Equivalent Control Group
Reis, N., Dias, M. J. C., Sousa, L., Canedo, F., Rico, M., Henriques, M. A., & Baixinho, C. (2023) · Healthcare · doi:10.3390/healthcare11182561 · Cited by 9
“…pilot pre-post with a non-equivalent control group study was justiﬁed because the major goals of this type of study are to assess the feasibility and to avoid the potentially disastrous consequences of embarking on a large study, which could potentially “drown” the whole research effort [21].”
Background
The effectiveness of technology‐based cognitive behavioral therapy on perinatal depression and anxiety: A systematic review and meta‐analysis
Ching, H., Chua, J. Y. X., Chua, J. S., & Shorey, S. (2023) · Worldviews on Evidence-Based Nursing · doi:10.1111/wvn.12673 · Cited by 11 · see citing works
Implementation Science: Increasing the Public Health Impact of Audiology Research
Studts, C. R. (2022) · American Journal of Audiology · doi:10.1044/2022_aja-21-00205 · Cited by 14 · see citing works
“…research, for others, rigorous quasi-experimental designs are better suited (e.g., pre–post designs with nonequivalent control groups, interrupted time series, stepped wedge, sequential multiple assignment trials [SMART], and others; Hull et al., 2019; Miller et al., 2020; Schliep et al., 2017).”
Meta-Analyses Do Not Establish Improved Mortality With Ivermectin Use in COVID-19
Rothrock, S. G., Weber, K. D., Giordano, P. A., & Barneck, M. D. (2021) · American Journal of Therapeutics · doi:10.1097/mjt.0000000000001461 · Cited by 13 · see citing works
Igniting creativity through travel: The roles of interest, cognitive flexibility, and destination information presentation style
Su, L., Chen, H., & Lin, Z. (2025) · Tourism Management · doi:10.1016/j.tourman.2025.105228 · Cited by 12 · see citing works
Multicomponent Deimplementation Strategy to Reduce Low-Value Preoperative Testing.
Antunez, A. G., Kazemi, R. J., Richburg, C. E., Pesavento, C. M., Vastardis, A. F., Kim, E., Kappelman, A. L., Nanua, D., Pediyakkal, H., Jacobson-Davies, F. E., et al. (2025) · JAMA Surgery · doi:10.1001/jamasurg.2024.6063 · Cited by 8
Self-citation
Prática da telessimulação no ensino e aprendizagem em saúde: revisão de escopo [Tele-simulation in health teaching and learning: a scoping review] [Práctica de la telesimulación en la enseñanza y el aprendizaje en salud: revisión del alcance]
Bernardinelli, F. C. P., Nascimento, J. D. S. G., do Nascimento, K. G., de Amorim, G. C., da Silva, S. A., & Chaváglia, S. R. R. (2022) · Revista Enfermagem UERJ · doi:10.12957/reuerj.2022.67137 · Cited by 6 · see citing works
A quasi-experimental study provides evidence that registered dietitian nutritionist care is aligned with the Academy of Nutrition and Dietetics evidence-based nutrition practice guidelines for type 1 and 2 diabetes
Lamers‐Johnson, E., Kelley, K., Knippen, K. L., Feddersen, K., Sánchez, D. M., Parrott, J. S., Colin, C. R., Papoutsakis, C., & Jimenez, E. Y. (2022) · Frontiers in Nutrition · doi:10.3389/fnut.2022.969360 · Cited by 13 · see citing works
“This quasi-experimental study used a controlled pre-post design and was conducted from May 2017 to June 2019 (see Supplementary material for study timeline) (12, 13).”
BackgroundMethod
The effect of students’ readiness and achievement in online learning integrates problem-based learning pedagogy during the COVID-19 pandemic
Ge, S., Leng, C.-H., & Shaharom, M. S. N. (2022) · International Journal of Chinese Education · doi:10.1177/2212585x221144901 · Cited by 8 · see citing works
“A quasi-experimental research design is for demonstrating the ef ﬁ cacy of interventions, which is a popular and rigorous experimental methodology (Miller et al., 2020).”
Method
Further insight into Student Learning Outcomes of Derivative Materials: Numbered Head Together and Expository Learning Model
Lumbantoruan, J. H. (2022) · Utamax Journal of Ultimate Research and Trends in Education · doi:10.31849/utamax.v4i2.9918 · Cited by 8
“This type of research is quantitative a QuasiExperimental (Miller et al., 2020).”
The Use of Augmented Reality in History Education: A Study on Conceptual Understanding Effects
Priyono, C., Sok, V., & Souza, F. (2025) · Journal Neosantara Hybrid Learning · doi:10.70177/jnhl.v2i3.1613 · Cited by 7
“To ensure a diverse and representative sample, 120 students are selected through stratified random sampling (Miller, 2020).”
A Scoping Review of Implementation Science Studies in the Field of Traumatic Brain Injury: State of the Science and Future Directions
Hyzak, K. A., Riccardi, J. S., Kinney, A. R., Esterov, D., Bogdanova, Y., & Bogner, J. A. (2024) · Journal of Head Trauma Rehabilitation · doi:10.1097/htr.0000000000000990 · Cited by 9 · see citing works
Efficacy of community groups as a social prescription for senior health—insights from a natural experiment during the COVID-19 lockdown
Chopra, R. C., Chakrabarthi, S., Narayan, I., & Chakraborty, S. (2024) · Scientific Reports · doi:10.1038/s41598-024-75262-y · Cited by 7 · see citing works
“To evaluate the suitability of our setup for a natural experiment, we note that in line with traditional randomized control trial (RCT) setups, our sample is cluster-randomized at the site (apartment) level 36 .”
Fostering university students’ online reading: effects of teacher-led strategy training embedded in a digital literacy course
Macedo‐Rouet, M., Saux, G., Potocki, A., Dujardin, É., Dyoniziak, Y., Pylouster, J., & Rouet, J. (2024) · Instructional Science · doi:10.1007/s11251-024-09676-6 · Cited by 6 · see citing works
“We used a quasi-experimental pre-post design with a non-equivalent control group (Miller et al., 2020).”
Implementation of experiential learning method in mechanical drawing course to enhance student understanding through real projects
Nasution, R., K., A., & Fernanda, Y. (2024) · Journal of Engineering Researcher and Lecturer · doi:10.58712/jerel.v3i2.127 · Cited by 7 · see citing works
“Using an experimental class and a control class, this method can identify differences in results between the treatment class and the control class (Miller et al., 2020).”
Method
The Impact of Gamification Depth on Higher Educational Students’ Intrinsic Motivation and Performance Levels
Yie, D. L., Sanmugam, M., Yahaya, W., & Khlaif, Z. N. (2024) · Higher Education for the Future · doi:10.1177/23476311241248994 · Cited by 7 · see citing works
“The approach involved participants selected into one of several distinct treatment groups, which ideally should be similar to compare the effectiveness of the interventions before and after the experiment (Maciejewski, 2020; Miller et al., 2020).”
The Effects of an Entrepreneurial Project on the Career-Choice Readiness, Metacognition, and Growth Mindset of Secondary Students
Brausch-Böger, M. E., & Förster, M. (2024) · Education Sciences · doi:10.3390/educsci14050485 · Cited by 5 · see citing works
Mutual Aid Groups for Loneliness, Psychosocial Disability, and Continuity of Care
Agudelo‐Hernández, F., Guapacha-Montoya, M., & Andrade, R. M. R. (2024) · Community Mental Health Journal · doi:10.1007/s10597-023-01216-9 · Cited by 6 · see citing works
“In addition, collecting a sufficient number of data points between interruptions is crucial to ensure that the effects of each intervention can be determined individually (Miller et al., 2020).”
Intervention Effect of New Tuberculosis Control Model on Tuberculosis Incidence in Xinjiang
Zhang, Y., Wang, X.-Q., Liu, N.-Q., Wang, S.-L., Wang, X.-M., & Cao, M.-Q. (2023) · Infection and Drug Resistance · doi:10.2147/idr.s441899 · Cited by 7 · see citing works
“5 In the real world, policy evaluation usually relies on evidence from natural experiments or observational studies, and quasi-experimental design can be used to identify the real causal relationship between policy variables and target variables.”
Background
Elevating Social Sciences Learning Outcomes: TGT Type Cooperative Learning Model
Pada, A., & Amir, F. (2022) · Jurnal Ilmiah Sekolah Dasar · doi:10.23887/jisd.v6i4.54046 · Cited by 7
Evaluating the impact of prescriptive authority for psychologists on the rate of deaths attributed to mental illness.
Hughes, P. M., McGrath, R., & Thomas, K. (2022) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2022.12.006 · Cited by 7 · see citing works
An Implementation Science Approach to Promote Optimal Implementation, Adoption, Use, and Spread of Continuous Clinical Monitoring System Technology
Dykes, P., Lowenthal, G., Faris, A., Leonard, M., Hack, R. R., Harding, D., Whnp, C. H., Hurley, A., & An, P. G. (2020) · Journal of patient safety · doi:10.1097/pts.0000000000000790 · Cited by 7
An Evaluation of the Implementation of a UK School-Based Running Program
Chalkley, A., Routen, A., Harris, J. P., Cale, L. A., Gorely, T., & Sherar, L. (2020) · Children · doi:10.3390/children7100151 · Cited by 7 · see citing works
“Furthermore, it was believed to yield naturally occurring differences in types or intensities of implementation strategies, as determined by the schools and their local contexts, in order to evaluate their influence on implementation [19].”
Background
Effect of an Educational Intervention on Nurses’ Knowledge Regarding Use of Modified Early Warning Score in Recognition of Critical Illness
Olowo, S., Luwaga, R., & Niyonzima, V. (2025) · Open Access Emergency Medicine · doi:10.2147/oaem.s521517 · Cited by 4 · see citing works
Examining the effectiveness of Prostatic hyperplasia education on the level of participant’s knowledge and awareness
Krismanuel, H., & Tjhin, P. (2025) · PLoS ONE · doi:10.1371/journal.pone.0325653 · Cited by 5 · see citing works
“Because the significance value is 0.309 > 0.05, it is possible to draw the conclusion that there is no meaningful relationship between the pre-test and post-test variables [24–27].”
Influential
Unveiling the 21st Century Pedagogical Implications of Dramatization and Lecture Approaches through the Lens of EFL College Students
Gonzales, J., & Estremera, M. L. (2025) · International Research and Innovation Journal on Education, Engineering, and Agriculture Journal · doi:10.62960/irijeea.v1i2.23 · Cited by 6
“Of the 76 respondents, 38 were subjected to experimental set-up and the remaining half as the control group were taught exploiting lecture method of teaching adopting principles suggested by Gopalan et al., (2020), and Miller et al., (2020) when deciding for experimental inquiry of research.”
The impact of a mixed reality technology-driven health enhancing physical activity program among community-dwelling older adults: a study protocol
Diño, M. J. S., Dion, K. W., Abadir, P. M., Budhathoki, C. B., Huang, C., Padula, W. V., Himmelfarb, C. D., & Davidson, P. M. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1383407 · Cited by 8 · see citing works
Effect of Jacobson’s Progressive Muscle Relaxation on Fatigue and Sleep Quality among Geriatric Patients Undergoing Hemodialysis
Boughdady, A., Elashri, N., & Ibrahim, S. (2024) · Zagazig Nursing Journal · doi:10.21608/znj.2024.343343 · Cited by 6
“The advantages of this research design that we can compare scores post intervention to scores on the same measure in the same participants prior to intervention (22) .”
Background
Enhancing Medical Students’ Confidence and Knowledge in Antibiotic Prescription and Administration through Virtual Education: A Quasi-Experimental Study
Malli, I. A., Mohamud, M. S., & Al-Nasser, S. (2023) · Antibiotics · doi:10.3390/antibiotics12101546 · Cited by 8 · see citing works
“Even though this study was conducted based on non-random selection, it is the only experimental design that can establish cause-and-effect relationships between an independent and dependent variable [36].”
Background
Effect evaluation of a tele-neurologic intervention in primary care in a rural area in Germany—the NeTKoH study protocol of a stepped-wedge cluster randomized trial
Wainwright, K., Mayer, I., Gonçalves, A. S. O., Schulz, R. S., Kiel, S., Chenot, J., Flöel, A., von Podewils, F., Angermaier, A., & Kurth, T. (2023) · BMC Health Services Research · doi:10.1186/s12913-023-09724-w · Cited by 8 · see citing works
“For such interventions in primary care, a CRT with settings as the unit of randomization and patients nested within those settings is particularly suited as it reduces the risk of contamination [28].”
Background
The Effect of Wordwall Based Media on Students' Procedure Text Achievement in Ninth Grade Students
Adnyana, I., & Dewi, K. S. (2022) · Jurnal Pendidikan Bahasa Inggris Undiksha · doi:10.23887/jpbi.v10i1.45143 · Cited by 6
“Typically, the researcher used this experimental design to see if there is a relationship between the two variables or if they have an effect on each other (Miller et al., 2020).”
Method
Challenges in the implementation and evaluation of a transmural palliative care pathway for acutely hospitalized older adults; lessons from the PalliSupport program: A qualitative process evaluation study.
van Doorne, I., van Schie, V. M. W., Parlevliet, J., Willems, D., van Rijn, M., & Buurman, B. (2022) · Archives of gerontology and geriatrics (Print) · doi:10.1016/j.archger.2022.104782 · Cited by 6 · see citing works
Real-world Methotrexate Use in a Prospective Cohort of Paediatric Patients with Plaque Psoriasis: Effectiveness, Adverse Events and Folic Acid Regimen
Bruins, F., van Acht, M. V., Bronckers, I. M., Groenewoud, H., Jong, E. D. D., & Seyger, M. (2022) · Acta Dermato-Venereologica · doi:10.2340/actadv.v102.1000 · Cited by 6
“Therefore we were able to compare FA regimens through a quasi-experimental design (18) with a more recent cohort (November 2016 to October 2020) using FA 6 times per week and a “historical” comparison cohort (September 2008 to November 2016) using FA once weekly from the same registry.”
Method
Probiotic Bifidobacterium animalis subsp. lactis consumption slows down orthodontic tooth movement in mice
Duffles, L. F., Menino, A. P., Taira, T. M., de Oliveira, S., Salvador, S. L. D. S., Messora, M. R., Vinolo, M. A. R., & Fukada, S. Y. (2021) · Archives of Oral Biology · doi:10.1016/j.archoralbio.2021.105324 · Cited by 6 · see citing works
Mixed reality technology for older adults: Evaluating the impact of a novel virtual humanoid coach in a community-based physical exercise program in the Philippines
Dino, M. J. S., Dion, K. W., Abadir, P. M., Budhathoki, C., Huang, C.-M., Ong, I., Vital, J. C., Cotter, V. T., Himmelfarb, C. R. D., & Davidson, P. M. (2024) · Health Informatics Journal · doi:10.1177/14604582241267793 · Cited by 5 · see citing works
“18 Speci ﬁ cally, a pre-post control group design was adopted 19 to compare the measurements from the control and experimental groups.”
An introduction to implementation science in rehabilitation medicine
Roseen, E. J., Welch, S. A., Stout, N. L., Bean, J. F., Friedly, J. L., & Mittman, B. S. (2025) · PM&R · doi:10.1002/pmrj.70040 · Cited by 5 · see citing works
“87,88 Effectiveness-oriented implementation trials are often conducted in health systems interested in achieving near-term improvements in implementation outcomes in addition to supporting the production of scientific knowledge.”

Authors: Miller, C. J., Smith, S. N., & Pugatch, M.
Affiliations: United States Department of Veterans Affairs (US)
University of Michigan (US)
Corresponding author: Miller, Christopher J.
ORCID: Miller, Christopher J.
Smith, Shawna N.
Pugatch, Marianne
Venue: Psychiatry Research
Publisher: Elsevier BV
Published: 2020-01
Volume: 283
Pages: 112452
ISSN: 0165-1781 (ISSN-L)
DOI: 10.1016/j.psychres.2019.06.027
PMC: PMC6923620
OpenAlex: W2951715721
Cited by: 373
References: 67
Citations by year: 2026: 63 · 2025: 89 · 2024: 67 · 2023: 63 · 2022: 56 · 2021: 20 · 2020: 14 · 2019: 1
FWCI: 4.18
Type: journal article
Language: en
Open access: hybrid
doi.org
Licence: cc-by-nc-nd
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
www.elsevier.com/legal/tdmrep-license (tdm)
creativecommons.org/licenses/by-nc-nd/4.0/ (vor, 114 d)
Locations: Psychiatry Research (publishedVersion, cc-by-nc-nd)
PubMed (publishedVersion)
Europe PMC (PubMed Central) (acceptedVersion)
PubMed Central (acceptedVersion)
Funders: National Institute of Mental Health
U.S. Department of Veterans Affairs
Topics: Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Keywords: implementation research, implementation science, experimental design, quasi-experimental design, randomized controlled trials, interrupted time series, stepped wedge design, pre-post design, research design selection, clinical adoption, intervention sustainability
MeSH terms: Implementation Science, Humans, Research Design, Randomized Controlled Trials as Topic, Control Groups, Biomedical Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Experimental and quasi-experimental designs in implementation research

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Self-check

Cites

Cited by

Record

About these checks