Single-case experimental designs: a systematic review of published research and current standards

“Single-case time-series experiments are an increasingly popular method of scientific inquiry (Smith, 2012), and, like RCTs, they can demonstrate the causal effects of treatments (APA Presidential Task Force on Evidence-Based Practice, 2006; Barlow & Hersen, 1984; D.”

“To this end, experimental studies could be used, as well as micro trials (Leijten et al., 2015) or single–case study designs (Smith, 2012).”

“plementary rather than mutually exclusive (Maggin & Odom, 2014; Parker & Brossart, 2003; White, Rusch, Kazdin, & Hartmann, 1989) and should, arguably, be used in combination (Davis et al., 2013; Smith, 2012).”

“This would effectively down-weight evidence from SCDs with very small numbers of cases, in a meta-analysis that included evidence from multiple types of designs.”

“…et al., 2015; Lee et al., 2014; MacPherson et al., 2016; McCrae et al., 2005; Norberg et al., 2014; Payne et al., 2013; Penica & Williams, 2008; Piven & Duran, 2014; Romero, 2010; Sevick et al., 2005; Smith, 2012b; Sonnier, 2002; Soroudi et al., 2008; Ward et al., 2010; Yngman-Uhlin et al., 2012).”

“Data for the reliability analysis When determining the data sample for conducting the present study, we set specific criteria based on the review studies conducted by Shadish and Sullivan (2011) and Smith (2012) to identify the characteristics of the single-case designs used in the articles.”

“, Journal of Behavioral Education in 2012, Remedial and Special Education in 2013, the Journal of School Psychology and Neuropsychological Rehabilitation in 2014, Aphasiology in 2015) and methodological quality recommendations (Horner et al., 2005; Kratochwill et al., 2013; Maggin et al., 2014; Smith, 2012; Tate et al…”

“Among all of the design elements discussed above, the characterization of participants appears to have received the most attention in discussion on SCEDs (Kazdin, 2011; Kratochwill & Stoiber, 2002; Smith, 2012) and is among the recommended reporting requirements of Single Case Reporting Guideline In Behavioral Interve…”

“Both Shadish and Sullivan (2011) and Smith (2012), for example, found that the average SCD had more cases and more time points than the minimum required by the SCD.”

“comparable frequency to between-groups research in the aphasiology, education, psychology, and neurorehabilitation literature (Beeson & Robey, 2006; Perdices & Tate, 2009; Shadish & Sullivan, 2011), evidence of inadequate and incomplete reporting is documented in multiple surveys of this literature in different popula…”

“Smith (2012) found that SCD was reported in 143 different journals that span a variety of fields such as behavior analysis, psychology, education, speech, and pain management; across these fields, multiple baselines account for 69% of SCDs.”

“Single case experimental designs (SCED) provide a pragmatic alternative that, when well-designed and executed, rival the scientific rigor of randomized controlled trials while requiring far fewer participants (Barlow, Nock, & Hersen, 2008; Smith, 2012; Vilardaga, 2014).”

“Email: jninci@tamu.edu Ninci et al. 511 In research and practice, visual analysis is the most widely used and recommended method for interpretation of single-case data (Gast, 2010; Kennedy, 2005; Kratochwill et al., 2010, 2013; Smith, 2012).”

“Thus, the lack of clear consensus (Kratochwill et al., 2010; Smith, 2012; Tate et al., 2013) and indications suggest that the current paper is necessary, as we consider that more discussion is needed apart from more research (Gage & Lewis, 2013).”

“This journal was selected because it is a leading journal on the topic used by applied researchers, and it strongly promotes the use of VA rather than quantitative analysis methods (Shadish & Sullivan, 2011; Smith, 2012).”

“Smith (2012) provides a synthesis of guidelines for single-case experimental designs (SCEDs).”

“…health providers who may not have support for a larger research effort, but are interested in examining specific clinical interventions/programs they are developing and implementing as a part of their PCBH model care delivery (see Smith, 2012 for more information on single-subject designs).”

“…is documented in multiple surveys of this literature in different populations (Barker, Mellalieu, McCarthy, Jones, & Moran, 2013; Didden, Korzilius, van Oorsouw, & Sturmey, 2006; Maggin, Chafouleas, Goddard, & Johnson, 2011; Smith, 2012; Tate, Perdices, McDonald, Togher, & Rosenkoetter, 2014).”

“We have not seen such an outcome used in SCD research in surveys of the literature (Smith, 2012).”

“This development is part of a renewed interest in single-case experimental methodology which Smith (2012) describes as “poised for resurgence”.”

“…statistics, such as deficient performance in the presence of data outliers in the baseline phase, insensitivity to data trends and variability in the data, and insensitivity to differences in the magnitude of effect (Maggin et al., 2011; Smith, 2012; Wolery, Busick, Reichow, & Barton, 2010).”

“To increase power and internal validity of within-site studies, interrupted time-series designs can be used [47].”

“Smith (2012) reported that authors in fewer than half of the SCR studies published between 2000 and 2010 (n = 409) identified the analytic method they used; only 28.1% explicitly stated that they used visual analysis.”

“all published SCD data because they have the ability to digitize published graphs, and who are therefore already developing and conducting analyses and metaanalyses of SCD data (Shadish, in press-b); and (f) increasing recommendations to use statistics by various entities providing standards for SCDs (Smith, 2011).”

“Hedge’s g is a standardized-mean differences approach used to compute effect size for single-case designs [31].”

“…participated in development of these standards or have published critiques and additional suggestions (Horner et al., 2005; Kratochwill & Stoiber, 2002; Reichow, Volkmar, & Cicchetti, 2008; Smith, 2012; Wendt & Miller, 2012), including developing research review protocols (Tate et al., 2008, 2014).”

“Furthermore, outcome measures that meet the design standards (with or without reservations) can be categorized as demonstrating ‘‘Strong Evidence,’’‘‘Moderate Evidence,’’ or ‘‘No Evidence’’ (see also Smith, 2012).”

“ATD is an ideal design for comparing the effects of two or more treatments in applied research, and can identify the presence or absence of a causal relationship between the independent variable (e.g., font type) and a change in the dependent variable (e.g., reading speed and accuracy (Smith, 2012).”

“In line with the findings of earlier systematic reviews (Hammond & Gast, 2010; Shadish & Sullivan, 2011; Smith, 2012), the multiple baseline design continues to be the most frequently used design, and despite the advancement of sophisticated statistical methods for the analysis of SCEDs, two thirds…”

“Second, the most appropriate design should be identified (see Kratochwill et al., 2010; Smith, 2012).”

“The review performed by Smith (2012) identified 409 articles in peer-reviewed journals in the period 2000-2010 reporting an SCED study, with the most frequent research areas being behavioral modification/interventions, developmental disorders, and school psychology.”

“…psychology, school psychology, special education, and medicine (Alnahdi, 2015; Chambless & Ollendick, 2001; Gabler, Duan, Vohra, & Kravitz, 2011; Hammond & Gast, 2010; Kratochwill & Stoiber, 2000; Leong, Carter, & Stephenson, 2015; Shadish & Sullivan, 2011; Smith, 2012; Swaminathan & Rogers, 2007).”

“A person-specific analysis aims to elucidate processes that exist within a single entity (e.g. a single patient) over time (see Ram, Brose, & Molenaar, 2013; Smith, 2012).”

“…patterns has so far been approached in different ways in the psychotherapy literature: The reported findings cover a broad spectrum, ranging from detailed single-case studies (Kykyri et al., 2017; Smith, 2012) to large-scale assessments of trajectories (Bar-Kalifa et al., 2016; Lutz et al., 2013).”

“We followed guidelines in the design, analysis and reporting of case series to ensure the method was applied optimally (Smith, 2012; Kratochwill et al., 2010, 2012; Tate et al., 2016).”

“Although single-case designs may be considered an appropriate alternative to group designs, there is a clear need for researchers to report effect sizes to provide stakeholders with a better understanding of the practical significance of the study’s findings (Smith, 2012).”

“Future studies using novel approaches including single case-control designs 88 might uncover individual differences in predictors across phenotypically comparable patients.”

“In single-case experimental design guidelines, three is the minimum number of data points required to establish a baseline, and the minimum number of data points needed in each phase [40].”

“Recent years have also seen a proliferation of quantitative methods to evaluate single-case design types (see Smith, 2012, for a review), and in conjunction with visual inspection, these techniques offer a wide range of supplemental tools for quantifying and communicating observed effects.”

“These standards have been used by the WWC to review more than 11,000 studies (Kratochwill, 2013), and are comparable in terms of research design, measurement, and data analysis to other standards that have been developed in education and psychology (Smith, 2012).”

“By now, the problems of naïve ANOVA models for the analysis of single-case experimental data are well documented and widely acknowledged: the assumptions of normality, homogeneity of variance, and independent residuals are implausible for this kind of data (Richards, 2019; Smith, 2012; Solomon, 2014).”

“Systematic reviews regularly indicate that visual analysis is by far the most preferred data analysis technique among applied SCED researchers (Smith, 2012; Tanious & Onghena, 2021, 2022) despite the availability of sophisticated statistical techniques (Busse et al., 2015; Manolov & Moeyaert, 2017).”

“One challenge appears to be the analysis of SCED data (Barlow et al., 2009; Smith, 2012).”

“Based on published meta-analyses of single-case research (Smith, 2012), the 350-participant threshold makes it particularly challenging for interventions validated with single-case designs to obtain a rating of strong or moderate evidence.”

“According to Smith’s literature review of the single-case design, this is one of the most cost-effective and efficient procedures to experimentally test the effect of an intervention [86].”

“An effect size index can also be computed from interrupted time series analysis via ARIMA (autoregressive integrate moving average) models, which allow controlling for trend and autocorrelation (Simonton, 1977).”

“However, program effects persisted after excluding agencies with significant baseline trends, making regression to the mean a less plausible explanation for our findings (Smith 2012).”

“Regarding quantitative analysis, there is a lack of consensus about which techniques are most appropriate (Busse et al., 2015; Smith, 2012).”

“A minimum of 5 points per phase is considered the standard in multiple baseline design (Smith 2012).”

“, Journal of Behavioral Education in 2012, Remedial and Special Education in 2013, the Journal of School Psychology and Neuropsychological Rehabilitation in 2014, Aphasiology in 2015) and methodological quality recommendations (Horner et al., 2005; Kratochwill et al., 2013; Maggin et al., 2014; Smith, 2012; Tate et al…”

“…for SCD literature reviews do not include guidelines for evaluating the researchers' statistical-analysis techniques—see reviews of the numerous appraisal rubrics for SCD research provided by Maggin, Briesch, Chafouleas, Ferguson, and Clark (2013), Smith (2012), and Wendt and Miller (2012).”

“Although there is growing interest in statistical analyses of data sets within SCEDs (e.g., Kazdin, 2011; Manolov et al., 2014; Shadish, 2014; Smith, 2012), it is likely that visual analysis remains the primary tool for SCED researchers and practitioners (e.”

“…sound (Horner et al., 2005; Kratochwill and Levin, 2010; Smith, 2012), capable of demonstrating effectiveness of interventions, and is especially suitable for the assessment of outcomes of psychological/behavioral interventions (Robey et al., 1999; Borckardt et al., 2008; Smith, 2012).”

“The combination of a recognition of the level of evidence provided by SCEDs and an emerging consensus on what constitutes methodologically strong SCEDs perhaps explains why Smith (2012) views SCEDs as ‘poised for resurgence’ (p. 510).”

“…or rubrics that suggest how an individual SCED study should be conducted and analyzed to favor both internal and external validity (e.g., Horner et al., 2005; Reichow et al., 2018; Tate et al., 2013; see also Maggin, Briesch, Chafouleas, Ferguson, & Clark, 2014; Smith, 2012, for reviews).”

“Moreover, SCRDs are traditionally accompanied by thoroughly documented parameters, such as response definitions, designs, and procedures (Smith, 2012; Wolery & Ezell, 1993) and specificity in these areas is thought to support later replications (Kazdin, 2011; Kratochwill et al., 2012).”

“, Journal of Behavioral Education in 2012, Remedial and Special Education in 2013, the Journal of School Psychology and Neuropsychological Rehabilitation in 2014, Aphasiology in 2015) and methodological quality recommendations (Horner et al., 2005; Kratochwill et al., 2013; Maggin et al., 2014; Smith, 2012; Tate et al…”

“In fact, researchers now have at their disposal a number of appraisal guidelines that they can use to assist in literature reviews for SCD research (see, for example, Smith, 2012; Wendt & Miller, 2012, for an overview of various standards including the WWC Standards).”

“Single-case designs (SCD) data are still commonly analyzed visually (Campbell & Herzinger, 2010; Kratochwill, Levin, Horner, & Swoboda, 2014; Parker, Cryer, & Byrns, 2006; Smith, 2012), either on exclusive basis or jointly with an objective quantification.”

“However, Smith’s review of current methods and standards for analysis suggests using standardized mean difference approaches because the effect sizes calculated with these were the least affected by autocorrelation [39].”

“Fortunately, there is general agreement about many of the features necessary to have sufficient internal validity for single-case research, including minimum levels of interobserver agreement for dependent variables and the need to collect repeated measures in the absence and presence of an intervention (Smith, 2012).”

“After establishing a stable baseline for each participant (i.e., minimal to no increasing trend; Smith, 2012), the independent variable is introduced to the first participant while the other participants remain in baseline.”