Measures Inherent to Treatments in Program Effectiveness Reviews

“As another example of the correlation between methodology and outcomes, Slavin and Madden (2011) examined effect sizes of reading and math studies that met the standards of the WWC with or without reservations.”

“…tests are usually less comprehensive and more likely to measure aspects of content inherent to treatment, but not control group instruction (Slavin & Madden, 2011), standardized tests may provide a more reliable measure of lasting differences between treatment and control groups.”

“Standardized achievement tests are rare in this corpus of studies, so each study has its own achievement test speci fi cally designed for the content at hand, and it might be that in some cases the achievement test was biased toward the innovative group (see Slavin & Madden, 2011).”

“In contrast, treatment-independent measures (Slavin & Madden, 2011) examine gains in skills that are presumably learned in both treatment and control classrooms, such as receptive language.”

“Slavin and Madden (2011) observed a higher effect size for treatment-inherent measures than for treatment-independent measures.”

“These characteristics make wise interventions particularly suitable for scaling up, even where the average size of their effect on the overall population is relatively small (Bakker et al., 2019; Kraft, 2020; Slavin & Madden, 2011).”

“Schools that did or did not adopt reading programs by Years 3 or 4 were compared on their reading test scores, controlling for baseline scores from before Year 1, to their matched counterparts (see Slavin & Madden, 2011, for details of these analyses).”

“Compared with those with large sample sizes, studies with small sample sizes tend to be more strictly controlled and thus there is a higher chance of obtaining positive results [72].”

“…not generalized content knowledge.8 The significant effects on researcher-developed outcomes for comprehension, vocabulary, and content knowledge are partially due to fact that researcher-developed measures more closely reflect the intervention than standardized measures (Slavin & Madden, 2011).”

“…controls, or instruments that measure outcomes closely related to the content of the intervention or based on teachers’/pupils’ self-reports of outcomes, tend to show suspiciously large apparent effect sizes (Slavin and Smith 2009; Slavin and Madden 2011; Van der Kleij, Feskens and Eggen 2015).”

“Slavin and Madden (2011) showed that among WWC studies of beginning reading programs, the outcome measures that were closely aligned with the intervention yielded weighted effect sizes of 0.51 SD, whereas the more general outcome measures had average effect sizes equal to 0.06.”

“…academic learning outcomes from independently created tests; researcher-designed measures were excluded because they may inflate effect sizes (Slavin & Madden, 2011); published between 1980 and our final searches in February 2020 (although we used updated published versions if released…”

“One possible explanation is that standardized tests tend to have higher reliability than measures developed by researchers or teachers and thus less subject to attenuation effects (Slavin & Madden, 2011).”

“This finding is surprising given prior research that found researcher-created assessments generally produced larger effect sizes than standardized measures (e.g., Bloom, Hill, Black, & Lipsey, 2008; Slavin & Madden, 2011).”

“…explanation for the smaller effects obtained for writing collectively as well as for specific writing outcomes in this review and the former one by Graham et al. (2023) is that methodologically stronger studies produce smaller effects than methodologically weaker ones (Slavin & Madden, 2011).”

“Based onWWC standards (WWC 2014) and study quality recommendations articulated by Slavin and colleagues (Slavin and Madden 2011; Slavin and Smith 2009), study quality was assessed according to adherence to three explicit criteria: Studies gained the Bhighest^ rating if they had samples of N=250 or more, included at le…”

“…to treatments, usually made by the experimenter or programme developer, have been found to be associated with much larger effect sizes than are measures that are independent of treatments (Slavin & Madden, 2011), and for this reason, effect sizes from treatment-inherent measures were excluded.”

“…literacy experts do not all agree with the WWC’s analysis and presentation of reading program effectiveness (e.g., McArthur, 2008; Slavin, 2008; Slavin & Madden, 2011; Slavin & Smith, 2009; Stockard, 2008; Stockard & Wood, 2013), it can provide an initial basis for guiding the selection of…”

“Treatment-inherent outcome measures are associated with larger effect sizes than independent measures [57, 67].”

“…developed tests are usually less comprehensive and more likely to measure aspects of content inherent to intervention but not control group instruction (Slavin & Madden, 2011), standardised tests should provide a more reliable measure of lasting differences between intervention and control groups.”

“Some scholarsargue that standardized testsarecurriculum-neutral outcome measures, which avoid the bias (i.e., inflated treatment effects) that can occur when investigators use outcome measures that are aligned with particular curricular interventions (e.g., Slavin & Madden, 2011).”

“…this commonsense connection between the degree to which the assessment is related to the intervention is a well-recognized phenomenon, and performance on distant, broad-band measures remains notoriously difficult to improve (e.g., Bloom, Hill, Black, & Lipsey, 2008; Slavin & Madden, 2011).”

“…or programdeveloped measures inherent to treatment and/or measures closely aligned with the intervention, such as the ones used in Cuevas et al. (2012), have been found to be associated with much larger effect sizes than measures that are independent of treatments (Slavin & Madden, 2011).”

“…Slavin, Lake, Chambers, Cheung & Davis, 2009a; Slavin, Lake, Cheung & Davis 2009b; Slavin, Lake, Chambers, Cheung & Davis,2009c; Slavin, Lake, Davis & Madden, 2010; Slavin & Madden, 2011) carried out large Best Evidence Encyclopedia syntheses of various reading programs in Kindergarten to Grade 5.”

“…designs to be smaller than effects from quasi-experimental designs, and effects from author-developed outcome measures to be larger than effects from outcome measures not developed by authors (Cheung & Slavin, 2016; Hill, Bloom, Black, & Lipsey, 2008; Slavin & Madden, 2011; Taylor et al., 2018).”

“This finding, however, contradicts previous meta-analytic studies (e.g., de Boer et al., 2014; Slavin & Madden, 2011).”

“This result was in line with the previous studies suggested that small sample sizes had the larger effect sizes than large sample sizes (Cheung and Slavin, 2013; Slavin and Madden, 2011).”

“Outcome measures may also be too tightly aligned and narrowly focused on the target skills or knowledge of the teacher preparation practice (i.e., treatment-inherent ; Slavin & Madden, 2011) to allow for generalizable findings that contribute to the broader evidence base (Zeichner, 2005).”

“The development team used the CCSS-M as a touchstone to provide guidelines for avoiding overalignment of the instrument to the speci fi c PD programs being evaluated (Slavin & Madden, 2011 ).”

“This may happen when outcome assessments are developed by the investigators or the curriculum developer (Slavin & Madden, 2011) and use methods that mirror pedagogical strategies employed in the intervention (e.g., an outcome measure that involves parents in a semistructured parent-child…”

“The test in itself may also form part of the experimental intervention (Slavin & Madden, 2011).”

“The IQA provided a treatment independent (Slavin & Madden, 2011) measure of classroom instruction, and it served as the primary outcome of interest for the confirmatory RQ1.”

“Less obvious examples of interventions capturing superficial shifts in teaching certainly exist (Slavin & Madden, 2011).”

“Slavin and Madden (2011) find that the What Works Clearinghouse (2008a, 2008b) reading and math reviews averaged effect sizes from measures that clearly produced different estimates.”

“For example, studies based on researcher-developed instruments tend to report significantly higher effects than those based on intervention-independent instruments (Cheung & Slavin, 2016; De Boer, Donker, & van der Werf, 2014; Slavin & Madden, 2011).”

“Treatment-inherent outcome measures yield larger effect sizes than outcome measures that are not treatment-inherent [39, 40].”

“It is indeed unfortunate that poor methodologies tend to report much higher effect size than those with more rigorous methods (Slavin and Smith, 2009: 500; Slavin and Madden, 2011: 370).”

“As the TOCA was designed by a developer of the GBG, and has primarily been used in GBG studies, there may be concerns that the measure is “inherent to treatment” (Slavin & Madden, 2011).”

“…this commonsense connection between the degree to which the assessment is related to the intervention is a well-recognized phenomenon, and performance on distant, broad-band measures remains notoriously difficult to improve (e.g., Bloom, Hill, Black, & Lipsey, 2008; Slavin & Madden, 2011).”

“Furthermore, national tests constitute global measures of reading comprehension and may therefore result in lower effect sizes compared to researcher-designed measures (Shadish et al., 2002; Slavin & Madden, 2011).”

“In education research, treatment-inherent measures are associated with larger effect sizes than treatment-independent measures, which measure content taught in both comparison groups [39].”

“Researcher‐developed measures risk inflating effect sizes (e.g., Slavin, 2011).”

“…target certain domains and not general performance skills, or that it may be easier to improve scores on tests of subdomains than on tests of more general skills, or that tests of subdomains may be more likely to be inherent to treatment (see Slavin & Madden, 2011 for a discussion of the latter).”

“Some (Cheung & Slavin, 2015; Slavin & Madden, 2011) have argued that intervention effects on experimenter-developed tests that are not backed up by effects on standardized tests reflect over-alignment of the experimenter-developed test to the treatment, giving the treatment group an unfair advantage.”

“The researcher-developed measures in this study produced significant correlations with these assessments, but using the measures to make interpretations about general vocabulary knowledge is inappropriate because of their inherent proximity to the treatment (Slavin & Madden, 2011).”

“…influenzano le dimensioni degli effetti nell’istruzione In questo recente lavoro di Cheung e Slavin (2016) sono riprese le conclusioni delle due precedenti ricerche svolte (Slavin & Smith, 2009; Slavin & Madden, 2011) e sono rinforzate attraverso l’analisi di un campione di meta-analisi più vasto.”

“Several investigations have shown that interventions have larger effects on treatment-inherent measures than on independent measures (Slavin & Madden, 2011).”

“Larger effects may be observed with more proximal assessments of classroom learning in comparison to standardised tests [45], or other, more reliable measures of sleep may show stronger associations with academic variables.”

“This may stem from the fact that, while standardized measures are generally more reliable (Slavin & Madden, 2011), unstandardized metrics incorporate multiple assessment methods (e.g., teacher and student formative evaluations as well as term-based course grades).”

“A treatment-inherent measure is one derived from a test of constructs not ordinarily taught and ‘inherent’ to the treatment or intervention (Slavin and Madden 2011).”

“In their analysis of seven mathematics research studies, Slavin and Madden (2011) reported that the effect size for homemade measurements is 0.45 and -0.03 for standardized measures.”

“…tests are usually less comprehensive and more likely to measure aspects of content inherent to treatment, but not control group instruction (Slavin & Madden, 2011), standardized tests may provide a more reliable measure of lasting differences between treatment and control groups.”

“…literature does indeed show that tests more closely aligned to the material taught in the intervention are associated with larger effect sizes (e.g., Cheung & Slavin, 2016; Slavin & Madden, 2011), it is worth examining whether there are alternative explanations which account for this issue.”