Effects of after-school programs with at-risk youth on attendance and externalizing behaviors: A systematic review and meta-analysis

May combine two claims.

The mean effect of after-school programs on attendance and externalizing behaviors is small and not statistically significant.

“Mean effects were small and non-significant for attendance and externalizing behaviors.”

May combine two claims.

There is moderate to large heterogeneity among effect sizes, and no tested moderator variable explains the variance between studies.

“A moderate to large amount of heterogeneity was present; however, no moderator variable tested explained the variance between studies.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Nearly half (45.8%) of the 24 included studies were from grey literature.

“Of the 24 studies included in this meta-analysis, nearly half (45.8 %) were found in the grey literature.”

May combine two claims.

Seven included studies were randomized controlled designs, while 70.8% were quasi-experimental.

“Seven of the studies were randomized controlled designs, while the majority (70.8 %) were quasi-experimental designs.”

May combine two claims.

Most studies (63%) were low risk for reporting bias, three high risk, and six unclear.

“We assessed most studies (63 %) as low risk for reporting bias as the authors appeared to have reported the expected outcomes, three studies as high risk as there appeared to be selective or incomplete reporting of expected outcomes, and six studies as unclear risk of reporting bias.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The comparison condition for 70.8% of studies was no treatment or waitlist.

“The comparison condition for the majority (70.8 %) of the studies was no treatment or waitlist.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The review included 24 studies that appeared in 31 reports.

“Twenty-four studies reported in 31 reports were included in the review.”

May combine two claims.

Only seven (29%) included studies used randomization, with only three describing randomization procedure and only three providing allocation concealment information.

“Only seven (29 %) of the included studies used randomization to assign students to the after-school program or control condition, with only three of these seven studies providing a clear description of their randomization procedure and only three providing information related to allocation concealment.”

Classified as probably not a substantial claim.

Sixteen studies with 16 effect sizes were analyzed for attendance outcomes.

“Of the included studies, 16 studies with 16 effect sizes were included in the analysis of attendance outcomes”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Almost every included study (95.8%) took place in the United States; one took place in Ireland.

“Despite efforts to search for studies conducted in a broad geographical area, nearly all of the included studies (95.8 %) were conducted in the United States, with one study conducted in Ireland.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The majority of externalizing behavior outcomes were self-reported (65%).

“The majority of externalizing behaviors included in this review were self-reported (65 %).”

May combine two claims.

Few included studies specified increasing school attendance as a primary goal or described a theory of change linking program mechanisms to attendance.

“Although 16 studies in the current review measured school attendance, few specified increasing school attendance as a primary goal of the after-school program or explicated a theory of change connecting the mechanisms of the after-school program to school attendance.”

May combine two claims.

Selection bias was rated high risk in 17 (71%) studies, uncertain in four, and low risk in two.

“Selection bias was rated as high risk in 17 (71 %) of the included studies, uncertain in four of the included studies, and low risk in two of the included studies.”

May combine two claims.

Attrition bias was high risk in nine (38%) studies, low risk in three, and unclear in ten.

“Attrition bias was assessed as high risk in nine (38 %) of the included studies.”

• Also stated in: § Characteristics of Included Studies and Programs · plain

  “Three studies were assessed as low risk of attrition bias as they reported either low or no overall or differential attrition or used missing data strategies to perform the analysis with data from all participants assigned to condition.”

• Also stated in: § Characteristics of Included Studies and Programs · plain

  “The remaining 10 studies were assessed as unclear risk as the studies did not clearly indicate the procedures used for the management of attrition or the attrition rates could not be reliably calculated.”

May combine two claims.

In this review, variation among programs and weak reporting on program elements and staffing prevented clear separation of effects by program or participant features.

“the heterogeneity of programs included in this review and the relatively poor reporting related to specific elements and staffing of the included after-school programs made it difficult to parse out differential effects across programs related to program or participant characteristics.”

May combine two claims.

Weak rigor and compromised internal validity in most included studies restricted causal inference and the review's conclusions.

“Most of the studies lacked rigor and internal validity of the studies was compromised, thus limiting the causal inferences that could be drawn from the studies and the conclusions that can be made from this review.”

May combine two claims.

Among included studies, 44% of those appearing after 2007 assigned students randomly, versus 20% of those appearing in or before 2007.

“Indeed, 44 % of the included studies in this review published after 2007 used randomization procedures to assign students to condition, while only 20 % of studies published before or during 2007 assigned students randomly.”

May combine two claims.

Across included studies, most assessed domains showed high risk of bias and methodological weaknesses, matching earlier reviews.

“Moreover, methodological flaws and high risk of bias on most of the domains assessed in this review were found across included studies, which is consistent with findings from past reviews of after-school programs.”

May combine two claims.

A comprehensive search resulted in 24 after-school program intervention studies, 14 of which had not been in prior reviews.

“A comprehensive search for published and unpublished literature resulted in the inclusion of 24 after-school program intervention studies, 14 of which have not been included in prior reviews.”

May combine two claims.

For at-risk primary and secondary students, the review found no clear evidence that after-school programs affect attendance or externalizing behaviors.

“this review found a lack of evidence of effects of after-school programs on school attendance and externalizing behaviors for at-risk primary and secondary students.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The rapid growth of after-school programming was not based on strong empirical findings.

“the rapid growth of after-school programming resulted from lobbying and grass roots efforts and was not based on strong empirical findings”

• cites Apsler 2009 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

After-school programs have the potential to curb juvenile crime and improve youth developmental outcomes.

“after-school programs have the potential to curb juvenile crime and positively impact youth developmental outcomes both short and long-term”

May combine two claims.

Because 65% of Durlak et al.'s included studies were quasi-experiments, lack of baseline adjustment could over- or under-estimate effects.

“This is problematic because, although all included studies in Durlak et al.’s review used a comparison group, 65 % of the included studies were quasi-experiments and thus potentially suffered from selection bias. Without adjusting for baseline differences, the effects could be over- or under-estimated.”

• cites Durlak et al. 2010 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Publication bias can result from journals not publishing non-significant primary outcomes or authors not submitting such studies.

“Publication bias may be the result of journals choosing not to publish papers with non-significant primary outcomes (Hopewell et al. 2009) or study authors choosing not to submit the study for publication (Cooper et al. 1997).”

• cites Hopewell et al. 2009Not checked yet

  Not checked in the free run.

• cites Cooper et al. 1997 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Kane (2004) found that three of four studies showed no significant effect on attendance, with mixed effects on grades and limited impact on child self-care.

“In (Kane 2004) review of the results of four after-school program evaluations, three of the four studies found no statistically significant effect on attendance, mixed effects on grades within and across studies, some evidence of improvement in homework completion, positive effects on parental engagement in school, and limited impact on child self-care.”

• cites Kane 2004Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Lauer et al. (2006) found small but significant impact on reading and math achievement, with one-on-one tutoring having larger effect sizes.

“The review found small but significant impact on both reading and math achievement across studies, with those using one-on-one tutoring to have larger effect sizes.”

• cites Lauer et al. 2006 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

On average, students in after-school programs did not show improved behavior or school attendance compared to comparison peers.

“On average, students participating in after-school programs did not demonstrate improved behavior or school attendance compared to their comparison group peers.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Roth et al. (2010) found little support that increased participation improved academic and behavior outcomes, but found participation levels decrease with age and more frequent attendees have improved attendance.

“The review found little support that increased participation resulted in improved academic and behavior outcomes, but it did find after-school participation levels to decrease with age and more frequent attendees to have improved school attendance outcomes.”

• cites Roth et al. 2010 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Durlak et al. (2010) found an overall positive and statistically significant effect of after-school programs across all outcomes examined.

“In a meta-analysis of after-school programs (Durlak et al. 2010) found an overall positive and statistically significant effect of after-school programs across all outcomes examined (d = 0.22, CI 0.16, 0.29).”

• cites Durlak et al. 2010 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The analysis adjusted for baseline differences by subtracting the baseline effect size from the post-test effect size.

“To control for pre-test difference between the intervention and control conditions, we subtracted the pre-test effect size from the post-test effect size (Lipsey and Wilson 2001).”

• cites Lipsey and Wilson 2001Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Risk of bias was assessed by two review authors independently using the Cochrane Collaboration's tool.

“To examine the risk of bias of included studies, two review authors independently rated each included study using the Cochrane Collaboration’s tool for assessing risk of bias.”

• cites Higgins et al. 2011 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

I2 quantifies heterogeneity beyond sample differences.

“(Higgins and Thompson 2002) suggested the calculation of I2, which quantifies the amount of heterogeneity beyond sample differences.”

• cites Higgins and Thompson 2002 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Positive-result studies reach publication more often and sooner than negative-result studies.

“Additionally, a review by (Hopewell et al. 2009) found studies with positive results to be published more frequently and more quickly than studies with negative results.”

• cites Hopewell et al. 2009Not checked yet

  Not checked in the free run.

May combine two claims.

Durlak et al. 2010 assigned an effect size of zero for non-significant outcomes lacking sufficient data, a procedure that can bias results.

“Also, Durlak and colleagues estimated an effect size of zero for all outcomes when the primary study authors reported the result as non-significant and did not report enough data to calculate a true effect size.”

• cites Durlak et al. 2010 sourceNot checked yet

  Not checked in the free run.

• cites Lipsey and Wilson 2001Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The review followed Campbell Collaboration standards for systematic reviews.

“We followed the Campbell Collaboration standards for the conduct of systematic reviews of interventions (Campbell Collaboration, 2014).”

• cites Campbell Collaboration, 2014Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Meta-analysis was used to synthesize effects quantitatively, and moderator analysis was conducted.

“Meta-analysis was used to synthesize the effects of interventions quantitatively, and moderator analysis was conducted to examine potential moderating variables.”

May combine two claims.

Durlak et al. 2010 sometimes failed to adjust for baseline differences and did not require baseline control or demonstrated equivalence.

“Durlak and colleagues, furthermore, did not adjust for pre-test differences in all cases and did not require that studies control for pre-test differences or demonstrate pre-test equivalence.”

• cites Durlak et al. 2010 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Coders initially agreed on the Risk of Bias tool 81% of the time, and disagreements were settled by consensus.

“Initial agreement between coders on the Risk of Bias tool was 81 %. Coders reviewed the coding agreement, and discrepancies were discussed and resolved by consensus.”

May combine two claims.

Many prior reviews lacked transparency in inclusion criteria, systematic searching, and data extraction methods.

“Many prior reviews were not transparent in their inclusion criteria (Fashola 1998; Hollister 2003; Simpkins et al. 2004; Apsler 2009) and lacked systematic searching (Fashola 1998; Scott-Little et al. 2002; Hollister 2003; Simpkins et al. 2004; Apsler 2009) and data extraction methods (Fashola 1998; Hollister 2003; Simpkins et al. 2004; Apsler 2009; Roth et al. 2010).”

• cites Fashola 1998Not checked yet

  Not checked in the free run.

• cites Hollister 2003Not checked yet

  Not checked in the free run.

• cites Apsler 2009 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The review used systematic review methodology for all aspects of search, selection, and coding.

“Systematic review methodology was used for all aspects of the search, selection, and coding of studies.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Zief et al. (2006) found limited program impact on academic and behavioral outcomes and a small positive impact on self-care.

“The review found limited program impact on academic and behavioral outcomes and a small positive impact on instance of self-care.”

• cites Zief et al. 2006Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Durlak et al. (2010) found support for moderation on four outcomes for programs using SAFE practices.

“They also examined several moderator variables and found support for moderation on four outcomes for programs that used specific practices (i.e., SAFE: sequenced, active, focused, and explicit) compared to those that did not use those practices.”

• cites Durlak et al. 2010 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Durlak et al. (2010) found no significant effects on drug use and school attendance.

“but no significant effects on drug use and school attendance”

• cites Durlak et al. 2010 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Imputing zero for non-significant results artificially decreases variability and is not recommended.

“Imputing zero for studies that report non-significant results is not recommended as it artificially decreases the variability of the variables.”

• cites Lipsey and Wilson 2001Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Prior reviews of after-school programs have yielded mixed and inconclusive findings, but most concur that more rigorous evaluations are needed.

“Overall, prior reviews of after-school programs have yielded mixed and inconclusive findings of effects on various outcomes, but most have concurred that more rigorous evaluations of after-school programs are needed.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The coding instrument was pilot tested by two authors and adjusted.

“The data extraction instrument, available from the authors, was pilot tested by two authors and adjustments to the coding form were made.”

May combine two claims.

The review's results conflict with Durlak et al. 2010's positive problem-behavior finding, but agree with their null drug-use and attendance findings and with Zief et al. 2006's null behavior and attendance findings.

“These results contradict (Durlak et al. 2010) findings of significant effects on problem behaviors [ES 0.19, CI (0.10, 0.27)], but corroborate Durlak and colleagues’ findings of non-significant effects on drug use and school attendance, and (Zief et al. 2006) findings of no effects on behavior and school attendance.”

• cites Durlak et al. 2010 sourceNot checked yet

  Not checked in the free run.

• cites Zief et al. 2006Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Descriptive data were coded by one author, with 20% of studies coded by a second, and agreement was 92.3%.

“Descriptive data related to study, sample, and intervention characteristics were coded by one author, with 20 % of the studies coded by a second author. Inter-rater agreement on descriptive items was 92.3 %.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Fashola (1998) cited the need for more rigorous study designs to increase confidence in after-school program effectiveness.

“(Fashola 1998) review across 34 studies inspected the effectiveness of programs and ease of replicability, again citing the need for more rigorous study designs to increase confidence in after-school program effectiveness.”

• cites Fashola 1998Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Apsler (2009) found serious methodological flaws in after-school program research.

“(Apsler 2009) literature review of after-school programs examined the quality of after-school program research; the review found serious methodological flaws, such as inappropriate comparison groups, sporadic participant attendance, and high attrition, which limited conclusions that could be drawn from the included studies.”

• cites Apsler 2009 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Robust variance estimation allows all estimated effect sizes from a study to be included and synthesized simultaneously, unlike traditional meta-analysis.

“Unlike traditional meta-analysis, robust variance estimation allows for the inclusion and synthesis of all estimated effect sizes simultaneously (Hedges et al. 2010; Tanner-Smith and Tipton 2013).”

• cites Hedges et al. 2010 sourceNot checked yet

  Not checked in the free run.

• cites Tanner-Smith and Tipton 2013Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Hollister (2003) found that among ten studies with more rigorous methodologies, some programs were effective.

“(Hollister 2003) review of after-school programs examined their impact on academic and risky behavior. Among the ten studies with more rigorous methodologies, some programs were found to be effective.”

• cites Hollister 2003Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Scott-Little et al. (2002) concluded that after-school programs appear to have positive impacts, but included studies lacked designs for causal inference.

“Scott-Little and colleagues (Scott-Little et al. 2002) concluded in their review of 23 studies that after-school programs appear to have positive impacts on participants, but the included studies lacked the type of research designs and information to allow for making causal inferences.”

• cites Scott-Little et al. 2002 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The potential for after-school programs to positively affect externalizing behavior through increased supervision is more complicated than expected.

“(Cross et al. 2009) noted, however, that the potential for after-school programs to impact externalizing behavior positively through increased supervision is more complicated than one might expect.”

• cites Cross et al. 2009 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Earlier reviews of after-school programs repeatedly called for greater rigor, yet progress has been limited.

“Although past after-school program reviews have consistently suggested improvements in the rigor of after-school program research, limited progress has been made.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some studies have found no support for moderating variables.

“while others have found no support for moderating variables (Roth et al. 2010).”

• cites Roth et al. 2010 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

It is possible that researchers measured attendance or externalizing outcomes but withheld nonsignificant results, which could inflate the effects in this meta-analysis.

“researchers could have measured these outcomes but chose not to report them if they were not significant, thus potentially inflating the effects of after-school programs reported in this meta-analysis.”

May combine two claims.

Any of the procedures used by Durlak et al. 2010 could have biased effect size estimates and explain the difference between the two reviews.

“Any of these procedures could have resulted in a bias of the effect size estimates and could explain the difference in the meta-analytic results between the two reviews.”

• cites Durlak et al. 2010 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Excluding studies without pre-test data or baseline adjustment may have created selection bias at the review level.

“Although unlikely, we could have introduced review level selection bias by excluding studies that did not provide pre-test data or adjust for baseline differences.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Earlier reviews and primary studies indicate that features of programs and participants could moderate the effects of after-school programs.

“Prior reviews and primary research suggest that program and participant characteristics may moderate effects of after-school programs, such as program quality and characteristics (Simpkins et al. 2004; Lauer et al. 2006; Durlak et al. 2010)”

• cites Simpkins et al. 2004Not checked yet

  Not checked in the free run.

• cites Lauer et al. 2006 sourceNot checked yet

  Not checked in the free run.

• cites Durlak et al. 2010 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The review required included studies to supply pre-test outcome data or adjust for baseline scores, partly to address selection bias.

“we limited studies included in this review to those that provided pre-test data of the outcomes of interest or adjusted for pre-test on the outcome, so we could control for selection bias to some extent.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Certain reviews and primary studies point to possible moderators for after-school programs and other organized youth activities, including study quality and program features.

“Some reviews and individual studies have identified potential moderators of after-school programs and other types of organized youth activities, such as study quality and characteristics of the program (i.e., length, intensity, presence of specific components; Bohnert et al. 2010).”

• cites Bohnert et al. 2010 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

After-school programs can reduce delinquency by providing adult supervision and structured activities.

“By providing adult supervision and structured activities, after-school programs have the potential to reduce delinquency (Cross et al. 2009, p. 394).”

• cites Cross et al. 2009 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The review limited the quantity of moderator tests to decrease the probability of spurious results.

“Given sufficient heterogeneity, we conducted moderator analyses; we limited the quantity of such tests to decrease the probability of spurious results (Polanin and Pigott 2014).”

• cites Polanin and Pigott 2014Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Final inclusion decisions were made by two authors, with discrepancies resolved by consensus and a third author when needed.

“Following initial screening, potentially eligible studies were further reviewed by two authors to determine final inclusion. Any discrepancies between authors were discussed and resolved through consensus, and when needed, a third author reviewed the study.”