Targeted school-based interventions for improving reading and mathematics for students with, or at risk of, academic difficulties in Grades 7–12: A systematic review

Abstract

BACKGROUND Low levels of numeracy and literacy skills are associated with a range of negative outcomes later in life, such as reduced earnings and health. Obtaining information about effective interventions for educationally disadvantaged youth is therefore important. OBJECTIVES The main objective was to assess the effectiveness of interventions targeting students with or at risk of academic difficulties in Grades 7–12. SEARCH METHODS We searched electronic databases from 1980 to July 2018. We searched multiple international electronic databases (in total 14), seven national repositories, and performed a search of the grey literature using governmental sites, academic clearinghouses, and repositories for reports and working papers, and trial registries (10 sources). We hand searched recent volumes of six journals and contacted international experts. Lastly, we used included studies and 23 previously published reviews for citation tracking. SELECTION CRITERIA Studies had to meet the following criteria to be included: - Population: The population eligible for the review included students attending regular schools in Grades 7–12, who were having academic difficulties, or were at risk of such difficulties. - Intervention: We included interventions that sought to improve academic skills, were performed in schools during the regular school year, and were targeted (selected/indicated). - Comparison: Included studies used a treatment‐control group design or a comparison group design. We included RCTs, quasirandomised controlled trials (QRCTs) and QESs. - Outcomes: Included studies used standardised tests in reading or mathematics. - Setting: Studies carried out in regular schools in an OECD country were included. DATA COLLECTION AND ANALYSIS Descriptive and numerical characteristics of included studies were coded by members of the review team. A review author independently checked coding. We used an extended version of the Cochrane Risk of Bias tool to assess risk of bias. We used random‐effects meta‐analysis and robust‐variance estimation procedures to synthesise effect sizes. We conducted separate meta‐analyses for tests performed within three months of the end of interventions (short‐run effects) and longer follow‐up periods. For short‐run effects, we performed subgroup and moderator analyses focused on instructional methods and content domains. Sensitivity of the results to effect size measurement, outliers, clustered assignment of treatment, missing values, risk of bias and publication bias was assessed. RESULTS We found 24,411 potentially relevant records and screened 4,244 in full text. In total 247 studies met our inclusion criteria and we included 71 studies in meta‐analyses. The reasons for not including studies in the meta‐analyses were that they had too high risk of bias (118), compared two alternative interventions (38 studies), lacked necessary information (13 studies), or used overlapping samples (7 studies). Of the 71 studies, 99 interventions, and 214 effect sizes included in the meta‐analysis, 76% were RCTs, and the rest QESs. The total number of student observations in the analysed studies was around 105,700. The target group consisted of, on average, 47% girls, 73% minority students, and 62% low income students. The mean grade was 8.3. Most studies included in the meta‐analysis had a moderate to high risk of bias. The average effect size for short‐run outcomes was positive and statistically significant (weighted average effect size [ES] = 0.22, 95% confidence interval [CI] = [0.148, 0.284]). The effect size corresponds to a 56% chance that a randomly selected score of a student who received the intervention is greater than the score of a randomly selected student who did not. All measures indicated substantial heterogeneity across effect sizes. Seven studies included follow‐up outcomes. The average effect size was small and not statistically significant (ES = 0.05, 95% CI = [−0.096, 0.192]), but there was substantial variation. We focused the analysis of comparative effectiveness on the short‐run outcomes and two types of intervention components: instructional methods and content domains. Interventions that included small group instruction (ES = 0.38, 95% CI = [0.211, 0.547]), peer‐assisted instruction (ES = 0.19, 95% CI = [0.061, 0.319]), progress monitoring (ES = 0.19, 95% CI = [0.086, 0.290]), CAI (ES = 0.17, 95% CI = [0.043, 0.309]) and coaching of personnel (ES = 0.10, 95% CI = [0.038, 0.166]) had positive and significant average effect sizes. Interventions that provided incentives for students did not have a significant average effect size (ES = 0.05, 95% CI = [−0.103, 0.194]). The average effect size of interventions that included none of the above components, but for example provided extra instructional time, instruction in groups smaller than whole class but larger than 5 students, or just changed the content had a relatively large, but statistically insignificant effect size (ES = 0.20