Not Too Late: Improving Academic Outcomes Among Adolescents

May combine two claims.

In the first RCT, participation increased grades in math and non-math courses.

“Our first randomized controlled trial (RCT) of Saga's tutoring model with 2,633 9th and 10th grade students in Chicago public schools found participation increased math test scores by 0.16 standard deviations (SDs) and increased grades in math and non-math courses.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Estimates for high school graduation are imprecise.

“These effects persist into future years, although estimates for high school graduation are imprecise.”

May combine two claims.

Survey data show no detectable effects on grit, conscientiousness, or locus of control.

“or on grit, conscientiousness, or locus of control.”

• AssumptionRests on an assumption: The survey measures of social-emotional skills and relationships with adults validly capture the constructs they are intended to measure.

May combine two claims.

The point estimate for graduation is positive, 1.9 percentage points, but imprecisely estimated; declines of up to 4.8 points or gains as large as 8.6 points cannot be ruled out.

“The point estimate for graduation is positive, 1.9 percentage points, but imprecisely estimated; we cannot rule out declines of up to 4.8 points or gains as large as 8.6 points.”

May combine two claims.

When items are rank-ordered by difficulty and effects are estimated for specific items, there are suggestive but imprecisely estimated indications that students with below-median baseline scores gain most on the easiest items, while above-median students gain most on harder items.

“When we rank-order items by difficulty and estimate effects for specific items we see suggestive (but imprecisely estimated) indications that students with below-median baseline scores gain most on the easiest items, while above-median students gain most on harder items.”

May combine two claims.

In study 2, the treatment-on-the-treated effect on math scores after one year is 0.37 SD, roughly twice as large as in study 1.

“The TOT effect on math scores after one year is roughly twice as large as in study 1 (0.37 SD).”

May combine two claims.

Results for 9th graders are similar to those from the pooled sample of 9th and 10th graders, consistent with findings suggesting the effects of CPS double dose algebra are limited.

“Appendix Table 16 shows results for 9 th graders are similar to those from the pooled sample of 9 th and 10 th graders, consistent with the findings of Nomi and Allensworth (2009) suggesting the effects of CPS double dose algebra are limited.”

• cites Nomi and Allensworth (2009)Not checked yet

  Not checked in the free run.

May combine two claims.

In study 1, some estimated effects on arrests are large relative to control means but are not statistically significant.

“However, some of the estimated effects on arrests are large relative to the control means, though they are not statistically significant.”

May combine two claims.

The estimated TOT effect on graduating on time from high school, pooling study 1 and 2 samples, is positive 1.9 percentage points relative to a control complier mean of 77.2%, but it is imprecisely estimated.

“Table 9 shows that the estimated TOT effect on graduating on time from high school, pooling the study 1 and 2 samples again, is positive 1.9 percentage points relative to a CCM of 77.2%, but this is imprecisely estimated.”

• AssumptionThe authors state this as an assumption: For the TOT IV estimate, treatment assignment has no effect on the outcomes of those assigned to treatment who do not participate.
• AssumptionThe authors state this as an assumption: The stable unit treatment value assumption (SUTVA) holds: treatment assignment has no spillover effects on control students' outcomes.

May combine two claims.

In 11th grade, the TOT effect on math test scores is 0.22 SD, about the same size as the pooled impact measured at the end of one year of programming.

“We see TOT effects on math test scores in 11 th grade of 0.22 SD, about the same size as the pooled impact measured at the end of one year of programming, and the TOT on math grades is 0.18 GPA points, about 35% of the year one effect.”

• AssumptionThe authors state this as an assumption: For the TOT IV estimate, treatment assignment has no effect on the outcomes of those assigned to treatment who do not participate.
• AssumptionThe authors state this as an assumption: The stable unit treatment value assumption (SUTVA) holds: treatment assignment has no spillover effects on control students' outcomes.

May combine two claims.

There are suggestive reductions in use of drugs other than marijuana and seriously hurting someone in a fight, but these are not statistically significant once multiple testing is accounted for.

“We also see suggestive reductions in use of drugs (other than marijuana) and seriously hurting someone in a fight, though these are not statistically significant once we account for the number of hypotheses being tested here.”

May combine two claims.

The academic subject matter of the tutoring was specifically math, and the study finds significant positive effects on math test scores but no effects on reading test scores.

“The academic subject matter of the tutoring was specifically math, and we find significant positive effects on math test scores, but no effects on reading test scores.”

May combine two claims.

In study 1, the TOT impact on math grades is 0.56 points on a 1-4 GPA scale, relative to a control complier mean of 1.63, representing a change of about C- to C+.

“So it is notable that we see changes in math grades as well, with a TOT impact of 0.56 points on a 1-4 GPA scale, relative to the control complier mean (CCM) of 1.63; this represents a change of about a C-to a C+.”

• AssumptionThe authors state this as an assumption: For the TOT IV estimate, treatment assignment has no effect on the outcomes of those assigned to treatment who do not participate.
• AssumptionThe authors state this as an assumption: The stable unit treatment value assumption (SUTVA) holds: treatment assignment has no spillover effects on control students' outcomes.

May combine two claims.

In 11th grade, the TOT effect on math grades is 0.18 GPA points, about 35% of the year one effect.

“We see TOT effects on math test scores in 11 th grade of 0.22 SD, about the same size as the pooled impact measured at the end of one year of programming, and the TOT on math grades is 0.18 GPA points, about 35% of the year one effect.”

• AssumptionThe authors state this as an assumption: For the TOT IV estimate, treatment assignment has no effect on the outcomes of those assigned to treatment who do not participate.
• AssumptionThe authors state this as an assumption: The stable unit treatment value assumption (SUTVA) holds: treatment assignment has no spillover effects on control students' outcomes.

May combine two claims.

In study 1 year 1, the participation rate was 40.2% for those assigned to treatment and 1.1% for controls.

“The participation rate of study 1 youth in year 1 (defined as receipt of any Saga tutoring at all) was 40.2% for those assigned to treatment and 1.1% for controls, and for study 2 was 36.9% for the treatment group and 7.8% for controls.”

May combine two claims.

For study 1, about 25% of control students in the schools received SES tutoring, which involves 21 hours of writing tutoring per year and 20 hours of math tutoring.

“For example, for study 1 we estimate about 25% of control students in our schools received SES tutoring, which involves 21 hours of writing tutoring per year and 20 hours of math tutoring”

May combine two claims.

In the pooled sample, the TOT estimate is a decline of 0.09 in percentage of math courses failed, equal to 47% of the control complier mean.

“In the pooled sample, the TOT estimate is a 0.26 SD increase in math test scores, a 0.52 point increase in math GPA relative to a CCM of 1.67, and a decline of 0.09 in percentage of math courses failed, equal to 47% of the CCM.”

• AssumptionThe authors state this as an assumption: For the TOT IV estimate, treatment assignment has no effect on the outcomes of those assigned to treatment who do not participate.
• AssumptionThe authors state this as an assumption: The stable unit treatment value assumption (SUTVA) holds: treatment assignment has no spillover effects on control students' outcomes.

May combine two claims.

The standard error of 3.4 percentage points means the study cannot rule out a decline in graduation rates as large as -4.8 percentage points or an increase as large as +8.6 points.

“The standard error of 3.4 percentage points means we cannot rule out a decline in graduation rates as large as -4.8 percentage points or an increase as large as +8.6 points.”

May combine two claims.

Baseline math scores are strikingly different across the PTE quartiles: the top quartile has scores 0.61 SD above the study sample average at baseline, and the bottom quartile 1.2 SD below average at baseline.

“But it's also the case that the baseline math scores are strikingly different across these quartiles as well, with those in the top quartile having scores 0.61 SD above our study sample average at baseline and those in the bottom quartile 1.2 SD below average at baseline.”

May combine two claims.

The 90% time-on-math figure is high relative to other published figures for regular public school classrooms, but similar observations could not be carried out for the specific CPS classrooms that serve as the control condition.

“This figure is high relative to other published figures for regular public school classrooms, but we could not carry out similar observations for the specific CPS classrooms that serve as the control condition in this study.”

May combine two claims.

In wave 1 of the survey, there are no statistically significant changes in measures of grit, conscientiousness, or, after multiple testing corrections, locus of control.

“Panels B, C, and D show there are no statistically significant changes in measures of”

May combine two claims.

Around one-quarter of the study sample has estimated math test score gains in the neighborhood of zero or negative, but positive and sometimes substantially positive gains in math GPA.

“For present purposes the most relevant feature of Figure 3 is the fact that around onequarter of our study sample has estimated math test score gains in the neighborhood of zero or negative, but positive (and sometimes substantially positive) gains in math GPA.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Majority (over 97%) of study students for whom we have grade data only take one math class each semester after excluding Math Lab and double-dose Algebra.

“majority (over 97%) of study students for whom we have grade data only take one math class each semester after excluding Math Lab and double-dose Algebra.”

May combine two claims.

Above-median students score higher than below-median students on every item, and in both groups the share who answer a question correctly increases with the P-score.

“Above-median students score higher than below-median students on every item, and in both groups, the share who answer a question correctly increases with the P-score (see appendix figure 3).”

May combine two claims.

In wave 1 of the survey, there are no statistically significant changes in the number of adults the student reports having available to talk to in school, the number of adults they think care about them, or their willingness to talk to adults in school.

“Panel A of Table 10 shows no statistically significant changes in the number of adults the student reports having available to talk to in their school, the number of adults they think care about them, or their willingness to talk to adults in the school.”

The quote only partly states this claim.

In the pooled Study 1 and 2 sample at Year 1, the treatment improved math GPA (ITT = 0.219, p < 0.01; TOT = 0.518).

“Math GPA 4277 1.803 0.219*** (0.029) 0.518***”

May combine two claims.

On average, Saga increases student's test scores by six percentile points in study 1 and by 14 percentile points for study 2.

“We find that, on average, Saga increases student's test scores by six percentile points in study 1 and by 14 percentile points for study 2.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Visually, below-baseline-median students seem to benefit the most on items in the bottom or middle of the difficulty distribution while above-median students may benefit the most on more difficult items.

“visually it looks like below-baseline-median students seem to benefit the most on items in the bottom or middle of the difficulty distribution while the above-median may benefit the most on more difficult items.”

May combine two claims.

In CPS administrative data, those students do not show particularly large gains or any gains in measures of effort such as school attendance or school disciplinary infractions; nor in survey data are there changes in measures like time spent on homework or share homework completed.

“But in the CPS administrative data we do not see those students showing particularly large gains (or any gains for that matter) in measures of effort such as school attendance or school disciplinary infractions.Nor in the survey data do we see any changes in measures like time spent on homework, or share homework completed.”

The quote only partly states this claim.

In the first wave of surveys, there is no statistically significant impact on the item asking if disruptions by others get in the way of learning.

“In addition, in the first wave of surveys we administered to students to measure mechanisms, we asked if”

The quote only partly states this claim.

In the pooled Study 1 and 2 sample at Year 1, the treatment had no statistically significant effect on total arrests (ITT = -0.048, SE = 0.029).

“Total Arrests 5343 0.414 -0.048 (0.029) -0.13”

May combine two claims.

In wave 2 of the survey, there are no detectable impacts on perceptions about supportive adults, grit, conscientiousness, locus of control, growth mindset, or social networks.

“Here again we see no detectable impacts on perceptions about supportive adults, grit, conscientiousness, locus of control, growth mindset, or social networks.”

May combine two claims. The quote only partly states this claim.

In Study 1 at Year 1, the treatment improved math GPA (ITT = 0.274, p < 0.01; TOT = 0.561, p < 0.01).

“| Intent to | Treat | Estimate | 0.082** | 0.274*** | -0.043*** | 0.119** | 1.344** |”

May combine two claims. The quote only partly states this claim.

In Study 1 at Year 1, the treatment reduced the percentage of math courses failed (ITT = -0.043, p < 0.01; TOT = -0.087, p < 0.01).

“| Intent to | Treat | Estimate | 0.082** | 0.274*** | -0.043*** | 0.119** | 1.344** |”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In Study 2, the 11th grade intent-to-treat effect on Math Test (z-score) was 0.017 (SE 0.041), with N=1445, control mean 0.000, and FDR Q-value 0.717.

“11th Grade Math Test (z-score) | 1445 | 0.000 | 0.017 | (0.041) | 0.046 | (0.126) | 0.046 | 0.717”

May combine two claims. The quote only partly states this claim.

In Study 1 at Year 2, the treatment improved math test scaled scores (ITT = 0.163, p < 0.01; TOT for Year 2 = 0.839, p < 0.01).

“| | ITT | Estimate | | | 0.163*** | 0.14*** | -0.021 | 0.067 | 0.984* |”

May combine two claims.

In Study 1, the 11th grade intent-to-treat effect on Math Test (z-score) was 0.158 (SE 0.04), with N=1528, control mean 0.010, and FDR Q-value 0.001.

“11th Grade Math Test (z-score) | 1528 | 0.010 | 0.158*** | (0.04) | 0.3*** | (0.074) | -0.214 | 0.001”

May combine two claims. The quote only partly states this claim.

In Study 2, Year 1, the intent-to-treat effect on Math GPA was 0.15 (SE 0.043), with N=2062, control mean 1.859, and permutation p<0.05.

“ITT Estimate | 0.126*** | 0.15*** | -0.029** | | -0.007 | 0.065* | -0.011 | | -0.002 | 0.756 | 0.07 | | -0.013 | -0.026 | 0 | -0.052* | -0.019 | -0.091** | | |”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In Study 1, the intent-to-treat effect on Graduated On-Time was 0.008 (SE 0.017), with N=1853, control mean 0.738, and FDR Q-value 0.74.

“Graduated On-Time | 1853 | 0.738 | 0.008 | (0.017) | 0.017 | (0.036) | 0.759 | 0.74”

May combine two claims. The quote only partly states this claim.

In Study 1 at Year 1, the treatment improved math test scaled scores (ITT = 0.082, p < 0.05; TOT = 0.16, p < 0.05).

“| Intent to | Treat | Estimate | 0.082** | 0.274*** | -0.043*** | 0.119** | 1.344** |”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the pooled analysis of both studies, the intention-to-treat effect on number of math courses is 0.007 (SE 0.009).

“Number of Math Courses | 4331 | 1.900 0.007 | (0.009) | 0.017 | (0.021) | 1.942 | 0.423”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In Study 2, the intent-to-treat effect on Graduated On-Time was 0.008 (SE 0.019), with N=1897, control mean 0.761, and FDR Q-value 0.919.

“Graduated On-Time | 1897 | 0.761 | 0.008 | (0.019) | 0.025 | (0.06) | 0.788 | 0.919”

May combine two claims.

In Study 1, Year 1, the intent-to-treat effect on Math GPA was 0.274 (SE 0.04), with N=2215, control mean 1.760, and permutation p<0.01.

“SY 2014 Math GPA 2215 1.760 0.274*** (0.04) 0***”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

In Study 2, Year 1, the intent-to-treat effect on Math Courses Failed (percent) was -0.029 (SE 0.013), with N=2062, control mean 0.149, and permutation p=0.159.

“ITT Estimate | 0.126*** | 0.15*** | -0.029** | | -0.007 | 0.065* | -0.011 | | -0.002 | 0.756 | 0.07 | | -0.013 | -0.026 | 0 | -0.052* | -0.019 | -0.091** | | |”

May combine two claims.

In the pooled Year 1 sample, the intent-to-treat effect on Drug Arrests was -0.013 (SE 0.009), and the interaction with whether the school was above median in CPS disciplinary incidents was 0.043 (SE 0.017), with N=5410.

“Drug Arrests | 5410 | 0.054 | -0.013 | (0.009) | 0.043** | (0.017)”

May combine two claims.

In the pooled Year 1 sample, the intent-to-treat effect on Out-of-School Suspensions was -0.09 (SE 0.075), and the interaction with whether the school was above median in CPS disciplinary incidents was 0.421 (SE 0.174), with N=5035.

“Out-of-School Incs | 5035 | 1.180 | -0.09 | (0.075) | 0.421** | (0.174)”

May combine two claims.

A comparison of Saga Tutors SAT scores from AY2016-17 and AY2017-18 in Chicago reveals that average math SAT scores were 707 compared with the average math SAT scores for new NYC teachers of 613 reported in Rockoff, Jacob, Kane, and Staiger (2008).

“A comparison of Saga Tutors SAT scores from AY2016-17 and AY2017-18 in Chicago reveals that average math SAT scores were 707 compared with the average math SAT scores for new NYC teachers of 613 reported inRockoff, Jacob, Kane, and Staiger (2008).”

• cites Rockoff, Jacob, Kane, and Staiger (2008)Not checked yet

  Not checked in the free run.

May combine two claims. The quote only partly states this claim.

In Study 1 at Year 1, the treatment reduced the percentage of non-math core courses failed (ITT = -0.027, p < 0.05; TOT = -0.056, p < 0.05).

“| Intent to | Treat | Estimate | 0.082** | 0.274*** | -0.043*** | 0.119** | 1.344** |”

May combine two claims.

In the pooled Year 1 sample, the intent-to-treat effect on Disciplinary Incidents was -0.171 (SE 0.082), and the interaction with whether the school was above median in CPS disciplinary incidents was 0.379 (SE 0.166), with N=5035.

“Disciplinary incidents | 5035 | 1.549 | -0.171** | (0.082) | 0.379** | (0.166)”

May combine two claims.

In the pooled Year 1 sample, the intent-to-treat effect on Days Absent was -0.281 (SE 0.719), and the interaction with whether the school was above median in CPS disciplinary incidents was 2.252 (SE 1.157), with N=5035.

“Days Absent | 5035 | 23.932 | -0.281 | (0.719) | 2.252* | (1.157)”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Study 1, Year 1, the intent-to-treat effect on Math Test Scaled Score (Z) was 0.082 (SE 0.036), with N=1852, control mean -0.000, and permutation p=0.252.

“SY 2014 Math Test Scaled Score (Z) 1852 -0.000 0.082** (0.036) 0.252”

May combine two claims. The quote only partly states this claim.

In Study 1 at Year 2, the treatment improved math GPA (ITT = 0.14, p < 0.01; TOT for Year 2 = 0.685, p < 0.01).

“| | ITT | Estimate | | | 0.163*** | 0.14*** | -0.021 | 0.067 | 0.984* |”

May combine two claims. The quote only partly states this claim.

In Study 2, Year 1, the intent-to-treat effect on Math Test Scaled Score (Z) was 0.126 (SE 0.036), with N=1865, control mean 0.008, and permutation p<0.05.

“ITT Estimate | 0.126*** | 0.15*** | -0.029** | | -0.007 | 0.065* | -0.011 | | -0.002 | 0.756 | 0.07 | | -0.013 | -0.026 | 0 | -0.052* | -0.019 | -0.091** | | |”

May combine two claims. The quote only partly states this claim.

In Study 1 at Year 1, the treatment improved non-math GPA (ITT = 0.081, p < 0.05; TOT = 0.201, p < 0.01).

“| Intent to | Treat | Estimate | 0.082** | 0.274*** | -0.043*** | 0.119** | 1.344** |”

May combine two claims.

In Study 1, Year 1, the intent-to-treat effect on Math Courses Failed (percent) was -0.043 (SE 0.013), with N=2215, control mean 0.191, and permutation p<0.01.

“SY 2014 Math Courses Failed (percent) 2215 0.191 -0.043*** (0.013) 0.01***”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the pooled Year 1 sample (Studies 1 and 2), the intent-to-treat effect of the program on Total Arrests was -0.048 (SE 0.029), with N=5343, control mean 0.414, and permutation p=0.169.

“Total Arrests 5343 0.414 -0.048 (0.029) 0.169”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Being a successful tutor requires fewer specialized skills and less on-the-job learning than does classroom teaching.

“The key insight behind the specific educational intervention we study here, developed by Saga Education, is that being a successful tutor requires fewer specialized skills and less on-thejob-learning than does classroom teaching.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Teachers report that the two most challenging parts of their job are individualizing instruction and classroom management.

“No wonder teachers report that the two most challenging parts of their job are individualizing instruction and classroom management.”

Probably not a substantial claim: may not pass S4 (based).

The challenge to widespread implementation of tutoring has not been a pedagogical problem so much as an economic one, namely cost.

“The challenge to widespread implementation has not been a pedagogical problem so much as an economic one-that is, cost.”

May combine two claims.

Math is important for success in school and later earnings.

“because math is so important for success in school and later earnings (Duncan et al., 2007)”

• cites Duncan et al., 2007Not checked yet

  Not checked in the free run.

May combine two claims.

Motivation becomes more challenging as students age, as reflected by increased prevalence of disruptive behaviors.

“Motivation becomes more challenging as students age, as reflected by increased prevalence of disruptive behaviors.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The failure to improve schooling outcomes at a pace to keep up with technological change has harmed economic growth, leading to a dramatic rise in the returns to schooling over time.

“The failure to improve schooling outcomes at a pace to keep up with technological change has harmed economic growth, leading to a dramatic rise in the returns to schooling over time (Goldin and Katz, 2008).”

• cites Goldin and Katz, 2008Not checked yet

  Not checked in the free run.

May combine two claims.

The insight that tutoring requires fewer specialized skills expands the applicant pool and means tutors can be successful right away rather than after the usual three-year learning curve for classroom teachers.

“This insight expands the applicant pool, and also means that the people hired-tutors-can be successful right away (rather than the usual threeyear learning curve for classroom teachers”

May combine two claims.

The 0.16 SD gain is about what the average high school student learns in a year, so the intervention roughly doubles the annual math test score gain.

“This gain is about what the average high school student learns in a year, so the intervention (roughly speaking) doubles the annual math test score gain.”

Classified as probably not a substantial claim.

This substitution makes high-dosage tutoring more feasible from a cost perspective.

“This substitution makes high-dosage tutoring more feasible from a cost perspective.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Hired tutors have higher SAT scores than what other studies have reported for big-city public school classroom teachers.

“Hired tutors have higher SAT scores than what other studies have reported for big-city public school classroom teachers (Jacob et al., 2018).”

• cites Jacob et al., 2018Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The findings echo Banerjee et al. (2007)'s study in India of gains from substituting a simpler teaching task for teacher skill.

“Our findings echo Banerjee et al. (2007)'s study in India of gains from substituting a simpler teaching task for teacher skill.”

• cites Banerjee et al., 2007Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Duflo et al. (2011) found that personalization by tracking improved learning outcomes for students in both the top and bottom halves of the achievement distribution.

“This suggestive result is consistent with Duflo et al.'s (2011) finding that personalization by tracking improved learning outcomes for students in both the top and bottom halves of the achievement distribution.”

• cites Duflo et al., 2011Not checked yet

  Not checked in the free run.

May combine two claims.

Compared to regular classroom instruction, one-on-one or two-on-one tutoring greatly simplifies the instructional task, expanding the set of people who can be successful instructors.

“Compared to regular classroom instruction, one-on-one (or two-on-one) tutoring greatly simplifies the instructional task, expanding the set of people who can be successful instructors.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Roughly half of tutors hired were Black or Latinx, and approximately 50% were female.

“Roughly half of tutors hired were Black or Latinx, and approximately 50% were female.”

Classified as probably not a substantial claim.

Saga Education's tutoring model was selected to study in part because of its low cost relative to the intensity of the intervention.

“We selected Saga Education's tutoring model to study in part because of its low cost relative to the intensity of the intervention.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some studies find classroom teachers orient material towards the top third of the classroom.

“Some studies find classroom teachers orient material towards the top third of the classroom (Bloom, 1984)”

• cites Bloom, 1984Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Saga hired 139 total tutors across both study years out of an estimated pool of approximately 1,200 applicants.

“Saga hired 139 total tutors across both study years out of an estimated pool of approximately 1,200 applicants.”

Classified as probably not a substantial claim.

Past efforts to improve metacognition and social-emotional skills for adolescents and young adults have been more encouraging than efforts to improve academic skills.

“Given that past efforts to improve metacognition and social-emotional skills for adolescents and young adults have been more encouraging than efforts to improve academic skills (for example Blattman et al., 2017, Heller et al., 2017)”

• cites Blattman et al., 2017Not checked yet

  Not checked in the free run.

• cites Heller et al., 2017Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The Saga intervention is similar in spirit to how charters often hire teachers with less experience and formal teaching credentials than normal public schools in exchange for smaller class sizes.

“The Saga intervention is similar in spirit to how charters often hire teachers with less experience and formal teaching credentials than normal public schools in exchange for smaller class sizes (Lake, Bowen, & Demeritt, 2012).”

• cites Lake, Bowen, & Demeritt, 2012Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Teachers learn how to individualize instruction or handle classroom management usually through extensive preservice pedagogical training as well as on-the-job learning over the first several years of classroom teaching.

“Teachers learn how to individualize instruction or handle classroom management usually through extensive preservice pedagogical training as well as on-the-job learning over the first several years of classroom teaching (Rockoff, 2004; Clotfelter et al., 2010; Henry et al., 2011).”

• cites Rockoff, 2004Not checked yet

  Not checked in the free run.

• cites Clotfelter et al., 2010Not checked yet

  Not checked in the free run.

• cites Henry et al., 2011Not checked yet

  Not checked in the free run.

May combine two claims.

Extrapolating the test score impacts to earnings gains based on the best estimates in the literature implies that the benefit-cost ratio of the program is comparable to exemplar early childhood model programs and larger-scale efforts to improve outcomes for younger children.

“Extrapolating these test score impacts to earnings gains based on the best estimates in the literature implies that the benefit-cost ratio of the program is comparable to both exemplar early childhood model programs, like the Abecedarian Project and the Perry Preschool Program, as well as larger-scale efforts to improve outcomes for younger children such as the Tennessee Star class size reduction”

Classified as probably not a substantial claim.

In summer 2014, the research team randomized a separate sample of 2,710 9th and 10th graders for study 2.

“Motivated in part to see if these results could be replicated at this large scale, in the summer of 2014 we randomized a separate sample of 2,710 9 th and 10 th graders (“study 2”).”

May combine two claims.

The intervention substantially improves learning of adolescents at a cost of $3,500 to $4,300 per year per student.

“One way to read this study is as a program evaluation of an intervention that substantially improves learning of adolescents, at a cost of $3,500 to $4,300 per year per student.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Saga tutors neither had formal teacher training nor were licensed Illinois teachers.

“But Saga tutors neither had formal teacher training nor were licensed Illinois teachers.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

A school in which classroom teachers cycle in and out after just one year would face significant challenges.

“A school in which classroom teachers cycle in and out after just one year would face significant challenges.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

The instructional technology of tutoring is quite different from that of a classroom, and so the set of skills and experiences required to be a successful tutor are plausibly different.

“One major innovation is the recognition that the instructional “technology” of tutoring is quite different from that of a classroom, and so the set of skills and experiences required to be a successful tutor are plausibly different.”

May combine two claims.

The learning gains in study 1 are not statistical flukes or the result of unusually good program implementation that cannot be replicated, because the effects are at least as large in study 2.

“Table 4 suggests the learning gains experienced by students in study 1 are not statistical flukes, or the result of some unusually good program implementation that cannot be replicated, since the effects are at least as large in study 2.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Post-randomization missingness for grades equals 14.6% for controls and 14.8% for treatment.

“and somewhat higher for grades (14.6% vs. 14.8%)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Post-randomization missingness for test scores equals 29.8% for controls and 29.5% for treatment.

“and test scores (29.8% vs. 29.5%)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Random assignment was conditional on school-by-grade randomization blocks, and the probability of assignment to treatment varied across randomization blocks.

“we carried out random assignment conditional on school-by-grade “randomization blocks” and varied the probability of assignment to the treatment condition across randomization blocks.”

May combine two claims.

The treatment effect seems to increase with larger share of individuals within a block randomized to treatment, opposite to what would be expected if control students were benefiting from positive spillovers; under this test there is no evidence for violation of the SUTVA assumption.

“We see that the treatment effect actually seems to increase (rather than decrease) with larger share of individuals within a block who are randomized to treatment.This is the opposite relationship we would expect if control group students were benefitting from positive spillovers, and so at least under this test we do not see evidence for violation of the SUTVA assumption.”

• AssumptionThe authors state this as an assumption: The stable unit treatment value assumption (SUTVA) holds: treatment assignment has no spillover effects on control students' outcomes.

May combine two claims.

For year 2, the experiment cleanly identifies the ITT effect but cannot by itself tell how much of the ITT effect at the end of year 2 is due to gains among treatment youth who received tutoring just in year 1 versus in both years 1 and 2, because dosage duration was not randomly assigned.

“One challenge is that, for year 2, the experiment cleanly identifies the ITT effect, but cannot by itself tell us how much of the ITT effect at the end of year 2 is due to gains among treatment youth who received tutoring just in year 1 versus in both years 1 and 2. This challenge stems from the fact that we did not randomly assign dosage duration.”

May combine two claims.

In study 1 schools, students were independently randomized to receive the BAM program, and controlling or not controlling for BAM assignment has no impact on the estimates of Saga effects reported here.

“Because that randomization was independent of assignment to Saga tutoring, controlling or not controlling for BAM assignment has no impact on the estimates of Saga effects reported here.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Post-randomization missingness for school attendance equals 4.4% for controls and 6.2% for treatment.

“which for school attendance equals 4.4% for controls and 6.2% for treatment”

May combine two claims.

The results suggest the effects are at least additive in years of participation, and perhaps more-than-additive as models of dynamic complementarities in learning would suggest.

“The results suggest the effects are at the least additive in years of participation, and perhaps more-than-additive as models of dynamic complementarities in learning would suggest.”

May combine two claims.

An obvious interpretation of the pattern of results is that the academic subject matter of the tutoring sessions matters; tutoring students in math helps them to learn math.

“An obvious interpretation of this pattern of results is that the academic subject matter of the tutoring sessions matters; tutoring students in math helps them to learn math.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Another wave of in-person surveys with 1,238 youth in the study 1 sample had effective response rates of 90.1% for the treatment group and 89.1% for controls.

“We carried out another wave of in-person surveys with 1,238 youth in the study 1 sample in the fall after the second intervention year (2014-15), with effective response rates of 90.1% for the treatment group and 89.1% for controls.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Attrition rates are similar for study 2.

“Attrition rates are similar for study 2.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The number of randomized students who showed up at study 2 schools comprised 36% of the 5,068 total 9th and 10th grade students enrolled in study 2 schools.

“The number of randomized students who showed up at these study schools comprised 36% of the 5,068 total 9 th and 10 th grade students enrolled in study 2 schools.”

Classified as probably not a substantial claim.

All analyses control for randomization-block fixed effects.

“All analyses control for randomization-block fixed effects.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For study 2, 2,645 students were randomized in 15 schools, 12 of which were also in study 1.

“For study 2, we randomized a sample of 2,645 students in 15 schools (12 of which were also in study 1).”

May combine two claims.

The implication is that the sample has scores below the CPS average, in contrast to many studies of no-excuses charters where applicants tend to have slightly better baseline scores than the host school system.

“The implication is our sample has scores below the CPS average, in contrast to many studies of 'no excuses' charters where applicants tend to have slightly better baseline scores than the host school system (e.g., Angrist et al., 2013).”

• cites Angrist et al., 2013Not checked yet

  Not checked in the free run.

The quote only partly states this claim.

Fryer (2014) found more pronounced effects on test scores among 9th grade students, on the order of 0.32 SD.

“but did appear to have more pronounced effects on test scores among 9 th grade students, on the order of 0.32 SD”

• cites Fryer, 2014Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

In Fryer (2014), high schools could not be randomly assigned.

“although high schools could not be randomly assigned.”

• cites Fryer, 2014Not checked yet

  Not checked in the free run.

May combine two claims.

The estimated effects on math outcomes are robust to a range of estimation decisions; the non-math GPA result, but not classes failed in non-math subjects, is somewhat more sensitive to these choices.

“The estimated effects on math outcomes we measure are robust to a range of estimation decisions; the non-math GPA result (but not classes failed in non-math subjects) is somewhat more sensitive to these choices.”

• AssumptionThe authors state this as an assumption: The multiple imputation methods assume outcomes are missing at random, i.e. values are unrelated to missingness conditional on observed covariates.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The first wave of surveys selected a sub-sample of 881 youth and surveyed 663, with effective response rates of 88.2% for treatment students and 90.6% for control students.

“We selected a sub-sample of 881 youth and surveyed 663 for effective response rates of 88.2% for treatment students and 90.6% for control students.”

May combine two claims.

Randomization appears to have been successful; baseline characteristics are jointly the same across treatment and control groups in study 1 (p=0.832) and study 2 (p=0.893).

“Randomization appears to have been successful.We carry out an F-test of the null hypothesis that baseline characteristics are jointly the same across treatment and control groups by regressing a treatment-group indicator against all variables in Table 2, separately for studies 1 and 2, controlling for randomization blocks.The p-value for study 1 is p=0.832, and for study 2 is p=0.893.”

May combine two claims.

CPS already receives around $300 million in Title 1 grants each year, and existing research suggests for the average district how these funds are deployed leads to few detectable improvements in student learning.

“CPS already receives around $300 million in Title 1 grants each year, and existing research suggests for the average district how these funds are deployed leads to few detectable improvements in student learning (Deke, et al., 2012, Dragoset, et al., 2017).”

• cites Deke, et al., 2012Not checked yet

  Not checked in the free run.

• cites Dragoset, et al., 2017Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The pattern that below-median-at-baseline students benefit more on easier items and above-median students may benefit more on harder items would be consistent with the extra personalization built into the Saga program design.

“The fact that below-median-at-baseline students seem to benefit more on relatively easier items from Saga tutoring and above-median students may benefit more on harder items would be consistent with the extra personalization built into the Saga program design, where tutors are instructed to devote some time each session to working with students where they are academically as determined by frequent”

May combine two claims.

With per-student costs of $3,500 to $4,300, this implies the benefit cost ratio is 2.1 to 2.6 in study 1 and 4.9 to 6.0 in study 2.

“With per-student costs of $3,500 to $4,300, this implies the benefit cost ratio is 2.1 to 2.6 in study 1 and 4.9 to 6.0 in study 2.”

May combine two claims.

The figure is at the very least consistent with, if not definitive proof for, floor effects with math tests.

“The figure is at the very least consistent with, if not definitive proof for, floor effects with math tests.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

As a thought exercise, that funding, if repurposed to Saga tutoring, would be enough to serve around 80,000 students per year, or 23% of the system's total enrollment of 355,000.

“As a thought exercise, that funding, if repurposed to Saga tutoring, would be enough to serve around 80,000 students per year, or 23% of the system's total enrollment of 355,000.”

Classified as probably not a substantial claim.

To account for individuals having multiple observations, standard errors are clustered on individuals.

“To account for individuals having multiple observations, standard errors are clustered on individuals.”

May combine two claims.

Assuming these gains persist from ages 25 to 59 and discounting at a 5% rate to age 15, the present discounted value of the gains is about $9,000 for study 1 and $21,000 for study 2.

“Assuming these gains persist from ages 25 to 59 and discounting at a 5% rate to age 15, the present discounted value of the gains is about $9,000 for study 1 and $21,000 for study 2.”

• AssumptionThe authors state this as an assumption: The gains from Saga tutoring persist from ages 25 to 59.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

This stands in contrast with how tutoring services are often provided to school systems like Chicago's, where a tutoring provider hires former teachers at prevailing full-time teacher wages, which for a given level of spending per pupil means students get just a fraction of the number of instructional hours as they do with the intervention studied here.

“This stands in contrast with how tutoring services are often provided to school systems like Chicago's, where a tutoring provider hires (say) former teachers at prevailing full-time teacher wages, which then for a given level of spending per pupil means students get just a fraction of the number of instructional hours as they do with the intervention we study here.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Figure 4 provides suggestive visual evidence that reduced 'academic mismatch' could play a role.

“Figure 4 provides suggestive visual evidence that reduced 'academic mismatch' could play a role.”

Classified as probably not a substantial claim. May combine two claims.

By way of comparison the benefit-cost ratios that have been estimated for model early childhood programs are on the order of 1.9 and 2.2 for the Abecedarian Project, and between 3.9 and 6.8 for the Perry Preschool Program, both estimated using a 5% discount rate.

“By way of comparison the benefit-cost ratios that have been estimated for model early childhood programs are on the order of 1.9 and 2.2 for the Abecedarian Project (Masse and Barnett, 2002), and between 3.9 and 6.8 for the Perry Preschool Program (Heckman et al., 2010), both estimated using a 5% discount rate.”

• cites Masse and Barnett, 2002Not checked yet

  Not checked in the free run.

• cites Heckman et al., 2010Not checked yet

  Not checked in the free run.

May combine two claims.

These studies have long been cited as arguments for investments in early childhood compared to later life stages; evidently, however, adolescence is not too late to also realize large social benefits from human capital investment.

“These studies have long been cited as arguments for investments in early childhood compared to later life stages; evidently, however, adolescence is not too late to also realize large social benefits from human capital investment.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

We report the smallest FDR q-value at which we could reject the null for each outcome using the method from Benjamini and Hochberg (1995); the results are qualitatively similar if we use the method from Benjamini, Krieger and Yekutieli (2006).

“We report the smallest FDR q-value at which we could reject the null for each outcome using the method from Benjamini and Hochberg (1995); the results are qualitatively similar if we use the method from Benjamini, Krieger and Yekutieli (2006).”

• cites Benjamini and Hochberg (1995)Not checked yet

  Not checked in the free run.

• cites Benjamini, Krieger and Yekutieli (2006)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Getting every student up to speed improves the degree to which regular classrooms are 'personalized.'

“That is, getting every student up to speed improves the degree to which regular classrooms are 'personalized.'”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some students (N=65) were randomized into Study 2 twice, and both assignments are retained in the models.

“Some students (N=65) were randomized into Study 2 twice.Both assignments are retained in the models above.”

May combine two claims.

The benefit-cost ratio of Saga tutoring seems to be comparable in magnitude to that of the most successful early childhood programs, like the Abecedarian Project and the Perry Preschool Program, as well as interventions at scale to improve student outcomes such as the Tennessee STAR experiment.

“The benefit-cost ratio of Saga tutoring seems to be comparable in magnitude to that of the most successful early childhood programs, like the Abecedarian Project and the Perry Preschool Program, as well as interventions at scale to improve student outcomes such as the Tennessee STAR experiment.”

May combine two claims.

Together, these two findings suggest Saga increases participants' adult earnings by $900 each year for study 1 and $2,100 each year for study 2.

“Together, these two findings suggest Saga increases participants' adult earnings by $900 each year for study 1 and $2,100 each year for study 2.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

For non-math core courses, selecting the appropriate 'grade-level' course in each is less straightforward, and there is also possibility that including Saga in students' schedules affects which other 'core' courses they can take.

“For non-math core courses (that is, courses in science, social studies, and reading/English), selecting the appropriate 'grade-level' course in each is less straightforward, and there is also possibility that including Saga in students' schedules affects which other 'core' courses they can take.”

May combine two claims.

Taken together, the results suggest that the subject matter focus on math was important to generating the treatment effects on learning, and that tutoring seems unlikely to have boosted learning solely through a mentoring effect.

“Taken together, these results seem to suggest that the subject matter focus on math was important to generating the treatment effects on learning, and that tutoring seems unlikely to have boosted learning solely through a mentoring effect.”

May combine two claims.

It is not too late or too difficult to substantially change the academic outcomes of children who are struggling academically even once they have reached adolescence.

“But there is another, perhaps more important way to read this study: As a demonstration that it is not too late or too difficult to substantially change the academic outcomes of children who are struggling academically even once they have reached adolescence.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The personalization mechanism cannot be documented directly because the study lacks direct measures of the academic level at which tutors devoted their time and of classroom teaching.

“Unfortunately, we cannot document this mechanism directly because we do not have direct measures of the academic level at which tutors devoted their time, nor do we have that for classroom teaching.”

May combine two claims.

It is possible to improve academic skills by accounting for the challenges of individualizing instruction, and these strategies can be effective even when implemented in regular public high schools to broad, representative samples of students.

“The lesson may be that it is possible to improve academic skills by accounting for the challenges of individualizing instruction-among other things-and that these strategies can be effective even when implemented in regular public high schools to broad, representative samples of students.”

Classified as probably not a substantial claim.

The models include baseline covariates: randomization block, treatment assignment, gender, age, learning disability, free and reduced lunch recipient, race, baseline grade level, GPA, days absent from school, disciplinary incidents, including suspensions, and arrests.

“Baseline covariates: Randomization block, treatment assignment, gender, age, learning disability, free and reduced lunch recipient, race, baseline grade level, GPA, days absent from school, disciplinary incidents, including suspensions, and arrests.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Because randomization block fixed-effects fully explain school fixed-effects, the main effect for the 'above median per capita disciplinary incidents' dummy variable cannot be estimated and is not reported, but the interaction effect can be recovered and reported.

“Given that randomization block fixed-effects fully explain school fixed-effects, we cannot estimate the main effect for the 'above median per capita disciplinary incidents' dummy variable, so we do not report these estimates in the table.However, we can recover and report the interaction effect and see if impacts differ between these groups.”

• AssumptionThe authors state this as an assumption: Randomization block fixed-effects fully explain school fixed-effects.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The control condition for study 2 participants was overwhelmingly an elective course, while the control condition for study 1 9th-graders (the majority of the study 1 sample) was a second hour or 'double-dose' Algebra.

“It is important to note that while the control condition for study 2 participants was overwhelmingly an elective course, the control condition for study 1 9th -graders-i.e., the majority of the study 1 sample-was a second hour or”

Probably not a substantial claim: may not pass S3 (load-bearing).

The smallest FDR q-value at which the null could be rejected for each outcome is reported using the method from Benjamini and Hochberg (1995).

“We report the smallest FDR q-value at which we could reject the null for each outcome using the method from Benjamini and Hochberg (1995)”

• cites Benjamini and Hochberg (1995)Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Baseline covariates and randomization block fixed effects are included in all models.

“Baseline covariates and randomization block fixed effects included in all models (see text).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In study 1, 15 students randomized to the control group received one or more session of Saga; in study 2, 88 students randomized to control (15 female and 73 male) participated in one or more session of Saga programming.

“In study 1, 15 students randomized to the control group received one or more session of Saga.In study 2, 88 students randomized to control (15 female students and 73 male students) participated in one or more session of Saga programming.”