The (Surprising) Efficacy of Academic and Behavioral Intervention with Disadvantaged Youth: Results from a Randomized Experiment in Chicago

May combine two claims.

The study sample consists of 106 male 9th and 10th graders in a public high school on the south side of Chicago, of whom 95% are black and 99% are free or reduced price lunch eligible.

“The study sample consists of 106 male 9th and 10th graders in a public high school on the south side of Chicago, of whom 95% are black and 99% are free or reduced price lunch eligible.”

• Also stated in: § 1 I. INTRODUCTION · plain

  “106 male youth enrolled in 9 th and 10 th grade in academic year 2012-13 in a public high school on the south side of Chicago.Of the youth in our study sample, 99% were eligible for free or reduced price lunch and 95% are black”

May combine two claims.

Participation increased math test scores by 0.65 of the control group standard deviation and 0.48 SD in the national distribution.

“Participation increased math test scores by 0.65 of a control group standard deviation(SD) and 0.48 SD in the national distribution”

• Also stated in: § 1 I. INTRODUCTION · plain

  “program participation (the effects of treatment on the treated, or TOT) increased math achievement test scores by 0.65 of the control group's standard deviation (equal to 0.48SD within the national distribution)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All but one youth in the study sample (99%) are eligible for free or reduced price lunch, and 94% are eligible for free lunch specifically.

“All but one of the youth in our study sample (99%) are eligible for free or reduced price lunch (fully 94% are eligible for free-lunch specifically).”

May combine two claims.

The study sample is entirely male and almost entirely African-American.

“Table 1 shows that our study sample is entirely male, and almost entirely African-American.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Over one-quarter of the study sample has an individualized education plan (IEP), with most of these diagnosing some sort of learning disability.

“Over one-quarter of our study sample has an individualized education plan (IEP), with most of these diagnosing some sort of learning disability.”

May combine two claims.

Control-group students missing end-of-year 2013 post-randomization test scores have lower prior-year grades and achievement test scores than those with valid test results, and also have higher rates of absences, suspensions, and IEP designations.

“Comparing columns 1 and 2 of Table 3 we see that students assigned to the control group who are missing end-of-year 2013 (post-randomization) test scores have lower prior-year grades and achievement test scores than those who have valid test results, and also have higher rates of absences, suspensions, and IEP designations.”

May combine two claims.

During fall 2012, before intervention services began, the average youth scored at the 26th percentile of the national distribution on the EXPLORE/PLAN reading test and at the 22nd percentile in math.

“During the fall of 2012, before we began providing intervention services, the average youth scored at the 26 th percentile of the national distribution on their EXPLORE / PLAN reading test and at the 22 nd percentile in math.”

May combine two claims. The quote only partly states this claim.

Participation had sizable but sometimes not quite statistically significant effects in reducing F's in math and non-math classes by two-thirds.

“and F's in math and non-math classes by two-thirds.”

May combine two claims.

During the previous year (AY2011-12), male youth in the study sample missed an average of 19.5 days of school, experienced 3.2 out-of-school suspensions, failed 1.7 classes, and had an overall GPA of 2.15 on a four-point scale.

“During the previous year (AY2011-12), male youth in our study sample missed an average of 19.5 days of school, experienced 3.2 outof-school suspensions, failed 1.7 classes, and had an overall GPA of 2.15 on a four-point scale.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study sample of 106 male youth represents about one-third of all males in 9th and 10th grade in the study high school.

“Given that the study high school's total enrollment is slightly less than 1,000, our study sample of N=106 male youth represents about one-third of all males in 9 th and 10 th grade in the school.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For days absent and out-of-school suspensions, a total of eight observations out of 106 youth are missing.

“For both of these variables we are missing a total of eight observations out of our sample of 106 youth.”

May combine two claims.

Of those assigned to BAM plus Match, 36/48 (75%) ever attended at least one BAM session, while 19/48 (40%) participated in tutoring.

“(Of those assigned to BAM+Match, 36/48, or 75%, ever attended at least one BAM session while 19/48 or 40% participated in tutoring).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The CCM for absences is 45, or an average of nine weeks of school missed over the 2012-13 academic year.

“The CCM for absences is equal to 45, or an average of nine weeks of school missed over the 2012-13 academic year.”

May combine two claims. The quote only partly states this claim.

Participation increases rates of being on track for graduation, and hence expected high school graduation rates, by nearly one-half.

“increases rates of being”

May combine two claims.

The effect on out-of-school suspensions is equal to one half of the CCM but is not statistically significant.

“The effect on out of school suspensions is equal to one half of the CCM but, given our sizable standard error, is not statistically significant.”

May combine two claims.

When using math achievement test scores as the outcome, not controlling for baseline covariates at all tends to reduce t-statistics somewhat by making point estimates about one-quarter smaller than in the preferred model and standard errors about one-quarter larger.

“The first panel shows that when we use math achievement test scores as our outcome of interest, not controlling for baseline covariates at all, as shown in the last column, tends to reduce the t-statistics somewhat by making the point estimates about one-quarter smaller than in our preferred model specification and makes the standard errors about one-quarter larger.”

Probably not a substantial claim: may not pass S3 (load-bearing).

If tutors were in the room at the same time and had excess capacity, they would work with the disruptive youth.

“If the tutors were there in the room at the same time and a tutor had excess capacity, he or she would work with that youth.”

May combine two claims.

The TOT effect is 12.9 fewer days missed, or just over one-quarter.

“The effect of participating (TOT) is equal to 12.9 fewer days missed, or just over one-quarter.”

May combine two claims.

The absence effect is statistically significant with the pairwise t-test but not quite significant when adjusting for multiple comparisons; FWER p=0.109 and FDR q=0.141.

“This is statistically significant with just the pairwise t-test but not quite significant when we adjust for multiple comparisons; the FWER controlled p-value = 0.109, while the FDR qvalue = 0.141.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Many youth in the study sample were very far behind.

“Many of the youth in our study sample were very far behind.”

May combine two claims.

The intervention does not have statistically significant effects on reading scores, and the estimate is also not very precise; the 95% confidence interval includes the point estimates for the effects from other interventions.

“While our intervention does not have statistically significant effects on reading scores, our estimate is also not very precise.The 95% confidence interval for our estimate (Figure 7) includes the point estimates for the effects from these other interventions.”

May combine two claims.

The treatment-on-the-treated effect in the national distribution using math scale scores equals 0.48 SD, or about 60 percent of the black-white test score gap in the national NAEP test for 13 year olds.

“The TOT effect in the national distribution (using the math scale scores) equals 0.48 SD, or about 60 percent of the black-white test score gap in the national NAEP test for 13 year olds (see Figure 3).”

May combine two claims.

The average control complier youth gets about a D in math class while the average treatment-group complier scores about a C, and this result is statistically significant at the 5% threshold under PCER, FWER, and FDR.

“Put differently, the average control complier youth gets about a D in their math class while the average treatmentgroup complier scored at about a C (this result is statistically significant at the usual 5% threshold regardless of whether we focus on the PCER, the FWER, or the FDR).”

May combine two claims.

The likelihood of failing math during the intervention year is reduced by about two-thirds (TOT effect of 0.42), which is large as a share of the control complier mean (0.68) but just statistically significant at p<.10 even with the pairwise test.

“The next row shows that the likelihood of failing math during our intervention year is reduced by about two-thirds (TOT effect of 0.42) which is quite large as a share of the CCM (0.68) but given the sizable standard error just statistically significant at the p<.10 level even with just the pairwise test.”

May combine two claims.

The TOT effect increases the on-track likelihood by 24 percentage points, or 46% of the CCM.

“Program participation (the TOT effect) increases that likelihood by fully 24 percentage points, or 46% of the CCM.”

May combine two claims.

Youth assigned to BAM plus Match tutoring did much better in school, particularly in math class.

“Youth assigned to BAM+Match tutoring did much better in school, particularly in math class, as we show below.”

May combine two claims.

While there is some indication that youth getting into the most trouble during the pre-program year benefited more from BAM+Match with respect to many outcome measures such as math grades and non-math grades, this is not true for math test scores, and it is also not true when using measures of getting into trouble during the program year itself.

“while there is some indication that youth who were getting into the most trouble during the pre-program year (AY2011-12) benefited more from BAM+Match with respect to many of our outcome measures (e.g.math grades, non-math grades), this is not true for math test scores, and it is also not true when we use measures of getting into trouble during the program year itself (AY2012-13).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Unlike structured Match-style tutoring, which always focused on math, this informal tutoring occurred on a range of subject areas.

“(Unlike with the structured Match-style tutoring, which always focused on math, this sort of informal tutoring occurred on a range of subject areas.)”

May combine two claims.

Participating in programming increases the ranking in the national distribution by nearly 15 percentile points, to about the 34th percentile.

“Participating in programming (the TOT effect) increases the ranking in the national distribution by nearly 15 percentile points, to about the 34 th percentile.”

May combine two claims.

The intention-to-treat effect on spring math test scores standardized using the control group distribution is 0.51 SD; the per-comparison error rate using regular OLS standard errors is p=0.013, and the permutation-test p-value is 0.016.

“The ITT measured this way is 0.51 SD; focusing just on this outcome variable (not adjusting for multiple comparisons) the per comparison error rate (PCER) using the regular OLS standard errors equals p=0.013, and equals p=0.016 when we instead use a permutation test with 100,000 replications.”

May combine two claims.

Youth who are more likely to get into trouble in school have higher math test scores than those who do not get into trouble; the interaction between disciplinary problems and treatment assignment is positive.

“The first panel shows that youth who are more likely to get into trouble in school have higher math test scores than those youth who do not get into trouble (that is, the interaction between our measures of disciplinary problems and treatment assignment are positive).”

May combine two claims.

The participation rate was 71% for the 24 youth assigned to BAM-only and 75% for the 48 youth assigned to BAM plus Match tutoring.

“The participation rate for the 24 youth assigned to the BAM-only treatment arm was 71%, and for the 48 youth assigned to the BAM+Match tutoring treatment arm equaled 75%.”

May combine two claims.

Compared to main results, point estimates for math test results are slightly smaller when using multiple imputation, standard errors are larger, but the point estimate still suggests a sizable change in math scores.

“Compared to our main results, the point estimates for our math test results are slightly smaller when we use multiple imputation to fill in missing values to the spring achievement test scores, and the standard errors are larger, but the point estimate itself still suggests a sizable change in math scores.”

May combine two claims.

Assigning arbitrarily low scores to missing values and using quantile regression implies an effect on the median score of 0.39 SD using the control group distribution, compared to an effect on the median of 0.42 when using just non-missing observations.

“Assigning arbitrarily low scores to missing values and using quantile regression implies an effect on the median score of 0.39 SD (using the control group's distribution), compared to an effect on the median of 0.42 when we use just non-missing observations.”

May combine two claims.

The participation rate was 71% for the 24 youth assigned to the BAM-only treatment arm and 75% for the 48 youth assigned to the BAM+Match tutoring treatment arm.

“The participation rate for the 24 youth assigned to the BAM-only treatment arm was 71%, and for the 48 youth assigned to the BAM+Match tutoring treatment arm equaled 75%.”

May combine two claims.

The program increased math achievement test scores (Explore/Plan) in spring 2013, with an intent-to-treat effect of 0.510 control-distribution standard deviations (p=0.013) and a treatment-on-treated effect of 0.648 control-distribution standard deviations (p<0.01).

“| Z Score (Control Distribution) | 0 | 0.510* | 0.648** | -0.306 | 0.013 | 0.0163 | 0.03571 | 0.033 |”

May combine two claims.

The program increased math GPA in 2012-2013, with an intent-to-treat effect of 0.425 points on a 1-4 scale (p=0.021) and a treatment-on-treated effect of 0.583 points (p<0.05).

“| Math GPA 2012-2013 (1-4 point scale) | 1.346 | 0.425* | 0.583* | 1.239 | 0.021 | 0.0261 | 0.0371 | 0.033 |”

May combine two claims.

The program significantly reduced the number of non-math courses failed in 2012-2013, with an intent-to-treat effect of -1.454 (p<0.01) and a treatment-on-treated effect of -2.004 (p<0.01).

“| Non-Math Courses Failed 2012-2013 | 3.176 | -1.454** | -2.004** | 3.542 | 0.007 | 0.0094 | 0.019 | 0.022 |”

May combine two claims.

The program significantly reduced days absent in AY12-13, with an intent-to-treat effect of -10.272 days (p=0.041) and a treatment-on-treated effect of -12.919 days (p=0.041).

“| Days Absent AY12-13, N=98 | 44.031 | -10.272* | -12.919* | 45.14 | 0.041 | 0.047 | 0.109 | 0.141 |”

Probably not a substantial claim: may not pass S2 (checkable).

The set of people capable of being effective tutors is much greater than the set of people who could succeed in teaching a large classroom of students.

“the set of people who are capable of being effective tutors (in terms of either their abilities or prior training) is presumably much greater than the set of people who could succeed in teaching a large classroom of students.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The quality of the existing empirical evidence on the value of explicitly addressing non-academic factors or skills among disadvantaged youth is currently not ideal.

“The quality of the existing empirical evidence on the value of explicitly addressing non-academic factors or skills among disadvantaged youth is currently not ideal.”

May combine two claims.

Comparing the control group to the BAM-only group (N=24), one baseline difference is statistically significant at p<0.05 (disciplinary incidents in the previous year, 0.79 incidents over the year versus a control mean of 1.82) and one at p<0.10 (fall 2012 reading scores, about four percentile points).

“If we compare the control group to those assigned to the BAM-only group (N=24), we see one pair-wise difference that is statistically significant at the p<0.05 threshold (disciplinary incidents during the previous year, i.e.AY2011-12, of 0.79 incidents over the year -which is large relative to the control mean of 1.82) and one difference that is statistically significant at the p<0.10 level (fall”

May combine two claims.

The differential rate of missing end-of-year test scores does not seem to be due to treatment effects that reduce school dropout, but could be due to treatment effects that reduce student school absences.

“The results that we present below suggest the differential rate of missing-ness for end-of-year test scores does not seem to be due to treatment effects that reduce school dropout, but could be due to treatment effects that reduce student school absences.”

May combine two claims.

A previous RCT by the authors found that the BAM intervention reduced rates of violence involvement by 44% and increased schooling engagement, but did not have detectable effects on test scores when delivered on its own.

“A previous RCT by our team found the intervention reduced rates of violence involvement (by 44%) and increased schooling engagement, but did not have detectable effects on test scores when delivered on its own (Heller et al., 2013).”

• cites Heller et al., 2013Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Because tutoring requires different skills from classroom teaching, Match can expand its tutor recruitment pool to people with strong math skills who are willing to devote a year to public service but who do not necessarily have extensive prior teacher training or experience.

“This enables Match to expand the pool of people to recruit to be tutors and focus on people who are talented with strong math skills and willing to devote a year to public service, but who do not necessarily have extensive prior training or experience as teachers.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Because Fryer's findings come from a quasi-experimental study rather than a randomized experiment, they are inevitably subject to some uncertainty.

“Because Fryer's findings come from a quasi-experimental study rather than a randomized experiment, they are inevitably subject to some uncertainty.”

• cites Fryer's (2011)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

To ensure that standard errors are not misleadingly small because of the modest sample size, the study also reports p-values from a non-parametric permutation test.

“To ensure that the standard errors we calculate are not misleadingly small as an artifact of the modest number of youth in our study sample, we also report p-values that come from a non-parametric permutation test (Efron and Tibshirani, 1993).”

• cites Efron and Tibshirani, 1993Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

The instructional technology of tutoring is quite different from that of a classroom, so the set of skills and experiences required to be a successful instructor are different.

“One major innovation of the Match model, and a key reason we selected it, is the recognition that the instructional “technology” of tutoring is quite different from that of a classroom and so the set of skills and experiences required to be a successful instructor are different.”

Classified as probably not a substantial claim. May combine two claims.

Compared to regular classroom instruction, two-on-one tutoring greatly simplifies the instructional task, makes it easier to individualize instruction, develop positive relationships with students, and maximize time-on-task.

“Compared to regular classroom instruction, two-on-one tutoring greatly simplifies the instructional task that the adult is asked to carry out.Working with just two students makes it much easier for the instructor to individualize instruction (both in terms of the level and pace) to what students need.The tutoring method also makes it much easier to develop positive relationships with students, and”

Classified as probably not a substantial claim.

Youth were randomly assigned to Control (N=34), BAM only (N=24), or BAM plus Match-style high-dosage tutoring (N=48).

“These youth were randomly assigned to one of three conditions: (1) Control (N=34); (2) BAM only (N=24); (3) BAM plus Match-style high-dosage tutoring (N=48).”

May combine two claims.

Among Chicago youth at highest risk for school failure and crime (those arrested and sent to the Cook County Jail), on average they were two years behind grade level in reading, with some up to seven years behind, and four years behind grade level in math, with some up to 10 years below grade level.

“on average these youth were two years behind grade level in reading, with some up to seven years behind, and four years behind grade level in math, with some having math skills fully 10 years below grade level.”

• cites Keeley (2011)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The initial match rate to CPS administrative records for the study sample is 100% by construction.

“So our initial match rate to the CPS administrative records for our study sample is 100% by construction.”

May combine two claims.

A computer-assisted program that also helped individualize instruction was found to increase test scores by up to 0.47 SD after the second year of intervention, although impacts from either strategy were short-lived.

“The effects of a computer-assisted program that also helped individualize instruction was found to increase test scores by up to 0.47 SD after the second year of intervention, although impacts from either strategy were short-lived.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The control complier mean can differ from the overall control mean if the type of person who would participate in programming if assigned to treatment is systematically different from nonparticipants among those assigned to treatment.

“The CCM can differ from the overall control mean if the type of person who would participate in programming if assigned to treatment is systematically different from the non-participants among those assigned to treatment.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All study procedures were approved by the University of Chicago IRB.

“All of our study procedures were approved by the University of Chicago IRB.”

Classified as probably not a substantial claim.

The study is a randomized controlled trial.

“This paper reports on a randomized controlled trial of a two-pronged intervention”

• Also stated in: § 1 I. INTRODUCTION · plain

  “we report on the results of a randomized controlled trial (RCT)”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study is too small to be the definitive word on how to reduce disparities in academic outcomes within the U.S.

“While such a small study cannot be the definitive word on how to reduce disparities in academic outcomes within the U.S.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The research team carried out the random assignment for the project themselves.

“Our research team carried out the random assignment for the current project ourselves.”

May combine two claims.

The study randomized some youth to receive just the non-academic intervention, other youth to receive both the non-academic and academic interventions, and others to receive status quo services.

“Our research design, described more in section IV below, randomized some youth to receive just the non-academic intervention described here, other youth to receive both the non-academic intervention together with the academic intervention, and others to receive status quo services.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Family-wise error rate control is more conservative than false discovery rate control, while pairwise comparisons are not conservative enough.

“FWER control is more conservative for this purpose than FDR control, while pairwise comparisons are not conservative enough.”

• cites Benjamini and Hochberg, 1995Not checked yet

  Not checked in the free run.

May combine two claims.

Tutoring by its nature increases the amount of feedback and correction between student and instructor and ensures that all students receive this attention, including those struggling in school.

“Tutoring by its nature was found to increase the amount of feedback and correction between student and instructor, a key characteristic of effective teaching, and also ensures that all students receive this attention -including those students who are struggling in school.”

• cites Bloom (1984)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A large body of correlational research shows that schooling and other key life outcomes are correlated with non-cognitive skills such as self-regulation, social information processing, conflict resolution, grit, and future orientation.

“A large body of correlational research shows that schooling and other key life outcomes are correlated with what economists have come to call”

• cites Dodge et al. (1986)Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Previous research finds that spending more time in high school seems to increase non-academic factors related to decision-making, trust, and risky behavior.

“Previous research does find that spending more time in high school seems to increase non-academic factors related to decision-making, trust, and risky behavior (Oreopoulos and Salvanes, 2011).”

• cites Oreopoulos and Salvanes, 2011Not checked yet

  Not checked in the free run.

May combine two claims.

Compared to regular classroom instruction, tutoring generates large increases in time-on-task (90+% versus 65%) and improved student attitudes and interests.

“Compared to regular classroom instruction, tutoring also generates large increases in time-on-task (90+% versus 65%) and improved student attitudes and interests.”

• cites Bloom (1984)Not checked yet

  Not checked in the free run.

May combine two claims.

The results come from a pilot RCT at a single Chicago high school.

“The results reported here come from a pilot RCT at a single Chicago high school.”

May combine two claims.

Providing this intervention universally would not necessarily cut the black-white test score gap by 60%, since effects could be different for different populations and it is unknown how white youth would benefit.

“This does not mean that providing this intervention universally would cut the black-white test score gap by 60%, since the effects could be different for different populations and in particular we have no idea at present how white youth would benefit from the program if they were enrolled.”

Classified as probably not a substantial claim. May combine two claims.

Compared to students not on track, those on track have four-year graduation rates 59 percentage points higher and five-year graduation rates 57 percentage points higher.

“Compared to students who are not”

• cites Allensworth and Easton, 2005Not checked yet

  Not checked in the free run.

May combine two claims.

The Match tutoring intervention is delivered to over 700 students per year across three schools in Boston, to 500-600 students in several public high schools in Lawrence, Massachusetts, and was delivered to 3,000 students across 13 schools in Houston.

“The Match tutoring intervention is delivered to over 700 students per year across three schools in Boston, to 500-600 students in several public high schools in Lawrence, Massachusetts, and was delivered to 3,000 students across 13 schools in Houston (Fryer, 2011).”

• cites Fryer, 2011Not checked yet

  Not checked in the free run.

May combine two claims.

The small sample size and the possibility of crossover and spillover effects across the two treatment arms complicate the ability to disentangle the relative value of the two components.

“Our small sample size and the possibility of crossover and spillover effects across the two treatment arms complicates our ability to disentangle the relative value of the two components.”

May combine two claims.

The effect measured relative to the nationwide test-score distribution is equal to about 60% of the black-white test score gap in math in NAEP among 13 year olds, which equals 0.80 SD.

“the effect measured relative to the nationwide test-score distribution is equal to about 60% of the black-white test score gap in math in the National Assessment of Educational Progress (NAEP) among 13 year olds (which equals 0.80 SD).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Because the high school was chosen for its outstanding school leadership team, all else equal the results may overstate the effects that would be seen if the interventions were delivered in a broader set of schools with lower average quality of school leadership.

“We chose to work with this particular high school because of the outstanding school leadership team, which all else equal might make us think that our results may overstate the effects that we would see if the interventions were delivered in a broader set of schools where the average quality of school leadership is somewhat lower.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Interpretation of separate treatment arm results is complicated by possible treatment crossover or spillovers between the two groups.

“although interpretation of these results is complicated somewhat by the fact that there may be some treatment crossover or spillovers between the two groups as described above.”

Classified as probably not a substantial claim. May combine two claims.

The results come from a small-scale pilot test and so the findings are necessarily not the last word on this subject; many questions remain.

“Nonetheless our results come from a small-scale pilot test and so these findings are necessarily not the last word on this subject; many questions remain.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This pattern is what would be expected if those measures are a function of student behavior or demeanor as well as learning.

“this is what we would expect if those measures are a function of student behavior or demeanor as well as learning.”

May combine two claims.

There was some informal treatment crossover between the BAM-only and BAM plus Match tutoring groups that is not captured by administrative records on tutoring participation, complicating interpretation of the BAM-only treatment effects.

“It turns out that there was some informal treatment crossover between the BAM-only and BAM+Match tutoring groups that is not captured by our administrative records on tutoring participation, which complicates the interpretation of the effects of the BAM-only treatment that we present below.”

Classified as probably not a substantial claim.

The conventional wisdom summarized by Barrow, Claessens and Schanzenbach (2013) is that the finding of no test score improvement but a strong improvement in school attainment is consistent with a growing literature suggesting that interventions aimed at older children are more effective at improving non-cognitive skills than cognitive skills.

“The conventional wisdom around efforts to help disadvantaged youth is nicely summarized by Barrow, Claessens and Schanzenbach (2013):”

• cites Barrow, Claessens and Schanzenbach (2013)Not checked yet

  Not checked in the free run.

May combine two claims.

The BAM program has been delivered to at least 700 students per year since 2009 in Chicago, including in some of the most distressed schools in the city, and has been previously evaluated at large scale and found to have impacts on behavioral outcomes, although not test scores.

“The BAM program has been delivered to at least 700 students per year since 2009 in Chicago, including in some of the most distressed schools in the city, and has been previously evaluated at large scale and found to have impacts on behavioral outcomes, although not test scores (Heller et al., 2013).”

• cites Heller et al., 2013Not checked yet

  Not checked in the free run.

May combine two claims.

Outcomes were measured just during the program year; at present nothing is known about the degree to which these impacts will persist.

“And we have evaluated outcomes measured just during the program year; at present we know nothing about the degree to which these impacts will persist.”

Probably not a substantial claim: may not pass S2 (checkable).

The impacts of the pilot intervention reported in this paper are large enough to raise the question of whether the field has given up prematurely on the possibility of improving academic outcomes for disadvantaged youth.

“The impacts of the pilot intervention reported on in this paper are large enough to raise the question of whether the field has given up prematurely on the possibility of improving academic outcomes for disadvantaged youth.”

Probably not a substantial claim: may not pass S2 (checkable).

The key to making intensive remediation affordable, particularly high-dosage two-on-one individualized instruction, is the recognition that the tutoring method of instruction substantially changes the set of skills and experiences required to succeed as an instructor.

“The key to making this intensive remediation affordable, particularly the high-dosage two-on-one individualized instruction, is the recognition that the tutoring method of instruction substantially changes the set of skills and experiences required to succeed as an instructor.”

• AssumptionRests on an assumption: The tutoring method of instruction substantially changes the set of skills and experiences required to succeed as an instructor.

May combine two claims.

Perry Preschool improves high school GPA by 0.42 SD, math scores at age 14 by 0.33 SD, reading scores at 14 by 0.34 SD, and language achievement by 0.63 SD, at a cost of $20,500 per child.

“The widely-cited Perry Preschool improves high school GPA by 0.42 SD, or 0.3 points on a 4-point scale, improves overall math scores at age 14 by 0.33 SD, improves reading scores at 14 by 0.34 SD, and language achievement by 0.63 SD, at a cost of $20,500 (in 2013 dollars) per child (Schweinhart et al., 2005, p. xvii, 62).”

• cites Schweinhart et al., 2005Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The findings highlight a systemic challenge for urban school districts: the need for a more intensive safety net to help students who fall behind and experience a mismatch between what they need and what regular classrooms deliver.

“Our findings also highlight a systemic challenge for so many urban school districts -the need for a more intensive safety net to help students who fall behind as they progress through school and wind up experiencing a mismatch between what they need and what regular classrooms deliver.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Using a higher participation threshold assumes that youth who participate in fewer than N sessions do not benefit at all from participating in those sessions, which is a strong assumption because little is currently known about the functional form of the treatment dosage/treatment response relationship for these types of programs.

“But using a higher participation threshold boosts the TOT estimate by assuming that youth who participate in fewer than N sessions do not benefit at all from participating in those sessions, which seems like a strong assumption since so little is currently known about the functional form of the treatment dosage / treatment response relationship for these types of programs.”

• AssumptionRests on an assumption: Little is currently known about the functional form of the treatment dosage/treatment response relationship for these types of programs.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

As a conceptual matter it is not obvious that early childhood is the optimal time to intervene even if there is declining developmental plasticity over the life course; because impacts tend to fade out, there are advantages to intervening temporally closer to when important outcomes are realized.

“Put differently, as a conceptual matter it is not obvious that early childhood is the optimal time to intervene even if there is declining developmental plasticity over the life course; precisely because the impact of interventions tend to fade out over time, there are some advantages to intervening temporally closer to when important outcomes are realized.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Another tradeoff between intervening early versus later comes from the fact that it is easier to target interventions on highest-risk students during adolescence than early childhood, because there is more of a track record to use to identify participants who might benefit most.

“Another tradeoff between intervening early versus later comes from the fact that it is easier to target interventions on the highestrisk students during adolescence than early childhood, because we have more of a”

• AssumptionRests on an assumption: It is easier to target interventions on highest-risk students during adolescence than early childhood because there is more of a track record to use to identify participants who might benefit most.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

This mismatch is a problem that many previous interventions largely ignore, instead focusing on changing the quality of grade-level instruction in the classroom or the incentives of students to learn it.

“This mismatch is a problem that many previous interventions largely ignore, instead focusing on changing the quality of grade-level instruction in the classroom or the incentives of students to learn it.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

These comparisons highlight a tradeoff between intervening early in childhood versus later during adolescence, which has not received enough attention in the current policy discussion.

“These comparisons highlight a tradeoff between intervening early on in childhood versus later during adolescence, which has not received enough attention in the current policy discussion.”

Probably not a substantial claim: may not pass S2 (checkable).

The growing pessimism about academic interventions for low-income youth is premature, now that the key underlying problems may be being diagnosed.

“Perhaps the growing pessimism about academic interventions for low-income youth is premature, now that we may be diagnosing the key underlying problems.”

Probably not a substantial claim: may not pass S2 (checkable).

Given that most social-policy interventions show fade out of impacts, if the goal is to improve long-term life outcomes of children growing up in disadvantaged circumstances, there is a tradeoff between fade-out and developmental plasticity.

“Given that most social-policy interventions show”

• AssumptionRests on an assumption: Most social-policy interventions show fade out of impacts.

Probably not a substantial claim: may not pass S4 (based).

The large gains in academic outcomes for disadvantaged youth reported here stand against a backdrop of few prior success stories in improving academic outcomes, particularly achievement test scores, for similarly disadvantaged adolescents.

“The large gains in academic outcomes for disadvantaged youth reported here stand against a backdrop of few prior success stories in improving academic outcomes, particularly achievement test scores, for similarly disadvantaged adolescents.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Treatment-on-treated effects are essentially rescaled intent-to-treat effects, so the paper displays ITT p-values.

“As treatment on treated (TOT) effects are essentially rescaled intent to treat (ITT) effects, we display ITT p-values in the interest of space and readibility.”