The DOI matches its Crossref record; 62 of the 74 references it cites were found in a registry or an index; 2 of the 20 sources its claims cite were checked for support.
Not Too Late: Improving Academic Outcomes among Adolescents
Abstract
Improving academic outcomes for economically disadvantaged students has proven challenging, particularly for children at older ages. We present two large-scale randomized controlled trials of a high-dosage tutoring program delivered to secondary school students in Chicago. One innovation is to use paraprofessional tutors to hold down cost, thereby increasing scalability. Participating in math tutoring increases math test scores by 0.18 to 0.40 standard deviations, and increases math and nonmath course grades. These effects persist into future years. The data are consistent with increased personalization of instruction as a mechanism. The benefit-cost ratio is comparable to many successful early childhood programs. (JEL H75, I21, I24, I26, I32, J13, J15)
Overview
CoChat Summary
TL;DR
Two large randomized trials of Saga Education’s in-school, high-dosage math tutoring for Chicago public high school students found substantial gains in math test scores and grades, with effects persisting into 11th grade. The authors argue that intensive tutoring can improve adolescent academic outcomes at a cost of $3,500–$4,300 per participant per year, with benefit-cost ratios comparable to model early-childhood programs.
Key findings
- In study 1, the treatment-on-the-treated (TOT) effect after one year was +0.16 SD on standardized math tests, +0.56 points on math GPA (control complier mean 1.63), and a 49% decline in math course failures. In study 2, TOT effects were +0.37 SD on math tests, +0.42 points on math GPA (CCM 1.79), and a 44% decline in math course failures. Pooled year-1 TOT effects were +0.26 SD on math tests, +0.52 math GPA points (CCM 1.67), and a 47% decline in math course failures.
- Pooled 11th-grade outcomes showed TOT effects of +0.22 SD on math tests and +0.18 math GPA points. The TOT estimate for on-time high school graduation was +1.9 percentage points relative to a CCM of 77.2%, but imprecise: the standard error of 3.4 percentage points means declines as large as −4.8 points or gains as large as +8.6 points cannot be ruled out.
- Mechanism tests found no detectable effects on adult supports, grit, conscientiousness, locus of control, or social networks, and no evidence that classroom management was a key driver. There were gains in liking math and caring about math grades, and suggestive evidence consistent with personalization/floor effects.
- Estimated benefit-cost ratios were 2.1–2.6 in study 1 and 4.9–6.0 in study 2, with per-participant costs of $3,500–$4,300 per year.
Methods
- Two large-scale randomized controlled trials in Chicago Public Schools. Study 1 randomized 2,633 rising 9th/10th grade male students in summer 2013 across 12 CPS high schools; study 2 randomized 2,710 students (2,645 unique) in summer 2014 across 15 schools, including 9th/10th grade girls in seven schools. Randomization was conditional on school-by-grade blocks.
- The intervention was Saga Education’s in-school tutoring: one class period daily, two students per tutor, up to 140 contact hours per year, using Saga’s Algebra curriculum. Study 1 provided up to two years of intervention; study 2 ran for one year. Data included CPS administrative records, Saga attendance and internal assessments, Chicago Police arrest data, and two waves of in-person ISR surveys with researcher-administered math tests.
- Analysis estimated intent-to-treat (ITT) and treatment-on-the-treated (TOT) effects using randomization as an instrument for participation, with randomization-block fixed effects, baseline controls, permutation tests, false discovery rate q-values, and robustness checks using multiple imputation and quantile regression.
Limitations
- The paper states that graduation estimates are imprecise: the TOT estimate is +1.9 percentage points with a standard error of 3.4 percentage points, so effects ranging from −4.8 to +8.6 points cannot be ruled out.
- The paper states that floor effects on CPS math tests complicate interpretation of heterogeneity and personalization; item-level tests are underpowered (p=.53), and the authors cannot directly document the instructional-level mechanism because they lack direct measures of the academic level at which tutors or classroom teachers target instruction.
- Post-randomization data have missingness (e.g., study 1 attendance 4.4% for controls vs. 6.2% for treatment; grades 14.6% vs. 14.8%; test scores 29.8% vs. 29.5%), though treatment-control differences are not statistically significant. The provided text is an NBER working paper and states it has not been peer-reviewed.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



![Figure 4 Treatment effects on individual ISR items against item-level difficulty for students above/below the median baseline standardized math test score. Above/Below median is calculated for all students taking the ISR math asessment. The lines shown are local polynomial fitted lines to see any underlying nonlinearities in the relationships between item-level impacts and difficulty. We also include [0.5, 0.95] confidence intervals on the fitted lines. This plot truncates the 3 easiest items (P-plus > 0.75) to better display the graph; the full version is included in the appendices.](https://figures.cochat.ai/paper-figures/94ffa389f3da499b93a73321321ecf7a/not-too-late-improving-academic-outcomes-among-adolescents-figure-4.png)



![Figure 4 Treatment effects on individual ISR items against item-level difficulty for students above/below the median baseline standardized math test score. Above/Below median is calculated for all students taking the ISR math asessment. The lines shown are local polynomial fitted lines to see any underlying nonlinearities in the relationships between item-level impacts and difficulty. We also include [0.5, 0.95] confidence intervals on the fitted lines.](https://figures.cochat.ai/paper-figures/2b8dd114da1746388470a12ba69849f7/not-too-late-improving-academic-outcomes-among-adolescents-figure-4.png)
Verification
Claims the paper makes
2 Valid · 3 Not valid · 15 Not checked · 215 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“We do not know how many tutors it would be possible to hire before effectiveness declines, but over this range there is little difference between tutors at the top vs. bottom of Saga's ranked hiring list (Davis et al., 2017).”
What the sources say
- Supportedcites Davis et al., 2017 source
Section 4.3 (p. 24) states that even nonparametrically 'we see little evidence for a downward sloping relationship between tutor rank-order and tutor value-added, at least over the range of program operations that we can test with these data,' and that the estimated slope is 'essentially flat' (95% CI -0.004 to 0.003). The same page adds that 'tutor quality winds up not varying much over the scale of tutors studied here,' and p. 25 notes the provider's predicted quality is 'mostly all noise' within the hired set. The citing sentence's hedge about not knowing how many tutors could be hired before effectiveness declines matches the paper's own restriction to the tested range. (Note: the paper does document large heterogeneity in value-added across tutors overall, but not aligned with rank.) (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“percentage of courses failed in non-math classes are cut by 23% (-0.056/0.22).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This gives us another way to see that a sizable share of students are estimated to have PTE's that are close to zero or even negative with respect to math test scores.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In-person observations we conducted of the Saga program suggest students were spending approximately 90% of their time on math.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that the difference in impacts on academic outcomes are not statistically significant across highdisruption and low-disruption schools.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
69 more claims
Create a free account to see all 235 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 70
- It may state two separate claims as one. 80
- The quote only partly states this claim. 62
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Table 18: (Appendix) Year 1 Effect Estimates - Female Students Only
“Non-math GPA refers to all non math courses, not just core subjects.”
§ Non-Math Core Courses
“Non-math GPA refers to all non math courses in core subject areas (English, Science, Social Science).”
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2009) · doi:10.1515/9781400829828 · Cited by 9,467
School readiness and later achievement.
Duncan, G. J., Dowsett, C. J., Claessens, A., Magnuson, K., Huston, A. C., Klebanov, P., Pagani, L. S., Feinstein, L., Engel, M., Brooks-Gunn, J., et al. (2007) · Developmental Psychology · doi:10.1037/0012-1649.43.6.1428 · Cited by 5,439
Teachers, Schools, and Academic Achievement
Rivkin, S. G., Hanushek, E. A., & Kain, J. F. (2005) · Econometrica · doi:10.1111/j.1468-0262.2005.00584.x · Cited by 4,501
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230
The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring
BLOOM, B. S. (1984) · Educational Researcher · doi:10.3102/0013189x013006004 · Cited by 2,734
The Role of Cognitive Skills in Economic Development
Hanushek, E. A., & Woessmann, L. (2008) · Journal of Economic Literature · doi:10.1257/jel.46.3.607 · Cited by 2,435
Research works printed but not linked: 5
- Gordon, Robert, Thomas J. Kane, and Douglas O. Staiger. 2006. Identifying Effective Teachers Using Performance on the Job. Washington, DC: Brookings Institution Press.
- Heckman, James J. 2013. Giving Kids a Fair Chance. Cambridge, MA: MIT Press.
- Kraft Matthew A, 2021, AERA Open, 7, 1
- Krueger, Alan B. 2003. "Inequality, Too Much of a Good Thing." In Inequality in America: What Role for Human Capital Policies? Edited by James J. Heckman and Alan B. Krueger, 1-76. Cambridge, MA: MIT Press.
- Pedregosa Fabian, 2011, Journal of Machine Learning Research, 12, 2825
Cited by
Measuring the Impacts of Teachers II: Teacher Value-Added and Student Outcomes in Adulthood
Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014) · American Economic Review · doi:10.1257/aer.104.9.2633 · Cited by 1,717
Experimental evidence on learning using low-tech when school is out
Angrist, N., Bergman, P., & Matsheng, M. (2022) · Nature Human Behaviour · doi:10.1038/s41562-022-01381-z · Cited by 105
Zooming to Class? Experimental Evidence on College Students’ Online Learning during COVID-19
Kofoed, M. S., Gebhart, L., Gilmore, D., & Moschitto, R. (2024) · American Economic Review: Insights · doi:10.1257/aeri.20230077 · Cited by 125
Nickow, A., Oreopoulos, P., & Quan, V. (2024) · American Educational Research Journal · doi:10.3102/00028312231208687 · Cited by 82
Online tutoring works: Experimental evidence from a program with vulnerable children
Gortazar, L., Hupkau, C., & Roldán-Monés, A. (2024) · Journal of Public Economics · doi:10.1016/j.jpubeco.2024.105082 · Cited by 76
Metadata
- Authors
- Guryan, J., Ludwig, J., Bhatt, M. P., Cook, P. J., Davis, J. M. V., Dodge, K., Farkas, G., Fryer, R. G., Mayer, S., Pollack, H., Steinberg, L., & Stoddard, G.
- Affiliations
- Northwestern University (US)
- University of Chicago (US)
- Duke University (US)
- University of Oregon (US)
- University of California, Irvine (US)
- Harvard University (US)
- Temple University (US)
- ORCID
- Venue
- American Economic Review
- Publisher
- American Economic Association
- Published
- 2023-03-01
- Volume
- 113
- Issue
- 3
- Pages
- 738-765
- ISSN
- 0002-8282 (ISSN-L)
- OpenAlex
- W3135190335
- Cited by
- 77
- References
- 74
- Citations by year
- 2026: 18 · 2025: 24 · 2024: 17 · 2023: 15
- FWCI
- 56.32
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Economic Review (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Education › School Choice and Performance
- Social Sciences › Education › Global Educational Reforms and Inequalities
- Social Sciences › Education › Early Childhood Education and Development
- Keywords
- academic tutoring, low-income students, adolescents, high school students, mathematics scores, course grades, scalability, benefit-cost ratio, late intervention, Chicago
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Not Too Late: Improving Academic Outcomes among Adolescents. https://app.cochat.ai/papers/10.1257/aer.20210434
- The paper
- Guryan, J., Ludwig, J., Bhatt, M. P., Cook, P. J., Davis, J. M. V., Dodge, K., Farkas, G., Fryer, R. G., Mayer, S., Pollack, H., Steinberg, L., & Stoddard, G. (2023). Not Too Late: Improving Academic Outcomes among Adolescents. American Economic Review. https://doi.org/10.1257/aer.20210434
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.