CoChat

The DOI matches its Crossref record; 26 of the 28 references it cites were found in a registry or an index; none of the 18 sources its claims cite is checked for support yet.

A Multisite Randomized Study of an Online Learning Approach to High School Credit Recovery: Effects on Student Experiences and Proximal Outcomes

Open access
Rickles, Jordan, Clements, Margaret, Brodziak de los Reyes, Iliana, Lachowicz, Mark, Lin, Shuqiong, Heppen, Jessica · 2024 · Journal of Research on Educational Effectiveness

Abstract

Online credit recovery will likely expand in the coming years as school districts try to address increased course failure rates brought on by the coronavirus pandemic. Some researchers and policymakers, however, raise concerns over how much students learn in online courses, and there is limited evidence about the effectiveness of online credit recovery. This article presents findings from a multisite randomized study, conducted prior to the pandemic, to expand the field’s understanding of online credit recovery’s effectiveness. Within 24 high schools from a large urban district, the study randomly assigned 1,683 students who failed Algebra 1 or ninth grade English to a summer credit recovery class that either used an online curriculum with in-class teacher support or the school’s business-as-usual teacher-directed class. The results suggest that online credit recovery had relatively insignificant effects on student course experiences and content knowledge, but significantly lower credit recovery rates for English. There was limited heterogeneity in effects across students and schools. Non-response on the study-administered student survey and test limit our confidence in the student experience and content knowledge results, but the findings are robust to different approaches to handling the missing data (multiple imputation or listwise deletion). We discuss how the findings add to the evidence base about online credit recovery and the implications for future research.

Overview

CoChat Summary

TL;DR

In a multisite randomized trial of summer credit recovery in a large urban district, online courses with in-class teacher support produced no significant average differences in student experiences or content knowledge but significantly lower English 9 credit recovery rates. Algebra 1 results were mixed and imprecise, with limited heterogeneity and low online course progression.

Key findings

  • English 9 online credit recovery rates were significantly lower: 52% vs. 67% in teacher-directed classes, with a model-estimated effect of −23 percentage points (Cox effect size ≈ −0.58). Algebra 1 rates were 62% vs. 69%, with a non-significant model-estimated effect of −6 percentage points (effect size ≈ −0.15).
  • Content knowledge differences were not statistically significant: Algebra 1 online students scored 0.15 SD higher on the end-of-course test (0.21 SD, p = .03 under listwise deletion) and −0.03 SD on PSAT math; English 9 online students scored −0.02 SD on the end-of-course test and +0.04 SD on PSAT reading.
  • Student experiences showed no significant Algebra 1 differences across five measures; for English 9, personalized instruction was 0.25 SD higher in online classes (significant), while other experience differences were not significant.
  • Online progression was low: only 10% of Algebra 1 and 12% of English 9 online students spent at least the recommended 40 hours, averaging about 24 hours; only 28% and 34%, respectively, completed at least three-fourths of the course, averaging 51% of content. Heterogeneity was limited, though Algebra 1 end-of-course effects were positive for higher prior GPA students (0.34 SD) vs. lower GPA students (−0.06 SD), a differential significant after Benjamini-Hochberg adjustment.

Methods

  • Multisite randomized trial in Los Angeles high schools during the 2018 and 2019 summer terms. Students who failed Algebra 1 or English 9 in their first high school year were randomly assigned within blocks by subject, cohort, and school to online credit recovery (Edgenuity curriculum plus credentialed in-class teacher support) or business-as-usual teacher-directed credit recovery.
  • The study included 1,683 unique students; analytic samples were 613 Algebra 1 students (305 online, 308 teacher-directed) and 1,124 English 9 students (564 online, 560 teacher-directed). Measures included an end-of-course student survey with five experience constructs, study-developed end-of-course tests (20 multiple-choice items each), district final course grade/pass, PSAT math/reading, and district extant covariates.
  • Intent-to-treat effects were estimated separately by subject using linear regression for survey and test outcomes and logistic regression for passing, with covariates and randomization-block fixed effects. Missing data were handled with multiple imputation (M = 20); exploratory analyses tested site-by-treatment heterogeneity and student-level moderators.

Limitations

  • The paper states that non-response on the student survey and test limits confidence in student experience and content knowledge results; missing data were high and differential, with Algebra 1 showing high attrition even under the optimistic WWC boundary and English 9 high under the cautious boundary. Students missing outcomes had lower ninth-grade attendance and GPA, and the Algebra 1 end-of-course result was sensitive to missing-data handling (0.15 SD, p = .08 with multiple imputation vs. 0.21 SD, p = .03 with listwise deletion).
  • The paper states that online engagement and progression were suboptimal, most students did not spend the recommended time online and completed less than half the course, and it is unclear how results would generalize if students spent recommended time or how this engagement compares with typical teacher-directed classes. It also states the study tested a limited version of an online model with in-class support, with less teacher supplementation than intended, using a single provider (Edgenuity), so findings may not generalize to other providers or virtual courses with little/no instructional support.
  • Evident from the text: this article reports only on summer terms, not the smaller school-year sample; the study is set in one large urban district; student experiences are self-reported; and wide 90% confidence intervals mean many null results cannot establish substantive equivalence.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1080/19345747.2023.2198524

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
CC-BY-NC-ND · From the PDF
Image 2
CC-BY-NC-ND · From the PDF
Image 3
CC-BY-NC-ND · From the PDF
Image 4
CC-BY-NC-ND · From the PDF
Figure 1 Dimensions of the online learning model. Note . For each dimension, the highlighted aspect represents a key feature of the online learning model for this study.
Figure 1 Dimensions of the online learning model. Note . For each dimension, the highlighted aspect represents a key feature of the online learning model for this study.CC-BY-NC-ND · From the PDF
Figure 2 Theory of action.
Figure 2 Theory of action.CC-BY-NC-ND · From the PDF
Figure 3 Estimated differences in student instructional experiences between online and teacherdirected Algebra 1 classes. Note. The 95% confidence interval visualizes the null hypothesis test ( a ¼ 0.05), where intervals that intersect with zero indicate that the effect is not statistically significant. The 90% confidence interval visualizes the two one-sided tests of equivalence ( a ¼ 0.05), where intervals represent the range of group differences within which we cannot conclude the groups are statistically equivalent. Standard errors are reported in parentheses. N ¼ 305 students in the online classes and 308 students in the teacher-directed classes.
Figure 3 Estimated differences in student instructional experiences between online and teacherdirected Algebra 1 classes. Note. The 95% confidence interval visualizes the null hypothesis test ( a ¼ 0.05), where intervals that intersect with zero indicate that the effect is not statistically significant. The 90% confidence interval visualizes the two one-sided tests of equivalence ( a ¼ 0.05), where intervals represent the range of group differences within which we cannot conclude the groups are statistically equivalent. Standard errors are reported in parentheses. N ¼ 305 students in the online classes and 308 students in the teacher-directed classes.CC-BY-NC-ND · From the PDF
Image 8
CC-BY-NC-ND · From the PDF
Figure 4 Estimated differences in credit recovery rates between online and teacher-directed Algebra 1 classes. Note. The effect on the course pass rate was estimated with a logistic regression model. We display the effect in percentage points (bottom panel) relative to the observed pass rate in the teacher-directed classes. The 95% confidence interval visualizes the null hypothesis test ( a ¼ 0.05), where intervals that intersect with zero indicate that the effect is not statistically significant. The 90% confidence interval visualizes the two one-sided tests of equivalence ( a ¼ 0.05), where intervals represent the range of group differences within which we cannot conclude the groups are statistically equivalent. The standard error is reported in parentheses. N ¼ 305 students in the online classes and 308 students in the teacher-directed classes. Inc: incomplete or no final grade assigned.
Figure 4 Estimated differences in credit recovery rates between online and teacher-directed Algebra 1 classes. Note. The effect on the course pass rate was estimated with a logistic regression model. We display the effect in percentage points (bottom panel) relative to the observed pass rate in the teacher-directed classes. The 95% confidence interval visualizes the null hypothesis test ( a ¼ 0.05), where intervals that intersect with zero indicate that the effect is not statistically significant. The 90% confidence interval visualizes the two one-sided tests of equivalence ( a ¼ 0.05), where intervals represent the range of group differences within which we cannot conclude the groups are statistically equivalent. The standard error is reported in parentheses. N ¼ 305 students in the online classes and 308 students in the teacher-directed classes. Inc: incomplete or no final grade assigned.CC-BY-NC-ND · From the PDF
Figure 5 Estimated differences in student content knowledge between online and teacher-directed Algebra 1 classes. Note. The 95% confidence interval visualizes the null hypothesis test ( a ¼ 0.05), where intervals that intersect with zero indicate that the effect is not statistically significant. The 90% confidence interval visualizes the two one-sided tests of equivalence ( a ¼ 0.05), where intervals represent the range of group differences within which we cannot conclude the groups are statistically equivalent. Standard errors are reported in parentheses. N ¼ 305 students in the online classes and 308 students in the teacher-directed classes.
Figure 5 Estimated differences in student content knowledge between online and teacher-directed Algebra 1 classes. Note. The 95% confidence interval visualizes the null hypothesis test ( a ¼ 0.05), where intervals that intersect with zero indicate that the effect is not statistically significant. The 90% confidence interval visualizes the two one-sided tests of equivalence ( a ¼ 0.05), where intervals represent the range of group differences within which we cannot conclude the groups are statistically equivalent. Standard errors are reported in parentheses. N ¼ 305 students in the online classes and 308 students in the teacher-directed classes.CC-BY-NC-ND · From the PDF
Figure 6 Estimated differences in student instructional experiences between online and teacherdirected English 9 classes. Note. The 95% confidence interval visualizes the null hypothesis test ( a ¼ 0.05), where intervals that intersect with zero indicate that the effect is not statistically significant. The 90% confidence interval visualizes the two one-sided tests of equivalence ( a ¼ 0.05), where intervals represent the range of group differences within which we cannot conclude the groups are statistically equivalent. Standard errors are reported in parentheses. N ¼ 564 students in the online classes and 560 students in the teacher-directed classes.  Average difference between the online and teacher-directed classes is statistically significant ( p value < 0.05).
Figure 6 Estimated differences in student instructional experiences between online and teacherdirected English 9 classes. Note. The 95% confidence interval visualizes the null hypothesis test ( a ¼ 0.05), where intervals that intersect with zero indicate that the effect is not statistically significant. The 90% confidence interval visualizes the two one-sided tests of equivalence ( a ¼ 0.05), where intervals represent the range of group differences within which we cannot conclude the groups are statistically equivalent. Standard errors are reported in parentheses. N ¼ 564 students in the online classes and 560 students in the teacher-directed classes.  Average difference between the online and teacher-directed classes is statistically significant ( p value < 0.05).CC-BY-NC-ND · From the PDF
Image 12
CC-BY-NC-ND · From the PDF
Figure 7 Estimated differences in credit recovery rates between online and teacher-directed English 9 classes. Note. The effect on the course pass rate was estimated with a logistic regression model. We display the effect in percentage points (bottom-right panel) relative to the observed pass rate in the teacher-directed classes. The 95% confidence interval visualizes the null hypothesis test ( a ¼ 0.05), where intervals that intersect with zero indicate that the effect is not statistically significant. The 90% confidence interval visualizes the two one-sided tests of equivalence ( a ¼ 0.05), where intervals represent the range of group differences within which we cannot conclude the groups are statistically equivalent. Standard errors are reported in parentheses. N ¼ 564 students in the online classes and 560 students in the teacher-directed classes. Inc: incomplete or no final grade assigned.  Average difference between the online and teacher-directed classes is statistically significant ( p value < 0.05).
Figure 7 Estimated differences in credit recovery rates between online and teacher-directed English 9 classes. Note. The effect on the course pass rate was estimated with a logistic regression model. We display the effect in percentage points (bottom-right panel) relative to the observed pass rate in the teacher-directed classes. The 95% confidence interval visualizes the null hypothesis test ( a ¼ 0.05), where intervals that intersect with zero indicate that the effect is not statistically significant. The 90% confidence interval visualizes the two one-sided tests of equivalence ( a ¼ 0.05), where intervals represent the range of group differences within which we cannot conclude the groups are statistically equivalent. Standard errors are reported in parentheses. N ¼ 564 students in the online classes and 560 students in the teacher-directed classes. Inc: incomplete or no final grade assigned.  Average difference between the online and teacher-directed classes is statistically significant ( p value < 0.05).CC-BY-NC-ND · From the PDF
Figure 8 Estimated differences in student content knowledge between online and teacher-directed English 9 classes. Note. The 95% confidence interval visualizes the null hypothesis test ( a ¼ 0.05), where intervals that intersect with zero indicate that the effect is not statistically significant. The 90% confidence interval visualizes the two one-sided tests of equivalence ( a ¼ 0.05), where intervals represent the range of group differences within which we cannot conclude the groups are statistically equivalent. Standard errors are reported in parentheses. N ¼ 564 students in the online classes and 560 students in the teacher-directed classes.
Figure 8 Estimated differences in student content knowledge between online and teacher-directed English 9 classes. Note. The 95% confidence interval visualizes the null hypothesis test ( a ¼ 0.05), where intervals that intersect with zero indicate that the effect is not statistically significant. The 90% confidence interval visualizes the two one-sided tests of equivalence ( a ¼ 0.05), where intervals represent the range of group differences within which we cannot conclude the groups are statistically equivalent. Standard errors are reported in parentheses. N ¼ 564 students in the online classes and 560 students in the teacher-directed classes.CC-BY-NC-ND · From the PDF
Figure 9 Estimated average effect of the online credit recovery classes on student content knowledge, by academic foundation moderators. Note. Effect estimates are in standard deviation units. The horizontal bars represent the 95% confidence interval, where intervals that intersect with zero indicate that the estimated effect is not statistically significant from zero. For Algebra 1, the achievement moderator is based on the Grade 8 mathematics state test score and the test outcomes are the studydeveloped algebra test and PSAT math. For English 9, the achievement moderator is based on the Grade 8 English language arts state test score and the test outcomes are the study-developed English test and PSAT reading. EOC: end-of-course test.  Difference between the average effect in the higher vs. lower group is statistically significant ( p value < 0.05). auro 9
Figure 9 Estimated average effect of the online credit recovery classes on student content knowledge, by academic foundation moderators. Note. Effect estimates are in standard deviation units. The horizontal bars represent the 95% confidence interval, where intervals that intersect with zero indicate that the estimated effect is not statistically significant from zero. For Algebra 1, the achievement moderator is based on the Grade 8 mathematics state test score and the test outcomes are the studydeveloped algebra test and PSAT math. For English 9, the achievement moderator is based on the Grade 8 English language arts state test score and the test outcomes are the study-developed English test and PSAT reading. EOC: end-of-course test.  Difference between the average effect in the higher vs. lower group is statistically significant ( p value < 0.05). auro 9CC-BY-NC-ND · From the PDF
Image 16
CC-BY-NC-ND · From the PDF

Verification

Claims the paper makes

0 Valid · 2 Not valid · 15 Not checked · 82 Own results and interpretations

41 more claims

Create a free account to see all 99 claims and every source verdict.

See all claims — free account
  • Borderline: one of the tests for a substantial claim is unclear. 2
  • It may state two separate claims as one. 38
  • Probably not a substantial claim. 22

Create a free account to see these claims.

Citations

Cites

Show all 26 references

Research works printed but not linked: 9

  1. Atchison, D., Rickles, J., Clements, P., Brodziak de los Reyes, I., & Heppen, J. (2020). Online credit recovery: Resources and costs. American Institutes for Research. Retrieved from https://www.air.org/sites/default/files/Online-Credit-Recovery-Study-Brief-4-Resources-Oct-2020.pdf
  2. Atkins D. Brown J. S. & Hammond A. (2007). A review of the open educational resources (OER) movement: Achievement challenges and new opportunities. Report for the William and Flora Hewlett Foundation.
  3. Bakia, M., Mislevy, J., Heying, E., Patton, C., Singleton, C., & Krumm, A. (2013). Supporting K–12 students in online learning: A review of online Algebra I courses. SRI International.
  4. Blackboard K–12 (2009). Credit recovery: Exploring answers to a national priority. Blackboard.
  5. Gemin, B., Pape, L., Vashaw, L., & Watson, J. (2015). Keeping pace with K–12 digital learning: An annual review of policy and practice. Evergreen Education Group.
  6. Kehrer, P., Kelly, K., & Heffernan, N. (2013). Does immediate feedback while doing homework improve learning? In Proceedings of the Twenty-Sixth International FLAIRS Conference (pp. 542–545).
  7. Schaeffer, C. E., & Konetes, G. D. (2010). Impact of learner engagement on attrition rates and student success in online learning. International Journal of Instructional Technology & Distance Learning, 7(5), 3–9.
  8. Taylor, S., Clements, P., Heppen, J., Rickles, J., Sorensen, N., Walters, K., Allensworth, E., & Michelman, V. (2016). Getting back on track: The role of in-person instructional support for students taking online credit recovery. American Institutes for Research. https://www.air.org/system/files/downloads/report/In-Person-Support-Credit-Recovery.pdf
  9. What Works Clearinghouse (2022). What Works Clearinghouse procedures and standards handbook, version 5.0. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance (NCEE).

Cited by

See all 6

Metadata

Authors
Rickles, J., Clements, M., Brodziak de los Reyes, I., Lachowicz, M., Lin, S., & Heppen, J.
Affiliations
  • American Institutes for Research (US)
  • American Institutes for Research, Arlington, Virginia, USA
Corresponding author
Rickles, Jordan
Venue
Journal of Research on Educational Effectiveness
Published
2023-04-13
Volume
17
Issue
3
Pages
467-490
ISSN
  • 1934-5739 (ISSN-L)
  • 1934-5747
OpenAlex
W4365458852
Cited by
6
References
28
Citations by year
2026: 3 · 2025: 1 · 2023: 2
FWCI
4.45
Type
journal article
Language
en
Open access
Locations
Journal of Research on Educational Effectiveness (publishedVersion, cc-by-nc-nd)
Funders
U.S. Department of Education’s Institute of Education Sciences
Topics
  • Social Sciences › Education › Online and Blended Learning
  • Social Sciences › Education › Education during COVID-19 pandemic
  • Social Sciences › Education › School Choice and Performance
Keywords
online learning, Algebra I, ninth-grade English, course failure, content knowledge, teacher support, summer school, urban schools, COVID-19 pandemic, missing data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Multisite Randomized Study of an Online Learning Approach to High School Credit Recovery: Effects on Student Experiences and Proximal Outcomes. https://app.cochat.ai/papers/10.1080/19345747.2023.2198524
The paper
Rickles, J., Clements, M., Brodziak de los Reyes, I., Lachowicz, M., Lin, S., & Heppen, J. (2024). A Multisite Randomized Study of an Online Learning Approach to High School Credit Recovery: Effects on Student Experiences and Proximal Outcomes. Journal of Research on Educational Effectiveness. https://doi.org/10.1080/19345747.2023.2198524

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.