Cited by 125Education during COVID-19 pandemicCOVID-19 Pandemic ImpactsCOVID-19 and Mental Health

The Impact of COVID‐19‐Related School Closures on Student Achievement—A Meta‐Analysis

Open access
König, Christoph, Frey, Andreas·2022·Educational Measurement: Issues and Practice

Abstract

COVID‐19 led to school closures and the necessity to use remote learning in 2020 and 2021 around the globe. This article provides results for a three‐level random‐effects meta‐analysis examining the average effect of the COVID‐19‐related school closures with respect to several moderator variables. The results showed a robust average effect of , . The moderator analysis was largely insignificant; however, the results tentatively point out that younger students in schools were more negatively affected compared to older students, and that the negative effect reduced with subsequent lockdowns in autumn and winter 2020/2021. The results are discussed with respect to potential explanations.

AI snapshot

Written from the paper’s full text.

TL;DR

This meta-analysis finds a small but statistically significant negative average effect of COVID-19-related school closures on student achievement. Moderator results are largely tentative: younger/primary students and earlier spring 2020 closures tended to show larger negative effects, while online-learning-app studies sometimes showed positive effects.

Key findings

  • The three-level random-effects model estimated an average effect of \(d = -0.175\) (\(SE = 0.063\), \(p = 0.013\), 95% CI \([-0.308, -0.041]\)).
  • The three-level model fit better than a two-level model with between-study heterogeneity fixed to zero (\(\chi^2_1 = 22.36\), \(p < 0.001\)); variance components were \(\tau_{L2} = 0.345\) and \(\tau_{L3} = 0.194\), with ICC = 0.241 and \(I^2_{L2} = 75.91\%\), \(I^2_{L3} = 24.06\%\).
  • Moderator analyses were largely insignificant, but the authors report tendencies that school-based learning was more negatively affected than learning apps, primary students were more affected than secondary students, and effects were more negative in spring 2020 than in later lockdowns.
  • Based on a typical spring 2020 closure duration of seven to eight weeks, the authors report a weekly learning loss of \(-0.022\), within the range of summer learning losses (\(d = -0.005\) to \(-0.050\) SD per week).

Methods

  • Systematic three-stage literature search: reviews/overviews, Web of Science for April 30, 2021–September 15, 2021, and preprint servers PsyArXiv, EdArXiv, and SocArXiv, yielding 425 potentially relevant studies.
  • Inclusion criteria required a focus on COVID-19-related school closures, primary/secondary education, student achievement as dependent variable, pre- and post-closure data, appropriate statistical analyses, and reported or calculable effect sizes; 18 studies with 109 effect sizes were included.
  • Main analysis was a three-level random-effects meta-analysis using the R package metafor with restricted maximum likelihood estimation, inverse-variance weighting, and cluster-robust covariance matrices; moderators were mode (school vs. learning app), school type (primary vs. secondary), and timing (spring 2020 vs. later lockdowns).

Limitations

  • The paper states that the cluster sample of 18 studies is small, below the usual minimum of 20 clusters for accurate variance components and standard errors, and that statistical power for moderator tests was low.
  • The paper states that moderator results were mostly insignificant and should be interpreted tentatively; publication-bias asymmetry tests were significant (\(\tau = -0.149\), \(p = 0.023\); \(z = -2.203\), \(p = 0.028\)), though the fail-safe \(N = 299\) suggested publication bias was unlikely.
  • The risk-of-bias assessment found the highest risk due to lack of inclusion of potential confounding variables, and the paper notes that no information could be obtained about deviations from intended interventions except for Clark et al. (2020).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1111/emip.12495

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

5 of 10 cited sources checked · 5 supported · 0 flagged

  • Findingcitation

    Positive effects of COVID-19-related school closures on student achievement were almost exclusively reported by studies using data from learning apps or online learning environments.

    “Positive effects of the COVID-19-related school closures on student achievement were almost exclusively reported by studies using data from learning apps or online learning environments (Meeter, 2021;Spitzer & Musslick, 2021;van der Velde et al., 2021).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Meeter, 2021 source
    • cites Spitzer & Musslick, 2021 source
    • cites van der Velde et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Interpretationcitation

    The results confirm the core conclusion of prior synthesizing research examining the first phase of school closures.

    “The results confirm the core conclusion of prior synthesizing research examining the first phase school closures.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Hammerstein et al., 2021 source
  • Study design and conductcitation

    Cluster-robust covariance matrices were used to account for dependencies between effect sizes stemming from the same study and the small number of studies.

    “To account for dependencies between effect sizes stemming from the same study and the small number of studies, we report estimates and standard errors based on cluster-robust covariance matrices (Pustejovsky & Tipton, 2018).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Pustejovsky & Tipton, 2018 source
  • Findingown data

    The average effect of COVID-19-related school closures on student achievement is d = -0.175 (SE = 0.063, p = 0.013, 95% CI [-0.308, -0.041]).

    “The results showed a robust average effect of d = -0.175(SE= 0.063, p = 0.013, 95%CI[-0.308,-0.041]).”
    • Also stated in: § Results · boosted
      “It yielded a robust estimate of the average effect of COVID-19-related school closures on student achievement of -0.175(SE = 0.063, p = 0.013, 95%CI[-0.308,-0.041]).”
    Quote in the paperSubstantial claim
  • Findingown data

    The three-level random-effects model provided a significantly better fit than a two-level model with between-study heterogeneity fixed to zero (χ2(1) = 22.36, p < 0.001).

    “We found that the three-level model provided a significantly better fit compared to the twolevel model (χ 2 1 = 22.36, p < 0.001).”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 4
  • May combine two claims. 5
  • The quote only partly states this claim. 2
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The risk of bias of the 18 included studies was assessed using the ROBINS-I tool.

    “The Cochrane Risk Assessment of the 18 studies was conducted using the “Risk Of Bias in Non-Randomized Studies of Interventions” tool (ROBINS-I; Sterne et al. 2016).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Sterne et al. 2016 source
  • May combine two claims.

    Findingown data

    The three-level random-effects model estimated variance components τL2 = 0.345 and τL3 = 0.194, with an intraclass correlation of ICC = 0.241, and I2L2 = 75.91% of total variation attributable to within-cluster and I2L3 = 24.06% to between-cluster heterogeneity.

    “The estimated variance components were τ L2 = 0.345 and τ L3 = 0.194, with an intraclass correlation of ICC = 0.241.”
    • Also stated in: § Results · plain
      “Thus, I 2 L2 = 75.91% of the total variation is attributable to within-cluster, and I 2 L3 = 24.06% to between-cluster heterogeneity.”
    Quote in the paperSubstantial claim
  • May combine two claims. The quote only partly states this claim.

    Findingown data

    Compared to learning apps, school learning tended to be negatively impacted by COVID-19-related school closures, and the disruption of learning was more severe in schools than in learning apps or online learning environments.

    “the results point into the direction that, compared to learning apps, school learning tended to be negatively impacted by the COVID-19-related school closures.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The suddenness of the switch from in-person to remote learning in spring 2020 may have made it impossible to adapt the necessary scaffolding measures adequately to a remote context, which may explain why younger students were affected more by the COVID-19-related school closures than older students in spring 2020.

    “The suddenness of the switch from in-person to remote learning in spring 2020 may have made it impossible to adapt the necessary scaffolding measures adequately to a remote context, which may explain why younger students were affected more by the COVID-19-related school closures than older students in spring 2020.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim.

    Premisecitation

    Younger students have fewer capabilities for self-regulated learning than older students do.

    “younger students have fewer capabilities for self-regulated learning than older students do”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Tomasik et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The negative effect of COVID-19-related school closures was primarily due to the immature measures of remote learning in spring 2020.

    “the metaregression tentatively points out the possibility that the negative effect was primarily due to the immature measures of remote learning in spring 2020.”
    Quote in the paperNot a substantial claim
  • May combine two claims. The quote only partly states this claim.

    Interpretationreasoning

    The results of the meta-regression provide a tentative indication for familiarization effects in the school context, such that later lockdowns were less severe because experiences from the first lockdown could be used to design and implement more effective remote learning measures.

    “The results of the meta-regression provide another tentative indication for familiarization effects in the school context.”
    • Also stated in: § Educational Measurement: Issues and Practice · hedged
      “This possibility is corroborated by the effect of the COVID-19-related school closures on student achievement being more severe in case of the early lockdown, compared to the lockdowns implement later.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Study design and conductown data

    Publication bias was assessed by testing funnel plot asymmetry with the rank-correlation and Egger's regression test, and the fail-safe N was N = 299, indicating that publication bias was unlikely.

    “the fail-safe N was N = 299, indicating the presence of publication bias being unlikely.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites van Aert and Niemeyer (2021)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

8 of 8
  • The paper relies on this without support.

    The typical duration of COVID-19-related school closures in spring 2020 was seven to eight weeks.

    “The typical duration of the COVID-19-related school closures in spring 2020 was seven to eight weeks.”
1 of 1

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • With an international standard deviation of 100, this is an average effect of 0.15 SD, which aligns remarkably well with results from several meta-analyses.

    “With an international standard deviation of 100, this would be an average effect of 0.15 SD which aligns remarkably well with results from several meta-analyses (e.g., Storey & Zhang, 2021;König & Frey, 2022;Betthäuser et al., 2023;Di Pietro, 2023).”

    From School closure policies and student reading achievement: evidence across countries (2023) · cites this paper as König & Frey, 2022

  • The findings from the RVE analysis are broadly consistent with those from previous meta-analyses (Storey and Zhang 2021a: 0.15 SD; König and Frey 2022: 0.175 SD; Betthäuser et al. 2023: 0.14 SD).

    “our findings from the RVE analysis are broadly consistent with those from previous meta-analyses. Storey and Zhang (2021a) concluded that due to Covid-19 students lost, on average, 0.15 standard deviations of learning, König and Frey (2022) found average losses of 0.175 standard deviations, and Betthäuser et al. (2023) estimated average losses at 0.14 standard deviations.”

    From The impact of Covid-19 on student achievement: Evidence from a recent meta-analysis (2023) · cites this paper as König and Frey (2022)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Reported variance components do not match the heterogeneity proportions

    Minor

    If 0.345 and 0.194 are variance components as stated, they imply approximately 64.0% within-study and 36.0% between-study heterogeneity, not 75.91% and 24.06%. The reported proportions instead approximately match the *squares* of those values, suggesting the values are standard deviations rather than variances.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 8

  1. Cook T.D., 1979, Quasi‐experimentation: Design and analysis issues
  2. Domingue B. W., 2021, Changing patterns of growth in oral reading fluency during the COVID‐19 pandemic—Policy brief
  3. De Witte K., 2021, Financing Education in the Context of COVID‐19
  4. Kogan V., 2021, The COVID‐19 pandemic and student achievement on Ohio's third‐grade English language arts assessment
  5. Kogan V., 2021, How the COVID‐19 pandemic affected student learning in Ohio: Analysis of Spring 2021 Ohio state tests
  6. Kuhfeld M., 2020, Technical appendix for: Learning during COVID‐19: Initial findings on students’ reading and math achievement and growth
  7. Rose S., 2021, Impact of school closures and subsequent support strategies on attainment and socio‐emotional wellbeing in key stage 1: Interim Paper 1
  8. Woessmann L., 2020, Folgekosten ausbleibenden Lernens: Was wir über die Corona‐bedingten Schulschließungen aus der Forschung lernen können [Follow‐up costs of an absence of learning: What research can teach us about corona‐related school closures], Ifo Schnelldienst, 73, 38

Cited by

Sorted by citations

Authors
König, C., & Frey, A.
Affiliations
Goethe University Frankfurt (DE)
Corresponding author
König, Christoph
Venue
Educational Measurement: Issues and Practice
Publisher
Wiley
Published
2022-02-03
Volume
41
Issue
1
Pages
16-22
ISSN
  • 0731-1745 (ISSN-L)
  • 1745-3992
OpenAlex
W4210705702
Cited by
125
References
27
Citations by year
2026: 15 · 2025: 24 · 2024: 23 · 2023: 41 · 2022: 17 · 2021: 1 · 2020: 1
FWCI
58.36
Type
journal article
Language
en
Open access
Locations
  • Educational Measurement Issues and Practice (publishedVersion, cc-by)
  • Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) (submittedVersion, cc-by)
Topics
  • Social Sciences › Education › Education during COVID-19 pandemic
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › COVID-19 Pandemic Impacts
  • Social Sciences › Psychology › Clinical Psychology › COVID-19 and Mental Health
Keywords
COVID-19 pandemic, school closures, academic performance, distance education, lockdown, grade level effects, education during pandemic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review