CoChat
Sign Up
Cited by 1,848Education during COVID-19 pandemicSchool Choice and PerformanceCOVID-19 Pandemic Impacts

Learning loss due to school closures during the COVID-19 pandemic

Open access
Engzell, Per, Frey, Arun, Verhagen, Mark D.·2021·Proceedings of the National Academy of Sciences
DOI 10.1073/pnas.2022376118PMC PMC8092566
Open this paper in CoChat — free account

Abstract

Significance: School closures have been a common tool in the battle against COVID-19. Yet, their costs and benefits remain insufficiently known. We use a natural experiment that occurred as national examinations in The Netherlands took place before and after lockdown to evaluate the impact of school closures on students’ learning. The Netherlands is interesting as a “best-case” scenario, with a short lockdown, equitable school funding, and world-leading rates of broadband access. Despite favorable conditions, we find that students made little or no progress while learning from home. Learning loss was most pronounced among students from disadvantaged homes.

AI snapshot

Written from the paper’s full text.

TL;DR

Using Dutch national assessments taken before and after an 8-week COVID-19 school closure, this study finds that primary students made little or no progress while learning from home. The Netherlands is a “best-case” setting, yet average learning loss was 3.16 percentile points (0.08 SD), with larger losses among students from disadvantaged homes.

Key findings

  • In the baseline difference-in-differences estimate for a composite of math, spelling, and reading, students lost on average 3.16 percentile points, equivalent to 0.08 SD; raw differences ranged from −0.76 percentiles in spelling to −2.15 percentiles in math.
  • Losses were concentrated among students from less-educated homes: the two lowest parental-education categories, together 8% of the population, suffered losses 40% larger than average (high, −3.07; low, −4.34; lowest, −4.25); within-family estimates showed a gap of 60% (high, −3.25; low, −4.67; lowest, −5.20).
  • There was little evidence that effects differed by sex, school grade, subject, or prior performance, but school-level variation was considerable, with some schools seeing losses of 10 percentile points or more and others no losses or small gains.
  • On a learning-readiness test not tied to curricular content, effects shrank by nearly two-thirds (main effect −1.19 vs. −3.16), suggesting most of the drop reflects knowledge learned; 3.16 percentiles translates to 7.9 to 10.5 weeks of lost learning depending on the benchmark.

Methods

  • Natural experiment/difference-in-differences design using biannual national standardized tests in The Netherlands from 2017 to 2020, covering 15% of Dutch primary schools (n ≈ 350,000) and students aged 8 to 11 years (grades 4 to 7); 2020 tests occurred just before and after 8 weeks of nationwide closures starting March 16.
  • Main outcomes were math, spelling, reading, and a composite score, plus an oral reading fluency/learning-readiness test; scores were transformed to percentiles, with baseline models adjusting for a linear year trend and days between tests and clustering standard errors at the school level.
  • Robustness included placebo analyses, year exclusions, loss-to-follow-up adjustments via regression, propensity-score and maximum-entropy weighting, school and family fixed effects, and a specification curve analysis of more than 2,000 alternative models.

Limitations

  • The paper states that only time will tell whether students rebound, remain stable, or fall farther behind; test scores do not capture psychosocial development or societal and parental costs; and results may underestimate the full costs of school closures.
  • The paper notes that The Netherlands is a best-case scenario due to short closures, high technological preparedness, and equitable school funding, making the estimates a likely lower bound; the short closure also gave little time to adapt, and remote learning might improve with time.
  • The text makes evident that the sample covers 15% of primary schools, loss to follow-up is addressed but only a subset of students took tests after lockdown, and the within-family fixed-effects specification reduces the sample by approximately 60% and shifts it toward larger families.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1073/pnas.2022376118

Cites

96 of 96 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

14 of 59 cited sources checked · 8 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    The World Bank assumes a yearly progress of 0.40 SD in its projections of learning loss due to the pandemic.

    “In their projections of learning loss due to the pandemic, the World Bank assumes a yearly progress of 0.40 SD (46).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [46] J. P. Azevedo, A. Hasan, D. Goldemberg, S. A. Iqbal, K. Geven (2020) Simulating the potential impacts of COVID-19 school closures on schooling and learning outcomes: A set of global estimates. World Bank Policy Research Working Paper (9284). https://openknowledge.worldbank.org/handle/10986/33945. Accessed 9 February 2021. source

      The Empirical methodology (p. 15) says the projections assume learning gains of 0.2–0.5 SD from one grade to the next, depending on country income. Annex A.1 (p. 44) specifies 40 PISA points, equivalent to 0.40 SD, as the baseline for upper-middle-income countries—not as a universal yearly rate. (LLM, conf=0.90)

      Experimental — accuracy not yet measured
  • Interpretationown data

    The results confirm the fears held by many that school closures would cause socioeconomic gaps to widen.

    “Our results show that learning loss was particularly pronounced for students from disadvantaged homes, confirming the fears held by many that school closures would cause socioeconomic gaps to widen (51 –55).”
    Quote in the paperSubstantial claim5 of 5 sources checked
    • cites [51] E. A. Hanushek, L. Woessmann, The Economic Impacts of Learning Losses (Organisation for Economic Co-operation and Development OECD, 2020). source
    • cites [52] F. Agostinelli, M. Doepke, G. Sorrenti, F. Zilibotti, When the Great Equalizer Shuts Down: Schools, Peers, and Parents in Pandemic Times (National Bureau of Economic Research, 2020). PMC8735857 35017763 source
    • cites [53] A. Bacher-Hicks, J. Goodman, C. Mulhern, Inequality in household adaptation to schooling shocks: Covid-induced online learning engagement in real time. J. Publ. Econ. 193, 104345 (2020). PMC8486492 34629567 source

      The Introduction reports that school closures widened socioeconomic gaps in searches for online learning resources. The Discussion says this pattern suggests academic gaps will be wider in future school years, but the paper does not establish that closures had already widened learning-loss gaps; its measured outcome is primarily search engagement. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [54] Z. Parolin, E. K. Lee, Large socio-economic, geographic and demographic disparities exist in exposure to school closures. Nat. Hum. Beh., 10.1038/s41562-021-01087-8. (2021). PMC8060162 33737734 source
    • cites [55] D. H. Bailey, G. J. Duncan, R. J. Murnane, N. A. Yeung (2021) Achievement gaps in the wake of COVID-19 (EdWorkingPaper No. 21-346, Annenberg Institute at Brown University). https://www.edworkingpapers.com/ai21-346. Accessed 9 February 2021. source
  • Interpretationown data

    The magnitude of the findings appears to validate scenarios projected by the European Commission and the World Bank.

    “Nevertheless, the magnitude of our findings appears to validate scenarios projected by bodies such as the European Commission (34) and the World Bank (46).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [34] G. Di Pietro, F. Biagi, P. Costa, Z. Karpinski, J. Mazza, The Likely Impact of COVID-19 on Education: Reflections Based on the Existing Literature and Recent International Datasets (Publications Office of the European Union, 2020).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [46] J. P. Azevedo, A. Hasan, D. Goldemberg, S. A. Iqbal, K. Geven (2020) Simulating the potential impacts of COVID-19 school closures on schooling and learning outcomes: A set of global estimates. World Bank Policy Research Working Paper (9284). https://openknowledge.worldbank.org/handle/10986/33945. Accessed 9 February 2021. source
  • Study design and conductcitation

    The propensity score and entropy weighting match treatment and control groups on a wider range of individual- and school-level characteristics.

    “Moreover, we match treatment and control groups on a wider range of individual- and school-level characteristics using reweighting on the propensity of treatment (68) and maximum-entropy balancing (69).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [68] G. W. Imbens, J. M. Wooldridge, Recent developments in the econometrics of program evaluation. J. Econ. Lit. 47, 5– 86 (2009). source
    • cites [69] J. Hainmueller, Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies. Polit. Anal., 20, 25– 46 (2012). source
  • Findingown data

    The largest gap in effect sizes between educational backgrounds is found in the within-family analysis, estimated at 60%.

    “The largest gap in effect sizes between educational backgrounds is found in our within-family analysis, estimated at 60% (parental education: high, −3.25; low, −4.67; lowest, −5.20).”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 19
  • May combine two claims. 30
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Dynamic models of learning stress how small losses can accumulate into large disadvantages with time.

    “Dynamic models of learning stress how small losses can accumulate into large disadvantages with time (56 –58).”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [56] T. A. DiPrete, G. M. Eirich, Cumulative advantage as a mechanism for inequality: A review of theoretical and empirical developments. Annu. Rev. Sociol. 32, 271– 297 (2006). source
    • cites [57] M. Kaffenberger, Modelling the long-run learning impact of the Covid-19 learning shock: Actions to (more than) mitigate loss. Int. J. Educ. Dev. 81, 102326 (2021). 33716394 PMC7923185 source
    • cites [58] N. Fuchs-Schündeln, D. Krueger, A. Ludwig, I. Popova, The Long-Term Distributional and Welfare Effects of COVID-19 School Closures (National Bureau of Economic Research, 2020). source

  • May combine two claims.

    Premisecitation

    Test scores do not consider children's psychosocial development, societal costs due to productivity decline, or heightened pressure among parents.

    “Test scores do not consider children’s psychosocial development (63, 64), either societal costs due to productivity decline or heightened pressure among parents (65, 66).”
    Quote in the paperNot a substantial claim4 of 4 sources checked
    • cites [63] A. Gassman-Pines, C. M. Gibson-Davis, E. O. Ananat, How economic downturns affect children’s development: An interdisciplinary perspective on pathways of influence. Child Dev. Perspect. 9, 233– 238 (2015). 31327980 PMC6641565 source

      The section on economic and psychological effects on continuously employed individuals reports increased financial and psychological stress within families and greater parental distress during downturns, supporting the point about pressure on parents. The paper also discusses children’s emotional well-being separately from test scores, but it does not say that test scores omit societal costs from declining productivity or explicitly assess what test scores capture. No page number was available in our copy. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [64] L. Shanahan, Emotional distress in young adults during the COVID-19 pandemic: Evidence of risk and resilience from a longitudinal cohort study. Psychol. Med., 10.1017/S003329172000241X (2020). PMC7338432 32571438 source

      The Methods (Measures; page unavailable) assess economic and lifestyle disruptions among young adults, and the Discussion (page unavailable) links those disruptions to emotional distress. Neither passage addresses what test scores omit, societal costs from declining productivity, or heightened pressure on parents. Searches for those specific points found no supporting passage in our copy. (LLM, conf=0.73)

      Experimental — accuracy not yet measured
    • cites [65] C. Collins, L. C. Landivar, L. Ruppanner, W. J. Scarborough, COVID-19 and the gender gap in work hours. Gender Work. Organ. 28, 101– 12. (2020). PMC7361447 32837019 source

      The Introduction describes acute time and financial pressures on working parents, and the Results (section 4.1; page unavailable) report that mothers reduced paid work hours amid childcare and homeschooling demands. These passages support heightened parental pressure and reduced work time, but the paper does not assess test scores or establish societal costs from a decline in productivity. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [66] P. Biroli (2020) “ Family life in lockdown” (IZA Discussion Paper 13398, Institute of Labor Economics, Bonn, Germany). source

      The Results section (Family Cooperation and Tensions; page unavailable) reports increased household tension associated with changes in childcare and other household tasks during lockdown, supporting the point about pressure on parents. The paper does not establish societal costs from declining productivity or state that test scores omit these effects. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    The control for pretreatment trends is the most influential model component on the magnitude of estimates, followed by the control for test timing and the inclusion of school and family fixed effects.

    “Doing so identifies the control for pretreatment trends as the most influential, followed by the control for test timing and the inclusion of school and family fixed effects.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The fixed-effects specification shifts the sample toward larger families, and effects in this subsample are similar using the baseline specification.

    “However, the fixed-effects specification shifts the sample toward larger families, and effects in this subsample are similar using our baseline specification ( SI Appendix, section 7.7).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Excluding adjustment for testing date decreases the effect size by 12%, while including fixed effects increases it by 1.6% (school level) or 6.3% (family level).

    “Excluding adjustment for testing date decreases the effect size by 12%, while including fixed effects increases it by 1.6% (school level) or 6.3% (family level).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Test scores improve by 0.30 to 0.40 percentiles per week, equivalent to 0.31 to 0.41 SD annually, in the comparison years.

    “We validate these benchmarks in our data by exploiting variation in testing dates during comparison years and show that test scores improve by 0.30 to 0.40 percentiles per week, equivalent to 0.31 to 0.41 SD annually ( SI Appendix, section 4.3).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Students lost on average 3.16 percentile points in the national distribution, equivalent to 0.08 standard deviations.

    “Students lost on average 3.16 percentile points in the national distribution, equivalent to 0.08 standard deviations (SD) ( SI Appendix, section 4.3).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The treatment sample is not skewed on sex, parental education, or prior performance.

    “In SI Appendix, Table S3, we show that the treatment sample is not skewed on sex, parental education, or prior performance.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    The raw difference between midyear and end-of-year test scores in 2020 ranged from −0.76 percentiles in spelling to −2.15 percentiles in math.

    “This graph reveals a raw difference ranging from −0.76 percentiles in spelling to −2.15 percentiles in math.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Learning losses are evident throughout the age range studied and across all three subject areas: math, spelling, and reading.

    “These losses are evident throughout the age range we study and across all of the three subject areas: math, spelling, and reading.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Reestimating the main specification dropping comparison years one at a time yields results estimated with less precision but otherwise in line with the main analysis.

    “We also reestimate our main specification dropping comparison years one at a time ( SI Appendix, section 7.3). These results are estimated with less precision but otherwise in line with those of our main analysis.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    The estimated treatment effect translates into 0.09 percentiles or 0.002 SD per school day, which is on the lower end of the summer-loss range.

    “Our estimated treatment effect translates into 3.16/35 = 0.09 percentiles or 0.002 SD per school day and is thus on the lower end of that range.*”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    Parental education groups make up 92%, 4%, and 4% of the student body and the sample.

    “These groups make up, respectively, 92, 4, and 4% of the student body and our sample ( SI Appendix, section 5.1).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites SI Appendix, section 5.1

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation

    The literature reports reductions in achievement ranging from 0.001 to 0.010 SD per school day lost during summer.

    “This literature reports reductions in achievement ranging from 0.001 to 0.010 SD per school day lost (10).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10] M. Kuhfeld, Projecting the potential impacts of COVID-19 school closures on academic achievement. Educ. Res. 49, 549– 565 (2020). source

  • May combine two claims.

    Interpretationreasoning

    Using the smaller benchmark, learning loss exceeds the period of school closures (3.16/0.30 = 10.5 wk), implying that students regressed during this time.

    “Using the smaller benchmark, learning loss exceeds the period of school closures (3.16/0.30 = 10.5 wk), implying that students regressed during this time.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Using the larger benchmark, a treatment effect of 3.16 percentiles translates into 7.9 wk of lost learning-nearly exactly the same period that schools in The Netherlands remained closed.

    “Using the larger benchmark, a treatment effect of 3.16 percentiles would translate into 3.16/0.40 = 7.9 wk of lost learning—nearly exactly the same period that schools in The Netherlands remained closed.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    Failure to adjust for pretreatment trends generates placebo estimates that are biased in a positive direction and is thus likely to underestimate treatment effects.

    “However, failure to adjust for pretreatment trends generates placebo estimates that are biased in a positive direction and is thus likely to underestimate treatment effects.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    Hypotheses and analysis protocols for this study were preregistered.

    “Hypotheses and analysis protocols for this study were preregistered ( SI Appendix, section 4.1).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The Netherlands presents a 'best-case' scenario for studying learning loss, providing a likely lower bound on learning loss elsewhere in Europe and the world.

    “The Netherlands is interesting as a “best-case” scenario, with a short lockdown, equitable school funding, and world-leading rates of broadband access.”
    • Also stated in: § Study Setting · hedged
      “For these reasons, The Netherlands presents a “best-case” scenario, providing a likely lower bound on learning loss elsewhere in Europe and the world.”
    Quote in the paperSubstantial claim0 of 6 sources checked
    • cites [30] H. A. Patrinos, “ School choice in the Netherlands” (CESifo DICE Report 2/2011, ifo Institute for Economic Research, Munich, Germany, 2011).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [31] H. F. Ladd, E. B. Fiske, Weighted student funding in The Netherlands: A model for the US? J. Pol. Anal. Manag. 30, 470– 498 (2011). source
    • cites [33] Statistics Netherlands (CBS), The Netherlands leads Europe in internet access. https://www.cbs.nl/en-gb/news/2018/05/the-Netherlands-leads-europe-in-internet-access. Accessed 9 February 2021.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [34] G. Di Pietro, F. Biagi, P. Costa, Z. Karpinski, J. Mazza, The Likely Impact of COVID-19 on Education: Reflections Based on the Existing Literature and Recent International Datasets (Publications Office of the European Union, 2020).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [36] Johns Hopkins Coronavirus Resource Center, COVID-19 case tracker. https://coronavirus.jhu.edu/data. Accessed 9 February 2021.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [37] P. Tullis, Dutch cooperation made an ‘intelligent lockdown’ a success. Bloomberg Businessweek (4 June 2020).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    If yearly progress is 0.80 SD, remote learning operated at 50% efficiency.

    “which would indicate that remote learning operated at 50% efficiency.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The baseline specification adjusts for a linear trend in year and the time elapsed between testing dates and clusters standard errors at the school level.

    “In our baseline specification, we adjust for a linear trend in year and the time elapsed between testing dates and cluster standard errors at the school level.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study restricts analysis to schools where at least 75% of students took their tests after lockdown.

    “Moreover, we restrict analysis to schools where at least 75% of students took their tests after lockdown ( SI Appendix, section 7.5).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Early influential studies found that summer is a time when socioeconomic learning gaps widen, but this finding has failed to replicate in more recent studies or in European samples.

    “Although early influential studies also found that summer is a time when socioeconomic learning gaps widen, this finding has failed to replicate in more recent studies (8, 9) or in European samples (48, 49).”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [8] P. T. von Hippel, C. Hamrock, Do test score gaps grow before, during, or between the school years?: Measurement artifacts and what we can know in spite of them. Soc. Sci. 6, 43 (2019). source
    • cites [9] M. Kuhfeld, Surprising new evidence on summer learning loss. Phi Delta Kappan 101, 25– 29 (2019). source
    • cites [48] P. Verachtert, J. Van Damme, P. Onghena, P. Ghesquière, A seasonal perspective on school effectiveness: Evidence from a Flemish longitudinal study in kindergarten and first grade. Sch. Effect. Sch. Improv. 20, 215– 233 (2009). source
    • cites [49] F. Meyer, K. Meissel, S. McNaughton, Patterns of literacy learning in German primary schools over the summer and the influence of home literacy practices. J. Res. Read. 40, 233– 253 (2017). source

  • May combine two claims.

    Study design and conductnone given

    The study uses externally validated tests, a representative sample, and techniques that allow for causal inference.

    “Our study represents one of the first attempts to quantify learning loss from COVID-19 using externally validated tests, a representative sample, and techniques that allow for causal inference.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    The Netherlands is close to the OECD average in school spending and reading performance, but among its top performers in math.

    “The country is close to the Organization for Economic Cooperation and Development (OECD) average in school spending and reading performance, but among its top performers in math (32).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [32] A. Schleicher, PISA 2018: Insights and Interpretations (OECD Publishing, 2018).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The short duration of school closures gave students, educators, and parents little time to adapt.

    “The short duration of school closures gave students, educators, and parents little time to adapt.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The study assessed more than 2,000 alternative models in a specification curve analysis.

    “To identify the model components that exert the most influence on the magnitude of estimates we assessed more than 2,000 alternative models in a specification curve analysis (44) ( SI Appendix, section 7.13).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [44] U. Simonsohn, J. P. Simmons, L. D. Nelson, Specification curve analysis. Nat. Hum. Behav. 4, 1208– 1214 (2020). 32719546 source

  • May combine two claims.

    Premisecitation

    Survey evidence from lockdown indicates high levels of dissatisfaction with remote learning and considerable disparities in help with schoolwork and learning resources.

    “Despite favorable conditions, survey evidence from lockdown indicates high levels of dissatisfaction with remote learning (38) and considerable disparities in help with schoolwork and learning resources (39).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [38] M. de Haas, R. Faber, M. Hamersma, How COVID-19 and the Dutch ‘intelligent lockdown’ changed activities, work and travel behaviour: Evidence from longitudinal data in the Netherlands. Transport. Res. Interdisciplinary Perspect. 6. 100150 (2020). PMC7284275 34171019 source
    • cites [39] T Bol, Inequality in homeschooling during the corona crisis in the Netherlands. First results from the LISS Panel. SocArXiv [Preprint] (2020). 10.31235/osf.io/hf32q (Accessed 9 February 2021). source

  • May combine two claims.

    Interpretationreasoning

    The high degree of school autonomy in The Netherlands likely created considerable variation in the pandemic response, possibly explaining the wide school-level variation in estimated learning loss.

    “The high degree of school autonomy in The Netherlands is also likely to have created considerable variation in the pandemic response, possibly explaining the wide school-level variation in estimated learning loss ( SI Appendix, section 7.9).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The difference-in-differences design discards students who did not take the tests, which might lead to bias if their performance trajectories differ from those observed.

    “Our difference-in-differences design discards those students who did not take the tests, which might lead to bias if their performance trajectories differ from those we observe.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Early estimates suggest that grades 3 to 8 students more than 6 mo into the pandemic underperformed by 7.5 percentile points in math but saw no loss in reading achievement.

    “Early estimates suggest that grades 3 to 8 students more than 6 mo into the pandemic underperformed by 7.5 percentile points in math but saw no loss in reading achievement (28).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28] M. Kuhfeld, B. Tarasawa, A. Johnson, E. Ruzek, K. Lewis, “ Learning during COVID-19: Initial findings on students’ reading and math achievement and growth” (NWEA Brief, Portland, OR, 2020).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The study addresses loss to follow-up using regression adjustment, rebalancing on propensity scores and maximum-entropy weights, and fixed-effects designs that compare students within the same schools and families.

    “We do so using a difference-in-differences design ( SI Appendix, section 4.2) and address loss to follow-up using various techniques: regression adjustment, rebalancing on propensity scores and maximum-entropy weights, and fixed-effects designs that compare students within the same schools and families.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    Following school reopenings, tests were taken in person under normal conditions and with minimal social distancing.

    “Following school reopenings, tests were taken in person under normal conditions and with minimal social distancing.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Studies of school days lost due to other causes are mixed-some find durable effects and spillovers to adult earnings, while others report a fadeout of effects over time.

    “Studies of school days lost due to other causes are mixed—some find durable effects and spillovers to adult earnings (59, 60), while others report a fadeout of effects over time (61, 62).”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [59] A. Ichino, R. Winter-Ebmer, The long-run educational cost of World War II. J. Labor Econ. 22, 57– 87 (2004). source
    • cites [60] D. Jaume, A. Willén, The long-run effects of teacher strikes: Evidence from Argentina. J. Labor Econ. 37, 1097– 1139 (2019). source
    • cites [61] S. Cattan, D. Kamhöfer, M. Karlsson, T. Nilsson “ The short- and long-term effects of student absence: Evidence from Sweden” (IZA Discussion Paper 10995, Institute of Labor Economics, Bonn, Germany, 2017). source
    • cites [62] B. Sacerdote, When the saints go marching out: Long-term outcomes for student evacuees from hurricanes Katrina and Rita. Am. Econ. J. Appl. Econ. 4, 109– 135 (2012). source

  • May combine two claims.

    Premisecitation

    School closures were short in comparative perspective, and the first wave of the pandemic had less of an impact than in other European countries.

    “School closures were short in comparative perspective ( SI Appendix, section 1), and the first wave of the pandemic had less of an impact than in other European countries (36, 37).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [36] Johns Hopkins Coronavirus Resource Center, COVID-19 case tracker. https://coronavirus.jhu.edu/data. Accessed 9 February 2021.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [37] P. Tullis, Dutch cooperation made an ‘intelligent lockdown’ a success. Bloomberg Businessweek (4 June 2020).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Social distancing measures may have altered factors such as seating arrangements or indoor climate that in turn can influence student performance.

    “Social distancing measures may have altered factors such as seating arrangements or indoor climate that in turn can influence student performance (41 –43).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [41] M. J. Mendell, G. A. Heath, Do indoor pollutants and thermal conditions in schools influence student performance?: A critical review of the literature. Indoor Air 15, 27– 52 (2005). 15660567 source
    • cites [42] P. D. Marshall, M. Losonczy-Marshall, Classroom ecology: Relations between seating location, performance, and attendance. Psychol. Rep. 107, 567– 577 (2010). 21117484 source
    • cites [43] R. J. Park, J. Goodman, A. P. Behrer, Learning is inhibited by heat exposure, both internationally and within the United States. Nat. Hum. Behav. 5, 19– 27 (2020). 33020588 source

  • May combine two claims.

    Premisecitation

    No other country has higher rates of broadband penetration, and efforts were made early in the pandemic to ensure access to home learning devices.

    “No other country has higher rates of broadband penetration (33, 34), and efforts were made early in the pandemic to ensure access to home learning devices (35).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [33] Statistics Netherlands (CBS), The Netherlands leads Europe in internet access. https://www.cbs.nl/en-gb/news/2018/05/the-Netherlands-leads-europe-in-internet-access. Accessed 9 February 2021.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [34] G. Di Pietro, F. Biagi, P. Costa, Z. Karpinski, J. Mazza, The Likely Impact of COVID-19 on Education: Reflections Based on the Existing Literature and Recent International Datasets (Publications Office of the European Union, 2020).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [35] F. M. Reimers, A. Schleicher, A Framework to Guide an Education Response to the COVID-19 Pandemic of 2020 (OECD Publishing, 2020).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    The Dutch school system combines centralized and equitable school funding with a high degree of autonomy in school management.

    “The Dutch school system combines centralized and equitable school funding with a high degree of autonomy in school management (30, 31).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [30] H. A. Patrinos, “ School choice in the Netherlands” (CESifo DICE Report 2/2011, ifo Institute for Economic Research, Munich, Germany, 2011).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [31] H. F. Ladd, E. B. Fiske, Weighted student funding in The Netherlands: A model for the US? J. Pol. Anal. Manag. 30, 470– 498 (2011). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The study balances treatment and control groups on a wider set of covariates, including at the school level, using maximum-entropy weights and the estimated propensity of treatment.

    “Next, we balance treatment and control groups on a wider set of covariates, including at the school level, using maximum-entropy weights and the estimated propensity of treatment ( SI Appendix, section 7.4).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The within-family design reduces the sample size by approximately 60%.

    “This step reduces the size of our sample by approximately 60%, as not every student has a sibling attending a sampled school within the years that we are able to observe.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The oral reading fluency test is interpreted as a test of learning readiness.

    “We interpret it as a test of learning readiness.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    The student monitoring system provides the test score data and comprises mandatory tests taken twice a year throughout primary school (ages 6 to 12).

    “The first one is the student monitoring system, which provides our test score data (40). This system comprises a series of mandatory tests that are taken twice a year throughout a child’s primary school education (ages 6 to 12 y).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [40] K. F. Vlug, Because every pupil counts: The success of the pupil monitoring system in The Netherlands. Educ. Inf. Technol. 2, 287– 306 (1997). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Throughout the analyses, confidence intervals are adjusted for clustering on schools using robust standard errors.

    “Throughout our analyses, we adjust confidence intervals for clustering on schools using robust standard errors.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Nationally standardized tests are taken across three main subjects: math, spelling, and reading.

    “Nationally standardized tests are taken across three main subjects: math, spelling, and reading ( SI Appendix, section 3.1).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites SI Appendix, section 3.1

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductcitation

    The sample covers 15% of all primary schools and is broadly representative of the national student body.

    “The sample covers 15% of all primary schools and is broadly representative of the national student body ( SI Appendix, section 5.1).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites SI Appendix, section 5.1

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The oral reading fluency test's goal is to assess technical reading ability, likely a mix of reading ability, cognitive processing, and verbal fluency.

    “In the terminology of the test producer, its goal is to assess “technical reading ability”—likely a mix of reading ability, cognitive processing, and verbal fluency.”
    Quote in the paperSubstantial claim
46 of 46

  • The paper relies on this without support.

    Students across The Netherlands take the same examination within a given year.

    “Students across The Netherlands take the same examination within a given year.”

  • The paper relies on this without support.

    The norm for transforming test results to percentile scores is the same across years, so absolute changes in performance over time are preserved.

    “Test results are transformed to percentile scores, but the norm for transformation is the same across years so absolute changes in performance over time are preserved.”

  • The paper relies on this without support.

    The oral reading fluency assessment does not test for the retention of curricular content.

    “As this part of the assessment does not test for the retention of curricular content, we would expect it to be less affected by school closures, which is indeed what we find.”

  • The authors state this as an assumption.

    Percentiles were drawn from an underlying normal distribution.

    “Assuming that percentiles were drawn from an underlying normal distribution, we use the following formula to convert between one and the other: d=Φ−10.50+δ100, [3] where δ is the treatment effect on the percentile scale, and Φ−1 is the inverse cumulative standard normal distribution.”

  • The paper relies on this without support.

    In the absence of the pandemic, the rate of progress in 2020 would have followed the same trend as in prior years.

    “We analyze the rate of progress in 2020 to that in previous years using a difference-in-differences design.”

  • The paper relies on this without support.

    Comprehension of the words is not needed and students and parents are discouraged to prepare for the oral reading fluency test.

    “Crucially, comprehension of the words is not needed and students and parents are discouraged to prepare for it.”

  • The paper relies on this without support.

    The test producer's translation keys are based on smaller samples than the study's sample.

    “However, as these keys are actually based on smaller samples than that at our disposal, we further impose a uniform distribution in our sample within cells defined by subject, grade, and testing occasion: midyear vs. end of year.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • School attendance is foundational to young people's educational attainment, social development, and future life opportunities.

    “School attendance is foundational to young people’s educational attainment, social development, and future life opportunities (Engzell et al., 2021; McDonald et al., 2022).”

    From Beyond the Classroom: Strengthening Pupil Engagement, Attendance and Belonging through Alternative Education and Community Partnerships (2025) · cites this paper as Engzell et al., 2021

    The Discussion (page not provided) reports that Dutch primary-school students learned less during school closures, supporting the educational part of the claim. It notes that test scores do not capture psychosocial development and that evidence on lasting effects and adult earnings is mixed; it does not establish that attendance is foundational to social development or future life opportunities. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The impact is 40 percent larger among those in the least educated homes, suggesting that the pandemic not only increased educational inequality, but that disadvantaged children's skills actually deteriorated.

    “The impact is 40 percent larger among those in the least educated homes, suggesting that the pandemic not only increased educational inequality, but that disadvantaged children's skills actually deteriorated.”

    From Educational Inequality* (2022) · cites this paper as Engzell, Frey, and Verhagen (2021)

    The Results (Baseline Specification; page unavailable) report losses 40% larger than average for the two lowest parental-education categories combined, not specifically for the least-educated category. The Discussion says the findings suggest widening socioeconomic gaps, but whether students actually regressed depends on the benchmark used; it does not establish that disadvantaged children’s skills deteriorated. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Results from Engzell, Frey, and Verhagen (2021) for an eight-week school closure imply that the PISA gap would further increase by around 0.05 of a standard deviation.

    “Results from Engzell, Frey, and Verhagen (2021) for an eight-week school closure imply that this gap would further increase by around 0.05 of a standard deviation.”

    From Educational Inequality* (2022) · cites this paper as Engzell, Frey, and Verhagen (2021)

    The Discussion reports an average loss of 0.08 SD after the Dutch school closure and says losses among children from less-educated households were up to 60% larger; the implied additional loss is about 0.05 SD. The study measured Dutch primary-school test scores, not a PISA gap, so applying that estimate to the PISA gap is an extrapolation rather than a finding of the cited paper. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Engzell, Frey, and Verhagen (2021) find that in the Netherlands, eight weeks of online rather than in-person learning led to 0.08 of a standard deviation lower test scores for students aged eight to eleven.

    “Engzell, Frey, and Verhagen (2021) find that in the Netherlands, eight weeks of online rather than in-person learning led to 0.08 of a standard deviation lower test scores for students aged eight to eleven.”

    From Educational Inequality* (2022) · cites this paper as Engzell, Frey, and Verhagen (2021)

  • The learning loss was up to 60% greater among students from disadvantaged backgrounds (less-educated households) in the Netherlands.

    “However, the learning loss was up to 60% greater among students from disadvantaged backgrounds (i.e., students from less-educated households), indicating the disproportionate impact of the pandemic on student learning.”

    From Impacts of the COVID-19 Pandemic on Student Learning and Opportunity Gaps Across the 2020–2021 School Year: A National Survey of Teachers (2022) · cites this paper as Engzell et al. (2021)

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Engzell, P., Frey, A., & Verhagen, M. D.
Affiliations
  • Stockholm University (SE)
  • University of Oxford (GB)
Corresponding author
  • Engzell, Per
  • Frey, Arun
  • Verhagen, Mark D.
ORCID
Venue
Proceedings of the National Academy of Sciences
Publisher
National Academy of Sciences
Published
2021-04-07
Volume
118
Issue
17
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
DOI
10.1073/pnas.2022376118
PMC
PMC8092566
OpenAlex
W3143968054
Cited by
1,848
References
75
Citations by year
2026: 154 · 2025: 211 · 2024: 312 · 2023: 476 · 2022: 456 · 2021: 231 · 2020: 7 · 2015: 1
FWCI
562.67
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • UCL Discovery (University College London) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • SocArXiv (OSF Preprints) (submittedVersion, cc-by-nc-nd)
  • SocArXiv (OSF Preprints) (submittedVersion)
  • Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
Funders
  • Leverhulme Trust
  • University of Oxford
  • Vetenskapsrådet (2016-07099)
  • Forskningsrådet om Hälsa, Arbetsliv och Välfärd (2016-07099)
  • Organisation de Coopération et de Développement Économiques
  • Economic and Social Research Council
Topics
  • Social Sciences › Education › Education during COVID-19 pandemic
  • Social Sciences › Education › School Choice and Performance
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › COVID-19 Pandemic Impacts
Keywords
learning loss, school closures, COVID-19 pandemic, lockdown, national examinations, academic performance, home background, parental education, home-based learning, educational inequality
MeSH terms
COVID-19, SARS-CoV-2, Child, Female, Humans, Learning, Male, Netherlands, Quarantine, Schools, Pandemics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review