CoChat
Sign Up
Cited by 118Education during COVID-19 pandemicCOVID-19 Pandemic ImpactsCOVID-19 and healthcare impacts

Effects of Pandemic-Related School Closures on Pupils’ Performance and Learning in Selected Countries: A Rapid Review

Open access
Zierer, Klaus·2021·Education Sciences
DOI 10.3390/educsci11060252
Open this paper in CoChat — free accountOpen PDF

Abstract

The coronavirus pandemic has led to drastic measures around the world, which have also affected the education system. Schools were closed in the spring of 2020 in almost every country in the world, and many children and young people are still involved in distance learning to this day. What effect these measures have on children’s and young people’s learning performance is important in view of the time pressure under which educational policy decisions have to be taken. The rapid review presented in this paper delivers evidence on the effects of school closures to contain the coronavirus pandemic in the spring of 2020 on children’s and young people’s learning performance in five countries (USA, Belgium, the Netherlands, Switzerland and Germany).

AI snapshot

Written from the paper’s full text.

TL;DR

This rapid review synthesizes seven studies from five countries on how spring 2020 pandemic school closures affected pupils’ learning performance. It reports an overall negative pooled effect of −0.14 standard deviations, with primary pupils and mathematics more affected, but high heterogeneity across studies.

Key findings

  • Across 12 effects from seven primary studies, the averaged effect size was −0.14 SD (95% CI: −0.19, −0.10); depending on assumptions about normal yearly learning, pupils lost between 23% and 35% of learning time.
  • Moderator analyses found larger negative effects for primary school pupils (d = −0.17, 95% CI: −0.25, −0.10) than secondary pupils (d = −0.10, 95% CI: −0.16, −0.03), and for mathematics (d = −0.17, 95% CI: −0.22, −0.11) than native language (d = −0.11, 95% CI: −0.15, −0.07).
  • Individual results were mixed: negative effects appeared in studies from the USA, Belgium, the Netherlands and Germany—e.g., Belgium mathematics −0.19 SD, Netherlands mathematics Cohen’s d = −0.142, Germany mathematics “Operations” d = −0.09—while some studies reported no significant differences, and one Netherlands study reported positive effects ranging from Cohen’s d ~0.045 to ~0.85.
  • Heterogeneity was high (Q = 594.798; p < 0.001), indicating that effects depended substantially on moderating variables.

Methods

  • Rapid review with literature searched on ResearchGate and Google Scholar until 10 March 2021 using the terms “learning performance”, “COVID” and “school closure”; 75 hits were screened, and seven studies from the USA, Belgium, the Netherlands, Switzerland and Germany were included.
  • Included studies covered primary and secondary pupils and learning performance in mathematics, native language, or several subjects combined.
  • Quantitative synthesis calculated Cohen’s d where possible, combined primary study effects into a precision-weighted averaged effect size based on sample size, and examined learner age group and school subject as moderators; heterogeneity was assessed with Q and I².

Limitations

  • The paper states that only two databases were used, so publication bias cannot be ruled out and further relevant work was likely missed; methodological quality of included studies was assessed only conditionally.
  • The paper states that data come from five countries and cannot be transferred to countries with different political, cultural and social conditions.
  • Evident from the text: high heterogeneity (Q = 594.798, p < 0.001; I² up to 98.54%) limits the interpretability of the averaged effect, and several included studies had measurement or participation issues, such as insufficient completed tasks, lower test participation, or use of a learning software that also served as the measurement tool.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/educsci11060252

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 74 cited sources checked · 0 supported · 0 flagged

  • Findingcitation

    In the Netherlands Snappet study, positive effects between Cohen's d ~0.045 and ~0.18 were noted for all pupil groups regarding attainment during closed schools.

    “Positive effects between Cohen's d ~0.045 and Cohen's d ~0.18 are to be noted for all groups of pupils also with regard to attainment, in the time of closed schools [7] (p.6).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] Meeter, M. (2021, May 20). Primary school mathematics during Covid-19: No evidence of learning gaps in adaptive practicing results. Available online: https://psyarxiv.com/8un6x/. source

      Not checked in the free run.

  • Findingcitation

    In the Netherlands, the time-measured learning loss corresponds to 7.9 to 10.5 weeks of non-learning relative to progress in a normal school year.

    “Engzell et al. [6] found that the time-measured learning loss in relation to the progress that would be achieved in a normal school year corresponds to a period of 7.9 to 10.5 weeks of non-learning [6] (p. 10).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [6] Engzell, 2021, Learning inequality during the COVID-19 pandemic, Proc. Natl. Acad. Sci. USA source

      Not checked in the free run.

  • Findingcitation

    In Flanders, Belgium, seven-week school closures were associated with a decrease in primary school pupils' mathematics test values by 0.19 standard deviations.

    “Moldonado and De Witte [5] investigated the effect of the seven-week-long school closures on primary school pupils of Flemish schools in Belgium and determined a decrease in the test values achieved in mathematics by 0.19 standard deviations [5] (p. 1).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5] Maldonado, J.E., and De Witte, K. (2021, May 20). The Effect of School Closures on Standardised Student Test Outcomes. Discussion Paper Series, DPS20.17. KU Leuven Department of Economics. Available online: https://feb.kuleuven.be/research/economics/ces/documents/DPS/2020/dps2017.pdf.

      Not checked in the free run.

    • Rests on an assumption: Observed differences in test performance between 2019 and 2020 cohorts are attributable to pandemic-related school closures rather than to other concurrent factors.
  • Findingcitation

    In the Flemish Belgium study, mathematics performance deficits were greater for pupils who were already disadvantaged.

    “It is noticeable that the performance deficits were greater for pupils who were already disadvantaged [5] (p. 1).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] Maldonado, J.E., and De Witte, K. (2021, May 20). The Effect of School Closures on Standardised Student Test Outcomes. Discussion Paper Series, DPS20.17. KU Leuven Department of Economics. Available online: https://feb.kuleuven.be/research/economics/ces/documents/DPS/2020/dps2017.pdf.

      Not checked in the free run.

  • Findingcitation

    In the Flemish Belgium study, heterogeneity of mathematics performance increased both within the schools examined and between institutions.

    “The heterogeneity of performance increased both within the schools examined and between the institutions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] Maldonado, J.E., and De Witte, K. (2021, May 20). The Effect of School Closures on Standardised Student Test Outcomes. Discussion Paper Series, DPS20.17. KU Leuven Department of Economics. Available online: https://feb.kuleuven.be/research/economics/ces/documents/DPS/2020/dps2017.pdf.

      Not checked in the free run.

  • May combine two claims. 21
  • Classified as probably not a substantial claim. 14
  • The quote only partly states this claim. 2

  • May combine two claims.

    Findingcitation

    In the Netherlands Snappet study, after reopening, attainment effects were smaller but still positive.

    “After the reopening, these turned out to be smaller, but still positive.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] Meeter, M. (2021, May 20). Primary school mathematics during Covid-19: No evidence of learning gaps in adaptive practicing results. Available online: https://psyarxiv.com/8un6x/. source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    In the Netherlands mathematics comparison, the reported mean difference corresponds to an effect size of Cohen's d = 0.142, with a 95% confidence interval of -0.15 to -0.135.

    “The reported mean difference corresponds to an effect size of Cohen's d = 0.142, with a 95% confidence interval of -0.15--0.135(own calculation according to Lenhard, W. & Lenhard, A., 2016).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    In the Netherlands, with a sample of n = 286,515, there was a significant difference in mean mathematics test values; pupils tested after school closures had significantly worse values than a comparison group without closures.

    “With a sample size of n = 286,515, a significant difference in mean values was found when comparing the control group and the test group.The group of pupils that was tested after the school closures showed significantly worse test values than a comparison group that did not experience any school closures [6] (p. 85).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [6] Engzell, 2021, Learning inequality during the COVID-19 pandemic, Proc. Natl. Acad. Sci. USA source

      Not checked in the free run.

    • Rests on an assumption: Observed differences in test performance between 2019 and 2020 cohorts are attributable to pandemic-related school closures rather than to other concurrent factors.

  • May combine two claims.

    Findingcitation

    In the Netherlands Snappet study, reported effects of school closures were all positive and ranged between Cohen's d ~0.045 and Cohen's d ~0.85.

    “The reported effects of the school closures were all positive and ranged between Cohen's d ~0.045 and Cohen's d ~0.85.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] Meeter, M. (2021, May 20). Primary school mathematics during Covid-19: No evidence of learning gaps in adaptive practicing results. Available online: https://psyarxiv.com/8un6x/. source

      Not checked in the free run.

    • Rests on an assumption: Observed differences in test performance between 2019 and 2020 cohorts are attributable to pandemic-related school closures rather than to other concurrent factors.

  • May combine two claims.

    Findingcitation

    In the USA, across all ethnic groups tested, mathematics percentile values in autumn 2020 were lower than in autumn 2019, with third and fourth grade mathematics performance each decreasing by 9 percentile positions.

    “Across all ethnic groups tested, lower percentile values were found in the test in the autumn of 2020 compared to a group of pupils of the same age who were tested in the autumn of 2019.In the third grade, mathematics performance decreased by 9 percentile positions; in the fourth grade, it decreased by 9 percentile positions [4] (p. 13).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [4] Kuhfeld, M., Ruzek, E., Johnson, A., Tarasawa, B., and Lewis, K. (2021, May 20). Technical appendix for: Learning during COVID-19. Initial Findings on Students’ Reading and Math Achievement and Growth. Available online: https://www.nwea.org/content/uploads/2020/04/COVID-technical-appendix_2020.04.08.pdf.

      Not checked in the free run.

    • Rests on an assumption: Observed differences in test performance between 2019 and 2020 cohorts are attributable to pandemic-related school closures rather than to other concurrent factors.

  • May combine two claims.

    Findingcitation

    In the Netherlands, eight-week school closures were associated with an overall learning loss of 3 percentile points or 0.08 standard deviations, with losses for pupils from less educated parents 60% greater.

    “Overall, a learning loss of 3 percentile points or 0.08 standard deviations was determined, whereby the losses in the case of pupils from less educated parents turned out to be 60% greater [6] (p. 1).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [6] Engzell, 2021, Learning inequality during the COVID-19 pandemic, Proc. Natl. Acad. Sci. USA source

      Not checked in the free run.

    • Rests on an assumption: Observed differences in test performance between 2019 and 2020 cohorts are attributable to pandemic-related school closures rather than to other concurrent factors.

  • May combine two claims.

    Findingcitation

    In the USA, school closures had considerable effects on mathematics performance, with grades three to five particularly affected.

    “Kuhfeld et al. [4], in their study, found considerable effects of school closures on mathematics performance, among other things.Grades three to five were particularly affected [4] (p. 8).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [4] Kuhfeld, M., Ruzek, E., Johnson, A., Tarasawa, B., and Lewis, K. (2021, May 20). Technical appendix for: Learning during COVID-19. Initial Findings on Students’ Reading and Math Achievement and Growth. Available online: https://www.nwea.org/content/uploads/2020/04/COVID-technical-appendix_2020.04.08.pdf.

      Not checked in the free run.

    • Rests on an assumption: Observed differences in test performance between 2019 and 2020 cohorts are attributable to pandemic-related school closures rather than to other concurrent factors.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Particularly inter-individual differences in learning increased sharply during the time of distance learning.

    “Tomasik et al. [13] point out that particularly inter-individual differences in learning increased sharply during the time of distance learning and that a decrease in the augmented learning can have negative long-term consequences [13] (p. 7).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] Tomasik, M.J., Helbling, L.A., and Moser, U. (2020). Educational gains of in-person vs. distance learning in primary and secondary schools: A natural experiment during the COVID-19 pandemic school closures in Switzerland. Int. J. Psychol. source

      Not checked in the free run.

  • The quote only partly states this claim.

    Findingcitation

    In Baden-Württemberg, school characteristics such as socio-economic capital and proportion of pupils with an immigrant background carried hardly any weight in relation to changes in competence values.

    “The effect of school characteristics, such as socio-economic capital and the proportion of pupils with an immigrant background, carried hardly any weight in relation to the changes of the competence values at the schools [9] (p.17).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9] Schult, J., Mahler, N., Fauth, B., and Lindner, M.A. (2021, May 20). Did Students Learn Less During the COVID-19 Pandemic? Reading and Mathematics Competencies Before and After the First Pandemic Wave. Available online: https://psyarxiv.com/pqtgf/. source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    The effect size calculated from Engzell et al.'s reported values is Cohen's d = -0.14, with a 95% confidence interval of -0.148 to -0.132.

    “The effect size calculated from this is Cohen's d = -0.14, at a 95% confidence interval of -0.148 to -0.132 (own calculation according to [12]).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12] Lenhard, W., and Lenhard, A. (2021, May 20). Berechnung von Effektstärken. Available online: https://www.psychometrica.de/effektstaerke.html.

      Not checked in the free run.

  • The quote only partly states this claim.

    Findingcitation

    In Hamburg secondary reading, analysis of differences between mean scores found no statistically significant differences.

    “In the course of analyzing the differences between the mean scores obtained with said test, no statistically significant differences were found [8] (p.67), Table 2.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [8] Depping, 2021, Kompetenzstände Hamburger Schüler*innen vor und während der Corona-Pandemie [Alternative pupils’ competence measurement in Hamburg during the Corona pandemic], DDS Die Dtsch. Sch. Beiheft

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    The difference in augmented learning between face-to-face and distance teaching is ∆χ2(1) = 8.86 and is significant (p < 0.001).

    “This difference of ∆χ 2 (1) = 8.86 is significant (p < 0.001) [13] (p. 6).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] Tomasik, M.J., Helbling, L.A., and Moser, U. (2020). Educational gains of in-person vs. distance learning in primary and secondary schools: A natural experiment during the COVID-19 pandemic school closures in Switzerland. Int. J. Psychol. source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    In Baden-Württemberg, average and weaker pupils in the 2020 test showed greater deficits in mathematical operations than previous cohorts, while this difference was not demonstrated in high-performance pupils.

    “The results of the investigations also suggest that average and weaker pupils in the 2020 test showed greater deficits in terms of mathematical operations than the cohorts from previous years.This difference could not be demonstrated in the high-performance pupils [9] (p.17).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [9] Schult, J., Mahler, N., Fauth, B., and Lindner, M.A. (2021, May 20). Did Students Learn Less During the COVID-19 Pandemic? Reading and Mathematics Competencies Before and After the First Pandemic Wave. Available online: https://psyarxiv.com/pqtgf/. source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    In the Netherlands Snappet study, an increase of 0.2 standard deviations for the 2019/20 school year was noted for all recorded domains, with n = 25,336 observations in the fourth grade.

    “Although some of the positive effects found were equalized after the reopening of the schools, an increase of 0.2 standard deviations for the school year 2019/20 can be noted for all domains recorded with a sample with n = 25,336 observations in the fourth grade.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] Meeter, M. (2021, May 20). Primary school mathematics during Covid-19: No evidence of learning gaps in adaptive practicing results. Available online: https://psyarxiv.com/8un6x/. source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    In the Netherlands Snappet study, practice effects were greater during school closures than before, and even greater after reopening.

    “In the area of practice, greater effects were demonstrated during school closures than before this measure was taken.After the institutes were reopened, these effects were even greater [7] (p. 7).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] Meeter, M. (2021, May 20). Primary school mathematics during Covid-19: No evidence of learning gaps in adaptive practicing results. Available online: https://psyarxiv.com/8un6x/. source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    In Ohio, the decrease in third grade language performance corresponds to about a third of the increase normally recorded in a year and affected economically disadvantaged and particularly pupils of color.

    “The decrease in performance corresponds to about a third of the increase in learning that is normally recorded in a year and affected economically disadvantaged and particularly pupils of color [10] (p. 2).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10] Kogan, V., and Lavertu, S. (2021, May 20). The COVID-19 Pandemic and Student Achievement on Ohio’s Third-Grade English Language Arts Assessment. Available online: http://glenn.osu.edu/educational-governance/reports/reports-attributes/ODE_ThirdGradeELA_KL_1-27-2021.pdf.

      Not checked in the free run.

  • May combine two claims.

    Study design and conductcitation

    In the Flemish Belgium study, the negative mathematics effect was robust to introduction of control variables and remained similar when the sample was restricted to schools that participated in each test year from 2015 to 2020.

    “The effect found was robust to the introduction of control variables into the model.Even if the sample was restricted to schools that participated in each year of the test (from 2015 to 2020 in total), the result remained similar [5] (p.24).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5] Maldonado, J.E., and De Witte, K. (2021, May 20). The Effect of School Closures on Standardised Student Test Outcomes. Discussion Paper Series, DPS20.17. KU Leuven Department of Economics. Available online: https://feb.kuleuven.be/research/economics/ces/documents/DPS/2020/dps2017.pdf.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    In the Kuhfeld et al. study, pupils from ethnic minorities were largely not tested in autumn 2020, which could lead to underestimation of the reported differences.

    “The authors point out that pupils from ethnic minorities were largely not tested in the autumn of 2020, which could lead to the extent of the differences reported here being underestimated [4] (p. 5).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [4] Kuhfeld, M., Ruzek, E., Johnson, A., Tarasawa, B., and Lewis, K. (2021, May 20). Technical appendix for: Learning during COVID-19. Initial Findings on Students’ Reading and Math Achievement and Growth. Available online: https://www.nwea.org/content/uploads/2020/04/COVID-technical-appendix_2020.04.08.pdf.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Access to technology alone is not enough to guarantee quality distance learning.

    “The authors also noted that access to technology alone is not enough to guarantee quality distance learning [6] (p. 11).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] Engzell, 2021, Learning inequality during the COVID-19 pandemic, Proc. Natl. Acad. Sci. USA source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    In Baden-Württemberg, dispersion of pupils' competence levels within individual schools could pose greater challenges for teachers than a slight pandemic-related weakening of the average competence level.

    “Schult et al. [9] stated that the dispersion of the competence levels of pupils within the individual schools could pose greater challenges for the teachers than a slight pandemic-related weakening of the average competence level at the school [9] (p.18).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9] Schult, J., Mahler, N., Fauth, B., and Lindner, M.A. (2021, May 20). Did Students Learn Less During the COVID-19 Pandemic? Reading and Mathematics Competencies Before and After the First Pandemic Wave. Available online: https://psyarxiv.com/pqtgf/. source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductcitation

    In the Hamburg primary reading study, in some cases no competence value could be calculated due to insufficiently completed tasks, which can distort the results.

    “Just like with the investigation of competence in mathematics, it should also be noted that in some cases no competence value could be calculated due to insufficiently completed tasks [8] (p.74).This can lead to a distortion of the results.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [8] Depping, 2021, Kompetenzstände Hamburger Schüler*innen vor und während der Corona-Pandemie [Alternative pupils’ competence measurement in Hamburg during the Corona pandemic], DDS Die Dtsch. Sch. Beiheft

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    In Ohio, school closures were not the only reason for the decline in language test performance; negative effects were also demonstrated for pupils who had hybrid or face-to-face lessons.

    “The authors pointed out that the school closures were not the only reason for the decline in performance in the language test.Negative effects were also demonstrated for pupils who had hybrid or face-to-face lessons during the study period [10] (p.3).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10] Kogan, V., and Lavertu, S. (2021, May 20). The COVID-19 Pandemic and Student Achievement on Ohio’s Third-Grade English Language Arts Assessment. Available online: http://glenn.osu.edu/educational-governance/reports/reports-attributes/ODE_ThirdGradeELA_KL_1-27-2021.pdf.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    In the Netherlands Snappet study, multi-level analyses were not possible for data protection reasons, increasing the probability of false-positive results.

    “Meeter [7] pointed out that for reasons of data protection, multi-level analyses were not possible, which in turn increases the probability of false-positive results [7] (p.10).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] Meeter, M. (2021, May 20). Primary school mathematics during Covid-19: No evidence of learning gaps in adaptive practicing results. Available online: https://psyarxiv.com/8un6x/. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationcitation

    In Baden-Württemberg, the more pronounced reading competence differences in high-performance pupils could be due to more time spent on basic reading tasks during school closures.

    “According to the authors of the study, this difference could be due to the fact that during school closures, more time is spent on basic reading tasks, such as reading e-mails and text in the chat [9] (p.17).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [9] Schult, J., Mahler, N., Fauth, B., and Lindner, M.A. (2021, May 20). Did Students Learn Less During the COVID-19 Pandemic? Reading and Mathematics Competencies Before and After the First Pandemic Wave. Available online: https://psyarxiv.com/pqtgf/. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    A decrease in augmented learning can have negative long-term consequences.

    “Tomasik et al. [13] point out that particularly inter-individual differences in learning increased sharply during the time of distance learning and that a decrease in the augmented learning can have negative long-term consequences [13] (p. 7).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] Tomasik, M.J., Helbling, L.A., and Moser, U. (2020). Educational gains of in-person vs. distance learning in primary and secondary schools: A natural experiment during the COVID-19 pandemic school closures in Switzerland. Int. J. Psychol. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    No statement can be made about the effects of school closures in other countries.

    “No statement can be made about the effects of school closures in other countries, especially since it can be assumed that political, social and cultural factors also have an influence.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Political, social and cultural factors influence the effects of school closures on learning performance.

  • Classified as probably not a substantial claim.

    Premisecitation

    Schult et al. stated that the data across all these studies and domains indicate an average learning loss of around 0.1 standard deviations during the first wave of the COVID-19 pandemic in 2020.

    “Schult et al. [9], referring to the studies by Depping et al. [8], Moldonado and De Witte [5], Meeter [7], Engzell et al. [6] and Kuhfeld et al. [4] stated that the data across all these studies and domains indicate an average learning loss of around 0.1 standard deviations during the first wave of the COVID 19 pandemic in 2020 [9] (p. 7).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [9] Schult, J., Mahler, N., Fauth, B., and Lindner, M.A. (2021, May 20). Did Students Learn Less During the COVID-19 Pandemic? Reading and Mathematics Competencies Before and After the First Pandemic Wave. Available online: https://psyarxiv.com/pqtgf/. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    When integrating published data, there is always the problem of publication bias, which manifests itself in the fact that significant effects are published more often than nonsignificant ones.

    “When integrating published data, there is always the problem of publication bias, which manifests itself in the fact that significant effects are published more often than nonsignificant ones [14,15].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [14] Lipsey, 1993, The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta-analysis, Am. Psychol. source

      Not checked in the free run.

    • cites [15] Schmidt, F.L., and Hunter, J.E. (2014). Methods of Meta-Analysis: Correcting Error and Bias in Research Findings, Sage. [3rd ed.]. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    Tomasik et al.'s analyses did not compare the best possible face-to-face teaching with the best possible distance teaching, which must be taken into account when interpreting the results.

    “The authors of the study point out that their analyses did not compare the best possible face-to-face teaching with the best possible distance teaching.This must be taken into account when interpreting the results [13] (p. 7), Table 3.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] Tomasik, M.J., Helbling, L.A., and Moser, U. (2020). Educational gains of in-person vs. distance learning in primary and secondary schools: A natural experiment during the COVID-19 pandemic school closures in Switzerland. Int. J. Psychol. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    In Tomasik et al.'s data, effects on learning progress could be found for primary education, in contrast to secondary education.

    “At this point, the results relating to primary education were of interest, where effects could be found on the basis of the data in contrast to secondary education [13] (p. 1).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] Tomasik, M.J., Helbling, L.A., and Moser, U. (2020). Educational gains of in-person vs. distance learning in primary and secondary schools: A natural experiment during the COVID-19 pandemic school closures in Switzerland. Int. J. Psychol. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Q test can be used to determine whether there is significant heterogeneity between primary study effects.

    “The Q test can be used to determine whether there is significant heterogeneity between primary study effects [18].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18] Cooper, H., and Hedges, L.V. (1994). Combining estimates of effect size. The Handbook of Research Synthesis, Russell Sage Foundation.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The test variable I2 is used to quantify the extent of the heterogeneity.

    “To quantify the extent of the heterogeneity, the test variable I 2 is used [19].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [19] Higgins, J.P., and Green, S. (2008). Analysing data and undertaking meta-analyses. Cochrane Handbook for Systematic Reviews of Interventions, John Wiley & Sons Ltd. source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductnone given

    The methodological quality of the studies included was only assessed conditionally, so possible biases may arise in the presentation of results.

    “In addition, the methodological quality of the studies included was only assessed conditionally, so possible biases may arise in the presentation of results.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    It is likely that further relevant work was not taken into account in this review.

    “It is likely that further relevant work was not taken into account.”
    Quote in the paperNot a substantial claim
34 of 34

  • The paper relies on this without support.

    Observed differences in test performance between 2019 and 2020 cohorts are attributable to pandemic-related school closures rather than to other concurrent factors.

    “The rapid review presented in this paper delivers evidence on the effects of school closures to contain the coronavirus pandemic in the spring of 2020 on children's and young people's learning performance in five countries (USA, Belgium, the Netherlands, Switzerland and Germany).”

  • The paper relies on this without support.

    The seven included studies are sufficient and representative to support conclusions about the effects of school closures in the five named countries.

    “After a subsequent full-text analysis, seven studies were included.They come from the USA, Belgium, the Netherlands, Switzerland and Germany, focus on learners in primary school and secondary school and examine learning performance in mathematics, native language or several subjects combined.”

  • The authors state this as an assumption.

    The unsystematic error variance of a primary study can be determined by the variance of the effect size.

    “The precision was determined on the basis of the respective sample size of the primary studies, under the assumption that the unsystematic error variance of a primary study can be determined by the variance of the effect size [17].”

  • The authors state this as an assumption.

    Political, social and cultural factors influence the effects of school closures on learning performance.

    “No statement can be made about the effects of school closures in other countries, especially since it can be assumed that political, social and cultural factors also have an influence.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Systematic reviews and meta-analyses suggest an average learning loss attributable to COVID-19 school closures of 0.17 of a standard deviation, at least one-third of a year's worth of learning.

    “Nevertheless, there are several systematic reviews and meta-analyses, and these suggest an average learning loss attributable to COVID-19 school closures of 0.17 of a standard deviation—that is, at least one-third of a year’s worth of learning16–20.”

    From COVID-19, school closures, and student learning outcomes. New global evidence from PISA (2025) · cites this paper as [19]

    Section 10 (page not provided) reports learning loss after school closures, but its mean effect is −0.14 standard deviations, not −0.17. It estimates lost learning time at 23%–35% of a school year, so the loss is not consistently at least one-third. The paper describes itself as a rapid review and says a comprehensive meta-analysis is still needed. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • The study's results are similar to what systematic reviews are finding.

    “Interestingly, our results are similar to what systematic reviews are finding learning16–20,44,45.”

    From COVID-19, school closures, and student learning outcomes. New global evidence from PISA (2025) · cites this paper as [19]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Spelling mean described as lower is higher

    Minor

    The test-group spelling mean is 0.71, higher than the control-group mean of 0.05, despite the text calling it significantly lower. The subsequently reported negative effect size also has the opposite direction from these means.

  • Mathematics effect size has opposite signs

    Minor

    The text gives Engzell et al.'s mathematics effect size as positive 0.142, but its stated confidence interval is entirely negative and Table 1 gives the effect as −0.142.

  • Secondary-school Swiss sample size differs between text and table

    Minor

    The text gives 15,551 secondary-school pupils, whereas the secondary-education row of Table 3 gives 13,134-the number elsewhere assigned to the primary-school sample.

  • Fourth-grade mathematics decline differs between text and table

    Minor

    The text reports a decline of 9 percentile positions for fourth graders, while Table 1 reports 11 percentile positions.

4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Depping, 2021, Kompetenzstände Hamburger Schüler*innen vor und während der Corona-Pandemie [Alternative pupils’ competence measurement in Hamburg during the Corona pandemic], DDS Die Dtsch. Sch. Beiheft, 17, 51
  2. Hedges, L.V., and Olkin, I. (1985). Statistical Methods for Meta-Analysis, Academic Press.

Cited by

Sorted by citations

Related

Authors
Zierer, K.
Affiliations
University of Augsburg (DE)
Corresponding author
Zierer, Klaus
ORCID
Zierer, Klaus
Venue
Education Sciences
Publisher
MDPI AG
Published
2021-05-22
Volume
11
Issue
6
Pages
252
ISSN
2227-7102 (ISSN-L)
DOI
10.3390/educsci11060252
OpenAlex
W3164533431
Semantic Scholar
7fe01464089dcd05c52f75484e29180efc0095fc
Cited by
118
References
11
Citations by year
2026: 4 · 2025: 16 · 2024: 22 · 2023: 32 · 2022: 36 · 2021: 8
FWCI
43.06
Type
journal article
Language
en
Open access
Licence
Locations
  • Education Sciences (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • OPUS (Augsburg University) (submittedVersion, cc-by)
Topics
  • Social Sciences › Education › Education during COVID-19 pandemic
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › COVID-19 Pandemic Impacts
  • Health Sciences › Medicine › Oncology › COVID-19 and healthcare impacts
Keywords
school closures, COVID-19 pandemic, student learning performance, distance education, educational disruption, learning loss, education policy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review