CoChat
Sign Up
Cited by 499Education during COVID-19 pandemicSchool Choice and PerformanceCOVID-19 Pandemic Impacts

The effect of school closures on standardised student test outcomes

Open access
Maldonado, Joana Elisa, De Witte, Kristof·2022·British Educational Research Journal
DOI 10.1002/berj.3754
Open this paper in CoChat — free accountOpen PDF

Abstract

The school closures owing to the 2020 COVID‐19 crisis resulted in a significant disruption of education provision, leading to fears of learning losses and of an increase in educational inequality. This article evaluates the effects of school closures based on standardised tests in the last year of primary school in the Dutch‐speaking Flemish region of Belgium. Using a 6‐year panel, we find that students of the 2020 cohort experienced significant learning losses in three out of five tested subjects, with a decrease in school averages of mathematics scores of 0.17 standard deviations and Dutch scores (reading, writing, language) of 0.19 standard deviations as compared to previous cohorts. This finding holds when accounting for school characteristics, standardised tests in Grade 4 and school fixed effects. Given the large observed effect sizes, the effect of school closures appears to be a combination of lost learning progress and learning loss. Moreover, we observe that inequality both within schools and across schools rises by 7% for mathematics and 8% for Dutch. The learning losses are correlated with observed school characteristics, as schools with a more disadvantaged student population experience larger learning losses.

AI snapshot

Written from the paper’s full text.

TL;DR

Using a 6-year panel of standardised Grade 6 tests in Dutch-speaking Flemish Catholic schools, the paper finds that the 2020 COVID-19 school closures reduced school-average mathematics scores by 0.17 SD and Dutch scores by 0.19 SD relative to previous cohorts. It also finds increased inequality within and across schools and larger losses in schools with more disadvantaged student populations.

Key findings

  • The 2020 cohort experienced significant learning losses in three of five tested subjects: mathematics −0.17 SD, Dutch −0.19 SD, and French −0.18 SD in the 2019–2020 comparison; science and social sciences decreases were not statistically significant throughout samples and specifications.
  • Inequality rose: within-school Gini increased by 0.01 for both mathematics and Dutch, corresponding to 7% and 8% increases; across-school Gini also increased by 0.01 in both subjects, equal to 7% for mathematics and 8% for Dutch.
  • Learning losses increased with the share of students with a low-educated mother and the share receiving financial support; losses decreased as Grade 4 GPA increased, and for mathematics losses increased with population size/urbanity.
  • Robustness checks using the same schools across all years, school fixed effects, and coarsened exact matching confirmed negative effects, with Dutch effects often larger (e.g., 0.31–0.32 SD in same-school and fixed-effects specifications).

Methods

  • Design: panel analysis of standardised tests administered in June to Grade 6 (last year of primary school) in Flemish Catholic schools from 2015 to 2020, combined with school-level administrative data, Grade 4 test scores, and COVID-19 school inspection survey data.
  • Sample: 401 schools in 2020, 1,164 in 2019, 1,150 in 2018, 1,062 in 2017, 1,033 in 2016, and 1,017 in 2015; participation was voluntary, so observations vary by subject and analysis.
  • Analysis: school-average subject scores regressed on a COVID-19 dummy for 2020 with school characteristics, Grade 6 characteristics, teacher characteristics, Grade 4 scores, and test-version fixed effects; robust standard errors; sub-samples 2019–2020, 2017–2020, and 2015–2020; inequality measured by Gini, 90/10 ratio, and generalised entropy, with marginal effects by SES, score quintile, and urbanity.

Limitations

  • The paper states that participating schools were not necessarily representative of the overall school network: they had lower shares of low-SES students and higher Grade 4 scores, so external validity could be limited; because losses were larger in more disadvantaged schools, general-population losses could be larger.
  • The paper states that Grade 6 students were among the first to re-enter school in Flanders, so observed effects for this cohort may be a lower bound for students in other grades who returned later.
  • The text makes evident that cross-year comparisons are at school level because different cohorts cannot be merged at student level; participation was voluntary with attrition, and 2020 within-school test participation was lowest for science (18%) and mathematics (40%) versus Dutch (90%), while test versions changed across years.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1002/berj.3754

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 14 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Within-school inequality, as measured by the Gini coefficient, is increasing as the share of students receiving financial support and the share of students with a mother with a low education level increase.

    “Figures B4 and B5 in Appendix B show that, similarly, inequality within schools, as measured by the Gini coefficient, is increasing as the share of students receiving financial support and the share of students with a mother with a low education level increase.”
    Quote in the paperNot a substantial claim
  • Findingown data

    For science, there is a non-significant learning loss of 0.15 standard deviations in the 2019-2020 sample.

    “For science, we observe in the fourth column of Table 1 a non-significant learning loss of 0.15 standard deviations in the 2019-2020 sample.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Learning losses are correlated with observed school characteristics, as schools with a more disadvantaged student population experience larger learning losses.

    “The learning losses are correlated with observed school characteristics, as schools with a more disadvantaged student population experience larger learning losses.”
    • Also stated in: § Introduction · plain
      “We also find learning losses to increase as the share of students with low socioeconomic status (SES) increases.”
    Quote in the paperSubstantial claim
  • Findingown data

    Schools with a higher share of students receiving financial support experienced larger average learning losses than schools with a lower share, for both mathematics and Dutch.

    “The bottom left graphs of Figures 1a and b show that this expectation is confirmed by the data for both mathematics and Dutch.In both subjects, the estimated effect of the school closures becomes increasingly negative as the share of students receiving financial support rises.”
    Quote in the paperSubstantial claim
  • Findingown data

    The 2020 cohort experienced an increase in inequality across schools.

    “Comparing inequality across schools over the years shows that the 2020 cohort also experienced an increase in inequality across schools.”
    Quote in the paperSubstantial claim
  • May combine two claims. 54
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 10

  • May combine two claims.

    Findingown data

    For Dutch, estimated learning losses are larger for schools with higher shares of students from a disadvantaged neighbourhood, although the observed effects are not significantly different from 0.

    “For Dutch, however, the top left graph in Figure 1b displays larger estimated learning losses for schools with higher shares of students from a disadvantaged neighbourhood, although the observed effects are not significantly different from 0.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    With schools that participated in all years, there are significant negative effects for both mathematics and Dutch, of 0.18 standard deviations for mathematics and 0.31 standard deviations for Dutch.

    “As observed before, there are significant negative effects for both mathematics and Dutch, of similar effect sizes of 0.18 standard deviations for mathematics and a larger effect size of 0.31 standard deviations for Dutch.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Learning losses increase as the share of students with a low-educated mother in the school increases, for both mathematics and Dutch.

    “In both subjects, learning losses increase as the share of students with a low-educated mother in the school increases.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    With school fixed effects, there are significant negative effects in both subjects, with effect sizes of 0.14 standard deviations for mathematics and 0.32 standard deviations for Dutch.

    “Again, the results prove to be robust with significant negative effects in both subjects.Effect sizes are similar to the main results in mathematics, with 0.14 standard deviations, and larger for Dutch, with 0.32 standard deviations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For mathematics, after accounting for all observed SES characteristics, learning losses increase as population size increases, with the largest learning losses and largest spread of scores in the biggest cities.

    “For mathematics, even after accounting for all observed SES characteristics, there appears to be a clear pattern of increasing learning losses as population size increases, with the largest learning losses, and the largest spread of scores, in the biggest cities.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The effect sizes are comparable in Dutch and French, and only slightly smaller in mathematics.

    “This means that the effect sizes are comparable in Dutch and French, and only slightly smaller in mathematics.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The 2020 cohort experienced decreases in the school averages of standardised test scores compared to previous cohorts, amounting to 0.17 standard deviations for mathematics and 0.19 standard deviations for Dutch.

    “The 2020 cohort experienced decreases in the school averages of standardised test scores as compared to previous cohorts, Effect of school closures on test outcomes 17 amounting to 0.17 standard deviations for mathematics and 0.19 standard deviations for Dutch.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The across-school Gini increase equals 7% for mathematics and 8% for Dutch.

    “This equals an increase of 7% for mathematics and 8% for Dutch, amounting to a similar effect size as the increase in within-school inequality in Dutch and a smaller effect in mathematics.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Learning losses appear to be increasing depending on most indicators for SES as well as population density, while they are decreasing as Grade 4 scores increase.

    “In addition, we find worrying results when considering marginal effects based on the indicators of SES: learning losses appear to be increasing depending on most indicators for SES as well as population density, while they are decreasing as Grade 4 scores increase.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The within-school Gini coefficient increase corresponds to an increase of 7% for mathematics and 8% for Dutch.

    “This corresponds to an increase by 7% and 8% for mathematics and Dutch, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The across-school 90/10 ratio had a mean of 2.03 for mathematics and 1.83 for Dutch and increased by 0.1 in mathematics; in Dutch there was a small negative effect of 0.03.

    “The 90/10 ratio has a mean of 2.03 for mathematics and 1.83 for Dutch and increases, respectively, by 0.1 in mathematics.In Dutch, there is a small negative effect of 0.03.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For mathematics, learning losses are slightly larger in the bottom percentiles but not significantly different from the top percentiles.

    “Figure B3 in Appendix B shows that for mathematics, learning losses are slightly larger in the bottom percentiles, but not significantly different from the top percentiles.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In social sciences, the effect is negative but not significant in the 2019-2020 sample.

    “The effect is found to be negative, but not significant in the 2019-2020 sample.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The distribution of scores in 2020 compared to 2019 is slightly more skewed to the left in most subjects, with a lower mean score among participating schools.

    “the distribution of scores in 2020 compared to 2019 is slightly more skewed to the left in most subjects, with a lower mean score among participating schools.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For Dutch, no clear pattern emerges as, irrespective of the quintile, all students underperform in 2020.

    “For Dutch, no clear pattern emerges as, irrespective of the quintile, all students underperform in 2020.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    With matching on school characteristics, the effects are negative in both subjects, with learning losses of 0.12 standard deviations for mathematics and 0.15 standard deviations for Dutch, although statistical significance decreases for mathematics.

    “The third column of Table 3 shows that the effects for the matched sample confirm the main results, with negative effects in both subjects.With a learning loss of 0.12 standard deviations for mathematics and 0.15 standard deviations for Dutch, the effect size is robust to matching schools as well, although the statistical significance decreases for mathematics.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The size and significance of the COVID-19 coefficient for the mathematics score appears to be robust across sub-samples, with an effect size of -0.15 to -0.17 standard deviations, significant at the 5% level.

    “the size and significance of the COVID-19 coefficient for the mathematics score appears to be robust across sub-samples, with an effect size of -0.15 to -0.17 standard deviations, significant at the 5% level.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the 2017-2020 sample, the effect of COVID-19 on social sciences becomes significant, with an effect size of 0.13 standard deviations, significant at the 10% level.

    “In the 2017-2020 sample, the effect of COVID-19 becomes significant.”
    • Also stated in: § Results for each subject · plain
      “However, even then, the effect size is, with 0.13 standard deviations, Effect of school closures on test outcomes 9 smaller than those for Dutch and mathematics and only significant at the 10% level.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The bottom quintiles performed less in 2020, while there is no significant decrease in mathematics scores for the top quintiles (nor a significant increase for the best-performing students).

    “In other words, the bottom quintiles performed less in 2020, while there is no significant decrease in mathematics scores for the top quintiles (nor is there a significant increase in the scores of the bestperforming students).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The within-school 90/10 ratio increased significantly by 0.15 for mathematics and 0.09 for Dutch.

    “With a mean of 2.07 in mathematics and 2.02 in Dutch, inequality as measured by the 90/10 ratio increases significantly by 0.15 for mathematics and 0.09 for Dutch.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    All inequality measures considered show an increase in inequality within schools, a widening of the spread in scores, in both mathematics and Dutch.

    “All inequality measures under consideration thus show an increase in inequality within schools, that is a widening of the spread in scores, in both mathematics and Dutch.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There are no significant differences in learning losses by the share of students with another home language, in either mathematics or Dutch.

    “The bottom right graphs of Figures 1a and b suggest that there are no significant differences in learning losses by the share of students with another home language, in either subject.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The within-school Gini coefficient increased due to the COVID-19 crisis by 0.01 for mathematics and Dutch, significant at the 1% level for mathematics and at the 5% level for Dutch.

    “Given a mean within-school Gini coefficient in 2019 of 0.14 for mathematics and 0.13 for Dutch, the Gini coefficient increases due to the COVID-19 crisis by 0.01 for mathematics and Dutch, significant at the 1% level for mathematics and at the 5% level for Dutch.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The average effect remains similar along the share of students from disadvantaged neighbourhoods and students with another home language, but the estimated learning loss increases considerably with increasing shares of students with a mother with a low education level and shares of students who receive financial support.

    “In summary, we observe that the average effect remains similar along the share of students from disadvantaged neighbourhoods and students with another home language, but the estimated learning loss increases considerably with increasing shares of students with a mother with a low education level and shares of students who receive financial support.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The within-school generalised entropy index increased by about 0.01 in both mathematics and Dutch, significant at the 1% level for mathematics and at the 10% level for Dutch.

    “With a mean of 0.05 in mathematics and Dutch, it increases by about 0.01 in both subjects, significant at the 1% level for mathematics and at the 10% level for Dutch.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Inequality both within schools and across schools rose by 7% for mathematics and 8% for Dutch.

    “we observe that inequality both within schools and across schools rises by 7% for mathematics and 8% for Dutch.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    With matching additionally on Grade 4 scores, the effects are negative for both subjects, with 0.12 standard deviations for mathematics and 0.16 standard deviations for Dutch, although statistical significance for mathematics again decreases.

    “The effect size is similar to the first matching approach, with 0.12 standard deviations for mathematics and 0.16 standard deviations for Dutch, although the statistical significance for mathematics again decreases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    French scores are significantly lower in 2020 compared to 2019, with an effect of -0.18 standard deviations, significant at the 1% level.

    “French scores are also found to be significantly lower in 2020 compared to 2019, with an effect of -0.18 standard deviations, significant at the 1% level.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The across-school Gini coefficient averaged 0.14 for mathematics and 0.13 for Dutch from 2015 to 2019, and increased significantly by 0.01 in both subjects.

    “The Gini coefficient is on average 0.14 for mathematics and 0.13 for Dutch from 2015 to 2019, and increases significantly by 0.01 in both subjects.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For Dutch, including a time trend results in smaller and less significant inequality estimates.

    “For Dutch, the inclusion of a time trend results in smaller and less significant estimates.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    In the 2019-2020 sample, the COVID-19 year is associated with a 0.15 SD lower science test score (not statistically significant at 10%).

    “COVID-19 | -0.17** | -0.19*** | -0.12 | -0.15 | -0.18***”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In the 2019-2020 sample, COVID-19 is associated with a 0.00 increase in the within-school entropy for Dutch (p<0.10).

    “COVID-19 | 0.01*** | 0.15*** | 0.01*** 0.01** | 0.09* | 0.00* |”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the 2019-2020 sample, the COVID-19 year is associated with a 0.12 SD lower social sciences test score (not statistically significant at 10%).

    “COVID-19 | -0.17** | -0.19*** | -0.12 | -0.15 | -0.18***”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The changes in inequality are driven by large learning losses in schools with large shares of disadvantaged students and small learning losses in schools with small shares of disadvantaged students.

    “The changes in inequality are hence driven by large learning losses in schools with large shares of disadvantaged students and small learning losses in schools with small shares of disadvantaged students.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The marginal effects remain robust when tested on the sub-samples of 2017-2020 and 2019-2020, as well as when using nonlinear specifications.

    “The marginal effects remain robust when tested on the sub-samples of 2017-2020 and 2019-2020, as well as when using nonlinear specifications.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Inequality both within and across schools increased as a result of the COVID-19 crisis.

    “Furthermore, we find that inequality both within and across schools increased as a result of the COVID-19 crisis.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The differences in test scores between the 2020 cohort and previous cohorts are attributable to the COVID-19 school closures rather than to other contemporaneous factors or pre-existing trends.

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given

    In 2020, the tests were communicated and organised in the same way as in previous years.

    “In 2020, the tests were communicated and organised in the same way as in previous years.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    School closures are associated with very high costs for students and do not affect all students equally.

    “For future policies in further management of the ongoing COVID-19 crisis, as well as other potential situations that could require school closures, this article clearly emphasises that school closures are associated with very high costs for students and do not affect all students equally.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The differences in test scores between the 2020 cohort and previous cohorts are attributable to the COVID-19 school closures rather than to other contemporaneous factors or pre-existing trends.

  • May combine two claims.

    Interpretationown data

    The school closures resulted in significant learning losses and a substantial increase in educational inequality.

    “We find that the school closures resulted in significant learning losses and a substantial increase in educational inequality.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The differences in test scores between the 2020 cohort and previous cohorts are attributable to the COVID-19 school closures rather than to other contemporaneous factors or pre-existing trends.

  • Classified as probably not a substantial claim.

    Premisecitation

    The literature about school interruptions shows learning losses to be larger in mathematics than in reading.

    “This is surprising, since the literature about school interruptions shows learning losses to be larger in mathematics than in reading (Kuhfeld et al., 2020).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Kuhfeld et al., 2020 source

  • May combine two claims.

    Study design and conductown data

    This finding holds when accounting for school characteristics and standardised tests in Grade 4, as well as when including school fixed effects.

    “This finding holds when accounting for school characteristics and standardised tests in Grade 4, as well as when including school fixed effects.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Raising student achievement by 0.2 standard deviations results, on average, in a 2.6% increase in annual lifetime earnings.

    “To put these observed effects into perspective, Chetty et al. (2014) observe that raising student achievement by 0.2 standard deviations results, on average, in a 2.6% increase in annual lifetime earnings.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Chetty et al. (2014) source

      Queued — the background check reaches it, usually within a day.

  • May combine two claims.

    Study design and conductown data

    Controlling for 2018 and 2020 being the second year with the same test leads to a similar result as the main results, meaning that the tests in 2018 and 2020 were similar to the respective previous year.

    “However, controlling for 2018 and 2020 being the second year with the same test leads to a similar result as the main results, meaning that the tests in 2018 and 2020 were similar to the respective previous year.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    Participating schools had, on average, lower shares of students with low SES than non-participating schools, with most differences statistically significant.

    “In all years, participating schools had, on average, lower shares of students with low SES, with most of these differences being statistically significant in t-tests.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    It is likely that groups of vulnerable students who recorded relatively larger learning losses will suffer disadvantages on the labour market and negative effects on earnings in the long-term.

    “It is therefore likely that the groups of vulnerable students who recorded relatively larger learning losses will suffer disadvantages on the labour market and negative effects on earnings in the long-term.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    Within schools, there was limited attrition in 2020; in 2020, attrition within schools was lowest for Dutch (90% of students participated) and highest for mathematics (40% participated) and science (18% participated).

    “Within schools, we observe limited attrition in 2020.In 2020, the attrition within schools is lowest for Dutch (90% of students participated in the test) and highest for mathematics (40% of students participated) and science (18% of students participated).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The impact of the school closures during the COVID-19 crisis can be identified by the year 2020, since no other changes took place in that year.

    “The impact of the school closures during the COVID-19 crisis can be identified by the year 2020, since no other changes took place in that year.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    Relative to the 2019 schools, schools participating in 2020 were smaller, had a lower share of students whose mothers are low-educated, more students with special needs, fewer Grade 6 students who experienced grade retention and higher Grade 4 scores in world studies.

    “Relative to the 2019 schools, schools participating in 2020 were smaller, had a lower share of students whose mothers are low-educated, more students with special needs, fewer Grade 6 students who experienced grade retention and higher Grade 4 scores in world studies.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The participating schools are not necessarily representative of the overall sample of schools in the school network and external validity could be limited.

    “Given the population consisting of more advantaged students in the participating schools, this means that the participating schools are not necessarily representative of the overall sample of schools in the school network and external validity could be limited.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    In line with job competition theory, the relative schooling of the current generation remains unaffected, so the impact on earnings and unemployment might be lower than expected based on pre-COVID research.

    “Hence, in line with the job competition theory, the relative schooling of the current generation of youngsters remains unaffected, such that the impact on earnings and unemployment might be lower than what can be expected based on pre-COVID research.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Job competition theory implies that relative schooling determines earnings and unemployment outcomes.

  • May combine two claims.

    Study design and conductreasoning

    The years 2019 and 2020 can easily be compared in all subjects, since the exact same test was administered in these 2 years, and thus represents the strongest sample.

    “the years 2019 and 2020 can easily be compared in all subjects, since the exact same test was administered in these 2 years, and thus represents the strongest sample.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    The results might be an upper bound for countries which experienced a period of shorter school closures or which relied on better distance learning strategies, while they serve as lower bound estimates for countries with longer school closures or less developed remote teaching strategies.

    “Our results might be an upper bound for countries which experienced a period of shorter school closures or which relied on better distance learning strategies (e.g. as in the Netherlands; Engzell et al., 2020), while they serve as lower bound estimates for countries with longer school closures or less developed remote teaching strategies (e.g. as in the USA; Slavin, 2020).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Engzell et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Slavin, 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    In previous years, the share of students participating in Dutch and mathematics tests was close to 100%, while this share was already lower for science, social sciences and French.

    “While in previous years, the share of students participating in Dutch and mathematics tests was close to 100%, this share was already lower for science, social sciences and French.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Study design and conductnone given

    The curriculum, funding, school organisation and overall setting in 2020 were the same as in the previous year.

    “The curriculum, funding, school organisation and overall setting in 2020 were the same as in the previous year.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    There is no overall pattern in the attrition to the 2020 sample, since most indicators of SES appear to be balanced across the samples.

    “There is, however, no overall pattern in the attrition to the 2020 sample, since, for example, most indicators of SES appear to be balanced across the samples.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The main results hold when limiting the sample to the same schools across all years and when using school fixed effects.

    “We demonstrate that the results are robust to limiting the sample using the same schools across all years as well as using school fixed effects.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    The attrition in 2020 is likely due to the tests being administered only in some classes, not others.

    “The attrition in 2020 is likely due to the tests being administered only in some classes, not others.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Less than a third of students in Germany, Switzerland and Austria had a high level of learning commitment with a study time of 5 hours or more per day during the school lockdown, while a substantial share studied for 2 hours or less per day.

    “According to a survey conducted in Germany, Switzerland and Austria, less than a third of students had a high level of learning commitment with a study time of 5 hours or more per day during the school lockdown, while a substantial share of students only studied for 2 hours or less per day (Huber & Helm, 2020).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Huber & Helm, 2020 source

      Queued — the background check reaches it, usually within a day.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given

    The cardinal properties of inequality measures such as the Gini coefficient and entropy are not well known, so comparison of their percentage changes should be interpreted with caution.

    “As the cardinal properties of inequality measures such as the Gini coefficient and entropy are not well known, the comparison of their percentage changes should be interpreted with sufficient caution.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    In 2018, attrited schools had significantly more teachers (mean 20.40) than participating schools (mean 15.97), p<0.001.

    “Number of teachers 339 20.40 1,150 15.97 0.00”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Fryer (2016) analyses 105 school-based RCTs and finds an average effect size of 0.05 standard deviations in mathematics and 0.07 standard deviations in reading.

    “Fryer (2016) analyses 105 schoolbased RCTs and finds an average effect size of 0.05 standard deviations in mathematics and 0.07 standard deviations in reading.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Fryer (2016)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

61 of 61

  • The authors state this as an assumption.

    Most teaching hours during distance learning were spent on the fundamental subjects (language, mathematics and reading).

    “Although there is no evidence on what schools taught exactly in this period, it can be assumed that, similar to the Netherlands, most teaching hours were spent on the fundamental subjects, that is language, mathematics and reading (Inspectie van het Onderwijs, 2020).”

  • The paper relies on this without support.

    The differences in test scores between the 2020 cohort and previous cohorts are attributable to the COVID-19 school closures rather than to other contemporaneous factors or pre-existing trends.

    “COVID-19 is a dummy variable for the year 2020.”

  • The paper relies on this without support.

    The Grade 6 students were among the first to re-enter school in Flanders.

    “Moreover, the Grade 6 students were among the first to re-enter school in Flanders.”

  • The paper relies on this without support.

    The complete generation of students was equally affected by the school closures.

    “On the one hand, this effect of reduced human capital accumulation due to learning losses could, at least partially, be mitigated as the complete generation of students was equally affected by the school closures.”

  • The paper relies on this without support.

    The sample under consideration represents a slightly advantaged population of schools in terms of grades on standardised tests and other background characteristics.

    “Regarding the external validity of our findings, it has to be kept in mind that the sample under consideration represents a slightly advantaged population of schools in terms of grades on standardised tests and other background characteristics.”

  • The paper relies on this without support.

    Job competition theory implies that relative schooling determines earnings and unemployment outcomes.

    “Hence, in line with the job competition theory, the relative schooling of the current generation of youngsters remains unaffected, such that the impact on earnings and unemployment might be lower than what can be expected based on pre-COVID research.”
6 of 6

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting definitions of the 90/10 ratio

    Minor

    The results section defines the ratio as the 90th-percentile score divided by the 10th-percentile score, while the table notes define it in the reverse direction.

  • Across-school table is described as containing standard deviations

    Minor

    The bottom panel of Table 2 reports Gini coefficients, 90/10 ratios and entropy, not standard deviations.

  • Wrong appendix-table reference for the time-trend analysis

    Minor

    The cited Table B8 lists numbers of test questions; the inequality analysis with a time trend is labeled Table B7.

  • Mathematics result is not significant at 5% in every subsample

    Minor

    The 2015-2020 mathematics estimate in Table 1 has one significance star, which the paper defines as p < 0.10; it does not have the two stars denoting p < 0.05.

  • Wrong appendix-table reference for excluding special-needs schools

    Minor

    The exclusion results are labeled Table B6, whereas Table B7 contains inequality regressions with a time trend.

  • Social-sciences conclusion denies a reported significant decrease

    Minor

    The conclusion says social-sciences scores did not decrease significantly, but the paper explicitly calls the 2017-2020 effect significant at the 10% level.

6 of 6

Cites

Flagged first, then by citations

Research works printed but not linked: 10

  1. Baert D., 2020, Weer naar school: Welke leerlingen, hoeveel dagen, is er opvang en wat met de veiligheid?
  2. Blainey K., 2020, The impact of lockdown on children’s education: A nationwide analysis
  3. Engzell P. Frey A.&Verhagen M.(2020)Learning inequality during the COVID‐19 pandemic paper presented at theJoint IZA & Jacobs Center Workshop: Consequences of COVID‐19 for Child and Youth Development[online] 19 October.
  4. Frenette M., 2020, School closures and the online preparedness of children during the COVID‐19 pandemic, .
  5. Grymonprez S., 2020, Jinnih Beels: 'Basisscholen bereiken 20 tot 30 procent van leerlingen niet’, De Standaard
  6. Heymans P. J., 2018, Eindrapport MICTIVO2018. Monitor ICT‐integratie in het Vlaamse onderwijs. Eindrapport van O&O‐opdracht: Meting ICT‐integratie in het Vlaamse onderwijs (MICTIVO)
  7. Inspectie van het Onderwijs, 2020, COVID‐19‐monitor Inspectie van het Onderwijs: Wat deden scholen en instellingen, in de periode van schoolsluiting tot aan 23 april, om het onderwijs zo goed mogelijk te continueren?
  8. OECD, 2017, PISA 2015 results (vol. III): Students’ well‐being
  9. Slavin R. E., 2020, How much have students lost in the COVID‐19 shutdowns?
  10. Vlaamse Onderwijsraad, 2018, Gevalideerde toetsen einde basisonderwijs: hefboom voor kwaliteit?

Cited by

Sorted by citations

Related

Authors
Maldonado, J. E., & De Witte, K.
Affiliations
  • KU Leuven (BE)
  • KU Leuven Belgium
Corresponding author
Maldonado, Joana Elisa
ORCID
Venue
British Educational Research Journal
Publisher
Wiley
Published
2021-07-21
Volume
48
Issue
1
Pages
49-94
ISSN
  • 0141-1926 (ISSN-L)
  • 1469-3518
DOI
10.1002/berj.3754
OpenAlex
W3186949619
Cited by
499
References
27
Citations by year
2026: 39 · 2025: 59 · 2024: 108 · 2023: 111 · 2022: 98 · 2021: 77 · 2020: 7
FWCI
150.31
Type
journal article
Language
en
Open access
Licence
onlinelibrary.wiley.com/termsAndConditions#vor (vor)
Locations
  • British Educational Research Journal (publishedVersion)
  • Lirias (KU Leuven) (submittedVersion, other-oa)
  • RePEc: Research Papers in Economics (submittedVersion)
  • Research Publications (Maastricht University) (submittedVersion, other-oa)
  • Lirias (KU Leuven) (submittedVersion)
Funders
Fonds Wetenschappelijk Onderzoek (S000617N)
Topics
  • Social Sciences › Education › Education during COVID-19 pandemic
  • Social Sciences › Education › School Choice and Performance
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › COVID-19 Pandemic Impacts
Keywords
school closures, COVID-19 pandemic, learning loss, standardized testing, primary education, educational inequality, mathematics achievement, school characteristics, disadvantaged students, Flemish education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review