School closure policies and student reading achievement: evidence across countries

In all three models, longer school closures are significantly associated with larger declines in average student reading performance.

“In all models, the estimates tell a similar story: longer school closures are significantly associated with larger declines in average student reading performance.”

The negative effect of school closures on student reading achievement remains after controlling for pandemic severity measures such as infection rates, vaccination policies, and lockdown stringency.

“This school closure effect remains even after controlling for measures of pandemic severity such as infection rates, vaccination policies, and a measure of lockdown stringency.”

Country-specific slope estimates also show a negative relationship between school closure duration and reading achievement.

“A similar exercise can be done where country-specific slopes are estimated which we present in the Appendix in Figs. 2 and 3. Results produce similar findings: a negative relationship between school closure duration and reading achievement.”

The negative effect of school closures on reading achievement is more pronounced for socioeconomically disadvantaged students.

“This effect is even more pronounced for socioeconomically disadvantaged students and those without home computer access.”

School closures have a significant and substantial negative effect on student reading achievement.

“We observe a significant and substantial negative effect of school closures on student reading achievement.”

The negative effect of school closures on reading achievement is more pronounced for students without home computer access.

“This effect is even more pronounced for socioeconomically disadvantaged students and those without home computer access.”

Student performance declined following school closures.

“The observed decline in student performance following school closures carries profound implications.”

The correlation between school closure duration and changes in average student reading achievement between PIRLS 2016 and 2021 is moderate (r = -0.44).

“The correlation between the two measures is moderate (r = -0.44).”

Using the COVID-19 severity measures as instruments, the estimated effect of school closures is -0.15.

“In addition, similar to Patrinos (2023), we use these measures as instruments, and estimate an effect of school closures as -0.15 (see first two columns of Table 5 for these results).”

• cites Patrinos (2023) sourceNot checked yet

  Not checked in the free run.

The effect of school closures on reading achievement is moderated by home resources: the interaction between school closure duration and the Home Resources for Learning scale is positive (0.021, SE 0.008).

“| Closed x Home Resources for Learning | 0.021* | (0.008) |”

An additional day of school closures is associated with a 0.14 PIRLS scale point decline in student reading performance.

“We estimate that an additional day of school closures is associated with a 0.14 PIRLS scale point decline in student reading performance.”

The only two coefficients that are significantly different from each other are the Country Trends and Global Trend model estimates.

“While the magnitude of each estimate varies, the only two coefficients that are significantly different from each other are the Country Trends and Global Trend model estimates.”

The estimated effect of one day of school closures on average student reading achievement ranges from -0.12 in the Country Trends model to -0.14 in the 16-21 model to -0.17 in the Global Trend model.

“The effect sizes range from -0.12 in the Country Trends model to -0.14 in the 16-21 model to -0.17 in the Global Trend model.”

Exclusion rates did not change much across cycles, with an average increase of about half a percentage point.

“Exclusion rates, which measures the percentage of schools and students from the target population that were excluded prior to sampling, did not change much across cycles (on average, about a half a percentage point increase).”

Egypt stands out as an outlier in the plot as it showed by far the largest positive change in average reading achievement.

“Egypt stands out as an outlier in the plot as it showed by far the largest positive change in average reading achievement.”

Excluding eight federal countries, the results remained consistent.

“Again, the results remained consistent (third column in Table 6).”

Including country-level measures of COVID-19 health impacts (case rates per capita and death rates per capita) did not change the overall estimates.

“We attempt to address part of this issue by including country-level measures of COVID-19 health impacts (i.e., case rates per capita and death rates per capita) to separate out the different effects and do not observe a change in the overall estimates.”

On average, countries in the sample closed schools for 110 days.

“On average, countries in our sample closed schools for 110 days.”

Including five COVID-19 severity measures as controls in the main model does not change the main takeaways (-0.15 versus -0.14).

“The inclusion of these five measures as controls into our main model does not change our main takeaways (-0.15 versus -0.14).”

The correlation between school closure duration and deviations from country-specific trends is moderate (r = -0.55).

“The correlation is moderate (r = -0.55)”

Excluding the five countries where PIRLS 2021 was postponed by one year, the main analysis results are qualitatively the same as for all 29 countries.

“The results of the main analysis are qualitatively the same as they are for all 29 countries (the second column in Table 6).”

Students without a computer in the home had a significantly more negative effect of school closures (-0.09 more negative) than those with a computer.

“In examining how the effect of school closures differs for students with and without a computer, we find that students without a computer in the home had a significantly more negative effect of school closures (-0.09 more negative) than those with a computer.”

There are questions about the reliability of the results in Egypt as between 15 and 25% of students had achievement too low for estimation.

“However, there are some questions about the reliability of the results in Egypt as between 15 and 25% of students had achievement too low for estimation(Mullis et al., 2023).”

• cites Mullis et al., 2023Not checked yet

  Not checked in the free run.

The estimated impact of one day of school closures on a student with socioeconomic status one standard deviation above the average is -0.104.

“Conversely, for a student with a socioeconomic status one standard deviation above the average, the estimated impact is -0.104 (-0.146 + 2 * 0.021).”

Removing Egypt from the sample leads to a slightly stronger correlation (r = -0.56).

“Removing Egypt from the sample leads to a slightly stronger correlation (r = -0.56)”

The estimated impact of one day of school closures on a student with socioeconomic status one standard deviation below the average is -0.186.

“Based on the analyses, this indicates that the impact of one day of school closures on a student with a socioeconomic status one standard deviation below the average is estimated to be -0.186(-0.146 -2 * 0.021).”

Using alternative weighting approaches for partial school closures (weights of 0 and 1), the main result remained robust.

“Using alternative measures of our main explanatory variable school closure revealed some impact on our analyses but the main result remained robust across these different weighting methods, as shown in Table 7.”

Patrinos (2023) estimates that a week of school closures leads to a decline of almost 1% of a standard deviation.

“He estimates that a week of school closures leads to a decline of almost 1% of a standard deviation.”

• cites Patrinos, 2023 sourceNot checked yet

  Not checked in the free run.

PIRLS provides internationally comparable data on reading achievement from five cycles spanning from 2001 to 2021.

“The first data source is the Progress in International Reading Literacy Study (PIRLS), which provides internationally comparable data on reading achievement from five cycles spanning from 2001 to 2021.”

Jack et al. (2022) found that school districts with full in-person learning had significantly smaller declines in standardized test pass rates than those utilizing remote learning strategies.

“Jack et al. (2022) study this district-level variation in 11 states and find that school districts with full inperson learning (i.e., no school closures or hybrid learning) had significantly smaller declines in standardized test pass rates than those utilizing remote learning strategies.”

• cites Jack et al., 2022Not checked yet

  Not checked in the free run.

The duration of school closures varied greatly internationally.

“The duration of school closures varied greatly internationally.”

• Also stated in: § 2.2 International variation in duration of school closures · plain

  “According to data collected from over 200 countries by UIS, the length of school closures (full or partial) varied between 0 and 93 weeks (UNESCO, 2022).”

• cites UNESCO, 2022Not checked yet

  Not checked in the free run.

The length of school closures (full or partial) varied between 0 and 93 weeks across over 200 countries.

“According to data collected from over 200 countries by UIS, the length of school closures (full or partial) varied between 0 and 93 weeks (UNESCO, 2022).”

• cites UNESCO, 2022Not checked yet

  Not checked in the free run.

Previous studies examining the impact of COVID-19 on education have relied mainly on national data.

“This knowledge gap arises from a limitation of prior research, as previous studies examining the impact of COVID-19 on education have relied mainly on national data.”

The factors driving across-country differences in learning deficits remain understudied.

“However, the factors driving these across-country differences remain understudied.”

These effect sizes are equivalent to approximately one-third to half a year's worth of learning.

“These effect sizes are equivalent to approximately one-third to half a year's worth of learning.”

PIRLS is the first international assessment to be administered after the onset of the pandemic.

“The Progress in International Reading Literacy Study (PIRLS, Mullis et al., 2023) provides a unique opportunity for this type of research as it is the first international assessment to be administered after the onset of the pandemic.”

• cites Mullis et al., 2023Not checked yet

  Not checked in the free run.

The differences in learning deficits between countries are striking.

“The differences in learning deficits between countries are striking, and further research is needed to understand why some countries have experienced greater deficits than others.”

• cites Betthäuser et al., 2023 sourceNot checked yet

  Not checked in the free run.

• cites Di Pietro, 2023Not checked yet

  Not checked in the free run.

The comparability of individual studies is limited due to differences in student populations, assessment domains, assessment instruments, and timing during the pandemic.

“However, it is also important to recognize that the comparability of individual studies is limited due to several factors.These include differences in student populations, assessment domains, and assessment instruments used, as well as variation in the timing of the studies during the pandemic.”

A year of school closures corresponds roughly to the loss of a little more than half a school year of learning.

“The estimated effect implies that a year of school closures corresponds roughly to the loss of a little more than half a school year of learning.”

Following the PIRLS study center, the authors removed countries with non-comparable data from the analysis to ensure the validity of their findings.

“Thus, following the PIRLS study center, we have removed these countries from our analysis to ensure the validity of our findings (Mullis et al., 2023).”

• cites Mullis et al., 2023Not checked yet

  Not checked in the free run.

The link between school closure policies and learning declines across countries remains unclear.

“As a result, the link between school closure policies and learning declines across countries remains unclear.”

The findings remain consistent with different weighting methods for school closure duration.

“However, our findings remain consistent with different weighting methods.”

Because most pandemic policies were implemented at the national level, there is often little variation within individual countries to evaluate the effects of these policies using national-level data.

“As most policies in response to the pandemic were implemented at the national level, there is often little variation within individual countries to evaluate the effects of these policies using national-level data.”

UIS data includes information on the length of school closures during the COVID-19 pandemic.

“This data includes information on the length of school closures during the COVID-19 pandemic.”

The true impact of school closure policies may be larger than the effects estimated in this study because other subjects (mathematics) were impacted more by the pandemic than reading.

“Evidence showing that other subjects (i.e., mathematics) were impacted more by the pandemic than reading makes it important to note that the true impact of the school closure policies may be larger than the effects estimated in this study (Betthäuser et al., 2023;Di Pietro, 2023).”

• cites Betthäuser et al., 2023 sourceNot checked yet

  Not checked in the free run.

• cites Di Pietro, 2023Not checked yet

  Not checked in the free run.

A decline in participation would likely lead to an underestimation of the school closure effect because non-participation is more likely among students hardest hit by the pandemic.

“It should be noted that a decline in participation would likely lead to an underestimation of the school closure effect given that non-participation is more likely among the students hardest hit by the pandemic (Werner & Woessmann, 2023).”

• cites Werner & Woessmann, 2023Not checked yet

  Not checked in the free run.

Learning deficits resulting from school closures can have lasting and long-term consequences for educational careers.

“suggesting that the learning deficits resulting from these closures can have lasting and long-term consequences for educational careers.”

The achievement gap between a student one standard deviation above and a student one standard deviation below the international average on the home resources for learning scale would grow by about 9 points in an average country (one that closed for 110 days).

“To put our results into context, our estimates would indicate that the achievement gap between a student one standard deviation above the international average on the home resources for learning scale and a student one standard deviation below would grow by about 9 points in an average country (i.e., one that closed for 110 days).”

The estimated effect implies that the average country's average student reading achievement declined about 15 points more than a country that did not close schools.

“To put our estimated effect size into context, this would imply that, the average country's average student reading achievement declined about 15 points more than a country that did not close schools.”

The measurement of school closures was conducted at the national level, which overlooks potential variations within countries.

“Third, our measurement of school closures was conducted at the national level, which overlooks potential variations within countries.”

The limited number of observations prevents accurately estimating quadratic or other non-linear trends with a fuller sample of countries.

“our limited number of observations prevents us from accurately estimating quadratic or other non-linear trends with a fuller sample of countries.”

• AssumptionThe authors state this as an assumption: The models used to analyze global and country-specific trends assume linearity.

Country-fixed effects control for time-invariant country factors that likely influence both student achievement and the decision to close schools.

“With the inclusion of multiple cycles of data, we are able to control for countryspecific time-invariant factors that likely influence both student achievement and their decision to close schools by including country-fixed effects.”

An increase of similar magnitude would also be observed between students with and without a computer in the home.

“An increase of a similar magnitude would also be observed between students with and without a computer in the home.”

Social inequalities were exacerbated by school closure policies.

“These findings suggest that there is evidence that social inequalities were exacerbated by school closure policies.”

• Also stated in: § 6.1 School closures are linked to declines in achievement and increased inequality · hedged

  “we observed evidence suggesting that social inequalities have been amplified due to school closure policies.”

A full year of school closures would entail a loss slightly above half (57%) of a school year's worth of learning.

“Considering that a standard school year comprises of around 180 days in various countries (OECD, 2021a), we can estimate that a full year of school closures would entail a loss slightly above half (57%) of a school year's worth of learning (calculation: 180 * 0.14/44 = 0.57).”

• cites OECD, 2021aNot checked yet

  Not checked in the free run.

• AssumptionThe authors state this as an assumption: An approximate improvement of 44 PIRLS scale points occurs over a typical school year, including both schooling and maturation effects.