The impact of Covid-19 on student achievement: Evidence from a recent meta-analysis

“Next, we compute the I 2 statistic to assess the heterogeneity of the results across studies (Higgins et al., 2003).”

“2, one would have expected Covid-19 school closures to have had a more negative impact on the achievement of younger students than older students, this effect could have been offset by the greater support in terms of parental involvement received by the former group of students during online learning. Bubb and Jones (…”

“(2021), the risk of bias in nonrandomized studies was assessed in 38(10) studies using the Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I) tool (Sterne et al., 2016).”

“G. Di Pietro clusters (i.e., 39), following Cameron and Miller (2015) we apply the correction for small of clusters by employing wild score bootstrapping (Kline & Santos, 2012).”

“In addition to problems in accessing appropriate technology (computers, reliable internet connection, etc.), not all students had a home environment free of disturbances and distractions, hence conducive to learning (Pokhrel & Chhetri, 2021).”

“15 This is calculated using the method-of-moments estimator provided in Hedges et al. (2010).”

“Finally, studies using a quasi-experimental or experimental design (e.g., Engzell et al., 2021) move closer to a causal interpretation of the relationship between Covid-19 and student performance. g) the publication year .”

“Publication bias has long been identified as a major problem in meta-analysis (Dwan et al., 2008).”

“LFK index values within ±1 suggest no asymmetry, LFK index values exceeding ±1 but within ±2 indicate minor asymmetry, while LFK index values exceeding ±2 denote major asymmetry (Furuya-Kanamori et al., 2018).”

“Moreover, there is evidence showing that Covid-19 and the related containment measures have had a detrimental effect on children ’ s wellbeing (Xie et al., 2020).”

“This finding is broadly consistent with that obtained by Hill et al. (2008) who conclude that a value of Cohen ’ s d of 0.4 (with a margin of error of ± 0.06) corresponds to the average annual reading achievement gains in fourth grade.”

“The appropriateness of the RE model is confirmed as I2 has a value of 100%.14 This suggests that all the variability in the effect-size estimates is due to heterogeneity as opposed to sampling error. Additionally, we also look at τ2 (between-study variance),15 which denotes the variability in the underlying true effec…”

“As observed by Stanley and Doucouliagos (2012), this is a more relevant problem in meta-analysis than in primary econometric studies given the higher risk of exhausting degrees of freedom in the former than in the latter.”

“Following school re-opening, teachers faced new requirements (e.g., hybrid teaching, more administrative tasks) that added to their already full workloads prior to Covid-19 (Pressley, 2022).”

“(2023) and Hammerstein et al. (2021), the risk of bias in nonrandomized studies was assessed in 3811 studies using the Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I) tool (Sterne et al.”

“Consistent with the findings of Betth ¨ auser et al. (2023), our results suggest that pandemic-related learning delays are similar across primary and secondary school students.”

“…as they use qualitative approaches (e.g., interviews), report teachers ’ /parents ’ views about the educational impact of Covid-19 (e.g., Kim et al., 2022; Lupas et al., 2021), or provide a theoretical discussion about how the pandemic is likely to affect education (e.g., Di Pietro et al., 2020).”

“Many studies (e.g., Bielinski et al., 2021; Borgonovi & Ferrara, 2022; Feng et al., 2021; Gambi & De Witte, 2021; Maldonado & De Witte, 2022) estimated the effect of Covid-19 on student performance in several subjects.”

“The reviews by Patrinos et al. (2022), Moscoviz and Evans (2022), Donnelly and Patrinos (2022), Hammerstein et al. (2021) and Zierer (2021) summarised the results of 35 studies, 29 studies, 8 studies, 11 studies and 9 studies, respectively.”

“Second, following Hanushek and Woessmann (2020), a learning deficit of about 0.186 standard deviations can be considered to be equivalent to the loss of just over half of a school year.”

“, 39), suggesting that the pvalue for the associated t-test accurately reflects the type I error (Tanner-Smith et al., 2016).”

“Tomasik et al. (2021) argued that the former group are more likely to have difficulties in coping with socio-emotional stressors associated with the pandemic.”

“In such a model, the intercept can be interpreted as the precision-weighted mean effect size adjusted for effect-size dependence (Friese et al., 2017).”

“Specifically, following the approach of Kaiser and Menkhoff (2020), we estimate the mean of the distribution of true effects using a RE meta-analysis based on a Robust Variance Estimation (RVE).”

“Kaffenberger (2021) estimates that if learning in grade 3 is reduced by one-third, the equivalent of about a three-month school closure, learning levels in grade 10 would be a full year lower.”

“Promoting the development of metacognition skills is, for instance, a powerful way to enhance student learning and performance (Stanton et al., 2021).”

“For instance, Havr ´ anek et al. (2020) indicate that researchers conducting meta-regression analysis in economics should consider data types.”

“Additionally, non-journal articles are more likely to contain typos in their regression tables (Cazachevici et al., 2020).”

“In the same vein, we removed studies using student self-reported outcome measures as well as those examining the educational impact of Covid-19 on specific subgroups of students (e.g., Agostinelli et al., 2022).”

“However, a fundamental issue with this approach lies in the dependence between multiple estimates from the same study given that effect sizes are assumed to be independent in meta-analysis (Cheung & Vijayakumar, 2016).”

“Similarly, a large number of studies (e.g., Ardington et al., 2021; Contini et al., 2021; Domingue et al., 2021; Gore et al., 2021) examined the impact of the pandemic on the achievement of students of different levels of education or even of students of different grades within the same level of education.”

“For instance, Sacerdote (2012) found that in the spring of 2006 students who were displaced by Katrina and Rita hurricanes saw their test scores fall by between 0.07 and 0.2 standard deviations.”

“Similarly, in the longitudinal data used by Ardington et al., 2021 attrition is significantly higher for the Covid-19 group and attrition is associated with poorer pre-pandemic reading proficiency levels. In Kuhfeld et al. (2022), between fall 2019 and fall 2021, the number of students testing in a grade dropped signi…”

“K ¨ onig and Frey (2022) as well as Betth ¨ auser et al. (2023) G. Di Pietro conclude that the presence of publication bias is”

“…towards online learning could have had a detrimental impact on the knowledge and skills of those students, mainly at secondary and tertiary levels, whose curriculum includes experiential learning experiences (e.g., field trips, hands-on activities) that cannot take place virtually (Tang, 2022).”

“The reviews by Patrinos et al. (2022), Moscoviz and Evans (2022), Donnelly and Patrinos (2022), Hammerstein et al. (2021) and Zierer (2021) summarised the results of 35 studies, 29 studies, 8 studies, 11 studies and 9 studies, respectively.”

“7 On the other hand, studies employing a correlational research design (e.g., Ludewig et al., 2022) attempt to isolate the effect of Covid-19 from that associated with other factors that could influence student achievement, but their results cannot be given a causal interpretation.”

“Mamolo (2022) found that the students ’ math motivation and self-efficacy decreased during the pandemic.”

“The reviews by Patrinos et al. (2022), Moscoviz and Evans (2022), Donnelly and Patrinos (2022), Hammerstein et al. (2021) and Zierer (2021) summarised the results of 35 studies, 29 studies, 8 studies, 11 studies and 9 studies, respectively.”

“Several studies from various countries (e.g., Bratti et al., 2004; Dabalen et al., 2001; Koda & Yuki, 2013) show that the final grade awarded to students successfully completing university is an important predictor of their labour market prospects.”

“For instance, Meeter (2021) analysed how Covid-19 affected the math performance of primary school children of grades 2 – 6.”

“During school closures, these students are less likely to have had access to a computer, an internet connection, and a space conducive to learning (Blask ´ o et al., 2022; Di Pietro et al., 2020).”

“Several studies from various countries (e.g., Bratti et al., 2004; Dabalen et al., 2001; Koda & Yuki, 2013) show that the final grade awarded to students successfully completing university is an important predictor of their labour market prospects.”

“Selecting incorrect variables leads to misspecification bias and invalid inference (Xue et al., 2021).”

“Using data from a survey addressed to math lecturers between May and June 2020, Ní Fhloinn and Fitzmaurice (2021) found that most of the respondents agreed that it is harder to teach math remotely.”

“Similarly, a large number of studies (e.g., Ardington et al., 2021; Contini et al., 2021; Gore et al., 2021; Domingue et al., 2021) examined the impact of the pandemic on the achievement of students of different levels of education or even of students of different grades within the same level of education.”

“Panaoura (2020) looked at parents ’ perception of how they have helped their children in math learning during the pandemic in Cyprus.”

“Using data on Chinese primary school students during Covid-19, Wang et al. (2022) concluded that parental involvement had a positive impact on children ’ s achievement in Chinese and English, but not in math.”

“Each coefficient is also given a Posterior Inclusion Probability (PIP), which is the sum of posterior model probabilities of the models including the relevant variable and indicates how likely such a variable is to be contained in the true model (Havr ´ anek et al., 2018).”

“As noted by Hammerstein et al. (2021), information on this domain is very rarely included in the relevant studies because Covid-19 school closures were not intended interventions.”

“For instance, Andrew et al. (2020) found that in the UK during the first Covid-19 lockdown while the proportion of primary school students reporting not to have a designated space to study at home was about 20%, the corresponding figure for secondary school students was approximately 10%.”

“Many studies (e.g., Bielinski et al., 2021; Borgonovi & Ferrara, 2022; Feng et al., 2021; Gambi & De Witte, 2021; Maldonado & De Witte, 2022) estimated the effect of Covid-19 on student performance in several subjects.”

“Its large value of 1.74 further corroborates the hypothesis of substantial heterogeneity of the effect sizes (Takase & Yoshida, 2021).”

“Longer periods of social isolation might have adversely affected students ’ mental health (e.g., anxiety and depression) and physical activity (Vaillancourt et al., 2021).”

“Finally, consistent with the approach taken in several studies (e.g., de Linde Leonard & Stanley, 2020), i) the effect size ’ s standard error is also included among our moderator variables.”

“This is also likely to have contributed to negatively impact their academic performance given the close association between mental and physical health and educational outcomes (Joe et al., 2009).”

“Finally, some studies (Domínguez-Álvarez et al., 2020; Gómez-Becerra et al., 2020) showed that during Covid-19 younger children present more emotional problems than older children.”

“In line with the approach followed by several recent meta-analyses (see, for instance, Di Pietro, 2022), we consider the year of the first appearance of a draft of the study in Google Scholar.”

“Furthermore, parents may be familiar with math methods different from the ones used by teachers (Shanley, 2016).”

“G. Di Pietro clusters (i.e., 39), following Cameron and Miller (2015) we apply the correction for small of clusters by employing wild score bootstrapping (Kline & Santos, 2012).”

“This measure is preferred to publication year on the ground that journals significantly differ with respect to the time between online availability date of an article and the date when the article is given a volume and issue number 8 (Al & Soydal, 2017).”

“Studies using a descriptive research design (e.g., Moliner & Alegre, 2022) provide information about the average gap in test scores between the Covid-19 and non-Covid-19 cohorts without accounting for differences between these two cohorts (for example in terms of individual characteristics such as…”

“In line with the current best practice in meta-analysis (Pol ´ ak, 2019), we use all relevant estimates included in the selected studies.”

“Several studies from various countries (e.g., Bratti et al., 2004; Dabalen et al., 2001; Koda & Yuki, 2013) show that the final grade awarded to students successfully completing university is an important predictor of their labour market prospects.”

“The reviews by Patrinos et al. (2022), Moscoviz and Evans (2022), Donnelly and Patrinos (2022), Hammerstein et al. (2021) and Zierer (2021) summarised the results of 35 studies, 29 studies, 8 studies, 11 studies and 9 studies, respectively.”

“…as they use qualitative approaches (e.g., interviews), report teachers ’ /parents ’ views about the educational impact of Covid-19 (e.g., Kim et al., 2022; Lupas et al., 2021), or provide a theoretical discussion about how the pandemic is likely to affect education (e.g., Di Pietro et al., 2020).”

“It controls for time-trend effects (Schütt, 2021).”

“Moreover, as argued by Ariyo et al. (2022), one would expect children of less educated parents to have received less parental support while learning at home than children of more educated parents.”

“Not only do students need to learn and accustom themselves to use technology in order to write mathematical symbols, but this is not always possible in online platforms such as chats (Mullen et al., 2021).”

“More in general, moderators are situational variables as well as characteristics of studies that might influence the effect estimate (Judd, 2015).”

“Similarly, Mendoza et al. (2021) and Arnal-Palaci ´ an et al. (2022) provided evidence about higher levels of math anxiety experienced by university and primary school students, respectively, during Covid-19.”

“Di Pietro (2023) conducted a meta-analysis of 38 observational studies (26 cross-sectional and 12 longitudinal) and one experimental study between 2020 and 2022 in 12 European countries, Australia, Brazil, China, Egypt, Mexico, South Africa, and the United States (six of which were rated to have a…”

“1 Substantial variation in learning deficits can be observed across individual studies and countries, as demonstrated by two recent meta-analyses (Betthäuser et al., 2023; Di Pietro, 2023).”

“This fits with the recent literature showing that social policies and economic conditions have been favourable to increased access and resources for education in these countries (Di Pietro, 2023; Tadesse & Muluye, 2020).”