Do test score gaps grow before, during, or between the school years?: Measurement artifacts and what we can know in spite of them

“Under a 3PL IRF, items differ not just in difficulty but in discrimination di and guessability ci (DeMars, 2010):”

“A classic question in the social and behavioral sciences is whether inequality in academic achievement comes primarily from inside or outside of schools (Coleman et al., 1966; Downey & Condron, 2016; Downey, von Hippel, & Broh, 2004; Jencks, 1972; Jennings, Deming, Jencks, Lopuch, & Schueler, 2015; Raudenbush & Eschma…”

“The entire summer learning literature reviewed in a 1996 meta-analysis was potentially vulnerable to artifacts related to scaling and changes of test form (Cooper et al. 1996).”

“The gap between children of high and low socioeconomic status (SES) has been reported to grow fastest during summer (Downey et al. 2004; Entwisle and Alexander 1992), but the gap between black and white children has been reported to grow fastest during school (Condron 2009; Downey et al.”

“The SES gap did not grow at all during the school years (Alexander et al. 2007a).”

“Other studies have reported that gaps grow dramatically between first and eighth grade, with practically all the growth occurring during summer vacations (Alexander et al. 2001; Hayes and Grether 1969, 1983).”

“But if score gaps grow fastest during summer vacations, then it would appear that, even after children start attending school, the major sources of inequality lie elsewhere (e.g., Alexander, Entwisle, & Olson, 2001; Downey et al., 2004; Hayes & Grether, 1969, 1983; Heyns, 1978; Murnane, 1975).”

“When the new scores were used, the timing and extent of gap growth was different than it was on the old scores, and results that had been obtained using the old score did not necessarily replicate with the new scores (Koretz, 2009; Quinn, 2015; Reardon, 2008).”

“Another difference between Thurstone and IRT models is that the Thurstone model assumes that ability is normally distributed (Thurstone, 1925; Yen et al., 1995b).”

“, von Hippel, P. T., & Hughes, M. (2008). Are “Failing” School Really Failing? Using Seasonal Comparisons to Evaluate School Effectiveness. Sociology of Education, 81(3), 242–270. Duncan, G. J., & Magnuson, K. (2011). The nature and impact of early achievement skills, attention skills, and behavior problems.”

“Despite its shortcomings, the number-right score was widely used in early longitudinal analyses of the ECLS-K (Condron, 2009; Downey et al., 2004; Downey, von Hippel, & Hughes, 2008; Fryer & Levitt, 2006; Reardon, 2003; von Hippel, 2009).”

“1093/aler/ahl003 Gershon, R. C. (2005). Computer Adaptive Testing. Journal of Applied Measurement, 6(1), 109– 127. Hanushek, E. A., & Rivkin, S. G. (2009). Harming the best: How schools affect the black-white achievement gap. Journal of Policy Analysis and Management, 28(3), 366–393. https://doi.org/10.1002/pam.20437…”

“And many studies attribute gaps and gap growth to schools (e.g., Condron, 2009; Hanushek & Rivkin, 2009; Heck, 2007).”

“When the CAT used Thurstone scaling, it appeared that the SD of ability was increasing with age, but when the CAT switched to a 3PL IRT model, it appeared that the SD of ability was constant or decreasing (Yen, 1986).”

“…classic question in the social and behavioral sciences is whether inequality in academic achievement comes primarily from inside or outside of schools (Coleman et al. 1966; Downey and Condron 2016; Downey, von Hippel, and Broh 2004; Jencks 1972; Jennings et al. 2015; Raudenbush and Eschmann 2015).”

“Likewise, the BSS finding that “a large portion of the [eighth-grade] achievement gap originates over the summer” (Alexander, Entwisle, and Olson 2007b) would be harder to support using an IRT-scaled test that showed little or no gap growth after children began school.”

“The P gap can be converted to a Z gap which represents the mean standardized difference that would exist between the scores of groups A and B if each group’s scores were normal and equally variable (Ho, 2009).”

“When the new scores were used, the timing and extent of gap growth was different than it was on the old scores, and results that had been obtained using the old score did not necessarily replicate with the new scores (Koretz, 2009; Quinn, 2015; Reardon, 2008).”

“They opined that “something’s awry in the state of test mark,” and argued that the old Thurstone scales were better than the new IRT scales (Clemans, 1993).”

“These arguments were refuted by psychometricians representing the CAT’s publisher, who argued that the new IRT scales were better than the old Thurstone scales (Yen et al. 1995a, 1995b).”

“These arguments were refuted by psychometricians representing the CAT’s publisher, who argued that the new IRT scales were better than the old Thurstone scales (Yen et al. 1995a, 1995b).”

“But defenders of the new CAT countered that IRT scales had better psychometric properties and could detect growth in the dispersion of students’ skills if growth were actually present (Clemans, 1995; Yen, Burket, & Fitzpatrick, 1995a, 1995b).”

“…the inconsistencies in the magnitude of summer loss, research using both recent data sources is consistent in failing to replicate the earlier finding of summer being the primary period in which socioeconomic inequalities widen (e.g., Kuhfeld, 2019; von Hippel, 2019; von Hippel & Hamrock, 2019).”

“Currently in the United States, there is a large and significant income achievement gap in reading skills when children enter kindergarten (Reardon, 2013; Von Hippel & Hamrock, 2016).”

“More recent studies, however, question not only the summer learning loss but also the social disparity effect (e.g., von Hippel & Hamrock, 2019) and argue with the methodological issues related to scaling, the use of different test forms, the choice between a regressor variable model or a change…”

“In terms of measuring growth, Von Hippel and Hamrock (2019) suggested that research on learning loss and gaps are subject to measurement artefacts.”

“Many recent studies refer to kindergarten children (e.g. Downey, Von Hippel, and Broh 2004 ; Von Hippel and Hamrock 2019 ), who are not our main focus here.”

“Further, recent research (von Hippel & Hamrock, 2019) points out flaws in the testing and scaling procedures used V101 N1 kappanonline.org 27 has been in school prior to testing.”

“…the Early Childhood Longitudinal Study: Kindergarten (ECLS-K) 1998–1999 and 2010–2011 cohorts to study SLL (Benson & Borman, 2010; Burkam et al., 2004; Downey et al., 2004; Downey et al., 2008; Quinn, 2014; Quinn et al., 2016; Quinn & Le, 2018; von Hippel et al., 2018; von Hippel & Hamrock, 2019).”

“Research shows that children from under-resourced communities are likely to have slower language development than their middle-and upper-SES peers (Durham, Farkas, Hammer, Tomblin, & Catts, 2007; Golinko ff et al., 2019; Hart & Risley, 1995; Reardon, 2013; von Hippel & Hamrock, 2016).”

“Von Hippel and Hamrock (2019) demonstrated empirically that conclusions about the growth of gaps are sensitive to whether one analyzes (an estimate of) the number of correct answers or an ability estimate from an item response theory model.”

“They have also found much higher variability in summer learning gains across students than during school years, which can contribute to widening race/ethnicity and socioeconomic achievement disparities in later school years (von Hippel & Hamrock, 2019).”

“…since socioeconomic and genetic differences in cognitive skills show up well before school age (Belsky et al. 2016; Ermisch 2008; Feinstein 2003; von Hippel and Hamrock 2019) and learning during early childhood is crucial for later achievement (Cunha et al. 2006), we examine whether children’s…”

“There is a puzzle in our results, one that von Hippel and Hamrock (2019) also observed.”

“These early studies drew attention to the influence of summer learning and were generative to the field; however, their focus was often limited to specific school districts or grade levels, and many used test scores that were not vertically linked (von Hippel & Hamrock, 2019).”

“While the magnitude of academic gains actually lost during the summer is debatable, as is the idea that summer loss is a primary factor in long-term achievement gaps, having access to educational experiences during the summer is meaningful (von Hippel & Hamrock, 2019).”

“Achievement gaps by socioeconomic status appear to widen during the summer, whereas gaps between White and Black students appear to primarily widen during the school year (von Hippel & Hamrock, 2019).”

“Without reliable estimates of the magnitude of learning loss for students with learning difficulties, some have turned to summer break studies for insight into the magnitude of learning loss during the pandemic (Alexander et al., 2007; Cooper et al., 1996; von Hippel & Hamrock, 2019).”

“Achievement gaps between poor and middle-class children open primarily during the first five years of life, when children’s contact with their families is greatest and schools have not yet had a chance to matter (von Hippel and Hamrock 2019; von Hippel, Workman, and Downey 2018).”

“This could be explained by two reasons: (1) a measurement effect, such that learning losses are over-estimated when measured immediately after a break (as in the summer learning loss, (von Hippel and Hamrock 2019; Von Hippel 2019)); (2) an actual catch-up following the reopening of schools.”

“Another measurement issue is test score scaling (von Hippel 2019; von Hippel and Hamrock 2019).”

“11 von Hippel and Hamrock (2019), find that cognitive losses deriving from summer vacations are reversed after variable lengths of time once back at school.”

“In this example, the researchers argued that the IRT scale measures students’ skills better than the number-right scale (von Hippel & Hamrock, 2019).”

“As compared with earlier work, recent studies have drawn more mixed conclusions regarding the seasonal dynamics of achievement inequality (Atteberry & McEachin, 2016; Quinn, Cooc, McIntyre, & Gomez, 2016; von Hippel & Hamrock, 2016).”

“The test scores in Entwisle and Alexander’s (1992, 1994) influential studies lacked this property, as did those in the 1998 ECLS-K cohort (for a discussion of this problem, see von Hippel and Hamrock 2019).”

“…evolve nonlinearly over the course of the year (e.g., Malmberg et al., 2010), yet many policy-relevant metrics like summer learning loss estimates are dependent on the assumption of within-year linearity (Downey et al., 2004; Quinn et al., 2016; von Hippel et al., 2018; von Hippel & Hamrock, 2019).”

“Regarding socioeconomic status (SES), von Hippel and Hamrock (2019) found that the reading gap between students in the top and bottom quintiles of SES shrank by 25% between the Fall of kindergarten and the Spring of second grade.”

“Achievement gaps are relatively stable over time (Morgan et al., 2009; von Hippel & Hamrock, 2019).”

“These studies find, at most, small associations of summer achievement loss with poverty or mothers’ education level (Hippel & Hamrock, 2019; Kuhfeld, 2019).”

“…to assess bias in ability group placement, because even at the start of kindergarten, reading and math skills are strongly correlated with SES and race/ethnicity, and reading skills are mildly correlated with gender (Duncan & Magnuson, 2011; von Hippel et al., 2018; von Hippel & Hamrock, 2019).”

“We examine academic growth separately for when school is in versus out of session, which is referred to as seasonal learning in prior literature (e.g., von Hippel & Hamrock, 2019).”

“…Study (ECLS), show that Black children, Hispanic children, and children from families with lower incomes enter kindergarten with substantially lower reading skills than do their White and higher-income peers (Fryer & Levitt, 2004; Quinn, 2015; Reardon & Portilla, 2016; von Hippel & Hamrock, 2019).”

“…nascent skills, it may seem unlikely that first graders could be tested with measures of written expression, but previous studies at this grade level successfully have included writing prompts (Berninger & Fuller, 1992; Coker et al., 2018; Kim et al., 2014; Kim et al., 2015; McMaster et al., 2009).”

“This issue of extending piecewise models to longer timeframes is corroborated by von Hippel and Hamrock (2019), who fit a piecewise model to all time points from the MAP assessment data.”

“…question whether modern data actually shows low‐income students to exhibit summer learning loss at any greater magnitude than higher‐income students (von Hippel and Hamrock, 2019; Von Hippel, 2019), and introduces the possibility that students whose achievement grows the most during the year…”

“However, schooling broadly — and classroom instruction speci fi cally — is often designed around grade-level standards (Polikoff, 2021), which do not account for the variability in learning readiness (Ansari, 2018; Engel et al., 2013; von Hippel & Hamrock, 2019).”

“…pointing to a Matthew effect that not only perpetuates itself but might also transfer to additional areas, putting at risk the development of other essential skills (Heckman, 2006; Lee & Burkam, 2002; Perfetti, 1988; Stanovich, 1988; von Hippel & Hamrock, 2019), 21st-century skills among them.”

“Examples of these critiques include the use of nonlinked test forms between spring and fall of adjacent grade levels in early SLL research (von Hippel & Hamrock, 2019), the role of teachers’ coaching to potentially boost spring performance and depress fall test scores (Slavin et al., 1989), and…”

“In contrast, gaps remained relatively constant among children of native-born parents throughout kindergarten to the end of 11th grade (see Figure 2A and B), which is consistent with the existing literature (e.g., Magnuson, Waldfogel and Washbrook 2012; von Hippel and Hamrock 2019).”

“Seasonal patterns of learning, where gains during the school year are followed by flattening or dropping of test scores during the summer, have been observed across a range of datasets (von Hippel & Hamrock, 2019).”

“…children as they move through school can shed light on whether schooling experiences exacerbate or ameliorate achievement gaps, though much has been written about the difficulties in getting these estimates right (Quinn, 2015; Quinn & McIntyre, 2017; Reardon, 2008; von Hippel & Hamrock, 2019).”

“Schools have been found to increase the Black-White gap (Kuhfeld et al. 2021; Quinn et al. 2016; von Hippel et al. 2018; von Hippel and Hamrock 2019), but findings for gaps between Hispanic and White students are inconsistent (von Hippel et al. 2018).”

“Variations in patterns of learning by grade level and season are well-documented in recent research (Kuhfeld, 2019; von Hippel & Hamrock, 2019).”

“In both the ECLS-K and NWEA datasets, scholars find that Black/White gaps grow faster during school periods than summers (Downey et al., 2004; von Hippel & Hamrock, 2019; von Hippel et al., 2018; Kuhfeld et al., 2020), although Workman et al. (2023) fail to replicate this pattern.”

“In contrast, the IRT theta scores used in this study are approximately interval scaled, which is more suitable to measure change over time (for more information about the use of ECLS-K scores, see Von Hippel & Hamrock, 2019).”

“As a result, children’s language and literacy skills can vary widely at kindergarten-level entry (von Hippel & Hamrock, 2019).”

“Furthermore, remediation of literacy skills requires signi fi cantly more time and resources compared with prevention of literacy dif fi culties (Torgesen, 2000 ; Torgesen et al., 2001 ; Vaughn & Wanzek, 2014 ; Vaughn et al., 2011 ; von Hippel & Hamrock, 2019 ).”

“This notably involved absenteeism (Santiba ~ nez and Guarino, 2021), holidays (von Hippel and Hamrock, 2019) or strikes (Jaume and Will (cid:1) en, 2019).”

“However, other works, including research by von Hippel and Hamrock (2019), have found that different gaps expand during different seasons and to varying degrees depending on data sources and measures of achievement.”

“See also, Von Hippel and Hamrock (2019).”

“…three data sets contains the same vertically scaled assessments, all of which may differ from one another in terms of scaling, content covered, or intended usage (see von Hippel & Hamrock, 2019, for a particularly relevant discussion of how measurement artifacts can affect inferences about SLL).”

“Furthermore, the results show that significant SES gaps already exist before children enter the school system, and these differences tend to remain relatively stable throughout the educational journey (Skopek & Passaretta, 2021; von Hippel & Hamrock, 2019).”

“Even if there is no full consensus on the size of this loss or on which type of students would be more affected (Atteberry & McEachin, 2021; von Hippel & Hamrock, 2019; Workman et al., 2023), a widespread concern exists regarding the risk that summer learning loss contributes to widening…”

“Such initiatives, therefore, have the potential to reduce or prevent gaps from emerging before children start formal schooling (Von Hippel & Hamrock, 2019).”

“Furthermore, it has been postulated that the magnitude of observed decreases in academic performance after summer vacation may be exaggerated by measurement choices such as grade - scaling and differences in test forms between prevacation and postvacation assessments ( 3 , 8 ).”