Performance Trajectories and Performance Gaps as Achievement Effect-Size Benchmarks for Educational Interventions
The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
Two complementary approaches to developing empirical benchmarks for achievement effect sizes in educational interventions are explored. The first approach characterizes the natural developmental progress in achievement made by students from one year to the next as effect sizes. Data for seven nationally standardized achievement tests show large annual gains in the early elementary grades followed by gradually declining gains in later grades. A given intervention effect will therefore look quite different when compared to the annual progress for different grade levels. The second approach explores achievement gaps for policy-relevant subgroups of students or schools. Data from national- and district-level achievement tests show that, when represented as effect sizes, student gaps are relatively small for gender and much larger for economic disadvantage and race/ethnicity. For schools, the differences between weak schools and average schools are surprisingly modest when expressed as student-level effect sizes. A given intervention effect viewed in terms of its potential for closing one of these performance gaps will therefore look very different depending on which gap is considered.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Under normal circumstances, students generally improve their performance by around 0.4 standard deviations per school year [10][11][12] .”
From A systematic review and meta-analysis of the evidence on learning during the COVID-19 pandemic (2023) · cites this paper as [11]
In the citing paper
“Typical estimates of yearly progress for primary school range between 0.30 and 0.60 SD (45).”
From Learning loss due to school closures during the COVID-19 pandemic (2021) · cites this paper as [45]
In the citing paper
“At the same time, some studies indicate a progress of up to 0.80 SD annually at the low extreme of our age range (45, 47)”
From Learning loss due to school closures during the COVID-19 pandemic (2021) · cites this paper as [45]
In the citing paper
“Bloom et al. (2008) demonstrate that, by Grade 5, student achievement improves by about 0.4 standard deviations over the course of an academic year, suggesting that the COVID-19 effect is larger than what could be expected from the loss of instruction time at school.”
From The effect of school closures on standardised student test outcomes (2022) · cites this paper as Bloom et al. (2008)
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
Philosophical transactions of the Royal Society of London. B
Press, H. (1886) · Cited by 10,080
(2000) · Cited by 9,198
Self-citationFisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
Meta-Analytic Procedures for Social Research
Rosenthal, R. (1991) · doi:10.4135/9781412984997 · Cited by 4,635
Department of Health, Education, and Welfare
Cooper, T. (1976) · Military Medicine · doi:10.1093/milmed/141.8.521 · Cited by 3,993
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,642
Statistical Methods for Meta-Analysis
(1985) · Elsevier eBooks · doi:10.1016/c2009-0-03396-0 · Cited by 2,521
IX. On the problem of the most efficient tests of statistical hypotheses
Neyman, J., & Pearson, E. S. (1933) · Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character · doi:10.1098/rsta.1933.0009 · Cited by 2,341
Parametric measures of effect size.
Rosenthal, R. (1994) · Cited by 1,847
Merrifield, P. R., Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1974) · American Educational Research Journal · doi:10.3102/00028312011001054 · Cited by 1,827
The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles
Merrifield, P. R., Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1974) · American Educational Research Journal · doi:10.2307/1162145 · Cited by 1,819
The Black-White Test Score Gap
(2011) · Rowman & Littlefield Publishers eBooks · doi:10.5771/9780815746119 · Cited by 1,715
ON THE PROBLEM OF THE MOST EFFICIENT TESTS OF STATISTICAL HYPOTHESES
(1967) · Joint Statistical Papers · doi:10.1525/9780520339897-008 · Cited by 1,504
The black-white test score gap
(1999) · Choice Reviews Online · doi:10.5860/choice.36-5211 · Cited by 1,471
Generalizability theory: a primer
(1992) · Choice Reviews Online · doi:10.5860/choice.29-4192 · Cited by 1,298
Brennan, R. L. (2001) · doi:10.1007/978-1-4757-3456-0 · Cited by 1,121
ON THE USE AND INTERPRETATION OF CERTAIN TEST CRITERIA FOR PURPOSES OF STATISTICAL INFERENCE
NEYMAN, J., & PEARSON, E. S. (1928) · Biometrika · doi:10.1093/biomet/20a.3-4.263 · Cited by 1,070
Estimation of effect size from a series of independent experiments.
Hedges, L. V. (1982) · Psychological Bulletin · doi:10.1037//0033-2909.92.2.490 · Cited by 1,062
The Handbook of Research Synthesis
Hanka, R. (1994) · BMJ · doi:10.1136/bmj.309.6952.488a · Cited by 1,052
Design Sensitivity: Statistical Power for Experimental Research.
Eliason, S. R., & Lipsey, M. W. (1991) · Contemporary Sociology · doi:10.2307/2072167 · Cited by 1,040
Self-citationContrasts and Effect Sizes in Behavioral Research: A Correlational Approach
Rosenthal, R., Rosnow, R. L., & Rubin, D. B. (1999) · Cited by 854
Intraclass Correlation Values for Planning Group-Randomized Trials in Education
Hedges, L. V., & Hedberg, E. C. (2007) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373707299706 · Cited by 847
Design Sensitivity: Statistical Power for Experimental Research.
Lachenbruch, P. A., & Lipsey, M. W. (1990) · Journal of the American Statistical Association · doi:10.2307/2289636 · Cited by 782
Self-citationEffect sizes for research: A broad practical approach.
Grissom, R. J., & Kim, J. J. (2005) · Cited by 732
Contrasts and Effect Sizes in Behavioral Research: A Correlational Approach
Rosenthal, R., Rosnow, R. L., & Rubin, D. B. (1999) · Cambridge University Press eBooks · doi:10.1017/cbo9780511804403 · Cited by 535
The Black-White Test Score Gap Through Third Grade
Fryer, R. G. (2006) · American Law and Economics Review · doi:10.1093/aler/ahl003 · Cited by 422 · cited 1×
“Although many studies evaluate such reforms (eg, Fryer & Levitt, 2006 ; Jencks & Phillips, 1998 ), little work has focused on how to assess whether their effects are large enough to be meaningful.”
BackgroundMeasures of effect size for categorical data.
Fleiss, J. L. (1994) · Cited by 248
Bloom, H. S., Richburg-Hayes, L., & Black, A. R. (2007) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373707299550 · Cited by 236
Self-citationMUNGER, A. M. (1963) · Contemporary Psychology: A Journal of Reviews · doi:10.1037/007027 · Cited by 193
The Benefits and Costs of Head Start
Ludwig, J., & Phillips, D. (2007) · National Bureau of Economic Research · doi:10.3386/w12973 · Cited by 148
Kane, T. J. (2004) · PsycEXTRA Dataset · doi:10.1037/e680712012-001 · Cited by 146
How Large an Effect Can We Expect from School Reforms?
Konstantopoulos, S., & Hedges, L. V. (2008) · Teachers College Record: The Voice of Scholarship in Education · doi:10.1177/016146810811000803 · Cited by 39
Penny Wise and Effect Size Foolish
Duncan, G. J., & Magnuson, K. (2007) · Child Development Perspectives · doi:10.1111/j.1750-8606.2007.00009.x · Cited by 26
Research works printed but not linked: 8
- Harris D. N., 2008, Toward new benchmarks for interpreting effect sizes: Combining effects with costs (Working Paper)
- 10.1037/0033-2909.92.2.490 · DOI · Paper page
- Lipsey M. W., Findings from prior studies as achievement effect size benchmarks for educational interventions
- Ludwig J., 2007, Society for Research on Child Development, Social Policy Report, 3
- Neyman J., 1928, Biometrika, 20, 175
- Nunnally J. C., 1967, Psychometric theory
- Tallmadge G. K., 1977, The joint dissemination review panel IDEABOOK
- 2002, TerraNova, The Second Edition
Cited by
PISA 2018 Results (Volume I): What Students Know and Can Do
OECD (2019) · PISA · doi:10.1787/5f07c754-en · Cited by 2,350
Learning loss due to school closures during the COVID-19 pandemic
Engzell, P., Frey, A., & Verhagen, M. D. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2022376118 · Cited by 1,848
“At the same time, some studies indicate a progress of up to 0.80 SD annually at the low extreme of our age range (45, 47), which would indicate that remote learning operated at 50% efficiency.”
InfluentialInterpreting effect sizes of education interventions
Kraft, M. A. (2020) · Educational Researcher · doi:10.3102/0013189x20912798 · Cited by 1,281
Empirical Benchmarks for Interpreting Effect Sizes in Research
Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008) · Child Development Perspectives · doi:10.1111/j.1750-8606.2008.00061.x · Cited by 950
Self-citationHeck, R. H., & Hallinger, P. (2009) · American Educational Research Journal · doi:10.3102/0002831209340042 · Cited by 585
Metadata
- Authors
- Bloom, H. S., Hill, C. J., Black, A. R., & Lipsey, M. W.
- Affiliations
- Manpower Demonstration Research Corporation (US)
- Georgetown University (US)
- Vanderbilt University (US)
- Venue
- Journal of Research on Educational Effectiveness
- Publisher
- Informa UK Limited
- Published
- 2008-10-15
- Volume
- 1
- Issue
- 4
- Pages
- 289-328
- ISSN
- 1934-5739 (ISSN-L)
- 1934-5747
- OpenAlex
- W1978049522
- Semantic Scholar
- ab9866b8754d68acbd25b4a477162bba21aed0ca
- Cited by
- 477
- References
- 36
- Citations by year
- 2026: 17 · 2025: 19 · 2024: 30 · 2023: 35 · 2022: 49 · 2021: 33 · 2020: 24 · 2019: 68 · 2018: 22 · 2017: 28 · 2016: 30 · 2015: 16 · 2014: 23 · 2013: 26 · 2012: 26
- FWCI
- 44.70
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Research on Educational Effectiveness (publishedVersion)
- Topics
- Social Sciences › Education › School Choice and Performance
- Social Sciences › Education › Educational Assessment and Improvement
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- educational intervention, performance trajectories, performance gap, achievement gap, standardized achievement tests, economic disadvantage, race and ethnicity, gender differences
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.