Making Sense of Effect Sizes: Systematic Differences in Intervention Effect Sizes by Outcome Measure Type
The DOI matches its Crossref record; 40 of the 41 references it cites were found in a registry or an index.
Abstract
One challenge in understanding “what works” in education is that effect sizes may not be comparable across studies, raising questions for practitioners and policymakers using research to select interventions. One factor that consistently relates to the magnitude of effect sizes is the type of outcome measure. This article uses study data from the What Works Clearinghouse to determine average effect sizes by outcome measure type. Outcome measures were categorized by whether the group who developed the measure potentially had a stake in the intervention (non-independent) or not (independent). Using meta-analysis and controlling for study quality and intervention characteristics, we find larger average effect sizes for non-independent measures than for independent measures. Results suggest that larger effect sizes for non-independent measures are not due to differences in implementation fidelity, study quality, or intervention or sample characteristics. Instead, non-independent and independent measures appear to represent partially but minimally overlapping latent constructs. Findings call into question whether policymakers and practitioners should make decisions based on non-independent measures when they are ultimately responsible for improving outcomes on independent measures.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Wolf and Harbatkin (2023) used data from the What Works Clearinghouse to categorize outcome measures in educational interventions and documented how average effect sizes varied for outcome measure type by whether the group who developed the measure potentially had a stake in the intervention (non-independent) or not (independent).”
From The state of evidence for social and emotional learning: A contemporary meta-analysis of universal school-based SEL interventions (2023) · cites this paper as Wolf & Harbatkin, 2023
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464 · cited 1×
“…conducted multivariate meta-analysis with robust variance estimation using the R packages metafor and clubsandwich (Olkin & Gleser, 2009; Pustejovsky, 2019; R Core Team, 2018; Viechtbauer, 2010).6 Tij is the WWC-calculated effect size, which is Hedges’ g (WWC Standards Handbook Version 4.1, 2020).”
MethodInfluentialSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · cited 1×
“…within the same study, we conducted multivariate meta-analysis with robust variance estimation using the R packages metafor and clubsandwich (Olkin & Gleser, 2009; Pustejovsky, 2019; R Core Team, 2018; Viechtbauer, 2010).6 Tij is the WWC-calculated effect size, which is Hedges’ g (WWC…”
MethodMoving to a World Beyond “ p < 0.05”
Wasserstein, R. L., Schirm, A. L., & Lazar, N. A. (2019) · The American Statistician · doi:10.1080/00031305.2019.1583913 · Cited by 2,976 · cited 1×
“Additionally, the typical convention of p .05 has long been criticized as inadequate for basing decisions about intervention effectiveness (Wasserstein et al., 2019).”
BackgroundMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · cited 1×
“…findings on independent measures is curious and may be related to WWC processes for selecting studies for review or WWC procedures when reviewing studies.14 The WWC reviews both peer-reviewed and non-peer-reviewed studies, which may limit publication bias (John et al., 2012; McBee et al., 2017).”
BackgroundRobust variance estimation in meta‐regression with dependent effect size estimates
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.5 · Cited by 2,086 · cited 1×
“However, robust variance estimation uses these weights for efficiency purposes only in estimating the model (Hedges et al., 2010).”
MethodEvaluation: A Systematic Approach
Wallace, D. P. (2005) · Library & Information Science Research · doi:10.1016/j.lisr.2004.09.009 · Cited by 1,922
Interpreting effect sizes of education interventions
Kraft, M. A. (2020) · Educational Researcher · doi:10.3102/0013189x20912798 · Cited by 1,281
Lipsey, M. W. (2009) · Victims & Offenders · doi:10.1080/15564880802612573 · Cited by 962 · cited 1×
“…hypothesis is that use of researcher and developer measures is confounded with greater implementation fidelity because researchers who develop their own measures are more invested in implementation in those studies, which could lead to higher average effect sizes (Li & Ma, 2010; Lipsey, 2009).”
Background
Research works printed but not linked: 2
- Statistics Website and Training. (2020). How education leaders use the What Works Clearinghouse website (WWC report) [Manuscript awaiting peer review]. American Institutes for Research (AIR).
- Statistics Website and Training (SWAT) Measurement Small Group. (2020). Preliminary analysis of effect sizes associated with researcher-developed measures (WWC report)[Manuscript awaiting peer review]. American Institutes for Research (AIR).
Cited by
Cipriano, C., Strambler, M. J., Naples, L. H., Ha, C., Kirk, M., Wood, M., Sehgal, K., Zieher, A. K., Eveleigh, A., McCarthy, M., et al. (2023) · Child Development · doi:10.1111/cdev.13968 · Cited by 636
“Wolf and Harbatkin (2023) used data from the What Works Clearinghouse to categorize outcome measures in educational interventions and documented how average effect sizes varied for outcome measure type by whether the group who developed the measure potentially had a stake in the intervention…”
Vembye, M. H., Weiss, F., & Bhat, B. H. (2023) · Review of Educational Research · doi:10.3102/00346543231186588 · Cited by 104
Effects of the 5E Instructional Model: A Systematic Review and Meta-Analysis
Polanin, J. R., Austin, M. J., Taylor, J. A., Steingut, R. R., Rodgers, M. A., & Williams, R. T. (2024) · AERA Open · doi:10.1177/23328584241269866 · Cited by 94
“Third, as others have found (Wolf & Harbatkin, 2023), findings from measures that researchers developed had larger effects compared to findings from standardized measures ( b = 0.37, SE = 0.18); the results were not statistically significant ( p = .07) but clearly aligned with the historical trend.”
InfluentialPollak, I., Stiehl, K. A. M., Birchwood, J., Schrank, B., Zechner, K. A., Wiesner, C., & Woodcock, K. A. (2023) · Journal of Youth and Adolescence · doi:10.1007/s10964-023-01871-x · Cited by 71
“Additionally, independent narrow measures, such as the ones used in this study, have been found to yield smaller effect sizes compared to broad or researcher-developed measures (Wolf & Harbatkin, 2023).”
BackgroundMethodVisscher, A. J., Dmoshinskaia, N., Pellegrini, M., & Naizaque, A. R. (2025) · Educational Research Review · doi:10.1016/j.edurev.2025.100742 · Cited by 75
Metadata
- Authors
- Wolf, B., & Harbatkin, E.
- Affiliations
- Institute of Education Sciences (US)
- Michigan State University (US)
- Corresponding author
- Wolf, Betsy
- Venue
- Journal of Research on Educational Effectiveness
- Publisher
- Informa UK Limited
- Published
- 2022-08
- Volume
- 16
- Issue
- 1
- Pages
- 134-161
- ISSN
- 1934-5739 (ISSN-L)
- 1934-5747
- OpenAlex
- W4289262332
- Cited by
- 45
- References
- 41
- Citations by year
- 2026: 12 · 2025: 10 · 2024: 14 · 2023: 9
- FWCI
- 4.98
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Research on Educational Effectiveness (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Social Sciences › Education › Educational Assessment and Improvement
- Keywords
- effect size, outcome measures, What Works Clearinghouse, educational intervention, intervention effectiveness, construct overlap, measurement bias, evidence-based decision making, implementation fidelity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Making Sense of Effect Sizes: Systematic Differences in Intervention Effect Sizes by Outcome Measure Type. https://app.cochat.ai/papers/10.1080/19345747.2022.2071364
- The paper
- Wolf, B., & Harbatkin, E. (2023). Making Sense of Effect Sizes: Systematic Differences in Intervention Effect Sizes by Outcome Measure Type. Journal of Research on Educational Effectiveness. https://doi.org/10.1080/19345747.2022.2071364
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.