CoChat

Making Sense of Effect Sizes: Systematic Differences in Intervention Effect Sizes by Outcome Measure Type

Wolf, Betsy, Harbatkin, Erica · 2023 · Journal of Research on Educational Effectiveness

The DOI matches its Crossref record; 40 of the 41 references it cites were found in a registry or an index.

Abstract

One challenge in understanding “what works” in education is that effect sizes may not be comparable across studies, raising questions for practitioners and policymakers using research to select interventions. One factor that consistently relates to the magnitude of effect sizes is the type of outcome measure. This article uses study data from the What Works Clearinghouse to determine average effect sizes by outcome measure type. Outcome measures were categorized by whether the group who developed the measure potentially had a stake in the intervention (non-independent) or not (independent). Using meta-analysis and controlling for study quality and intervention characteristics, we find larger average effect sizes for non-independent measures than for independent measures. Results suggest that larger effect sizes for non-independent measures are not due to differences in implementation fidelity, study quality, or intervention or sample characteristics. Instead, non-independent and independent measures appear to represent partially but minimally overlapping latent constructs. Findings call into question whether policymakers and practitioners should make decisions based on non-independent measures when they are ultimately responsible for improving outcomes on independent measures.

Overview

Integrity signals

Registry

Crossref · 10.1080/19345747.2022.2071364

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 40 references

Research works printed but not linked: 2

  1. Statistics Website and Training. (2020). How education leaders use the What Works Clearinghouse website (WWC report) [Manuscript awaiting peer review]. American Institutes for Research (AIR).
  2. Statistics Website and Training (SWAT) Measurement Small Group. (2020). Preliminary analysis of effect sizes associated with researcher-developed measures (WWC report)[Manuscript awaiting peer review]. American Institutes for Research (AIR).

Cited by

See all 53

Metadata

Authors
Wolf, B., & Harbatkin, E.
Affiliations
  • Institute of Education Sciences (US)
  • Michigan State University (US)
Corresponding author
Wolf, Betsy
Venue
Journal of Research on Educational Effectiveness
Published
2022-08
Volume
16
Issue
1
Pages
134-161
ISSN
  • 1934-5739 (ISSN-L)
  • 1934-5747
OpenAlex
W4289262332
Cited by
45
References
41
Citations by year
2026: 12 · 2025: 10 · 2024: 14 · 2023: 9
FWCI
4.98
Type
journal article
Language
en
Open access
closed
Locations
Journal of Research on Educational Effectiveness (publishedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Public Administration › Evaluation and Performance Assessment
  • Social Sciences › Education › Educational Assessment and Improvement
Keywords
effect size, outcome measures, What Works Clearinghouse, educational intervention, intervention effectiveness, construct overlap, measurement bias, evidence-based decision making, implementation fidelity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Making Sense of Effect Sizes: Systematic Differences in Intervention Effect Sizes by Outcome Measure Type. https://app.cochat.ai/papers/10.1080/19345747.2022.2071364
The paper
Wolf, B., & Harbatkin, E. (2023). Making Sense of Effect Sizes: Systematic Differences in Intervention Effect Sizes by Outcome Measure Type. Journal of Research on Educational Effectiveness. https://doi.org/10.1080/19345747.2022.2071364

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.