How Generalizable Is Your Experiment? An Index for Comparing Experimental Samples and Populations
The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
Although a large-scale experiment can provide an estimate of the average causal impact for a program, the sample of sites included in the experiment is often not drawn randomly from the inference population of interest. In this article, we provide a generalizability index that can be used to assess the degree of similarity between the sample of units in an experiment and one or more inference populations on a set of selected covariates. The index takes values between 0 and 1 and indicates both when a sample is like a miniature of the population and how well reweighting methods may perform when differences exist. Results of simulation studies are provided that develop rules of thumb for interpretation as well as an example.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 4×
“Interestingly, this rule of thumb corresponds roughly results from Rubin (2001), which suggest that balance is achieved when |d| .”
BackgroundMethodResultInfluentialMatching Methods for Causal Inference: A Review and a Look Forward
Stuart, E. A. (2010) · Statistical Science · doi:10.1214/09-sts313 · Cited by 5,478 · cited 4×
“Unfortunately, this dual randomization procedure is exceedingly rare in practice; for example, Olsen, Orr, Bell, and Stuart (2013) found that random site selection was implemented in only 7 of 273 experiments in the Digest of Social Experiments (Greenburg & Schroder, 2004).”
BackgroundMethodInfluentialUsing Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020363010465 · Cited by 2,228 · cited 3×
“Interestingly, this rule of thumb corresponds roughly results from Rubin (2001), which suggest that balance is achieved when |d| .25 and ½ ω2 2 (i.e., b > .96).”
BackgroundMethodResultAverage Values of Mean Squares in Factorials
Cornfield, J., & Tukey, J. W. (1956) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177728067 · Cited by 292 · cited 3×
“The fact that the generalization problem has many similarities to the observational studies problem led Hedges and O’Muircheartaigh (2011) to propose to use post-stratification methods with population information to reweight units in the experiment so that the experimental sample is compositionally similar to the infe…”
MethodResultInfluential
Cited by
A national experiment reveals where a growth mindset improves achievement
Yeager, D. S., Hanselman, P., Walton, G. M., Murray, J. S., Crosnoe, R., Muller, C., Tipton, E., Schneider, B., Hulleman, C. S., Hinojosa, C. P., et al. (2019) · Nature · doi:10.1038/s41586-019-1466-y · Cited by 1,513
“The generalizability index (Tipton, 2014) is a summary measure that provides the degree of distributional similarity between the schools in the analytic sample and the inference population, conditional on a set of covariates.”
BackgroundMethodInfluentialSelf-citationUse of Quasi-Experimental Research Designs in Education Research: Growth, Promise, and Challenges
Gopalan, M., Rosinger, K., & Ahn, J. B. (2020) · Review of Research in Education · doi:10.3102/0091732x20903302 · Cited by 411
A Review of Generalizability and Transportability
Degtiar, I., & Rose, S. (2023) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042522-103837 · Cited by 295
Conformal Inference of Counterfactuals and Individual Treatment Effects
Lei, L., & Candès, E. J. (2021) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/rssb.12445 · Cited by 166
Where and with whom does a brief social-belonging intervention promote progress in college?
Walton, G. M., Murphy, M. C., Logel, C., Yeager, D. S., Goyer, J. P., Brady, S. T., Emerson, K. T. U., Paunesku, D., Fotuhi, O., Blodorn, A., et al. (2023) · Science · doi:10.1126/science.ade4420 · Cited by 131
Metadata
- Authors
- Tipton, E.
- Venue
- Journal of Educational and Behavioral Statistics
- Published
- 2014-12
- Volume
- 39
- Issue
- 6
- Pages
- 478-501
- ISSN
- 1076-9986 (ISSN-L)
- 1935-1054
- OpenAlex
- W2041627061
- Cited by
- 134
- References
- 4
- FWCI
- 8.25
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How Generalizable Is Your Experiment? An Index for Comparing Experimental Samples and Populations. https://app.cochat.ai/papers/10.3102/1076998614558486
- The paper
- Tipton, E. (2014). How Generalizable Is Your Experiment? An Index for Comparing Experimental Samples and Populations. Journal of Educational and Behavioral Statistics. https://doi.org/10.3102/1076998614558486
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.