Covariate Balance in Simple, Stratified and Clustered Comparative Studies
Open accessAbstract
In randomized experiments, treatment and control groups should be roughly the same—balanced—in their distributions of pretreatment variables. But how nearly so? Can descriptive comparisons meaningfully be paired with significance tests? If so, should there be several such tests, one for each pretreatment variable, or should there be a single, omnibus test? Could such a test be engineered to give easily computed p-values that are reliable in samples of moderate size, or would simulation be needed for reliable calibration? What new concerns are introduced by random assignment of clusters? Which tests of balance would be optimal? To address these questions, Fisher’s randomization inference is applied to the question of balance. Its application suggests the reversal of published conclusions about two studies, one clinical and the other a field experiment in political participation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The results of Moving to Opportunity have been mixed.
“The results have been mixed.”
From Reducing Violence by Transforming Neighborhoods: A Natural Experiment in Medellin, Colombia (2012) · cites this paper as [23]
The passages from Section 1.2 and the additional searches discuss the Vote’98 voter-turnout experiment, not Moving to Opportunity. Searches for “Moving to Opportunity” and its housing-voucher outcomes found no passage in our copy addressing whether its results were mixed. (LLM, conf=0.72)
Experimental — accuracy not yet measuredAlthough male and female youth from families that received vouchers to relocate to lower-poverty areas experienced declining involvement in violent crimes, male youth also experienced more conduct disorder, other problem behaviors, and property crime arrests.
“Although male and female youth from families that received vouchers to relocate to lowerpoverty areas experienced declining involvement in violent crimes, male youth also experienced more conduct disorder, other problem behaviors, and property crime arrests (23).”
From Reducing Violence by Transforming Neighborhoods: A Natural Experiment in Medellin, Colombia (2012) · cites this paper as [23]
Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 40 · Cited by 414
Research works printed but not linked: 13
- BOWERS, J. and HANSEN, B. B. (2006). RItools, an add-on pack- age for R. BRAZZALE, A. R., DAVISON, A. C. and REID, N. (2006). Ap- plied Asymptotics. Cambridge Univ. Press.
- COCHRAN, W. G. and RUBIN, D. B. (1973). Controlling bias in observational studies: A review. Sankhyā Ser. A: Indian J. Sta- tist. 35 417-446.
- DAVISON, A. (2003). Statistical Models. Cambridge Univ. Press.
- FELLER, W. (1971). An Introduction to Probability Theory and Its Applications. II, 2nd ed. Wiley, New York.
- FLEISS, J. (1973). Statistical Methods for Rates and Proportions Rates and Proportions. Wiley, New York.
- GERBER, A. S. and GREEN, D. P. (2000). The effects of canvass- ing, telephone calls, and direct mail on voter turnout: A field experiment. American Political Science Review 94 653-663.
- GERBER, A. S. and GREEN, D. P. (2005). Correction to Gerber and Green (2000), replication of disputed findings, and reply to Imai (2005). American Political Science Review 99 301-313.
- HÁJEK, J. and ŠIDÁK, Z. (1967). Theory of Rank Tests. Academic Press, New York.
- HARRELL, F. E. (2001). Regression Modeling Strategies: With Ap- plications to Linear Models, Logistic Regression, and Survival Analysis. Springer, New York.
- HIGHTON, B. and WOLFINGER, R. (2001). The first seven years of the political life cycle. American Journal of Political Science 45 202-209.
- HILL, J. L., THOMAS, N. and RUBIN, D. B. (2000). The design of the New York schools choice scholarship program evaluation. In Research Design: Donald Campbell's Legacy (L. Bickman, ed) 155-180. Sage Publications, Thousand Oaks, CA.
- IMAI, K. (2005). Do get-out-the-vote calls reduce turnout? The im- portance of statistical methods for field experiments. American Political Science Review 99 283-300.
- MURRAY, D. M. (1998). Design and Analysis of Group- randomized Trials. Oxford Univ. Press.
- Authors
- Hansen, B. B., & Bowers, J.
- Venue
- Statistical Science
- Publisher
- Institute of Mathematical Statistics
- Published
- 2008-05-01
- Volume
- 23
- Issue
- 2
- Pages
- 219-236
- ISSN
- 0883-4237 (ISSN-L)
- 2168-8745
- OpenAlex
- W1547147213
- Cited by
- 414
- References
- 40
- Citations by year
- 2026: 6 · 2025: 11 · 2024: 20 · 2023: 17 · 2022: 28 · 2021: 34 · 2020: 41 · 2019: 38 · 2018: 26 · 2017: 35 · 2016: 35 · 2015: 26 · 2014: 22 · 2013: 21 · 2012: 21
- FWCI
- 10.76
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Statistical Science (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Project Euclid (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Quality Engineering
- Funders
- Yale University
- Case Western Reserve University
- National Institutes of Health
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
- Keywords
- covariate balance, stratified randomization, cluster randomization, omnibus test, field experiments, political participation, clinical trials
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
