Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments
The DOI matches its Crossref record; 47 of the 48 references it cites were found in a registry or an index.
Abstract
A key justification for using nonrandomized experiments is that, with proper adjustment, their results can well approximate results from randomized experiments. This hypothesis has not been consistently supported by empirical studies; however, previous methods used to study this hypothesis have confounded assignment method with other study features. To avoid these confounding factors, this study randomly assigned participants to be in a randomized experiment or a nonrandomized experiment. In the randomized experiment, participants were randomly assigned to mathematics or vocabulary training; in the nonrandomized experiment, participants chose their training. The study held all other features of the experiment constant; it carefully measured pretest variables that might predict the condition that participants chose, and all participants were measured on vocabulary and mathematics outcomes. Ordinary linear regression reduced bias in the nonrandomized experiment by 84–94% using covariate-adjusted randomized results as the benchmark. Propensity score stratification, weighting, and covariance adjustment reduced bias by about 58–96%, depending on the outcome measure and adjustment method. Propensity score adjustment performed poorly when the scores were constructed from predictors of convenience (sex, age, marital status, and ethnicity) rather than from a broader set of predictors that might include these. Please see the online supplements for a Letter to the Editor.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“They should be used only under the following conditions:(a) Initial group differences are minimized, especially by identifying intact comparison groups that are local to the treatment group and demonstrably heavily overlap with it on at least pretest measures of the outcome (Cook et al. 2008 ); (b) the process by which treatment subjects select into the intervention group (or are selected into it)”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
In the citing paper
“Direct comparisons of ordinary least squares and propensity score matching methods have not shown much of a difference to date (Glazerman et al. 2003; Bloom et al. 2005; Shadish et al. 2008; Cook et al. 2009)”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
In the citing paper
“In nonexperimental studies, the choice of data analysis technique is not very important for reducing selection bias.”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
Citations
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
(2002) · Cited by 13,211
Self-citationEstimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 3×
“As has been argued in causal inference in general (Rubin 1974; Holland 1986), the best approximation to this true counterfactual may be a group of participants whose assignment method (random or nonrandom) was itself randomly assigned to them, with all other features of the experiment held equal.”
BackgroundMethodMatching as an econometric evaluation estimator: Evidence from evaluating a job training programme
Heckman, J. J., Ichimura, H., & Todd, P. E. (1997) · The Review of Economic Studies · doi:10.2307/2971733 · Cited by 5,486 · cited 9×
“This was not done by Dehejia and Wahba (1999), Heckman et al. (1997), or Hill et al. (2004), or in any other such studies.”
BackgroundResultInfluentialStatistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · cited 2×
“As has been argued in causal inference in general (Rubin 1974; Holland 1986), the best approximation to this true counterfactual may be a group of participants whose assignment method (random or nonrandom) was itself randomly assigned to them, with all other features of the experiment held equal.”
BackgroundDoes matching overcome lalonde’s critique of nonexperimental estimators?
A. Smith, J., & E. Todd, P. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.011 · Cited by 3,176
Reducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082 · cited 4×
“Table 1 reports four propensity score adjustments for the nonrandomized experiment: (a) stratification on propensity score quintiles (Rosenbaum and Rubin 1984), (b) use of the propensity score as a covariate in an analysis of covariance (ANCOVA), (c) propensity score ANCOVA including nonlinear…”
BackgroundMethodInfluentialCausal effects in nonexperimental studies: Reevaluating the evaluation of training programs
Dehejia, R. H., & Wahba, S. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473858 · Cited by 2,395
Variable Selection for Propensity Score Models
Brookhart, M. A., Schneeweiss, S., Rothman, K. J., Glynn, R. J., Avorn, J., & Stürmer, T. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj149 · Cited by 2,336
Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020363010465 · Cited by 2,228 · cited 7×
“score matches to a smaller group of treatment group participant scores (Rosenbaum and Rubin, 1985; Rubin, 2001 ).”
BackgroundMethodInfluential
Cited by
Matching Methods for Causal Inference: A Review and a Look Forward
Stuart, E. A. (2010) · Statistical Science · doi:10.1214/09-sts313 · Cited by 5,478
Harder, V. S., Stuart, E. A., & Anthony, J. C. (2010) · Psychological Methods · doi:10.1037/a0019623 · Cited by 802
“…a covariate to the final regression model is also an option, although there is mixed evidence regarding its performance (Rubin, 2001; Rubin, 2004; Shadish et al., 2008); while simply including the PS itself does not generally work well, very recent work has indicated that including non-linear PS…”
BackgroundResultInfluentialGottfredson, D. C., Cook, T. D., Gardner, F. E. M., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015) · Prevention Science · doi:10.1007/s11121-015-0555-x · Cited by 627
“…ordinary least squares and propensity score matching methods have not shown much of a difference to date (Glazerman et al. 2003; Bloom et al. 2005; Shadish et al. 2008; Cook et al. 2009), though the latter is theoretically preferable because it is nonparametric and requires demonstrated overlap…”
BackgroundMaking a Difference in Science Education: The Impact of Undergraduate Research Programs
Eagan, M. K., Hurtado, S., Chang, M. J., Garcia, G. A., Herrera, F. A., & Garibay, J. C. (2013) · American Educational Research Journal · doi:10.3102/0002831213482038 · Cited by 551
“This finding supports work by Shadish et al. (2008) who found that more standard estimation techniques that included rich sets of covariates that extended beyond demographic characteristics performed just as well as propensity score models.”
BackgroundResultAverage causal effects from nonrandomized studies: A practical guide and simulated example.
Schafer, J. L., & Kang, J. (2008) · Psychological Methods · doi:10.1037/a0014268 · Cited by 529
Metadata
- Authors
- Shadish, W. R., Clark, M. H., & Steiner, P. M.
- Affiliations
- Southern Illinois University Carbondale (US)
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2008-12-01
- Volume
- 103
- Issue
- 484
- Pages
- 1334-1344
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2068144165
- Cited by
- 479
- References
- 48
- Citations by year
- 2026: 6 · 2025: 22 · 2024: 9 · 2023: 19 · 2022: 24 · 2021: 29 · 2020: 31 · 2019: 35 · 2018: 31 · 2017: 31 · 2016: 32 · 2015: 33 · 2014: 51 · 2013: 35 · 2012: 29
- FWCI
- 19.93
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- randomized experiments, assignment method, bias mitigation, propensity score adjustment, covariate adjustment, stratification, weighting, mathematical training, vocabulary training, confounding, convenience sampling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.