Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments
The DOI matches its Crossref record; 47 of the 48 references it cites were found in a registry or an index.
Abstract
A key justification for using nonrandomized experiments is that, with proper adjustment, their results can well approximate results from randomized experiments. This hypothesis has not been consistently supported by empirical studies; however, previous methods used to study this hypothesis have confounded assignment method with other study features. To avoid these confounding factors, this study randomly assigned participants to be in a randomized experiment or a nonrandomized experiment. In the randomized experiment, participants were randomly assigned to mathematics or vocabulary training; in the nonrandomized experiment, participants chose their training. The study held all other features of the experiment constant; it carefully measured pretest variables that might predict the condition that participants chose, and all participants were measured on vocabulary and mathematics outcomes. Ordinary linear regression reduced bias in the nonrandomized experiment by 84–94% using covariate-adjusted randomized results as the benchmark. Propensity score stratification, weighting, and covariance adjustment reduced bias by about 58–96%, depending on the outcome measure and adjustment method. Propensity score adjustment performed poorly when the scores were constructed from predictors of convenience (sex, age, marital status, and ethnicity) rather than from a broader set of predictors that might include these. Please see the online supplements for a Letter to the Editor.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“They should be used only under the following conditions:(a) Initial group differences are minimized, especially by identifying intact comparison groups that are local to the treatment group and demonstrably heavily overlap with it on at least pretest measures of the outcome (Cook et al. 2008 ); (b) the process by which treatment subjects select into the intervention group (or are selected into it)”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
In the citing paper
“Direct comparisons of ordinary least squares and propensity score matching methods have not shown much of a difference to date (Glazerman et al. 2003; Bloom et al. 2005; Shadish et al. 2008; Cook et al. 2009)”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
In the citing paper
“In nonexperimental studies, the choice of data analysis technique is not very important for reducing selection bias.”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
Citations
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
(2002) · Cited by 13,211
Self-citationEstimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 3×
“As has been argued in causal inference in general (Rubin 1974; Holland 1986), the best approximation to this true counterfactual may be a group of participants whose assignment method (random or nonrandom) was itself randomly assigned to them, with all other features of the experiment held equal.”
BackgroundMethodMatching as an econometric evaluation estimator: Evidence from evaluating a job training programme
Heckman, J. J., Ichimura, H., & Todd, P. E. (1997) · The Review of Economic Studies · doi:10.2307/2971733 · Cited by 5,486 · cited 9×
“This was not done by Dehejia and Wahba (1999), Heckman et al. (1997), or Hill et al. (2004), or in any other such studies.”
BackgroundResultInfluentialStatistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · cited 2×
“As has been argued in causal inference in general (Rubin 1974; Holland 1986), the best approximation to this true counterfactual may be a group of participants whose assignment method (random or nonrandom) was itself randomly assigned to them, with all other features of the experiment held equal.”
BackgroundDoes matching overcome lalonde’s critique of nonexperimental estimators?
A. Smith, J., & E. Todd, P. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.011 · Cited by 3,176
Reducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082
Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs
Dehejia, R. H., & Wahba, S. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473858 · Cited by 2,395
Variable Selection for Propensity Score Models
Brookhart, M. A., Schneeweiss, S., Rothman, K. J., Glynn, R. J., Avorn, J., & Stürmer, T. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj149 · Cited by 2,336
Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020363010465 · Cited by 2,228 · cited 7×
“score matches to a smaller group of treatment group participant scores (Rosenbaum and Rubin, 1985; Rubin, 2001 ).”
BackgroundMethodInfluentialLipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · cited 2×
“whether they yield the same average effect size. Lipsey and Wilson (1993) use the simplest”
BackgroundMethodScreening Depressed Patients in Family Practice: A Rapid Technic
Beck, A. T., & Beck, R. W. (1972) · Postgraduate Medicine · doi:10.1080/00325481.1972.11713319 · Cited by 1,823
Estimating the Effect of Training Programs on Earnings
Ashenfelter, O. (1978) · The Review of Economics and Statistics · doi:10.2307/1924332 · Cited by 1,515 · cited 1×
“…to as the “Ashenfelter dip”) that individuals who participate in job training programs tend to experience a “dip” in earnings just before entering, and thus it is not sufficient to control for only one period of pretreatment earnings when evaluating job training programs (Ashenfelter 1978).”
(2018) · The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation · doi:10.4135/9781506326139.n223 · Cited by 1,384
McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.4.403 · Cited by 1,317 · cited 1×
“Other methods for creating propensity scores exist, including classification trees, boosted regression, random forests, and boosted regression (e.g., Stone et al. 1995; McCaffrey, Ridgeway, and Morral 2004).”
MethodDiamond, A., & Sekhon, J. S. (2013) · Review of Economics and Statistics · doi:10.1162/rest_a_00318 · Cited by 1,102
Rubin, D. B. (2007) · Statistics in Medicine · doi:10.1002/sim.2739 · Cited by 1,047
Hirano, K., & Imbens, G. W. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020371312283 · Cited by 1,039
Causal Inference With General Treatment Regimes: Generalizing the Propensity Score
Imai, K., & van Dyk, D. A. (2004) · Journal of the American Statistical Association · doi:10.1198/016214504000001187 · Cited by 868
For objective causal inference, design trumps analysis
Rubin, D. B. (2008) · The Annals of Applied Statistics · doi:10.1214/08-aoas187 · Cited by 831
Hernán, M. A., Alonso, A., Logan, R., Grodstein, F., Michels, K. B., Willett, W. C., Manson, J. E., & Robins, J. M. (2008) · Epidemiology · doi:10.1097/ede.0b013e3181875e61 · Cited by 818
Cook, T. D., Shadish, W. R., & Wong, V. C. (2008) · Journal of Policy Analysis and Management · doi:10.1002/pam.20375 · Cited by 604
Self-citationPropensity Score Matching: A Statistical Method
Kane, L. T., Fang, T., Galetta, M. S., Goyal, D. K. C., Nicholson, K. J., Kepler, C. K., Vaccaro, A. R., & Schroeder, G. D. (2020) · Clinical Spine Surgery: A Spine Publication · doi:10.1097/bsd.0000000000000932 · Cited by 603
Ahmed, A., Husain, A., Love, T. E., Gambassi, G., Dell'Italia, L. J., Francis, G. S., Gheorghiade, M., Allman, R. M., Meleth, S., & Bourge, R. C. (2006) · European Heart Journal · doi:10.1093/eurheartj/ehi890 · Cited by 467
The Bias Due to Incomplete Matching
Rosenbaum, P. R. (2006) · Matched Sampling for Causal Effects · doi:10.1017/cbo9780511810725.020 · Cited by 422
Practical propensity score matching: a reply to Smith and Todd
Dehejia, R. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.012 · Cited by 390
The effects of psychological therapies under clinically representative conditions: a meta-analysis
Shadish, W. R., Navarro, A. M., Matt, G. E., & Phillips, G. (2000) · Psychological Bulletin · doi:10.1037/0033-2909.126.4.512 · Cited by 381
Self-citationNonexperimental Versus Experimental Estimates of Earnings Impacts
Glazerman, S., Levy, D. M., & Myers, D. (2003) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/0002716203254879 · Cited by 352 · cited 5×
“Unfortunately, whatever feasibility or ethical benefits sometimes accrue to nonrandomized experiments, they yield effect estimates that either are demonstrably different from those from randomized experiments (Glazerman, Levy, and Myers 2003) or are at best of unknown accuracy (Rosenbaum 2002).”
BackgroundResultInfluentialEstimating and Using Propensity Scores with Partially Missing Data
D'Agostino, R. B., & Rubin, D. B. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10474263 · Cited by 300
Mathematics Anxiety Effects in Simple and Complex Addition
Faust, M. W. (1996) · Mathematical Cognition · doi:10.1080/135467996387534 · Cited by 210
Does Alcoholics Anonymous Work? The Results from a Meta-Analysis of Controlled Experiments
Kownacki, R. J., & Shadish, W. R. (1999) · Substance Use & Misuse · doi:10.3109/10826089909039431 · Cited by 184
Self-citationInterval estimation for treatment effects using propensity score matching
Hill, J., & Reiter, J. P. (2006) · Statistics in Medicine · doi:10.1002/sim.2277 · Cited by 149
Random versus nonrandom assignment in controlled experiments: Do you get the same answer?
Shadish, W. R., & Ragsdale, K. (1996) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.64.6.1290 · Cited by 126 · cited 2×
“In an attempt to lessen reliance on this assumption, other metaanalyses have coded such confounding factors and included them as covariates to get an adjusted difference between randomized and nonrandomized experiments (e.g., Heinsman and Shadish 1996; Shadish and Ragsdale 1996; Glazerman et al. 2003).”
BackgroundSelf-citationHeinsman, D. T., & Shadish, W. R. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.154 · Cited by 125
Self-citationThe role of choice in health education intervention trials: a review and case study
Janevic, M. R., Janz, N. K., Dodge, J. A., Lin, X., Pan, W., Sinco, B. R., & Clark, N. M. (2003) · Social Science & Medicine · doi:10.1016/s0277-9536(02)00158-2 · Cited by 117
A two‐stage trial design for testing treatment, self‐selection and treatment preference effects
Rücker, G. (1989) · Statistics in Medicine · doi:10.1002/sim.4780080411 · Cited by 116
Comparing Experimental and Matching Methods Using a Large-Scale Voter Mobilization Experiment
Arceneaux, K., Gerber, A. S., & Green, D. P. (2006) · Political Analysis · doi:10.1093/pan/mpj001 · Cited by 116
Aiken, L. S., West, S. G., Schwalm, D. E., Carroll, J. L., & Hsiung, S. (1998) · Evaluation Review · doi:10.1177/0193841x9802200203 · Cited by 104
A Comparison of Experimental and Observational Data Analyses
Hill, J. L., Reiter, J. P., & Zanutto, E. L. (2004) · Wiley Series in Probability and Statistics · doi:10.1002/0470090456.ch5 · Cited by 98
Outcomes Research, PORTs, and Health Care Reform
Wennberg, J. E., Barry, M. J., Fowler, F. J., & Mulley, A. (1993) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1993.tb26335.x · Cited by 81
Concordance Between ACT Assessment and Recentered SAT I Sum Scores.
Dorans, N. J., Lyu, C. F., Pommerich, M., & Houston, W. M. (1997) · College and university · Cited by 80
Bloom, H. S., Michalopoulos, C., Hill, C. J., & Lei, Y. (2002) · Cited by 64
Causal Inference in Hybrid Intervention Trials Involving Treatment Choice
Long, Q., Little, R. J., & Lin, X. (2008) · Journal of the American Statistical Association · doi:10.1198/016214507000000662 · Cited by 64
Misunderstandings among experimentalists and observationalists about causal inference
Imai, K., King, G., & Stuart, E. A. (2007) · Cited by 40
Identification of maladjusted college students.
Kleinmuntz, B. (1960) · Journal of Counseling Psychology · doi:10.1037/h0047989 · Cited by 37
Estimating the causal effects of smoking
Rubin, D. B. (2001) · Statistics in Medicine · doi:10.1002/sim.677 · Cited by 34
Akins, T., Hollandsworth, J. G., & O'Connell, S. J. (1982) · Behaviour Research and Therapy · doi:10.1016/0005-7967(82)90144-9 · Cited by 16
Cited by
Matching Methods for Causal Inference: A Review and a Look Forward
Stuart, E. A. (2010) · Statistical Science · doi:10.1214/09-sts313 · Cited by 5,478
Harder, V. S., Stuart, E. A., & Anthony, J. C. (2010) · Psychological Methods · doi:10.1037/a0019623 · Cited by 802
“…a covariate to the final regression model is also an option, although there is mixed evidence regarding its performance (Rubin, 2001; Rubin, 2004; Shadish et al., 2008); while simply including the PS itself does not generally work well, very recent work has indicated that including non-linear PS…”
BackgroundResultInfluentialGottfredson, D. C., Cook, T. D., Gardner, F. E. M., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015) · Prevention Science · doi:10.1007/s11121-015-0555-x · Cited by 627
“…ordinary least squares and propensity score matching methods have not shown much of a difference to date (Glazerman et al. 2003; Bloom et al. 2005; Shadish et al. 2008; Cook et al. 2009), though the latter is theoretically preferable because it is nonparametric and requires demonstrated overlap…”
BackgroundMaking a Difference in Science Education: The Impact of Undergraduate Research Programs
Eagan, M. K., Hurtado, S., Chang, M. J., Garcia, G. A., Herrera, F. A., & Garibay, J. C. (2013) · American Educational Research Journal · doi:10.3102/0002831213482038 · Cited by 551
“This finding supports work by Shadish et al. (2008) who found that more standard estimation techniques that included rich sets of covariates that extended beyond demographic characteristics performed just as well as propensity score models.”
BackgroundResultAverage causal effects from nonrandomized studies: A practical guide and simulated example.
Schafer, J. L., & Kang, J. (2008) · Psychological Methods · doi:10.1037/a0014268 · Cited by 529
Metadata
- Authors
- Shadish, W. R., Clark, M. H., & Steiner, P. M.
- Affiliations
- Southern Illinois University Carbondale (US)
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2008-12-01
- Volume
- 103
- Issue
- 484
- Pages
- 1334-1344
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2068144165
- Cited by
- 479
- References
- 48
- Citations by year
- 2026: 6 · 2025: 22 · 2024: 9 · 2023: 19 · 2022: 24 · 2021: 29 · 2020: 31 · 2019: 35 · 2018: 31 · 2017: 31 · 2016: 32 · 2015: 33 · 2014: 51 · 2013: 35 · 2012: 29
- FWCI
- 19.93
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- randomized experiments, assignment method, bias mitigation, propensity score adjustment, covariate adjustment, stratification, weighting, mathematical training, vocabulary training, confounding, convenience sampling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.