Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments
The DOI matches its Crossref record; 47 of the 48 references it cites were found in a registry or an index.
Abstract
A key justification for using nonrandomized experiments is that, with proper adjustment, their results can well approximate results from randomized experiments. This hypothesis has not been consistently supported by empirical studies; however, previous methods used to study this hypothesis have confounded assignment method with other study features. To avoid these confounding factors, this study randomly assigned participants to be in a randomized experiment or a nonrandomized experiment. In the randomized experiment, participants were randomly assigned to mathematics or vocabulary training; in the nonrandomized experiment, participants chose their training. The study held all other features of the experiment constant; it carefully measured pretest variables that might predict the condition that participants chose, and all participants were measured on vocabulary and mathematics outcomes. Ordinary linear regression reduced bias in the nonrandomized experiment by 84–94% using covariate-adjusted randomized results as the benchmark. Propensity score stratification, weighting, and covariance adjustment reduced bias by about 58–96%, depending on the outcome measure and adjustment method. Propensity score adjustment performed poorly when the scores were constructed from predictors of convenience (sex, age, marital status, and ethnicity) rather than from a broader set of predictors that might include these. Please see the online supplements for a Letter to the Editor.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“They should be used only under the following conditions:(a) Initial group differences are minimized, especially by identifying intact comparison groups that are local to the treatment group and demonstrably heavily overlap with it on at least pretest measures of the outcome (Cook et al. 2008 ); (b) the process by which treatment subjects select into the intervention group (or are selected into it)”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
In the citing paper
“Direct comparisons of ordinary least squares and propensity score matching methods have not shown much of a difference to date (Glazerman et al. 2003; Bloom et al. 2005; Shadish et al. 2008; Cook et al. 2009)”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
In the citing paper
“In nonexperimental studies, the choice of data analysis technique is not very important for reducing selection bias.”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
Citations
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
(2002) · Cited by 13,211
Self-citationEstimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 3×
“As has been argued in causal inference in general (Rubin 1974; Holland 1986), the best approximation to this true counterfactual may be a group of participants whose assignment method (random or nonrandom) was itself randomly assigned to them, with all other features of the experiment held equal.”
BackgroundMethodMatching as an econometric evaluation estimator: Evidence from evaluating a job training programme
Heckman, J. J., Ichimura, H., & Todd, P. E. (1997) · The Review of Economic Studies · doi:10.2307/2971733 · Cited by 5,486 · cited 9×
“This was not done by Dehejia and Wahba (1999), Heckman et al. (1997), or Hill et al. (2004), or in any other such studies.”
BackgroundResultInfluentialStatistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · cited 2×
“As has been argued in causal inference in general (Rubin 1974; Holland 1986), the best approximation to this true counterfactual may be a group of participants whose assignment method (random or nonrandom) was itself randomly assigned to them, with all other features of the experiment held equal.”
BackgroundDoes matching overcome lalonde’s critique of nonexperimental estimators?
A. Smith, J., & E. Todd, P. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.011 · Cited by 3,176
Reducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082
Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs
Dehejia, R. H., & Wahba, S. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473858 · Cited by 2,395
Variable Selection for Propensity Score Models
Brookhart, M. A., Schneeweiss, S., Rothman, K. J., Glynn, R. J., Avorn, J., & Stürmer, T. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj149 · Cited by 2,336
Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020363010465 · Cited by 2,228 · cited 7×
“score matches to a smaller group of treatment group participant scores (Rosenbaum and Rubin, 1985; Rubin, 2001 ).”
BackgroundMethodInfluential
Cited by
Items 401–479 of 479
Madriz, L., & Alonso, F. (2014) · Cited by 1
Dong, N., & Lipsey, M. W. (2014) · Society for Research on Educational Effectiveness · Cited by 1
Wong, V. C., & Steiner, P. M. (2015) · Society for Research on Educational Effectiveness · Cited by 2
Self-citationTeacher Effectiveness and Causal Inference in Observational Studies
Rose, R. A. (2013) · Carolina Digital Repository (University of North Carolina at Chapel Hill) · doi:10.17615/gkgd-7t91 · Cited by 1
Rose, R. A., & Stone, S. I. (2017) · Journal of the Society for Social Work and Research · doi:10.1086/693046 · Cited by 1
“Shadish, Clark, and Steiner (2008) went further, randomly assigning participants to an RCT or an observational study.”
BackgroundKeeping youth in the community through an intensive case management and family partnerships model
Winokur, M., Orsi, R., Chen, P. C. B., & Alessi, L. (2020) · Journal of Public Child Welfare · doi:10.1080/15548732.2020.1802386 · Cited by 1
Meta-analyzing logistic regression slopes: A partial effect size for categorical outcomes
Anderson, N. (2021) · doi:10.7916/d8-37cs-mp34 · Cited by 2
Assessing the Employment Effects of California’s Paid Family Leave Program
Schenck, S. M. (2021) · Eastern Economic Journal · doi:10.1057/s41302-021-00193-9 · Cited by 1
Yeaton, W. H. (2023) · International Journal of Research & Method in Education · doi:10.1080/1743727x.2023.2220649 · Cited by 1
“While WSCs take on several forms, two-stage randomization (Shadish et al. 2008) bears some semblance to a SMART design, so it is important to distinguish them.”
BackgroundProceedings of the International Ambulatory Drug Safety Symposium: Munich, Germany, June 2023
Alexander, G. C., Budnitz, D. S., Hughes, C. M., Maas, R., Mair, A. R., McDonald, E. G., Meid, A. D., Payne, R., Seidling, H. M., Shakir, S., et al. (2023) · Drug Safety · doi:10.1007/s40264-023-01362-9 · Cited by 1
Which control variables do we really need for matching based evaluations of labour market programmes
Lechner, M., & Wunsch, C. (2010) · Econstor (Econstor) · Cited by 1
Contalbrigo, L., Santis, M. D., Toson, M., Montanaro, M., Farina, L., Costa, A. M., & Nava, F. A. (2020) · Springer eBooks · doi:10.1007/978-3-658-27143-5_12 · Cited by 1
Bifulco, R. (2011) · SSRN Electronic Journal · doi:10.2139/ssrn.1805890 · Cited by 1
Diffusion as a Validation Process: Learning from Patient Innovators
Fidelis, T., Zejnilović, L., & Oliveira, P. (2014) · Academy of Management Proceedings · doi:10.5465/ambpp.2014.17138abstract · Cited by 1
A national study comparing charter and traditional public schools using propensity score analysis
Bryer, J. (2014) · Scholars Archive - University at Albany (University at Albany, State University of New York) · doi:10.54014/h2br-2xkn · Cited by 4
Essays in health economics and public policy
Cannonier, C. (2011) · Louisiana State University · doi:10.31390/gradschool_dissertations.2226 · Cited by 1
Lurking Inferential Monsters? Quantifying bias in non-experimental evaluations of school programsPreprint
Weidmann, B., & Miratrix, L. W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07091 · Cited by 1
“3 In rare cases, such as Shadish, Clark, and Steiner (2008), researchers have been able to use an independent withinstudy comparison, in which participants are randomly allocated to an RCT arm or an observational study arm (Wong, Steiner, & Anglin, 2018).”
BackgroundInnovations in Systemic Family Therapy Effectiveness Research
Miller, R. B., & Jaurequi, M. E. (2020) · Wiley eBooks · doi:10.1002/9781119790181.ch19 · Cited by 1
Why Attentions May Not Be Interpretable?Preprint
Bai, B., Liang, J., Zhang, G., Li, H., Bai, K., & Wang, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05656 · Cited by 10
Izdebski, A., Thoral, P. J., Lalisang, R. C. A., McHugh, D., Gommers, D. A. M. P. J., Cremer, O. L., Bosman, R. J., Rigter, S., Wils, E., Frenzel, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.06707 · Cited by 1
Schmitt, N. (2013) · Oxford University Press eBooks · doi:10.1093/obo/9780199846740-0029 · Cited by 1
Epstein, I. (2022) · Research on Social Work Practice · doi:10.1177/10497315221121769 · Cited by 1
“Thinking along with similar lines, however, Shadish et al. (2008) conducted an experiment with original data to determine whether randomization produces the same results as nonrandomized samples.”
ResultSOUnD Experimental Public Administration: A Systematic Review and a TypologyPreprint
Belle, N., & Belardinelli, P. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5115070 · Cited by 4
Measurement effects between CAPI and Web questionnaires in the UK Household Longitudinal StudyPreprint
Vannieuwenhuyze, J., & Lynn, P. J. (2014) · RePEc: Research Papers in Economics · Cited by 1
Comparing Treatment Effects in Parallel ExperimentsPreprint
Jerit, J., Barabas, J., & Clifford, S. (2011) · SSRN Electronic Journal
Shi, K., & Nie, L. (2014) · Asian Journal of Finance & Accounting · doi:10.5296/ajfa.v6i2.5822
Steiner, P. M. (2011) · RePEc: Research Papers in Economics
Self-citationRandolph, A. G. (2013) · Critical Care · doi:10.1186/cc13166
“It has been argued that rigorous epidemiology studies can be equally important in creating a deeper understanding of the problem under study, often informing clinical practice [10,11].”
BackgroundEssays on Experimental and Quasi-Experimental Policy Design and Evaluation
Montero, M. (2012) · Digital Archive @ GSU · doi:10.57709/3139095
Can Quasi-Experiments Yield Causal Inferences?
Maciejewski, M. L., & Va, D. (2012)
Najab, J. (2014) · doi:10.13021/mars/6081
Assessing the Accuracy of Generalized Inferences from Comparison Group StudiesPreprint
Jaciw, A. P. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2516462
Informal Leaders How External Institutions Penetrate Schools Through Formal and
Sun, M., Frank, K. A., Penuel, W. R., & Kim, C. M. (2014)
AssessingtheEffectivenessof Recovery-OrientedACTinReducing StatePsychiatricHospitalUse
MORRISSEY, J. P., Domino, M. E., & Cuddeback, G. S. (2013)
Review of causal inference designs and analytic methods
Hong, G. (2015) · Wiley eBooks · doi:10.1002/9781119030638.ch3
William Raymond Shadish Jr. Will Shadish's Intellectual Accomplishments
Cook, T. D. (2016) · American Journal of Evaluation · doi:10.1177/1098214016666714
Mark, M. M. (2016) · American Journal of Evaluation · doi:10.1177/1098214016666710
Nash, C. J. (2015) · CU Scholar (University of Colorado Boulder)
Patterson, D. L. (2016) · eScholarship (California Digital Library)
Kramer, D. A., Lane, M., & Tanner, M. (2017) · Journal of School Choice · doi:10.1080/15582159.2016.1257894 · Cited by 1
“Shadish, Clark, and Steiner (2008) stated that using multilevel covariates in a matching approach more closely approximates estimates from a randomized control trial and that is not necessary to conduct a multilevel match to achieve balance.”
BackgroundXu, R. (2016) · Michigan State University Libraries · doi:10.25335/m52q4f
No Credibility without Plausibility: a Response to Lewis and Lanier
Rose, R. A. (2019) · Journal of Family Violence · doi:10.1007/s10896-019-00080-0
Abdulaziz, N. A. S. (2019) · University of North Texas Digital Library (University of North Texas) · doi:10.12794/metadc1538753
Our Research on Comparing Idea-Sharing Versus Unmindcuffing the Crowd
Majchrzak, A., & Malhotra, A. (2019) · Springer eBooks · doi:10.1007/978-3-030-25557-2_2
Using Experimental Data to Evaluate Methods for Observational Causal Inference.Preprint
Gentzel, A., Clarke, J., & Jensen, D. D. (2020) · arXiv (Cornell University)
López-Pedersen, A. (2020) · NVA (Nasjonalt vitenarkiv)
A Simulation-Based Test of Identifiability for Bayesian Causal Inference.Preprint
Witty, S., Jensen, D. D., & Mansinghka, V. K. (2021) · arXiv (Cornell University)
Neuropsychologie en psychiatrie
Philippot, P. (2019) · De Boeck Supérieur eBooks · doi:10.3917/dbu.amiev.2019.01.0011 · Cited by 1
Huitema, B. E. (2011) · Wiley series in probability and statistics · doi:10.1002/9781118067475.refs
Weisberg, H. I. (2010) · Wiley series in probability and statistics · doi:10.1002/9780470631102.biblio
Shintani, A., & Eden, S. (2012) · Faculty Opinions – Post-Publication Peer Review of the Biomedical Literature · doi:10.3410/f.13972971.15429075
How Within-Study Comparisons can Inform Research and Evaluation Design for the Third Sector
Anderson, K. P., & Wolf, P. J. (2022) · VOLUNTAS International Journal of Voluntary and Nonprofit Organizations · doi:10.1007/s11266-022-00464-9
“For example, when assessing the achievement impacts of educational programs, propensity score matching (PSM) using student demographic variables but without baseline test scores often fails to approximate experimental estimates (Bifulco, 2012; Shadish et al., 2008; Wilde & Hollister, 2007).”
BackgroundResultInfluentialInnovations in Systemic Family Therapy Effectiveness Research
Miller, R. B., & Jaurequi, M. E. (2020) · Wiley eBooks · doi:10.1002/9781119438519.ch19
Pasek, J., & Ahn, S. J. (2013) · Oxford University Press eBooks · doi:10.1093/obo/9780199756841-0138
Mellenbergh, G. J. (2019) · Springer eBooks · doi:10.1007/978-3-030-12272-0_6
Boruch, R. F., Weisburd, D. L., Berk, R. A., & Cave, B. (2014) · Springer eBooks · doi:10.1007/978-1-4614-5690-2_53
Family Income and School Readiness Skills
Portnow, S. (2017) · doi:10.18130/v3gw6w
Free Meals for All: The Effects of the Community Eligibility Provision Program in Tennessee
Kho, A. (2019) · American Educational Research Association Annual Meeting (AERA) · doi:10.3102/1446131
“In models assessing test scores as an outcome, I also include a lagged test score control variable as previous research finds that including pretreatment test scores can dramatically reduce bias in quasi-experimental methods (Bifulco, 2012; Shadish, Clark & Steiner, 2008).”
Yu, Y., & Ma, M. (2022) · Advances in transdisciplinary engineering · doi:10.3233/atde221030
The effect of open space maintenance spending on house prices
Brasington, D. M. (2023) · Journal of Regional Science · doi:10.1111/jors.12655
Does Owning a Home Make Us More Generous?
Bian, X., & Zhu, F. (2023) · Journal of Housing Research · doi:10.1080/10527001.2023.2225929
The Economic Inequality of Lockdowns:A Study of COVID-19 and Mobile Money in GhanaPreprint
Klobodu, E. K. M., Grewal, R., & Basuroy, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4863342
Turn the camera on to get better grade: Evidence from a field experiment
Anderson, J. L., & Krasnozhon, L. (2024) · International Review of Economics Education · doi:10.1016/j.iree.2024.100301
Extending the reach of the common cause design using meta-analytic methods: Applications and issues
Thompson, C. G., Yeaton, W. H., Velasquez, G., Petrova, K., & Becker, B. J. (2024) · Methodology · doi:10.5964/meth.10591
Lee, S., Umeda, C., Tucker-Roghi, G., Rosenberg, B. D., Gutierrez, K., Dale, S. N., & Kirkland, L. H. (2025) · Journal of American College Health · doi:10.1080/07448481.2025.2479702
“34 Given that any causal claims would only be suggestive, we paired qualitative data to provide greater depth to students’ and faculty members’ experiences.”
National Evidence of the Relationship Between SNAP Receipt and College Students’ Academic Outcomes
Voorhees, N., & Ortagus, J. C. (2025) · The Journal of Higher Education · doi:10.1080/00221546.2025.2488717 · Cited by 1
“To confirm the effectiveness of our IPTW strategy in balancing covariates, we used standardized mean differences and variance ratios for the pooled sample and sub-samples of historically underserved students (Rubin, 2001; Shadish et al., 2008).”
Sibling Models Can Test Causal Claims without Experiments: Applications for PsychologyPreprint
Garrison, S. M., Trattner, J. D., Lyu, X., Prillaman, H. R., McKinzie, L., Thompson, S., & Rodgers, J. L. (2025) · medRxiv · doi:10.1101/2025.08.25.25334395
Bryer, J., Andrade, H., Cleary, T. J., Lui, A. M., Franklin, D. W., & Akhmedjanova, D. R. (2025) · Educational Technology Research and Development · doi:10.1007/s11423-025-10547-1 · Cited by 2
“PSA is a quasi-experimental design that has been shown to provide causal estimates from observational data consistent with randomized control trials (Lonjon et al., 2014; Shadish et al., 2008; Stuart et al., 2017).”
Nouri, I., Bouallègue, S., Ibrahim, S., & Bouzaabia, R. (2025) · Journal of Humanities and Applied Social Sciences · doi:10.1108/jhass-11-2024-0190 · Cited by 1
Li, A., & Davies, S. (2025) · Sociological Perspectives · doi:10.1177/07311214251390957
“Controlling for prior outcomes has been shown to generally reduce the bias in estimated effects by 60 to 90 percent compared to an analogous randomized experiment in educational studies (Shadish, Clark, and Steiner 2008).”
Additional file 1 of Modelling hospital outcome: problems with endogeneity
Moran, J. L., Santamaria, J. D., & Duke, G. J. (2021) · Figshare · doi:10.6084/m9.figshare.14821162
Tripathi, K. K., & Bhattacharya, S. (2025) · Multidisciplinary Research Journal · doi:10.63635/mrj.v1i4.191
Vitry, G. (2021) · theses.fr (ABES) · doi:10.70675/9ca77eb1zbd3az4379zbd87zd7d8bf641d4c · Cited by 2
Naurin, D., Šadl, U., & Zglinski, J. (2026) · Cambridge University Press eBooks · doi:10.1017/9781009672580.008 · Cited by 2
Experimental Pathways in EU Law
Mayoral, J. A. (2026) · Cambridge University Press eBooks · doi:10.1017/9781009672580.015
Sibling Models Can Test Causal Claims without Experiments: Applications for PsychologyPreprint
Rodgers, J. L., Garrison, S. M., McKinzie, L., & Trattner, J. D. (2025) · OSF Preprints (OSF Preprints) · Cited by 1
Evidence for Evidence-Based Medicine — A Call for Some Theory and a New Problem
Trikalinos, T. A. (2026) · NEJM Evidence · doi:10.1056/evide2500311
Metadata
- Authors
- Shadish, W. R., Clark, M. H., & Steiner, P. M.
- Affiliations
- Southern Illinois University Carbondale (US)
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2008-12-01
- Volume
- 103
- Issue
- 484
- Pages
- 1334-1344
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2068144165
- Cited by
- 479
- References
- 48
- Citations by year
- 2026: 6 · 2025: 22 · 2024: 9 · 2023: 19 · 2022: 24 · 2021: 29 · 2020: 31 · 2019: 35 · 2018: 31 · 2017: 31 · 2016: 32 · 2015: 33 · 2014: 51 · 2013: 35 · 2012: 29
- FWCI
- 19.93
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- randomized experiments, assignment method, bias mitigation, propensity score adjustment, covariate adjustment, stratification, weighting, mathematical training, vocabulary training, confounding, convenience sampling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.