Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments
The DOI matches its Crossref record; 47 of the 48 references it cites were found in a registry or an index.
Abstract
A key justification for using nonrandomized experiments is that, with proper adjustment, their results can well approximate results from randomized experiments. This hypothesis has not been consistently supported by empirical studies; however, previous methods used to study this hypothesis have confounded assignment method with other study features. To avoid these confounding factors, this study randomly assigned participants to be in a randomized experiment or a nonrandomized experiment. In the randomized experiment, participants were randomly assigned to mathematics or vocabulary training; in the nonrandomized experiment, participants chose their training. The study held all other features of the experiment constant; it carefully measured pretest variables that might predict the condition that participants chose, and all participants were measured on vocabulary and mathematics outcomes. Ordinary linear regression reduced bias in the nonrandomized experiment by 84–94% using covariate-adjusted randomized results as the benchmark. Propensity score stratification, weighting, and covariance adjustment reduced bias by about 58–96%, depending on the outcome measure and adjustment method. Propensity score adjustment performed poorly when the scores were constructed from predictors of convenience (sex, age, marital status, and ethnicity) rather than from a broader set of predictors that might include these. Please see the online supplements for a Letter to the Editor.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“They should be used only under the following conditions:(a) Initial group differences are minimized, especially by identifying intact comparison groups that are local to the treatment group and demonstrably heavily overlap with it on at least pretest measures of the outcome (Cook et al. 2008 ); (b) the process by which treatment subjects select into the intervention group (or are selected into it)”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
In the citing paper
“Direct comparisons of ordinary least squares and propensity score matching methods have not shown much of a difference to date (Glazerman et al. 2003; Bloom et al. 2005; Shadish et al. 2008; Cook et al. 2009)”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
In the citing paper
“In nonexperimental studies, the choice of data analysis technique is not very important for reducing selection bias.”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
Citations
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
(2002) · Cited by 13,211
Self-citationEstimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 3×
“As has been argued in causal inference in general (Rubin 1974; Holland 1986), the best approximation to this true counterfactual may be a group of participants whose assignment method (random or nonrandom) was itself randomly assigned to them, with all other features of the experiment held equal.”
BackgroundMethodMatching as an econometric evaluation estimator: Evidence from evaluating a job training programme
Heckman, J. J., Ichimura, H., & Todd, P. E. (1997) · The Review of Economic Studies · doi:10.2307/2971733 · Cited by 5,486 · cited 9×
“This was not done by Dehejia and Wahba (1999), Heckman et al. (1997), or Hill et al. (2004), or in any other such studies.”
BackgroundResultInfluentialStatistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · cited 2×
“As has been argued in causal inference in general (Rubin 1974; Holland 1986), the best approximation to this true counterfactual may be a group of participants whose assignment method (random or nonrandom) was itself randomly assigned to them, with all other features of the experiment held equal.”
BackgroundDoes matching overcome lalonde’s critique of nonexperimental estimators?
A. Smith, J., & E. Todd, P. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.011 · Cited by 3,176
Reducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082
Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs
Dehejia, R. H., & Wahba, S. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473858 · Cited by 2,395
Variable Selection for Propensity Score Models
Brookhart, M. A., Schneeweiss, S., Rothman, K. J., Glynn, R. J., Avorn, J., & Stürmer, T. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj149 · Cited by 2,336
Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020363010465 · Cited by 2,228 · cited 7×
“score matches to a smaller group of treatment group participant scores (Rosenbaum and Rubin, 1985; Rubin, 2001 ).”
BackgroundMethodInfluential
Cited by
Items 101–200 of 479
Living on the Edge: Neighborhood Boundaries and the Spatial Dynamics of Violent Crime
Legewie, J. (2018) · Demography · doi:10.1007/s13524-018-0708-1 · Cited by 52
“In computer vision, image processing, and machine learning, edge-detection algorithms are essential tools with applications ranging from image sharpening to robotics and driverless cars (Shapiro and Stockman 2001).”
BackgroundMasaki, T. (2015) · World Development · doi:10.1016/j.worlddev.2015.11.004 · Cited by 49
Bayram, A. B. (2016) · Journal of European Public Policy · doi:10.1080/13501763.2015.1114659 · Cited by 52
“The sample is suitable for this purpose because there is no obvious theoretical reason to think that the process linking European identity to compliance attitudes would be different among those actors who did not participate in the survey (Druckman and Kam 2011; Shadish et al. 2008; Rubin 1974).”
BackgroundAlternatives to Randomized Experiments
West, S. G. (2009) · Current Directions in Psychological Science · doi:10.1111/j.1467-8721.2009.01656.x · Cited by 46
Observational research rigour alone does not justify causal inference
Ejima, K., Li, P., Smith, D. L., Nagy, T. R., Kadish, I., van Groen, T., Dawson, J., Yang, Y., Patki, A., & Allison, D. B. (2016) · European Journal of Clinical Investigation · doi:10.1111/eci.12681 · Cited by 50
Culver, K. C., & Bowman, N. A. (2019) · Research in Higher Education · doi:10.1007/s11162-019-09558-8 · Cited by 54
Rethinking the Role of Political Information
Levendusky, M. S. (2011) · Public Opinion Quarterly · doi:10.1093/poq/nfq070 · Cited by 61
Dorie, V., Hill, J., Shalit, U., Scott, M. A., & Cervone, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.02641 · Cited by 45
“Yet another option for data to use as a testing ground are constructed observational studies (see, for example, LaLonde and Maynard, 1987; Hill, Reiter and Zanutto, 2004; Shadish, Clark and Steiner, 2008).”
Hennegan, J. M., Henderson, J., & Redshaw, M. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-008616 · Cited by 44
“This rating was used to provide a more nuanced appraisal of study quality and is consistent with recent evidence that the results of nonrandomised designs more closely approximate those of randomised studies when adjustment is made for relevant area-specific characteristics, rather than for characteristics of convenie…”
ResultForbes, S. P., & Dahabreh, I. J. (2020) · Journal of General Internal Medicine · doi:10.1007/s11606-020-05713-5 · Cited by 57
Quantifying the robustness of causal inferences: Sensitivity analysis for pragmatic social science
Frank, K. A., Lin, Q., Xu, R., Maroulis, S. J., & Mueller, A. S. (2022) · Social Science Research · doi:10.1016/j.ssresearch.2022.102815 · Cited by 54
Arceneaux, K., Gerber, A. S., & GREEN, D. P. (2010) · Sociological Methods & Research · doi:10.1177/0049124110378098 · Cited by 48
“Comparing treated individuals to untreated individuals via matching, we obtain 258 Sociological Methods & Research 39(2) an observational estimate and see whether it recovers the experimental benchmark (cf. Arceneaux, Gerber, and Green 2006; LaLonde 1986; Shadish et al. 2009).”
BackgroundMethodAdministrative Climate and Novices' Intent to Remain Teaching
Pogodzinski, B., Youngs, P. A., Frank, K., & Belman, D. L. (2012) · The Elementary school journal · doi:10.1086/667725 · Cited by 49
Ing, M. M., Burnette, J. M., Azzam, T., & Wessler, S. R. (2020) · Educational Researcher · doi:10.3102/0013189x20968097 · Cited by 47
“However, along with a large sample size and consistent measures (Shadish, 2013), the design of this study included a carefully selected control group of first-year students from the same institution, who were taking the same courses for the entire year (Cook et al., 2008; Shadish et al., 2008).”
BackgroundAlvarado, S. E., & Turley, R. N. L. (2012) · Social Science Research · doi:10.1016/j.ssresearch.2012.05.017 · Cited by 51
Administrative Context and Novice Teacher-Mentor Interactions.
Pogodzinski, B. (2015) · Journal of Educational Administration · doi:10.1108/jea-06-2013-0073 · Cited by 41
Hadley, P. A., Rispoli, M., Holt, J. K., Papastratakos, T., Hsu, N., Kubalanza, M. C., & McKenna, M. M. (2016) · Language Learning and Development · doi:10.1080/15475441.2016.1193020 · Cited by 51
A Causal Replication Framework for Designing and Assessing Replication Efforts
Steiner, P. M., Wong, V. C., & Anglin, K. (2019) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000385 · Cited by 52
Self-citationTesting the Validity of the Single Interrupted Time Series Design
D, K. B. A., & Svoronos, T. (2019) · National Bureau of Economic Research · doi:10.3386/w26080 · Cited by 57
Furquim, F., Corral, D., & Hillman, N. (2020) · Higher education · doi:10.1007/978-3-030-31365-4_5 · Cited by 45
Estimates of External Validity Bias When Impact Evaluations Select Sites Nonrandomly
Bell, S. H., Olsen, R. B., Orr, L. L., & Stuart, E. A. (2016) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373715617549 · Cited by 52
Steiner, P. M., & Wong, V. C. (2018) · Evaluation Review · doi:10.1177/0193841x18773807 · Cited by 47
“The second WSC approach was introduced by Shadish, Clark, and Steiner (2008) and is called the independent arm design (first panel of Figure 1).”
MethodSelf-citationHevel, M. S., Martin, G. L., Weeden, D. D., & Pascarella, E. T. (2015) · Journal of college student development · doi:10.1353/csd.2015.0046 · Cited by 50
Dahabreh, I. J., Matthews, A., Steingrimsson, J. A., Scharfstein, D. O., & Stuart, E. A. (2024) · Epidemiologic Reviews · doi:10.1093/epirev/mxac011 · Cited by 55
Kuster, S., Jordan, S., Elhaï, M., Held, U., Steigmiller, K., Bruni, C., Cacciapaglia, F., Vettori, S., Siegert, E., Rednic, S., et al. (2022) · RMD Open · doi:10.1136/rmdopen-2022-002477 · Cited by 45
Unravelling gender composition effects on rule-breaking at school: a focus on study attitudes
Demanet, J., Vanderwegen, P., Vermeersch, H., & Houtte, M. V. (2013) · Gender and Education · doi:10.1080/09540253.2013.772567 · Cited by 43
Kalachaveedu, M., Senthilnathan, R. D., Azhagiyamanavalan, S., Ravi, R., Meenakshisundaram, H., & Dharmarajan, A. (2023) · Phytotherapy Research · doi:10.1002/ptr.7747 · Cited by 48
Finding a Comparison Group: Is Online Crowdsourcing a Viable Option?
Azzam, T., & Jacobson, M. R. (2013) · American Journal of Evaluation · doi:10.1177/1098214013490223 · Cited by 44
“The effectiveness of comparison groups in quasi-experimental designs has been examined by multiple scholars (Bloom, Michalopoulos, & Carolyn, 2005; Cook, Shadish, & Wong, 2008; Shadish, Clark, & Steiner, 2008).”
BackgroundHallberg, K., Cook, T. D., Steiner, P. M., & Clark, M. H. (2016) · Prevention Science · doi:10.1007/s11121-016-0732-6 · Cited by 49
“The data come from Shadish, Clark, and Steiner (2008).”
Self-citationContalbrigo, L., Santis, M. D., Toson, M., Montanaro, M., Farina, L., Costa, A. M., & Nava, F. A. (2017) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph14070683 · Cited by 50
“It is quite well known that psychological treatments are most successful when patients can choose them because they are more motivated to comply with treatments they prefer [84] and this is the case.”
BackgroundHartanto, A., & Yang, H. (2019) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/xlm0000581 · Cited by 48
Hübner, N., Wagner, W., Zitzmann, S., & Nagengast, B. (2023) · Educational Psychology Review · doi:10.1007/s10648-023-09724-6 · Cited by 51
Do Standard Measures of Preschool Quality Used in Statewide Policy Predict School Readiness?
Sabol, T. J., & Pianta, R. C. (2014) · Education Finance and Policy · doi:10.1162/edfp_a_00127 · Cited by 40
Improving the Success of Reentry Programs: Identifying the Impact of Service–Need Fit on Recidivism
Gill, C. E., & Wilson, D. B. (2016) · Criminal Justice and Behavior · doi:10.1177/0093854816682048 · Cited by 69
“…the results from propensity score models can be comparable to those from a randomized controlled trial when a rich set of covariates is included (Shadish et al., 2008), the degree to which the model accounted for the systematic aspects of why some offenders have a better service–need fit than…”
ResultThe Effect of Auditor Characteristics on the Value of Diversification
Lai, S., & Liu, C. (2017) · Auditing A Journal of Practice & Theory · doi:10.2308/ajpt-51831 · Cited by 44
Zhang, Y. W., Wang, J., & Fang, T. H. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.953363 · Cited by 51
High School Environments, STEM Orientations, and the Gender Gap in Science and Engineering DegreesPreprint
Legewie, J., & DiPrete, T. A. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2008733 · Cited by 50
“A number of recent studies that compare experimental with regression and matching estimates, however, have shown that these theoretical advantages are typically negligible in practice (Cook et al. 2009; Shadish et al. 2008; Angrist and Pischke 2008).”
ResultMatching Methods for Causal Inference
King, G. (2011) · Cited by 35
Quasi-experimental study designs series—paper 9: collecting data from quasi-experimental studies
Aloe, A. M., Becker, B. J., Duvendack, M., Valentine, J. C., Shemilt, I., & Waddington, H. S. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.02.013 · Cited by 43
Propensity score methods for causal inference: an overview
Pan, W., & Bai, H. (2018) · Behaviormetrika · doi:10.1007/s41237-018-0058-8 · Cited by 45
Bhushan, N., Steg, L., & Albers, C. J. (2018) · Energy Research & Social Science · doi:10.1016/j.erss.2018.07.027 · Cited by 40
Does Joint Physical Custody “Cause” Children’s Better Outcomes?
Braver, S. L., & Votruba, A. M. (2018) · Journal of Divorce & Remarriage · doi:10.1080/10502556.2018.1454203 · Cited by 34
Moss, B. G., Yeaton, W. H., & LIoyd, J. E. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373713504988 · Cited by 44
“randomization: the first-stage randomized participants to designs (Shadish, Clark, & Steiner, 2008); the second utilized randomization in the RE and either an RD or a nonequivalent control group design (Shadish et al., 2008; Shadish, Galindo, Wong, Steiner, & Cook, 2011).”
MethodFederal Nutrition Program Revisions Impact Low-income Households’ Food Purchases
Ng, S. W., Hollingsworth, B. A., Busey, E., Wandell, J., Miles, D. R., & Poti, J. M. (2018) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2017.12.003 · Cited by 38
Music therapy for the treatment of anxiety: a systematic review with multilevel meta-analyses
de Witte, M., Aalbers, S., Vink, A. C., Friederichs, S. A. H., Knapen, A., Pelgrim, T. C., Lampit, A., Baker, F. A., & van Hooren, S. (2025) · EClinicalMedicine · doi:10.1016/j.eclinm.2025.103293 · Cited by 71
“By incorporating non-randomized studies, which are often conducted under clinically representative conditions, this study enhances the external validity of findings, 45,46 offering a more nuanced understanding of the effects of music therapy on anxiety.”
Chu, C., Umanski, G., Blank, A. E., Grossberg, R. M., & Selwyn, P. A. (2010) · AIDS Care · doi:10.1080/09540121.2010.484456 · Cited by 33
“Because this was a non-randomized/observational study, we incorporated propensity scores (Shadish, Clark, & Steiner 2008 ) in an attempt to account for selection bias that potentially affected where individuals sought care.”
MethodMaxwell, C. D., Davis, R. C., & Taylor, B. G. (2010) · Journal of Experimental Criminology · doi:10.1007/s11292-010-9106-4 · Cited by 33
Bailey, D. H., Fuchs, L. S., Gilbert, J. K., Geary, D. C., & Fuchs, D. (2018) · Child Development · doi:10.1111/cdev.13175 · Cited by 39
Telep, C. W., & Weisburd, D. L. (2016) · Springer series on evidence-based crime policy · doi:10.1007/978-1-4939-3477-5_5 · Cited by 33
Zhou, X., & Xie, Y. (2014) · Sociological Methods & Research · doi:10.1177/0049124114555199 · Cited by 36
“More recently, Shadish, Clark, and Steiner (2008) explored the performances of ordinary least squares (OLS) adjustment and PS adjustment with different sets of predictors in an experimental setting.”
BackgroundBias Reduction in Quasi-Experiments With Little Selection Theory but Many Covariates
Steiner, P. M., Cook, T. D., Li, W., & Clark, M. H. (2014) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2014.978058 · Cited by 33
“This Shadish et al. (2008) design ensures that (a) assessments during the pretest period can function as covariates because they were collected before assigning treatments; (b) the treatment and testing procedures are identical for the randomized and nonrandomized groups in both mathematics and…”
MethodResultSelf-citationHeissel, J. A. (2016) · Economics of Education Review · doi:10.1016/j.econedurev.2016.05.001 · Cited by 42
Frank, K. A., Kim, J., Salloum, S. J., Bieda, K. N., & Youngs, P. A. (2020) · American Educational Research Journal · doi:10.3102/0002831220911065 · Cited by 43
“…covariates (including controls for prior practices, district membership, professional development, and mathematics knowledge for teaching [MKT]) that have proven critical to reducing bias in observational studies (Chetty et al., 2014; Shadish et al., 2008; Steiner et al., 2010; Wong et al., 2017).”
BackgroundMatching Methods for Selection of Participants for Follow-Up
Stuart, E. A., & Lalongo, N. S. (2010) · Multivariate Behavioral Research · doi:10.1080/00273171.2010.503544 · Cited by 36
“See Stuart (in press) for more discussion of this issue and that paper; Schafer and Kang (2008); and Shadish, Clark, and Steiner (2008) for more details on matching methods.”
The MABIC project: An effectiveness trial for reducing risk factors for eating disorders
Sánchez-Carracedo, D., Fauquet, J., López-Guimerà, G., Leiva, D., Puntí, J., Trepat, E., Pàmias, M., & Palao, D. (2016) · Behaviour Research and Therapy · doi:10.1016/j.brat.2015.11.010 · Cited by 30
Bias and High-Dimensional Adjustment in Observational Studies of Peer EffectsPreprint
Eckles, D., & Bakshy, E. (2022) · DSpace@MIT (Massachusetts Institute of Technology) · doi:10.6084/m9.figshare.12851349.v6 · Cited by 34
Mueller, C. E., Gaus, H., & Rech, J. (2013) · American Journal of Evaluation · doi:10.1177/1098214013503182 · Cited by 34
“Although the ideal procedure in doing this would have been to apply a within-study design (Shadish et al., 2008) with random assignment into CSEPP, control and experimental treatment condition, collecting new experimental data was not possible due to resource constraints.”
MethodDesigns of Empirical Evaluations of Nonexperimental Methods in Field Settings
Wong, V. C., & Steiner, P. M. (2018) · Evaluation Review · doi:10.1177/0193841x18778918 · Cited by 37
“Fourth, although the RCT sample sizes for the LaLonde example were comparable to those obtained in Shadish et al. (2008), the PSID and CPS provided much larger pools for drawing comparison cases, which had two potential advantages: First, it may have been easier to find comparable matches for…”
ResultSelf-citationFortson, K., Gleason, P., Kopa, E., & Verbitsky‐Savitz, N. (2014) · Economics of Education Review · doi:10.1016/j.econedurev.2014.11.001 · Cited by 37
Designing Studies That Would Address the Multilayered Nature of Health Care
Murray, D. M., Pennell, M. L., Rhoda, D. A., Hade, E. M., & Paskett, E. D. (2010) · JNCI Monographs · doi:10.1093/jncimonographs/lgq014 · Cited by 28
An International Meta-Analysis of Reading Recovery
D’Agostino, J. V., & Harmey, S. J. (2016) · Journal of Education for Students Placed at Risk (JESPAR) · doi:10.1080/10824669.2015.1112746 · Cited by 40
Early Colleges at Scale: Impacts on Secondary and Postsecondary Outcomes
Lauen, D. L., Barrett, N., Fuller, S. C., & Janda, L. (2017) · American Journal of Education · doi:10.1086/692664 · Cited by 39
“…is as high in practice as in the RCTs. Prospective WSC studies were designed to validate specific QE practices and have shown proof of concept: high-quality educational experiments and well-designed quasiexperiments generate quite close causal estimates (Aiken et al. 1998; Shadish et al. 2008).”
Estimation Based on Nearest Neighbor Matching: From Density Ratio to Average Treatment Effect
Lin, Z., Ding, P., & Han, F. (2023) · Econometrica · doi:10.3982/ecta20598 · Cited by 81
Causal Inference and Generalization in Field Settings: Experimental and Quasi-Experimental Designs
West, S. G., Cham, H., & Liu, Y. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9780511996481.007 · Cited by 32
What Can Be Learned From Empirical Evaluations of Nonexperimental Methods?
Wong, V. C., Steiner, P. M., & Anglin, K. (2018) · Evaluation Review · doi:10.1177/0193841x18776870 · Cited by 32
“ITS Political science Arceneaux, Gerber, and Green (2010) Dependent NECG Green, Leong, Kern, Gerber, and Larimer (2009) Dependent RDD Keele and Titiunik (2015) Dependent RDD Note.”
BackgroundMethodInfluentialSelf-citationKilgo, C. A., & Pascarella, E. T. (2015) · Higher Education · doi:10.1007/s10734-015-9925-3 · Cited by 36
“sense, they are consistent with a growing body of research indicating that traditional covariate adjustment and propensity score matching approaches tend to produce very similar estimates of causal effects—particularly when the data being analyzed were collected with a pretest–posttest longitudinal design (e.g., Foste…”
ResultDolničar, S., Viglia, G., & Kurtaliqi, F. (2025) · Annals of Tourism Research · doi:10.1016/j.annals.2025.104080 · Cited by 31
A comparison of alternative strategies for choosing control populations in observational studies
Steventon, A., Grieve, R., & Sekhon, J. S. (2015) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-014-0135-8 · Cited by 28
“A review of these also concluded that local controls should be preferred (Cook et al. 2008), though the number of constituent studies was small and dominated by labor market interventions (Shadish et al. 2008).”
BackgroundBalancing Treatment and Control Groups in Quasi‐Experiments: An Introduction to Propensity Scoring
Connelly, B. S., Sackett, P. R., & Waters, S. D. (2013) · Personnel Psychology · doi:10.1111/peps.12020 · Cited by 38
Randomized Experiments and Quasi-Experimental Designs in Educational Research
Steiner, P. M., Wroblewski, A., & Cook, T. D. (2009) · Sage eBooks · doi:10.4135/9781452226606.n5 · Cited by 42
Self-citationOrtagus, J. C. (2018) · Research in Higher Education · doi:10.1007/s11162-018-9495-1 · Cited by 29
Challenges and Dilemmas in Implementing Random Assignment in Educational Research
Ong-Dean, C., Huie Hofstetter, C., & Strick, B. R. (2011) · American Journal of Evaluation · doi:10.1177/1098214010376532 · Cited by 26
“To test the effectiveness of propensity scores in controlling for selection bias, Shadish, Clark, and Steiner (2008) assigned students to different training sessions, one on mathematics and the other on vocabulary, by first randomly assigning students to two different assignment methods—one random…”
MethodCook, T. D., Zhu, N., Klein, A., Starkey, P., & Thomas, J. (2020) · Psychological Methods · doi:10.1037/met0000260 · Cited by 41
Padgett, R. D., Salisbury, M. H., An, B. P., & Pascarella, E. T. (2010) · New Directions for Institutional Research · doi:10.1002/ir.370 · Cited by 30
Bias and High-Dimensional Adjustment in Observational Studies of Peer Effects
Eckles, D., & Bakshy, E. (2020) · Journal of the American Statistical Association · doi:10.1080/01621459.2020.1796393 · Cited by 60
“Despite these limitations, of the six observational–experimental comparisons they review, Cook et al. (2008) categorize Diaz and Handa (2006), along with Shadish et al. (2008), as one of the two less ambiguous comparisons.”
BackgroundMethodResultInfluentialLe, Q. A., Doctor, J. N., Zoellner, L. A., & Feeny, N. C. (2018) · Quality of Life Research · doi:10.1007/s11136-018-1833-4 · Cited by 27
Effects of Participant Preferences in Unblinded Randomized Controlled Trials
Floyd, A. H. L., & Moyer, A. E. (2010) · Journal of Empirical Research on Human Research Ethics · doi:10.1525/jer.2010.5.2.81 · Cited by 26
“Second, in the Shadish, Clark, and Steiner (2008) study, the vocabulary and mathematics training may have appealed to some people who are motivated for self-enrichment, but any real desire to improve these skills was limited to the experiment itself.”
BackgroundMethodInfluentialThe Consequences of Fatherhood Transition Among Disadvantaged Male Offenders: Does Timing Matter?
Na, C. (2016) · Journal of Developmental and Life-Course Criminology · doi:10.1007/s40865-015-0022-1 · Cited by 27
“It is known to produce “doubly robust” estimators of ATT ([52]: 215) regardless of the matching methods employed [67, 68].”
BackgroundTanner‐Smith, E. E., & Lipsey, M. W. (2014) · Peabody Journal of Education · doi:10.1080/0161956x.2014.895647 · Cited by 24
“For example, prior to a study where undergraduates were allowed to self-select into a mathematics or vocabulary training program, Shadish, Clark, and Steiner (2008) interviewed student counselors about variables that might be associated with students’ choice of program, which helped identify…”
BackgroundColman, S., Nichols-Barrer, I., Redline, J. E., Devaney, B., Ansell, S. V., & Joyce, T. (2012) · RePEc: Research Papers in Economics · Cited by 24
Sengewald, M.-A. D.-P., Steiner, P. M., & Pohl, S. (2018) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12146 · Cited by 34
Self-citationHow often does random assignment fail? Estimates and recommendations
Goldberg, M. H. (2019) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2019.101351 · Cited by 27
Terzian, M., Li, J., Fräser, M. W., Day, S. H., & Rose, R. A. (2014) · Research on Social Work Practice · doi:10.1177/1049731514534898 · Cited by 25
Design and Analysis in College Impact Research: Which Counts More?
Pascarella, E. T., Salisbury, M. H., & Blaich, C. F. (2013) · Journal of college student development · doi:10.1353/csd.2013.0032 · Cited by 29
Little, S., Stewart, A., & Ryan, N. R. (2018) · International Journal of Offender Therapy and Comparative Criminology · doi:10.1177/0306624x18764524 · Cited by 28
Random Forests Approach for Causal Inference with Clustered Observational Data
Suk, Y., Kang, H., & Kim, J. (2020) · Multivariate Behavioral Research · doi:10.1080/00273171.2020.1808437 · Cited by 27
“As a rule of thumb, a good balance for a covariate is when its absolute standardized mean difference is smaller than 0.1 and its variance ratio is in between 4/5 and 5/4 (Rubin, 2001; Shadish et al., 2008; Steiner et al., 2010).”
BackgroundInfluentialLockyer, A. (2012) · Australian Journal Of International Affairs · doi:10.1080/10357718.2011.637450 · Cited by 21
“…becoming the ‘gold standard’ for causal inference in the social sciences and present many opportunities for policy makers to test their own assumptions about the world (Gerber and Green 2008; Harrison and List 2004; Heckman and Smith 1995; Imai et al. 2008; Paluck 2009; Shadish et al. 2008).”
BackgroundFishbein, D. H., & Dariotis, J. K. (2017) · Prevention Science · doi:10.1007/s11121-017-0851-8 · Cited by 26
Ainembabazi, J. H., Abdoulaye, T., Feleke, S., Alene, A. D., Nguezet, P. M. D., Ndayisaba, P. C., Hicintuka, C., Mapatano, S., & Manyong, V. M. (2018) · World Development · doi:10.1016/j.worlddev.2018.03.013 · Cited by 30
Englis, B. G., & Frederiks, A. J. (2023) · Entrepreneurship Education and Pedagogy · doi:10.1177/25151274231161102 · Cited by 35
Mayhew, M. J., Pascarella, E. T., Trolian, T. L., & Selznick, B. S. (2014) · Research in Higher Education · doi:10.1007/s11162-014-9348-5 · Cited by 25
“study—when a pretest measure of the outcome is specified in the analytic model (e.g., Foster et al. 2009; Hanson et al. 2012; Padgett et al. 2010; Pascarella et al. 2013; Salisbury et al. 2013; Shadish et al. 2008; Shah et al. 2005).”
MethodKramer, D. A., Ortagus, J. C., & Lacy, T. A. (2017) · Research in Higher Education · doi:10.1007/s11162-017-9475-x · Cited by 23
“Shadish et al. (2008) stated that using multi-level covariates in a matching approach more closely approximates estimates from a randomized control trial, but it is not necessary to conduct a multi-level match to achieve balance.”
MethodSundell, K., & Olsson, T. (2017) · Oxford Bibliographies in Social Work · doi:10.1093/obo/9780195389678-0254 · Cited by 21
Qin, X., Wormington, S. V., Guzman-Alvarez, A., & Wang, M. (2021) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2021.1894520 · Cited by 29
“Because we have adjusted for most of the theoretically important pretreatment confounders in the initial analysis, we believe that the results are not highly sensitive to unmeasured pretreatment confounding (Hong et al., 2018; Shadish et al., 2008).”
BackgroundEvaluating the Impacts of an Enhanced Family Self-Sufficiency Program
Santiago, A. M., Galster, G., & Smith, R. J. (2017) · Housing Policy Debate · doi:10.1080/10511482.2017.1295093 · Cited by 23
Hu, X., & Ortagus, J. C. (2019) · Community College Review · doi:10.1177/0091552119850321 · Cited by 24
Is There a Magnet-School Effect? A Multisite Study of MSAP-Funded Magnet Schools
Wang, J., Schweig, J. D., & Herman, J. L. (2017) · Journal of Education for Students Placed at Risk (JESPAR) · doi:10.1080/10824669.2017.1292853 · Cited by 25
Experimental investigation on hole quality in drilling of composite pipes
Tibadia, R., Patwardhan, K., Shah, D. J., Shinde, D., Chaudhari, R., & Kalita, K. (2018) · Transactions of the Canadian Society for Mechanical Engineering · doi:10.1139/tcsme-2017-0045 · Cited by 19
Busenbark, J. R., Frank, K. A., Maroulis, S. J., Xu, R., & Lin, Q. (2021) · Emerald Group Publishing Limited eBooks · doi:10.1108/s1479-838720210000013010 · Cited by 29
Nagengast, B., Marsh, H. W., & Hau, K. (2013) · Sex Roles · doi:10.1007/s11199-013-0261-8 · Cited by 22
“These approaches can be highly effective in removing bias (Rosenbaum and Rubin 1985; Rubin 2001; Rubin and Thomas 1996) under some circumstances (Shadish et al. 2008), and overcome many of the drawbacks of conventional ANCOVA analysis (see Schafer and Kang 2008; Stuart 2010).”
BackgroundMethodResultInfluential
Metadata
- Authors
- Shadish, W. R., Clark, M. H., & Steiner, P. M.
- Affiliations
- Southern Illinois University Carbondale (US)
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2008-12-01
- Volume
- 103
- Issue
- 484
- Pages
- 1334-1344
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2068144165
- Cited by
- 479
- References
- 48
- Citations by year
- 2026: 6 · 2025: 22 · 2024: 9 · 2023: 19 · 2022: 24 · 2021: 29 · 2020: 31 · 2019: 35 · 2018: 31 · 2017: 31 · 2016: 32 · 2015: 33 · 2014: 51 · 2013: 35 · 2012: 29
- FWCI
- 19.93
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- randomized experiments, assignment method, bias mitigation, propensity score adjustment, covariate adjustment, stratification, weighting, mathematical training, vocabulary training, confounding, convenience sampling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.