Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments
The DOI matches its Crossref record; 47 of the 48 references it cites were found in a registry or an index.
Abstract
A key justification for using nonrandomized experiments is that, with proper adjustment, their results can well approximate results from randomized experiments. This hypothesis has not been consistently supported by empirical studies; however, previous methods used to study this hypothesis have confounded assignment method with other study features. To avoid these confounding factors, this study randomly assigned participants to be in a randomized experiment or a nonrandomized experiment. In the randomized experiment, participants were randomly assigned to mathematics or vocabulary training; in the nonrandomized experiment, participants chose their training. The study held all other features of the experiment constant; it carefully measured pretest variables that might predict the condition that participants chose, and all participants were measured on vocabulary and mathematics outcomes. Ordinary linear regression reduced bias in the nonrandomized experiment by 84–94% using covariate-adjusted randomized results as the benchmark. Propensity score stratification, weighting, and covariance adjustment reduced bias by about 58–96%, depending on the outcome measure and adjustment method. Propensity score adjustment performed poorly when the scores were constructed from predictors of convenience (sex, age, marital status, and ethnicity) rather than from a broader set of predictors that might include these. Please see the online supplements for a Letter to the Editor.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“They should be used only under the following conditions:(a) Initial group differences are minimized, especially by identifying intact comparison groups that are local to the treatment group and demonstrably heavily overlap with it on at least pretest measures of the outcome (Cook et al. 2008 ); (b) the process by which treatment subjects select into the intervention group (or are selected into it)”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
In the citing paper
“Direct comparisons of ordinary least squares and propensity score matching methods have not shown much of a difference to date (Glazerman et al. 2003; Bloom et al. 2005; Shadish et al. 2008; Cook et al. 2009)”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
In the citing paper
“In nonexperimental studies, the choice of data analysis technique is not very important for reducing selection bias.”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
Citations
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
(2002) · Cited by 13,211
Self-citationEstimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 3×
“As has been argued in causal inference in general (Rubin 1974; Holland 1986), the best approximation to this true counterfactual may be a group of participants whose assignment method (random or nonrandom) was itself randomly assigned to them, with all other features of the experiment held equal.”
BackgroundMethodMatching as an econometric evaluation estimator: Evidence from evaluating a job training programme
Heckman, J. J., Ichimura, H., & Todd, P. E. (1997) · The Review of Economic Studies · doi:10.2307/2971733 · Cited by 5,486 · cited 9×
“This was not done by Dehejia and Wahba (1999), Heckman et al. (1997), or Hill et al. (2004), or in any other such studies.”
BackgroundResultInfluentialStatistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · cited 2×
“As has been argued in causal inference in general (Rubin 1974; Holland 1986), the best approximation to this true counterfactual may be a group of participants whose assignment method (random or nonrandom) was itself randomly assigned to them, with all other features of the experiment held equal.”
BackgroundDoes matching overcome lalonde’s critique of nonexperimental estimators?
A. Smith, J., & E. Todd, P. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.011 · Cited by 3,176
Reducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082
Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs
Dehejia, R. H., & Wahba, S. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473858 · Cited by 2,395
Variable Selection for Propensity Score Models
Brookhart, M. A., Schneeweiss, S., Rothman, K. J., Glynn, R. J., Avorn, J., & Stürmer, T. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj149 · Cited by 2,336
Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020363010465 · Cited by 2,228 · cited 7×
“score matches to a smaller group of treatment group participant scores (Rosenbaum and Rubin, 1985; Rubin, 2001 ).”
BackgroundMethodInfluential
Cited by
Items 201–300 of 479
Formative Evaluation: Estimating Preliminary Outcomes and Testing Rival Explanations
Henry, G. T., Smith, A. A., Kershaw, D. C., & Zulli, R. A. (2013) · American Journal of Evaluation · doi:10.1177/1098214013502577 · Cited by 27
“…model, implemented for the falsification test and estimating effects, used a large number of covariates which prior research had shown to be beneficial for significantly reducing bias in matched sample studies (Bifulco, 2012; Cook et al., 2008; Glazerman et al., 2003; Shadish et al., 2008).”
MethodAssessing the Effectiveness of Recovery-Oriented ACT in Reducing State Psychiatric Hospital Use
MORRISSEY, J. P., Domino, M. E., & Cuddeback, G. S. (2012) · Psychiatric Services · doi:10.1176/appi.ps.201200095 · Cited by 24
Tumlinson, S. E., Sass, D. A., & Cano, S. M. (2014) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jst143 · Cited by 19
Villar, P. F., & Waddington, H. S. (2019) · Campbell Systematic Reviews · doi:10.1002/cl2.1027 · Cited by 23
“The way that researchers manage this problem may be another factor that effects bias (Shadish et al., 2008) Grouped versus individual data Authors, such as Fraker and Maynard (1987), have compared whether compiling nonrandomised estimates on grouped data (e.g., social security cells), as opposed to…”
BackgroundSørlie, M., & Ogden, T. (2014) · International Journal of School & Educational Psychology · doi:10.1080/21683603.2014.881309 · Cited by 22
“High-validity NECs are repeatedly found to generate approximately the same effect estimates as RCTs (Ragsdale, 1996; Shadish et al., 2008; Shadish, Galindo, Wong, Steiner, & Cook, 2011; Steiner, Cook, Shadish, & Clark, 2010).”
BackgroundHenry, G. T. (2015) · Wiley eBooks · doi:10.1002/9781119171386.ch6 · Cited by 20
Learning from friends: measuring influence in a dyadic computer instructional setting
DeLay, D., Hartl, A. C., Laursen, B. P., Denner, J., Werner, L., Campe, S., & Ortiz, E. (2013) · International Journal of Research & Method in Education · doi:10.1080/1743727x.2013.784961 · Cited by 19
“It is of note that estimates from observational studies that control for pre-test similarity closely approximate estimates from randomized experiments (Shadish, Clark, and Steiner 2008; Steiner et al. 2010).”
BackgroundCausal Disentanglement with Network Information for Debiased Recommendations
Sheth, P., Guo, R., Ding, K., Cheng, L., Candan, K. S., & Liu, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17849-8_21 · Cited by 21
Rialon, K. L., Englum, B. R., Gulack, B. C., Guevara, C. J., Bhattacharya, S. D., Shapiro, M. L., Rice, H. E., Scarborough, J. E., & Adibe, O. O. (2015) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2015.06.009 · Cited by 18
Sánchez‐Carracedo, D., López‐Guimerà, G., Fauquet, J., Barrada, J. R., Pàmias, M., Puntí, J., Querol, M., & Trepat, E. (2013) · BMC Public Health · doi:10.1186/1471-2458-13-955 · Cited by 34
The political consequences of transitions out of marriage in Great Britain
Kern, H. (2010) · Electoral Studies · doi:10.1016/j.electstud.2010.01.004 · Cited by 20
RealCause: Realistic Causal Inference BenchmarkingPreprint
Neal, B., Huang, C.-W., & Raghupathi, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.15007 · Cited by 80
Trends in methodological rigor in intervention research published in school psychology journals
Burns, M. K., Klingbeil, D. A., Ysseldyke, J. E., & Petersen‐Brown, S. (2012) · Psychology in the Schools · doi:10.1002/pits.21637 · Cited by 17
Boruch, R. F., Weisburd, D. L., & Berk, R. A. (2009) · Springer eBooks · doi:10.1007/978-0-387-77650-7_23 · Cited by 17
“Shadish et al. (2008) provided a persuasive illustration that is especially compelling because comparisons between a randomized experiment and an observational study were anticipated as part of their study’s design and before the data were collected.”
BackgroundResultRickles, J. H., & SELTZER, M. H. (2014) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998614559748 · Cited by 24
Gaines, B. J., & Kuklinski, J. H. (2011) · Cambridge University Press eBooks · doi:10.1017/cbo9780511921452.031 · Cited by 20
Kaliba, A. R. M., Gongwe, A. G., Mazvimavi, K., & Yigletu, A. A. (2021) · SAGE Open · doi:10.1177/2158244020979992 · Cited by 18
Thoomes, E. J., Graaf, M. T.-D., Cleland, J. A., Gallina, A., & Falla, D. (2022) · Physical Therapy · doi:10.1093/ptj/pzab312 · Cited by 19
An Experiment of Community-Based Learning Effects on Civic Participation
Lee, T., An, J., Sohn, H., & Yoo, I. T. (2018) · Journal of Political Science Education · doi:10.1080/15512169.2018.1498793 · Cited by 19
“Although randomized experiments are ideal in all cases, when implementing full randomization is impossible, researchers can employ quasi-experimental designs and achieve comparably legitimate results regarding causal relationships (Shadish, Clark, and Steiner 2008).”
BackgroundMcMahon, K., Berger, M. T., Khalsa, K. K., Harden, E., & Khalsa, S. B. S. (2021) · Journal of Child and Family Studies · doi:10.1007/s10826-021-01911-9 · Cited by 28
Discussion of “Are Related Party Transactions Red Flags?”
Jorgensen, B. N., & Morley, J. (2017) · Contemporary Accounting Research · doi:10.1111/1911-3846.12304 · Cited by 18
Methods to Address Confounding and Other Biases in Meta-Analyses: Review and RecommendationsPreprint
Mathur, M., & VanderWeele, T. (2021) · Annual Review of Public Health · doi:10.31219/osf.io/v7dtq · Cited by 81
Suk, Y., & Kang, H. (2022) · Psychometrika · doi:10.1007/s11336-021-09805-x · Cited by 17
Johara, F. T., Benedetti, A., Platt, R. W., Menzies, D., Viiklepp, P., Schaaf, H. S., & Chan, E. D. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01452-1 · Cited by 15
Luo, X., Dasgupta, T., Xie, M., & Liu, R. Y. (2021) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/rssb.12429 · Cited by 29
“We consider an adopted example from Shadish et al. (2008) involving estimating the effect of a vocabulary training course on vocabulary test scores to demonstrate the proposed methodology.”
MethodTeachers Without Borders: Consequences of Teacher Labor Force Mobility
Bastian, K. C., & Henry, G. T. (2015) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373714535662 · Cited by 25
Reducing selection bias in quasi-experimental educational studies
Brooks, C. A., Chavez, O., Tritz, J., & Teasley, S. D. (2015) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2723576.2723614 · Cited by 16
“[4] have found when it comes to interventions variables on proxy-pretests and topic preference together with demographics reduce nearly all bias for language-related outcomes and variables related to demographics, pretests.”
BackgroundCovariate Balance in Bayesian Propensity Score Approaches for Observational Studies
Chen, J., & Kaplan, D. E. (2014) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2014.911396 · Cited by 15
“Applications of this approach can be found in Kurth et al. (2006) and Shadish et al. (2008).”
BackgroundTrends in Methodological Quality in Controlled Trials of Psychological and Social Interventions
Sundell, K., & Åhsberg, E. (2016) · Research on Social Work Practice · doi:10.1177/1049731516633297 · Cited by 18
“This evolution in methodological quality is probably a reflection of advances in methodological theory (Des Jarlais et al., 2004; Flay et al., 2005; Moher et al., 2010; Shadish, Cook, & Campell, 2002; Shadish et al., 2008; Wright & Sim, 2003), statistical methods, analytic software, and the enforcement of guidelines s…”
MethodA critical look at methodologies used to evaluate charter school effectiveness
Ackerman, M., & Egalite, A. J. (2017) · Educational Assessment Evaluation and Accountability · doi:10.1007/s11092-017-9266-0 · Cited by 33
Rehabilitation of persons with dementia: using technology to improve participation
Zarit, S. H., Chiusano, C., Harrison, A. S., Sewell, L., Krause, C. A., & Liu, Y. (2020) · Aging & Mental Health · doi:10.1080/13607863.2020.1711864 · Cited by 17
Rock the MIC: The Matrix of Implied Causation, a Tool for Experimental Design and Model Checking
Brick, T. R., & Bailey, D. H. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920922775 · Cited by 17
Incorporating Both Randomized and Observational Data into a Single Analysis
Kaizar, E. E. (2015) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-010814-020249 · Cited by 19
Shadish, W. R., & Galindo, R. J. (2010) · The Corsini Encyclopedia of Psychology · doi:10.1002/9780470479216.corpsy0765 · Cited by 15
Self-citationComparing Training Approaches for Technological Skill Development in Introductory Statistics Courses
Baglin, J., & Costa, C. D. (2013) · Technology Innovations in Statistics Education · doi:10.5070/t571014007 · Cited by 15
Haphazard intentional allocation and rerandomization to improve covariate balance in experiments
Lauretto, M. D. S., Stern, R. B., Morgan, K. L., Clark, M. H., & Stern, J. M. (2017) · AIP conference proceedings · doi:10.1063/1.4985356 · Cited by 14
“[15], the same dataset used in the paper of Morgan and Rubin [12].”
MethodSelf-citationThoomes, E. J., Graaf, M. T.-D., Cleland, J. A., Gallina, A., & Falla, D. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-043021 · Cited by 17
How Do Propensity Score Methods Measure Up in the Presence of Measurement Error? A Monte Carlo Study
de Gil, P. R., Bellara, A. P., Lanehart, R. E., Lee, R. S., Kim, E., & Kromrey, J. D. (2015) · Multivariate Behavioral Research · doi:10.1080/00273171.2015.1022643 · Cited by 16
Collegiate Diversity Experiences and Students’ Views Regarding Social and Political Involvement
Parker, E. T., & Trolian, T. L. (2015) · NASPA Journal · doi:10.1080/19496591.2015.1080718 · Cited by 14
Held, U., Steurer, J., Pichierri, G., Wertli, M. M., Farshad, M., Brunner, F., Guggenberger, R., Porchet, F., Fekete, T. F., Schmid, U. D., et al. (2019) · Journal of Neurosurgery Spine · doi:10.3171/2019.1.spine181098 · Cited by 15
Internal control quality and relationship‐specific investments by suppliers and customers
Lai, S. (2019) · Journal of Business Finance & Accounting · doi:10.1111/jbfa.12396 · Cited by 14
LURKING INFERENTIAL MONSTERS? QUANTIFYING SELECTION BIAS IN EVALUATIONS OF SCHOOL PROGRAMS
Weidmann, B., & Miratrix, L. W. (2020) · Journal of Policy Analysis and Management · doi:10.1002/pam.22236 · Cited by 25
Rerandomization to Improve Covariate Balance in Randomized ExperimentsPreprint
Morgan, K. L. (2011) · SSRN Electronic Journal · doi:10.2139/ssrn.1885584 · Cited by 14
“This is a small part of a larger study by Shadish et al. (2008) that will be examined in more detail in Sections 3.5 and 4.4.”
InfluentialAssessing the Performance of the “Counterfactual as Self-Estimated by Program Participants”
Mueller, C. E., & Gaus, H. (2014) · American Journal of Evaluation · doi:10.1177/1098214014538487 · Cited by 13
“…approach is capable of delivering reliable estimates of treatment effects, the study depicted in this article followed the example of Cook, Shadish, and Wong (2008) and Shadish, Clark, and Steiner (2008) and employed a randomized experiment as a benchmark to examine the performance of the CSEPP.”
MethodSee, B. H., Morris, R., Gorard, S. A. C., Siddiqui, N., Easterbrook, M. J., Nieuwenhuis, M., Fox, K. J., Harris, P. R., & Banerjee, R. A. (2022) · Educational Research and Evaluation · doi:10.1080/13803611.2021.2022317 · Cited by 13
“We can conclude that our study has successfully conceptually replicated earlier studies in the US, partly because of the good study design (e.g., randomised controlled trial) and large sample sizes in the earlier studies (Patil et al., 2016; Shadish et al., 2008; Steiner et al., 2019).”
ResultTelep, C. W., & Weisburd, D. L. (2014) · Journal of Experimental Criminology · doi:10.1007/s11292-014-9206-7 · Cited by 14
“For example, regression discontinuity designs (e.g., see Cook et al. 2008), or propensity score matching with systematic knowledge of underlying causal structures and rich data to account for them (e.g., see Shadish et al. 2008) have been found to provide outcomes similar to randomized designs.”
BackgroundGaudiano, B. A., Hughes, J. A., & Miller, I. W. (2012) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2012.06.003 · Cited by 16
“Although there are potential methodological and statistical disadvantages to using these designs [34,35], incorporating preferences into clinical trials may improve the external validity of studies and more accurately reflect the effectiveness of treatments when they are implemented in the community [36].”
BackgroundShadish, W. R., Rindskopf, D., & Boyajian, J. G. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.01.035 · Cited by 13
Self-citationWolniak, G. C., Chen-Bendle, E. C., & Tackett, J. L. (2023) · AERA Open · doi:10.1177/23328584231183665 · Cited by 22
“Although there are many advantages of multi-institutional, longitudinal data with repeated measures for studying college student development (Mayhew et al., 2016; Pascarella et al., 2013; Shadish et al., 2008), the present study is not without limitations.”
BackgroundStockard, J. (2011) · Quality & Quantity · doi:10.1007/s11135-011-9652-5 · Cited by 13
Propensity Score Estimates in Multilevel Models for Causal Inference
Eckardt, P. A. (2012) · Nursing Research · doi:10.1097/nnr.0b013e318253a1c4 · Cited by 11
Kim, J., Sun, M., & Youngs, P. A. (2019) · Teachers College Record · doi:10.1177/016146811912100301 · Cited by 13
Lin, Q., Frank, K. A., Bathia, S., Draney, K., Thomas, J., & Anderson, C. (2022) · Journal of Research in Science Teaching · doi:10.1002/tea.21745 · Cited by 14
The Role and Influence of Exclusively Online Degree Programs in Higher Education
Ortagus, J. C., Hughes, R., & Allchin, H. (2024) · American Educational Research Journal · doi:10.3102/00028312231222264 · Cited by 20
“In addition, we employed a covariate balancing approach using standardized mean differences and variance ratios in alignment with recommendations by Shadish et al. (2008) and Rubin (2001).”
MethodEmpirical Innovations in Policy Analysis
Blume, G. H., Scott, T. A., & Pirog, M. A. (2014) · Policy Studies Journal · doi:10.1111/psj.12050 · Cited by 12
Rickles, J. H. (2011) · Evaluation Review · doi:10.1177/0193841x11428644 · Cited by 15
“Similarly, Shadish, Clark, and Steiner (2008, 1341) concluded ‘‘that attention to careful measurement of the selection process can be crucial to the success of subsequent analyses.’’”
BackgroundHallberg, K., & Eno, J. P. (2015) · Elsevier eBooks · doi:10.1016/b978-0-08-097086-8.10504-5 · Cited by 12
In Transit: The Well-Being of Migrants from Transition and Post-Transition CountriesPreprint
Nikolova, M., & Graham, C. L. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2508575 · Cited by 13
Waddington, H. S., Wilson, D. B., Pigott, T., Stewart, G. B., Aloe, A. M., Tugwell, P., & Welch, V. A. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.11.005 · Cited by 11
“[10] or, for an example in health systems, Fretheim et al.”
BackgroundGender, teaching style, classroom composition and alienation from learning: an exploratory study
Hadjar, A., & Backes, S. (2022) · Educational Research · doi:10.1080/00131881.2022.2143388 · Cited by 19
Pike, G. R., & Robbins, K. (2019) · Journal for STEM Education Research · doi:10.1007/s41979-019-00013-y · Cited by 14
Lukesch, V., & Zwick, T. (2020) · Empirical research in vocational education and training · doi:10.1186/s40461-020-00104-w · Cited by 17
“15 It has been shown, that the selection of relevant covariates is more important than the selection of the propensity score matching method (Cook et al. 2008; Pohl et al. 2009; Shadish et al. 2008).”
BackgroundEstimating causal effects with optimization-based methods: A review and empirical comparison
Cousineau, M., Verter, V., Murphy, S. A., & Pineau, J. (2022) · European Journal of Operational Research · doi:10.1016/j.ejor.2022.01.046 · Cited by 18
“If they incorrectly identify these covariates, do not have such covariates at hand or only rely on a convenient set of covariates, then the methods may show poor performance (e.g., Shadish, Clark, & Steiner, 2008 ).”
Curran, M. K., Walker, E. A., Roush, P., & Spratford, M. (2019) · Journal of Speech Language and Hearing Research · doi:10.1044/2018_jslhr-l-astm-18-0238 · Cited by 13
Modelling hospital outcome: problems with endogeneity
Moran, J. L., Santamaria, J. D., & Duke, G. J. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01251-8 · Cited by 12
Furtwengler, S. R. (2015) · Journal of Advanced Academics · doi:10.1177/1932202x15603365 · Cited by 20
Leow, C., & Wen, X. (2016) · Early Education and Development · doi:10.1080/10409289.2016.1208600 · Cited by 11
Preparing for the STEM Pathways? Dual Enrollment and College Major Choice in STEM
Hu, X., & Chan, H. (2023) · The Journal of Higher Education · doi:10.1080/00221546.2023.2241331 · Cited by 15
“As indicated in Table 1 and Figure 2, the standardized mean differences before and after weighting indicated the data met the balancing property of PSM (Rubin, 2001; Shadish et al., 2008).”
MethodRindskopf, D., Shadish, W. R., & Clark, M. H. (2018) · Evaluation Review · doi:10.1177/0193841x18789532 · Cited by 10
“In the present case, the likelihood is the probability density of observing the difference in Shadish et al. (2008) for a given value of y. Finally, we use Bayes’s theorem to combine the prior and the likelihood to get the posterior distribution pðyjyÞ, that is, the probability density of the true…”
BackgroundMethodResultInfluentialSelf-citationMatos, C., van Hunsel, F. P. A. M., Ribeiro, R. T., Ó, D. D., & Raposo, J. F. (2020) · Therapeutic Advances in Drug Safety · doi:10.1177/2042098620953935 · Cited by 11
Campbell, C. M., & Labrecque, R. M. (2022) · Journal of Experimental Criminology · doi:10.1007/s11292-022-09532-y · Cited by 11
“The findings from Dong and Lipsey’s (2018) study were somewhat similar to those found in other studies noting that PSM can approximate RCT results (e.g., Shadish et al., 2008).”
InfluentialWan, S., Brick, T. R., Alvarez-Vargas, D., & Bailey, D. H. (2022) · Developmental Psychology · doi:10.1037/dev0001490 · Cited by 12
Shadish, W. R. (2011) · New Directions for Evaluation · doi:10.1002/ev.369 · Cited by 10
Self-citationSetting up social experiments: the good, the bad, and the ugly
Barnow, B. S. (2010) · Journal for Labour Market Research · doi:10.1007/s12651-010-0042-6 · Cited by 10
“In another recent study, Shadish et al. (2008) conducted an intriguing experiment by randomly assigning one group of individuals to be randomly assigned to treatment status and the other to self-select one of the two treatment options (mathematics or vocabulary training).”
BackgroundMethodological and Analytical Issues in the Psychology of Aging
Ferrer, E. E., & Ghisletta, P. (2010) · Elsevier eBooks · doi:10.1016/b978-0-12-380882-0.00002-4 · Cited by 16
Whittaker, T. A. (2019) · Multivariate Behavioral Research · doi:10.1080/00273171.2019.1663136 · Cited by 11
The Role of College-Bound Friends in College Enrollment Decisions by Race, Ethnicity, and Gender
Alvarado, S. E. (2021) · American Educational Research Journal · doi:10.3102/0002831221996669 · Cited by 14
“The results from PSM may yield similar results as traditional ordinary least squares (OLS), controlling for the same covariates (Bifulco, 2012; Cook et al., 2008; Guarino et al., 2014; Shadish et al., 2002; Shadish et al., 2008).”
ResultRenneberg, B., Schulze, J., Böhme, S., West, S. G., & Schüz, B. (2021) · Applied Psychology Health and Well-Being · doi:10.1111/aphw.12322 · Cited by 11
Jaciw, A. P. (2016) · Evaluation Review · doi:10.1177/0193841x16664456 · Cited by 15
“For example, in education, Shadish, Clark, and Steiner (2008) have compellingly demonstrated that when participants are randomized either into a randomized trial with two treatment conditions (vocabulary or mathematics training) or a quasi-experiment, where individuals are allowed to self-select…”
BackgroundResultInfluentialDickason, C. N., Heinrich, C. J., & Smith, M. E. (2023) · The Journal of Higher Education · doi:10.1080/00221546.2023.2195770 · Cited by 14
“Research on quasi-experimental analyses using administrative data points to the importance of knowing the selection process into programs and having data on the key predictors of selection (Cook et al., 2008; Mueser et al., 2007; Shadish et al., 2008).”
BackgroundHevel, M. S., Weeden, D. D., Pasquesi, K., & Pascarella, E. T. (2015) · NASPA Journal · doi:10.1080/19496591.2015.1018270 · Cited by 10
Wedderhoff, N., & Bošnjak, M. (2020) · Psychologische Rundschau · doi:10.1026/0033-3042/a000484 · Cited by 10
Anderson, J. L., & Krasnozhon, L. (2021) · Decision Sciences Journal of Innovative Education · doi:10.1111/dsji.12237 · Cited by 12
A multivariate strategy to measure and test global imbalance in observational studies
D’Attoma, I., & Camillo, F. (2010) · Expert Systems with Applications · doi:10.1016/j.eswa.2010.08.132 · Cited by 10
Staines, G. L., & Cleland, C. M. (2012) · Review of General Psychology · doi:10.1037/a0026493 · Cited by 9
“By precluding noncompliance, Shadish et al. (2008a, 2008b) eliminated any difference in causal estimators; that is, the TOT analysis in the OS was no different from the ITT analysis in the RCT.”
BackgroundResultInfluentialSurgical Versus Nonsurgical Treatment of Adults With Displaced Fractures of the Proximal Humerus
Keudell, A. G. V., Vrahas, M. S., & Weaver, M. J. (2016) · Journal of Orthopaedic Trauma · doi:10.1097/bot.0000000000000550 · Cited by 9
“We recognize the inherent difficulty in enrolling patients in a randomized surgical trial; however, the low recruitment and possible selection bias makes it more difficult to apply the results of this study to the general population with proximal humerus fractures.(2)”
ResultEvaluating School Vouchers: Evidence from a Within-Study ComparisonPreprint
Anderson, K. P., & Wolf, P. J. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2952967 · Cited by 12
Kim, J., Salloum, S. J., Lin, Q., & Hu, S. (2022) · Teaching and Teacher Education · doi:10.1016/j.tate.2022.103779 · Cited by 8
Nagengast, B., Marsh, H. W., Chiorri, C., & Hau, K. (2014) · Journal of Educational Psychology · doi:10.1037/a0035615 · Cited by 13
Shadish, W. R., Steiner, P. M., & Cook, T. D. (2013) · Journal of Methods and Measurement in the Social Sciences · doi:10.2458/v3i2.16475 · Cited by 7
Are There Cognitive Consequences of Binge Drinking During College?
Trolian, T. L., An, B. P., & Pascarella, E. T. (2016) · Journal of college student development · doi:10.1353/csd.2016.0096 · Cited by 10
Lac, A., & Luk, J. W. (2018) · Psychological Assessment · doi:10.1037/pas0000669 · Cited by 14
Geyer, J., Freiman, L., Lubell, J. M., & Villarreal, M. (2019) · Journal of Consumer Affairs · doi:10.1111/joca.12246 · Cited by 8
Braga, A. A., & Weisburd, D. L. (2013) · Evaluation Review · doi:10.1177/0193841x14524208 · Cited by 8
Evaluating the performance of propensity score matching methods: A simulation study
Jacovidis, J. (2017) · JMU Scholoraly Commons (James Madison University) · Cited by 8
Comparative Regression Discontinuity: A Stress Test With Small Samples
Kisbu‐Sakarya, Y., Cook, T. D., Tang, Y., & Clark, M. H. (2018) · Evaluation Review · doi:10.1177/0193841x18776881 · Cited by 7
“However, such blinding 124 Evaluation Review 42(1) is only partial, since the RE’s ATE estimates were known from Shadish et al. (2008).”
BackgroundMethodSelf-citationFrameworks for Credible Causal Inference in Observational Studies of Family Violence
Rose, R. A. (2018) · Journal of Family Violence · doi:10.1007/s10896-018-0011-3 · Cited by 7
Zhang, Y., Lin, L., Starkopf, L., Chen, J., & Wang, W. W. B. (2021) · Contemporary Clinical Trials · doi:10.1016/j.cct.2021.106492 · Cited by 9
Davis, R. C., & Auchter, B. (2010) · Journal of Experimental Criminology · doi:10.1007/s11292-010-9105-5 · Cited by 7
“Recent empirical research has suggested that, at least in highly controlled psychological experiments, carefully conducted quasi-experiments can reduce sample bias to a level approximating random assignment ( Shadish and Steiner 2008 ).”
BackgroundElectrification of mountainous rural areas and development: A case study of Eastern Pamirs
Sadik‐Zada, E. R., Gatto, A., & Sodatshoeva, N. (2023) · The Electricity Journal · doi:10.1016/j.tej.2023.107307 · Cited by 7
Metadata
- Authors
- Shadish, W. R., Clark, M. H., & Steiner, P. M.
- Affiliations
- Southern Illinois University Carbondale (US)
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2008-12-01
- Volume
- 103
- Issue
- 484
- Pages
- 1334-1344
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2068144165
- Cited by
- 479
- References
- 48
- Citations by year
- 2026: 6 · 2025: 22 · 2024: 9 · 2023: 19 · 2022: 24 · 2021: 29 · 2020: 31 · 2019: 35 · 2018: 31 · 2017: 31 · 2016: 32 · 2015: 33 · 2014: 51 · 2013: 35 · 2012: 29
- FWCI
- 19.93
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- randomized experiments, assignment method, bias mitigation, propensity score adjustment, covariate adjustment, stratification, weighting, mathematical training, vocabulary training, confounding, convenience sampling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.