Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments
The DOI matches its Crossref record; 47 of the 48 references it cites were found in a registry or an index.
Abstract
A key justification for using nonrandomized experiments is that, with proper adjustment, their results can well approximate results from randomized experiments. This hypothesis has not been consistently supported by empirical studies; however, previous methods used to study this hypothesis have confounded assignment method with other study features. To avoid these confounding factors, this study randomly assigned participants to be in a randomized experiment or a nonrandomized experiment. In the randomized experiment, participants were randomly assigned to mathematics or vocabulary training; in the nonrandomized experiment, participants chose their training. The study held all other features of the experiment constant; it carefully measured pretest variables that might predict the condition that participants chose, and all participants were measured on vocabulary and mathematics outcomes. Ordinary linear regression reduced bias in the nonrandomized experiment by 84–94% using covariate-adjusted randomized results as the benchmark. Propensity score stratification, weighting, and covariance adjustment reduced bias by about 58–96%, depending on the outcome measure and adjustment method. Propensity score adjustment performed poorly when the scores were constructed from predictors of convenience (sex, age, marital status, and ethnicity) rather than from a broader set of predictors that might include these. Please see the online supplements for a Letter to the Editor.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“They should be used only under the following conditions:(a) Initial group differences are minimized, especially by identifying intact comparison groups that are local to the treatment group and demonstrably heavily overlap with it on at least pretest measures of the outcome (Cook et al. 2008 ); (b) the process by which treatment subjects select into the intervention group (or are selected into it)”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
In the citing paper
“Direct comparisons of ordinary least squares and propensity score matching methods have not shown much of a difference to date (Glazerman et al. 2003; Bloom et al. 2005; Shadish et al. 2008; Cook et al. 2009)”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
In the citing paper
“In nonexperimental studies, the choice of data analysis technique is not very important for reducing selection bias.”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
Citations
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
(2002) · Cited by 13,211
Self-citationEstimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 3×
“As has been argued in causal inference in general (Rubin 1974; Holland 1986), the best approximation to this true counterfactual may be a group of participants whose assignment method (random or nonrandom) was itself randomly assigned to them, with all other features of the experiment held equal.”
BackgroundMethodMatching as an econometric evaluation estimator: Evidence from evaluating a job training programme
Heckman, J. J., Ichimura, H., & Todd, P. E. (1997) · The Review of Economic Studies · doi:10.2307/2971733 · Cited by 5,486 · cited 9×
“This was not done by Dehejia and Wahba (1999), Heckman et al. (1997), or Hill et al. (2004), or in any other such studies.”
BackgroundResultInfluentialStatistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · cited 2×
“As has been argued in causal inference in general (Rubin 1974; Holland 1986), the best approximation to this true counterfactual may be a group of participants whose assignment method (random or nonrandom) was itself randomly assigned to them, with all other features of the experiment held equal.”
BackgroundDoes matching overcome lalonde’s critique of nonexperimental estimators?
A. Smith, J., & E. Todd, P. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.011 · Cited by 3,176
Reducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082
Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs
Dehejia, R. H., & Wahba, S. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473858 · Cited by 2,395
Variable Selection for Propensity Score Models
Brookhart, M. A., Schneeweiss, S., Rothman, K. J., Glynn, R. J., Avorn, J., & Stürmer, T. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj149 · Cited by 2,336
Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020363010465 · Cited by 2,228 · cited 7×
“score matches to a smaller group of treatment group participant scores (Rosenbaum and Rubin, 1985; Rubin, 2001 ).”
BackgroundMethodInfluential
Cited by
Matching Methods for Causal Inference: A Review and a Look Forward
Stuart, E. A. (2010) · Statistical Science · doi:10.1214/09-sts313 · Cited by 5,478
Harder, V. S., Stuart, E. A., & Anthony, J. C. (2010) · Psychological Methods · doi:10.1037/a0019623 · Cited by 802
“…a covariate to the final regression model is also an option, although there is mixed evidence regarding its performance (Rubin, 2001; Rubin, 2004; Shadish et al., 2008); while simply including the PS itself does not generally work well, very recent work has indicated that including non-linear PS…”
BackgroundResultInfluentialGottfredson, D. C., Cook, T. D., Gardner, F. E. M., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015) · Prevention Science · doi:10.1007/s11121-015-0555-x · Cited by 627
“…ordinary least squares and propensity score matching methods have not shown much of a difference to date (Glazerman et al. 2003; Bloom et al. 2005; Shadish et al. 2008; Cook et al. 2009), though the latter is theoretically preferable because it is nonparametric and requires demonstrated overlap…”
BackgroundMaking a Difference in Science Education: The Impact of Undergraduate Research Programs
Eagan, M. K., Hurtado, S., Chang, M. J., Garcia, G. A., Herrera, F. A., & Garibay, J. C. (2013) · American Educational Research Journal · doi:10.3102/0002831213482038 · Cited by 551
“This finding supports work by Shadish et al. (2008) who found that more standard estimation techniques that included rich sets of covariates that extended beyond demographic characteristics performed just as well as propensity score models.”
BackgroundResultAverage causal effects from nonrandomized studies: A practical guide and simulated example.
Schafer, J. L., & Kang, J. (2008) · Psychological Methods · doi:10.1037/a0014268 · Cited by 529
The link between high-impact practices and student learning: some longitudinal evidence
Kilgo, C. A., Ezell Sheets, J. K., & Pascarella, E. T. (2015) · Higher Education · doi:10.1007/s10734-014-9788-z · Cited by 498
“…the same effect estimates, particularly—as in the present study—when a pretest measure of the outcome is specified in the analytic model (e.g., Foster et al. 2009; Hanson et al. 2012; Padgett et al. 2010; Pascarella et al. 2013; Salisbury et al. 2013; Shadish et al. 2008; Shah et al. 2005).”
ResultThe importance of covariate selection in controlling for selection bias in observational studies.
Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H. (2010) · Psychological Methods · doi:10.1037/a0018719 · Cited by 490
“We use data from a within-study comparison (Shadish et al., 2008) in which volunteer undergraduate students from introductory psychology classes at a large mid-south public university were randomly assigned to be in a randomized experiment (N ϭ 235) or in a nonrandomized study (N ϭ 210).”
BackgroundMethodInfluentialSelf-citationHoang, A. T., Nižetić, S., Olcer, A. I., Ong, H. C., Chen, W.-H., Chong, C. T., Thomas, S., Bandh, S. A., & Nguyen, X. P. (2021) · Energy Policy · doi:10.1016/j.enpol.2021.112322 · Cited by 485
Randomisation to protect against selection bias in healthcare trials
Odgaard-Jensen, J., Vist, G. E., Timmer, A., Kunz, R., Akl, E. A., Schünemann, H., Briel, M., Nordmann, A. J., Pregno, S., & Oxman, A. D. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000012.pub3 · Cited by 439
Music therapy for stress reduction: a systematic review and meta-analysis
de Witte, M., Pinho, A. D. S., Stams, G.-J., Moonen, X., Bos, A. E. R., & van Hooren, S. (2022) · Health Psychology Review · doi:10.1080/17437199.2020.1846580 · Cited by 444
“The inclusion of quasi-experimental studies, which have been conducted under clinically representative conditions, increases external validity of meta-analytic findings and substantially increases statistical power of a meta-analysis (Shadish et al., 2002; Shadish et al., 2008).”
BackgroundProcedural Justice and Legal Compliance
Nagin, D. S., & Telep, C. W. (2017) · Annual Review of Law and Social Science · doi:10.1146/annurev-lawsocsci-110316-113310 · Cited by 409
Frank, K. A., Maroulis, S. J., Duong, M. Q., & Kelcey, B. M. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373713493129 · Cited by 406
Terrorist Events and Attitudes toward Immigrants: A Natural Experiment
Legewie, J. (2013) · American Journal of Sociology · doi:10.1086/669605 · Cited by 322
“While the practical benefits of matching remain in dispute (Shadish et al. 2008), matching offers potential advantages that might increase the balance between the treatment and control group.”
BackgroundMethodInfluentialHow to do a good systematic review of effects in international development: a tool kit
Waddington, H., White, H., Snilstveit, B., Hombrados, J. G., Vojtkova, M., Davies, P., Bhavsar, A., Eyers, J., Koehlmoos, T. P., Petticrew, M., et al. (2012) · Journal of Development Effectiveness · doi:10.1080/19439342.2012.711765 · Cited by 278
“Most authors in the quantitative ‘evidence movement’ (for example, Rubin 1974, Cook et al. 2008, Duflo et al. 2008, Shadish et al. 2008, Higgins and Green 2011, Gertler et al. 2011) have agreed that, where an experimental approach based on randomised allocation to the intervention is possible and…”
BackgroundThe High School Environment and the Gender Gap in Science and Engineering
Legewie, J., & DiPrete, T. A. (2014) · Sociology of Education · doi:10.1177/0038040714547770 · Cited by 273
“A number of recent studies that compare experimental with observational estimates show that such a comprehensive set of pretreatment variables is essential to reduce bias in estimates based on regression or matching methods (Shadish et al. 2008).”
BackgroundShadish, W. R., Hedges, L. V., & Pustejovsky, J. E. (2014) · Journal of School Psychology · doi:10.1016/j.jsp.2013.11.005 · Cited by 269
Self-citationDorie, V., Hill, J., Shalit, U., Scott, M., & Cervone, D. (2019) · Statistical Science · doi:10.1214/18-sts667 · Cited by 251
“Yet another option for data to use as a testing ground are constructed observational studies (see, for example, LaLonde and Maynard, 1987; Hill, Reiter and Zanutto, 2004; Shadish, Clark and Steiner, 2008).”
Propensity score matching in SPSS
Thoemmes, F. J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1201.6385 · Cited by 151
“Covariates of convenience (e.g. gender, age, income) are usually not sufficient (Shadish, Clark, & Steiner, 2008) instead researchers should strive to build a convincing case that no unobserved confounders are omitted.”
BackgroundBridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
(2010) · PsycEXTRA Dataset · doi:10.1037/e549902010-001 · Cited by 216
The Effect of Study Abroad on Intercultural Competence Among Undergraduate College Students
Salisbury, M. H., An, B. P., & Pascarella, E. T. (2013) · Journal of Student Affairs Research and Practice · doi:10.1515/jsarp-2013-0001 · Cited by 215
Effects of Parents' Migration on the Education of Children Left Behind in Rural China
Zhou, M., Murphy, R., & Tao, R. (2014) · Population and Development Review · doi:10.1111/j.1728-4457.2014.00673.x · Cited by 212
The Renaissance of Field Experimentation in Evaluating Interventions
Shadish, W. R., & Cook, T. D. (2009) · Annual Review of Psychology · doi:10.1146/annurev.psych.60.110707.163544 · Cited by 208
Self-citationDiemer, M. A., & Rapa, L. J. (2016) · Child Development · doi:10.1111/cdev.12446 · Cited by 192
“Assuming that this key observed confounder may correlate with some unobserved confounding variables that cannot be measured and controlled, this approach may also partially address omitted variables bias (Shadish, Clark, & Steiner, 2008).”
Karkou, V., Aithal, S., Zubala, A., & Meekums, B. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00936 · Cited by 186
Optimal Nonbipartite Matching and Its Statistical Applications
Lu, B., Greevy, R., Xu, X., & Beck, C. (2011) · The American Statistician · doi:10.1198/tast.2011.08294 · Cited by 181
“The ideal setup for conducting causal inference is a well-executed randomized experiment (Shadish, Clark, and Steiner 2008).”
MethodField Experiments in Organizations
Eden, D. (2017) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-041015-062400 · Cited by 167
Report of the AAPOR task force on non-probability sampling
Baker, R., Brick, J. M., Bates, N., Battaglia, M. A., Couper, M. P., Dever, J. A., Gile, K. J., & Tourangeau, R. (2013) · Cited by 198
Steiner, P. M., Cook, T. D., & Shadish, W. R. (2011) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998610375835 · Cited by 148
“In Shadish et al. (2008), volunteer undergraduate students from introductory psychology classes at a large mid-southern public university were randomly assigned to be in a randomized experiment (N ¼ 235) or a nonrandomized study (N ¼ 210).”
BackgroundMethodResultInfluentialSelf-citationA practical guide to propensity score analysis for applied clinical research
Lee, J., & Little, T. D. (2017) · Behaviour Research and Therapy · doi:10.1016/j.brat.2017.01.005 · Cited by 154
Shaping Professional Development to Promote the Diffusion of Instructional Expertise Among Teachers
Sun, M., Penuel, W. R., Frank, K. A., Gallagher, H. A., & Youngs, P. A. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373713482763 · Cited by 199
Valentine, J. C., & Thompson, S. G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1064 · Cited by 130
Sensitivity of matching-based program evaluations to the availability of control variables
Lechner, M., & Wunsch, C. (2013) · Labour Economics · doi:10.1016/j.labeco.2013.01.004 · Cited by 184
Frank, K. A., Zhao, Y., Penuel, W. R., Ellefson, N., & Porter, S. (2011) · Sociology of Education · doi:10.1177/0038040711401812 · Cited by 162
“This allows us to make a stronger case for the inference of effects of our primary factors (Cook, Shadish, and Wong 2008; Shadish, Clark, and Steiner 2008).”
BackgroundQuantifying the substitutability and complementarity between high-speed rail and air transport
Zhang, F., Graham, D. J., & Wong, M. S. C. (2018) · Transportation Research Part A Policy and Practice · doi:10.1016/j.tra.2018.08.004 · Cited by 132
Shadish, W. R. (2010) · Psychological Methods · doi:10.1037/a0015916 · Cited by 143
Self-citationThompson, A. J., & Pickett, J. T. (2019) · Journal of Quantitative Criminology · doi:10.1007/s10940-019-09436-7 · Cited by 139
Conducting Real-world Evidence Studies on the Clinical Outcomes of Diabetes Treatments
Schneeweiß, S., & Patorno, E. (2021) · Endocrine Reviews · doi:10.1210/endrev/bnab007 · Cited by 135
“In the end, all discussions of data quality culminate in the same question: Are the data appropriate for this specific study question? It has been shown in a double randomized experiment that if measurements are sufficient, then nonrandomized studies can produce unbiased estimates like an RCT (43).”
BackgroundSaunders, J., Parast, L., Babey, S. H., & Miles, J. V. (2017) · PLOS ONE · doi:10.1371/journal.pone.0179494 · Cited by 141
“This method has been found to be successful at matching groups and obtaining valid treatment effect estimates [42].”
BackgroundMethodWeisburd, D. L. (2010) · Journal of Experimental Criminology · doi:10.1007/s11292-010-9096-2 · Cited by 129
“These studies suggest that especially when techniques such as regression discontinuity designs or propensity score matching are used the results gained are similar to those from randomized controlled trials (e.g., see Cook et al. 2008; Shadish et al. 2008 ).”
ResultA Primer on Propensity Score Analysis
Shadish, W. R., & Steiner, P. M. (2010) · Newborn and Infant Nursing Reviews · doi:10.1053/j.nainr.2009.12.010 · Cited by 136
Self-citationBifulco, R. (2012) · Journal of Policy Analysis and Management · doi:10.1002/pam.20637 · Cited by 112
Wang, X. (2015) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373714552561 · Cited by 102
Clair, T. S., Cook, T. D., & Hallberg, K. (2014) · American Journal of Evaluation · doi:10.1177/1098214014527337 · Cited by 103
Lu, Y., & Treiman, D. J. (2011) · Social Forces · doi:10.1093/sf/89.4.1119 · Cited by 101
“Recent experimental studies demonstrate that PSM can reduce bias by 58 to 96 percent (Shadish et al. 2008).”
BackgroundText and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates
Keith, K. A., Jensen, D. D., & O’Connor, B. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.474 · Cited by 170
Assessing the Effectiveness of a College Freshman Seminar Using Propensity Score Adjustments
Clark, M. H., & Cundiff, N. L. (2011) · Research in Higher Education · doi:10.1007/s11162-010-9208-x · Cited by 138
“Although it is not clear which propensity score adjustment method is most effective (Kurth et al. 2006; Shadish et al. 2002, 2008 ), many studies have found that propensity score adjustments are effective in reducing selection bias regardless of the adjustment method used (Shadish et al.; Stu ¨rmer et al. 2006).”
MethodSelf-citationKuß, O., Legler, T., & Börgermann, J. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.005 · Cited by 107
A Call for Examining Replication and Bias in Special Education Research
Cook, B. G. (2014) · Remedial and Special Education · doi:10.1177/0741932514528995 · Cited by 108
“For example, Shadish, Clark, and Steiner (2008) randomly assigned university students to participate in either a randomized or non-randomized trial on the effects of math and vocabulary training.”
BackgroundWermink, H. T., BLOKLAND, A. A. J., Nieuwbeerta, P., Nagin, D. S., & Tollenaar, N. (2010) · Journal of Experimental Criminology · doi:10.1007/s11292-010-9097-1 · Cited by 132
“Furthermore, results from propensity score adjustments have been shown to approximate results from randomized experiments (see, for more information, Shadish et al. 2008).”
ResultRerandomization to Balance Tiers of Covariates
Morgan, K. L., & Rubin, D. B. (2015) · Journal of the American Statistical Association · doi:10.1080/01621459.2015.1079528 · Cited by 97
“We first focus on the initial randomization of the Shadish, Clark, and Steiner (2008) data, randomizing 445 participants into the randomized experiment (nC = 235) and the observational study (nT = 210), matching the sample sizes obtained in the actual study.”
BackgroundMethodInfluentialRen, J., Sha, A., Han, D., Li, P., Geng, J., & Ma, C. (2014) · Fertility and Sterility · doi:10.1016/j.fertnstert.2014.03.030 · Cited by 93
Matching and Propensity Scores
Steiner, P. M., & Cook, D. (2013) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199934874.013.0013 · Cited by 124
“Given such a PS and an additional covariance adjustment in the outcome model, the impact of choosing a specific matching or PS methods on the treatment effect and its standard error is relatively small (Schafer & Kang, 2008; Shadish, Clark & Steiner, 2008).”
BackgroundMethodSelf-citationCritical action as a pathway to social mobility among marginalized youth.
Rapa, L. J., Diemer, M. A., & Bañales, J. (2018) · Developmental Psychology · doi:10.1037/dev0000414 · Cited by 89
Propensity score analysis: promise, reality and irrational exuberance
Shadish, W. R. (2012) · Journal of Experimental Criminology · doi:10.1007/s11292-012-9166-8 · Cited by 100
Self-citationChaffin, M. J., Bard, D., BigFoot, D. S., & Maher, E. J. (2012) · Child Maltreatment · doi:10.1177/1077559512457239 · Cited by 99
A randomized experiment comparing random and cutoff-based assignment.
Shadish, W. R., Galindo, R., Wong, V. C., Steiner, P. M., & Cook, T. D. (2011) · Psychological Methods · doi:10.1037/a0023345 · Cited by 86
Self-citationKalita, K., Shivakoti, I., & Ghadai, R. K. (2017) · Materials and Manufacturing Processes · doi:10.1080/10426914.2017.1303156 · Cited by 85
“Additionally, a wide body of mathematical theory discovers the significances of making such allocation of units and their treatments via random mechanism [24, 25].”
BackgroundThe Impact of First-Year Seminars on College Students’ Life-long Learning Orientations
Padgett, R. D., Keup, J. R., & Pascarella, E. T. (2013) · NASPA Journal · doi:10.1515/jsarp-2013-0011 · Cited by 122
Cook, T. D., Steiner, P. M., & Pohl, S. (2009) · Multivariate Behavioral Research · doi:10.1080/00273170903333673 · Cited by 81
Self-citationFerraro, P. J., & Miranda, J. J. (2014) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2014.03.008 · Cited by 96
Marital Dissolution and Major Depression in Midlife: A Propensity Score Analysis
Sbarra, D. A., Emery, R. E., Beam, C. R., & Ocker, B. L. (2014) · Clinical Psychological Science · doi:10.1177/2167702613498727 · Cited by 81
Hedman, E., Mörtberg, E., Hesser, H., Clark, D. M., Lekander, M., Andersson, E., & Ljótsson, B. (2013) · Behaviour Research and Therapy · doi:10.1016/j.brat.2013.07.006 · Cited by 75
Internal Control Quality and Investment Efficiency
Lai, S., Liu, C., & Chen, S. (2020) · Accounting Horizons · doi:10.2308/horizons-12-148 · Cited by 88
Raker, E. J. (2020) · Demography · doi:10.1007/s13524-020-00862-y · Cited by 88
“Tj, t − 1 and Tj, t + 1 are lead and lag treatment (dummy) variables, respectively, which identify changes before (lead) a tornado occurs and after (lag) a tornado occurs (Shadish et al. 2008).”
BackgroundHousing Affordability and Child Well-Being
Newman, S. J., & Holupka, C. S. (2014) · Housing Policy Debate · doi:10.1080/10511482.2014.899261 · Cited by 96
Berk, R. A., Barnes, G. C., Ahlman, L., & Kurtz, E. (2010) · Journal of Experimental Criminology · doi:10.1007/s11292-010-9095-3 · Cited by 88
Armstrong, E., Eggins, E., Reid, N., Harnett, P. H., & Dawe, S. (2017) · Journal of Experimental Criminology · doi:10.1007/s11292-017-9290-6 · Cited by 98
“Given that effect sizes are inconsistently affected by the methodological quality of evaluation studies more generally, and the type of research design specifically (for reviews, see e.g., Higgins et al. 2013; Shadish et al. 2008), it was anticipated that study effects may differ depending onwhether studies used rando…”
Evaluation Methods and Measures for Causal Learning Algorithms
Cheng, L., Guo, R., Moraffah, R., Sheth, P., Candan, K. S., & Liu, H. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3150264 · Cited by 122
“settings—have been used in many areas, such as economics [3], marketing [148], and education [149].”
BackgroundCOVID-19 and the Global Shift Progress to Clean Energy
Hoang, A. T., Nguyễn, X. P., Tuấn, L. A., Huynh, T. T., & Pham, V. V. (2021) · Journal of Energy Resources Technology · doi:10.1115/1.4050779 · Cited by 74
In transit: The well-being of migrants from transition and post-transition countries
Nikolova, M., & Graham, C. L. (2015) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2015.02.003 · Cited by 83
Can the Learning Health Care System Be Educated With Observational Data?
Dahabreh, I. J., & Kent, D. M. (2014) · JAMA · doi:10.1001/jama.2014.4364 · Cited by 71
Head Start at Ages 3 and 4 Versus Head Start Followed by State Pre-K: Which Is More Effective?
Jenkins, J. M., Farkas, G., Duncan, G. J., Burchinal, M., & Vandell, D. L. (2015) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373715587965 · Cited by 94
Social norms motivate COVID-19 preventive behaviorsPreprint
Goldberg, M. H., Gustafson, A., Maibach, E. W., van der Linden, S., Ballew, M. T., Bergquist, P., Kotcher, J., Marlon, J. R., Rosenthal, S. A., & Leiserowitz, A. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/9whp4 · Cited by 78
Shager, H. M., Schindler, H. S., Magnuson, K. A., Duncan, G. J., Yoshikawa, H., & Hart, C. M. D. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373712462453 · Cited by 65
“, 2008; Shadish et al., 2008; Shadish et al., 2011). Our meta-analysis is a useful and robust method to summarize studies from Head Start and to improve our understanding of how research design predicts effect sizes. Nevertheless, this study should serve only as an important first step in this line of inquiry. Future…”
BackgroundExploring Colleagues’ Professional Influence on Mathematics Teachers’ Learning
Sun, M., Wilhelm, A. G., Larson, C. J., & Frank, K. A. (2014) · Teachers College Record The Voice of Scholarship in Education · doi:10.1177/016146811411600604 · Cited by 74
“Recently, T. Cook, Shadish and Wong (2008) and Shadish, Clark, and Steiner (2008) showed that estimates from nonrandomized studies that included controls for the precondition of the outcome variable closely approximated estimates from randomized experiments.”
BackgroundKomro, K. A., Flay, B. R., Biglan, A., & Wagenaar, A. C. (2016) · Translational Behavioral Medicine · doi:10.1007/s13142-015-0358-4 · Cited by 62
“Even with the use of propensity score methods, great care must be taken to avoid Bhidden bias^ [19].”
BackgroundMethodFell, M. J., Pagel, L., Chen, C., Goldberg, M. H., Herberz, M., Huebner, G. M., Sareen, S., & Hahnel, U. J. J. (2020) · Energy Research & Social Science · doi:10.1016/j.erss.2020.101646 · Cited by 85
“Researchers can address this concern in the way that is typically recommended for addressing confounding: anticipate how the pandemic might affect your variables of interest, measure this set of variables, and test whether they affect the study’s primary results [24,25].”
BackgroundHow External Institutions Penetrate Schools Through Formal and Informal Leaders
Sun, M., Frank, K. A., Penuel, W. R., & Kim, C. M. (2013) · Educational Administration Quarterly · doi:10.1177/0013161x12468148 · Cited by 73
Unbiased Causal Inference From an Observational Study: Results of a Within-Study Comparison
Pohl, S., Steiner, P. M., Eisermann, J., Soellner, R., & Cook, T. D. (2009) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373709343964 · Cited by 62
“We adopted the 20-item math posttest from Shadish et al. (2008) that required applying rules for transforming exponential equations.”
BackgroundMethodResultInfluentialSelf-citationMokhtarian, P. L., & van Herick, D. (2016) · Journal of Transport and Land Use · doi:10.5198/jtlu.2016.788 · Cited by 73
“2, they use a different denominator than Zhou and Kockelman (2008) and Cao (2009).”
BackgroundDay, J. C., Zahn, M. A., & Tichavsky, L. P. (2014) · Journal of Research in Crime and Delinquency · doi:10.1177/0022427814538033 · Cited by 74
“…selection, pretest measures of the outcome, and other measures that are highly correlated with both selection and the outcome may be particularly important for attaining accurate estimates of the propensity score and reducing selection bias (Shaddish, Clark, and Steiner 2008; Steiner et al. 2010).”
BackgroundThe Effects of Homeownership on Children's Outcomes: Real Effects or Self‐Selection?
Holupka, S., & Newman, S. J. (2012) · Real Estate Economics · doi:10.1111/j.1540-6229.2012.00330.x · Cited by 76
Zarit, S. H., Bangerter, L. R., Liu, Y., & Rovine, M. J. (2016) · Aging & Mental Health · doi:10.1080/13607863.2015.1128881 · Cited by 71
PECVD process parameter optimization: towards increased hardness of diamond-like carbon thin films
Ghadai, R. K., Kalita, K., Mondal, S. C., & Swain, B. P. (2018) · Materials and Manufacturing Processes · doi:10.1080/10426914.2018.1512114 · Cited by 60
“This contributes in distinguishing a ‘true and rigorous experiment’ from an observational study or a quasi-experiment [21,22].”
BackgroundWhy Attentions May Not Be Interpretable?
Bai, B., Liang, J., Zhang, G., Li, H., Bai, K., & Wang, F. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467307 · Cited by 99
Moser, S. E., West, S. G., & Hughes, J. N. (2012) · Journal of Educational Psychology · doi:10.1037/a0027571 · Cited by 85
“Shadish, Clark, and Steiner (2008) showed in an experiment in which participants were randomized to an experiment or a parallel observational study in which participants chose their treatment condition that the estimates of the treatment effect in the randomized and non-randomized arms did not…”
BackgroundLee, B. R., Ebesutani, C. K., Kolivoski, K. M., Becker, K. D., Lindsey, M. A., Brandt, N. E., Cammack, N. L., Strieder, F. H., Chorpita, B. F., & Barth, R. P. (2014) · American Journal of Orthopsychiatry · doi:10.1037/h0099811 · Cited by 67
The Organization as a Filter of Institutional Diffusion
Penuel, W. R., Frank, K. A., Sun, M., Kim, C. M., & Singleton, C. (2013) · Teachers College Record The Voice of Scholarship in Education · doi:10.1177/016146811311500105 · Cited by 70
“…instruction (as was done in the models we report in Table 4), the impacts of highest grade taught and perceived value of evidence-based reform strategies were essentially zero (or negative); controlling for the prior negates much of the impact of observed covariates (e.g., Shadish et al., 2008).”
BackgroundCollins, R. L., Martino, S. C., Elliott, M. N., & Miu, A. (2011) · Developmental Psychology · doi:10.1037/a0022563 · Cited by 69
Empirical Performance of Covariates in Education Observational Studies
Wong, V. C., Valentine, J. C., & Miller-Bains, K. (2016) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2016.1164781 · Cited by 56
Legewie, J. (2012) · KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie · doi:10.1007/s11577-012-0158-5 · Cited by 86
Marcus, S. M., Stuart, E. A., Wang, P., Shadish, W. R., & Steiner, P. M. (2012) · Psychological Methods · doi:10.1037/a0028031 · Cited by 55
“The Shadish et al. (2008) vocabulary (Treatment 1) versus mathematics training (Treatment 0) data were carefully constructed to have little noncompliance or missing data.”
BackgroundMethodResultInfluentialSelf-citationEstimation of causal effects of binary treatments in unconfounded studies
Gutman, R., & Rubin, D. B. (2015) · Statistics in Medicine · doi:10.1002/sim.6532 · Cited by 61
“[44], which was designed to estimate the effects of a specific training program on mathematics and vocabulary scores of undergraduates; thus, the potential outcomes were two-dimensional vectors of the two scores.”
BackgroundShadish, W. R. (2011) · Research on Social Work Practice · doi:10.1177/1049731511403324 · Cited by 55
“The first demonstration of this was brief in Shadish et al. (2008), showing that statistical adjustments eliminated nearly all bias when all 23 covariates were used but little or no bias when only a limited set of demographic covariates were used.”
BackgroundMethodResultInfluentialSelf-citationWeighting-Based Sensitivity Analysis in Causal Mediation Studies
Hong, G., Qin, X., & Yang, F. (2018) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998617749561 · Cited by 51
Furquim, F., Corral, D., & Hillman, N. (2019) · Higher education · doi:10.1007/978-3-030-11743-6_5-1 · Cited by 60
Teacher Preparation Policies and Their Effects on Student Achievement
Henry, G. T., Bastian, K. C., Fortner, C. K., Kershaw, D. C., Purtell, K. M., Thompson, C. L., & Zulli, R. A. (2014) · Education Finance and Policy · doi:10.1162/edfp_a_00134 · Cited by 76
Zhang, Y., Cole, S. T., Ricci, P., & Gao, J. (2017) · Journal of Travel Research · doi:10.1177/0047287517741004 · Cited by 51
Short-Term Effects of Imprisonment Length on Recidivism in the Netherlands
Wermink, H. T., Nieuwbeerta, P., Ramakers, A. A. T., Keijser, J. W. D., & Dirkzwager, A. (2017) · Crime & Delinquency · doi:10.1177/0011128716687290 · Cited by 59
The Impact of Diversity Courses on College Students’ Moral Development
Parker, E. T., Barnhardt, C. L., Pascarella, E. T., & McCowin, J. A. (2016) · Journal of college student development · doi:10.1353/csd.2016.0050 · Cited by 61
Metadata
- Authors
- Shadish, W. R., Clark, M. H., & Steiner, P. M.
- Affiliations
- Southern Illinois University Carbondale (US)
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2008-12-01
- Volume
- 103
- Issue
- 484
- Pages
- 1334-1344
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2068144165
- Cited by
- 479
- References
- 48
- Citations by year
- 2026: 6 · 2025: 22 · 2024: 9 · 2023: 19 · 2022: 24 · 2021: 29 · 2020: 31 · 2019: 35 · 2018: 31 · 2017: 31 · 2016: 32 · 2015: 33 · 2014: 51 · 2013: 35 · 2012: 29
- FWCI
- 19.93
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- randomized experiments, assignment method, bias mitigation, propensity score adjustment, covariate adjustment, stratification, weighting, mathematical training, vocabulary training, confounding, convenience sampling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.