Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments
The DOI matches its Crossref record; 47 of the 48 references it cites were found in a registry or an index.
Abstract
A key justification for using nonrandomized experiments is that, with proper adjustment, their results can well approximate results from randomized experiments. This hypothesis has not been consistently supported by empirical studies; however, previous methods used to study this hypothesis have confounded assignment method with other study features. To avoid these confounding factors, this study randomly assigned participants to be in a randomized experiment or a nonrandomized experiment. In the randomized experiment, participants were randomly assigned to mathematics or vocabulary training; in the nonrandomized experiment, participants chose their training. The study held all other features of the experiment constant; it carefully measured pretest variables that might predict the condition that participants chose, and all participants were measured on vocabulary and mathematics outcomes. Ordinary linear regression reduced bias in the nonrandomized experiment by 84–94% using covariate-adjusted randomized results as the benchmark. Propensity score stratification, weighting, and covariance adjustment reduced bias by about 58–96%, depending on the outcome measure and adjustment method. Propensity score adjustment performed poorly when the scores were constructed from predictors of convenience (sex, age, marital status, and ethnicity) rather than from a broader set of predictors that might include these. Please see the online supplements for a Letter to the Editor.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“They should be used only under the following conditions:(a) Initial group differences are minimized, especially by identifying intact comparison groups that are local to the treatment group and demonstrably heavily overlap with it on at least pretest measures of the outcome (Cook et al. 2008 ); (b) the process by which treatment subjects select into the intervention group (or are selected into it)”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
In the citing paper
“Direct comparisons of ordinary least squares and propensity score matching methods have not shown much of a difference to date (Glazerman et al. 2003; Bloom et al. 2005; Shadish et al. 2008; Cook et al. 2009)”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
In the citing paper
“In nonexperimental studies, the choice of data analysis technique is not very important for reducing selection bias.”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Shadish et al. 2008
Citations
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
(2002) · Cited by 13,211
Self-citationEstimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 3×
“As has been argued in causal inference in general (Rubin 1974; Holland 1986), the best approximation to this true counterfactual may be a group of participants whose assignment method (random or nonrandom) was itself randomly assigned to them, with all other features of the experiment held equal.”
BackgroundMethodMatching as an econometric evaluation estimator: Evidence from evaluating a job training programme
Heckman, J. J., Ichimura, H., & Todd, P. E. (1997) · The Review of Economic Studies · doi:10.2307/2971733 · Cited by 5,486 · cited 9×
“This was not done by Dehejia and Wahba (1999), Heckman et al. (1997), or Hill et al. (2004), or in any other such studies.”
BackgroundResultInfluentialStatistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · cited 2×
“As has been argued in causal inference in general (Rubin 1974; Holland 1986), the best approximation to this true counterfactual may be a group of participants whose assignment method (random or nonrandom) was itself randomly assigned to them, with all other features of the experiment held equal.”
BackgroundDoes matching overcome lalonde’s critique of nonexperimental estimators?
A. Smith, J., & E. Todd, P. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.011 · Cited by 3,176
Reducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082
Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs
Dehejia, R. H., & Wahba, S. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473858 · Cited by 2,395
Variable Selection for Propensity Score Models
Brookhart, M. A., Schneeweiss, S., Rothman, K. J., Glynn, R. J., Avorn, J., & Stürmer, T. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj149 · Cited by 2,336
Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020363010465 · Cited by 2,228 · cited 7×
“score matches to a smaller group of treatment group participant scores (Rosenbaum and Rubin, 1985; Rubin, 2001 ).”
BackgroundMethodInfluential
Cited by
Items 301–400 of 479
Evaluating Free-Choice Climate Education Interventions Applying Propensity Score Matching
Gaus, H., & Mueller, C. E. (2011) · Evaluation Review · doi:10.1177/0193841x12439703 · Cited by 7
“The application of radius matching represents a compromise between reducing the bias resulting from the differing PS between intervention participants and nonparticipants and the reduction in variance of the estimator resulting from the inclusion of additional information in the computation of treatment effects.1 Furt…”
ResultInfluentialValidity of Survey Response Propensity Indicators: A Behavior Genetics Approach
Littvay, L., Popa, S. A., & Fazekas, Z. (2012) · Social Science Quarterly · doi:10.1111/j.1540-6237.2012.00938.x · Cited by 7
Benefit–Cost Analysis of an Innovative Program for Self-Sufficiency and Homeownership
Galster, G., Santiago, A. M., Smith, R., & Leroux, J. (2019) · Evaluation Review · doi:10.1177/0193841x19846697 · Cited by 9
Collagen Allografts Compared With Autologous Tissue in Tympanoplasty
Cass, N. D., Patten, L. W., & Cass, S. P. (2019) · Otology & Neurotology · doi:10.1097/mao.0000000000002254 · Cited by 7
Self-selection and purchase value of research shoppers
Pallant, J. I., Sands, S., Ferraro, C., & Pallant, J. L. (2020) · International Journal of Retail & Distribution Management · doi:10.1108/ijrdm-12-2019-0387 · Cited by 8
The effect of a “surveillance nurse” telephone support intervention in a home care program
Kelly, R., & Godin, L. (2014) · Geriatric Nursing · doi:10.1016/j.gerinurse.2014.11.004 · Cited by 6
Steele, J., Watzinger‐Tharp, J., Slater, R. O., Roberts-Aguirre, G., & Bowman, K. (2024) · The B E Journal of Economic Analysis & Policy · doi:10.1515/bejeap-2022-0241 · Cited by 9
Biodex© training post-stroke for postural stability in the upper trunk: A pilot study
Nel, H. W., Mudzi, W., van Vuuren, E. C. J., & Musenge, E. (2020) · South African Journal of Physiotherapy · doi:10.4102/sajp.v76i1.1416 · Cited by 6
Brown, S., Song, M., Cook, T. D., & Garet, M. S. (2022) · American Educational Research Journal · doi:10.3102/00028312221136565 · Cited by 6
“In one study, the authors (Shadish et al., 2008; Steiner et al., 2010) examined a comparison group of students who were taking the same course at the same university at the same time as treated students.”
BackgroundVon Real-World-Daten zur Real-World-Evidenz: eine praktische Anleitung
Schneeweiß, S. (2023) · Prävention und Gesundheitsförderung · doi:10.1007/s11553-023-01026-7 · Cited by 7
Canegem, T. V., Vandelannote, I., Houtte, M. V., & Demanet, J. (2023) · Social Psychology of Education · doi:10.1007/s11218-023-09875-y · Cited by 11
“In fact, Shadish et al., (2008; p. 1342) conclude that “ the practice of propensity score analysis in applied research may yield adjustments of unknown or highly variable accuracy ”, and that " their results do not support the preferential use of propensity scores over ordinary linear regression…”
BackgroundLeow, C., Wen, X., & Korfmacher, J. (2014) · Applied Developmental Science · doi:10.1080/10888691.2014.977995 · Cited by 6
“…Issues Regarding the methodological examination, this article demonstrates that, in general, the two methods ended up with similar conclusions and this is consistent with a few other studies that also compared both methods in estimating treatment impacts (e.g., Shadish et al., 2008; Zanutto, 2006).”
BackgroundMethodResultInfluentialJaciw, A. P. (2016) · Evaluation Review · doi:10.1177/0193841x16664457 · Cited by 6
“In education, Shadish, Clark, and Steiner (2008) show that bias from self-selection can be almost completely reduced by modeling covariates that reflect both theory and a common sense determination of why individuals self-select into programs.”
BackgroundDenny, S., Grant, S., Galbreath, R., Utter, J., Fleming, T. M., & Clark, T. C. (2019) · Health Services Research · doi:10.1111/1475-6773.13136 · Cited by 15
Ortagus, J. C., & Hu, X. (2019) · Community College Review · doi:10.1177/0091552119867126 · Cited by 8
Mode Effects on Variances, Covariances, Standard Deviations, and Correlations
Vannieuwenhuyze, J. (2015) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smv009 · Cited by 7
Lessons Learned from Large-Scale Randomized Experiments
Slavin, R. E., & Cheung, A. (2017) · Journal of Education for Students Placed at Risk (JESPAR) · doi:10.1080/10824669.2017.1360774 · Cited by 8
Collier, D., Fitzpatrick, D., Houston, D. A., & Archer, E. (2018) · Journal of College Student Retention Research Theory & Practice · doi:10.1177/1521025120942892 · Cited by 6
“Propensity score weighting can reduce bias by 66% or more (Shadish et al., 2008).”
BackgroundFortson, K., Verbitsky‐Savitz, N., Kopa, E., & Gleason, P. (2012) · National Center for Education Evaluation and Regional Assistance · Cited by 17
Henry, D., Fok, C. C. T., & Allen, J. (2015) · Prevention Science · doi:10.1007/s11121-015-0577-4 · Cited by 6
“evidence suggests conditionswherein methods departing from Btrue^ experiments can produce comparable results (Cook et al. 2008; Shadish et al. 2008; Shadish and Cook 2009).”
Padgett, R. D. (2011) · doi:10.17077/etd.emxxrt83 · Cited by 7
Veteran Cultural Competence Training: Initial Effectiveness and National-Level Implementation
Geraci, J. C., Edwards, E. R., May, D., Halliday, T., Smith‐Isabell, N., El‐Meouchy, P., Lowell, S., Armstrong, N. J., Cantor, G., DeJesus, C., et al. (2023) · Psychiatric Services · doi:10.1176/appi.ps.202100437 · Cited by 7
Causal Inference for Dummies: A Tutorial on Directed Acyclic Graphs and Balancing Weights
Primbs, M. A., Bijlstra, G., Holland, R. W., & Thoemmes, F. J. (2025) · Social Cognition · doi:10.1521/soco.2025.43.3.217 · Cited by 7
Comment: Will Competition-Winning Methods for Causal Inference Also Succeed in Practice?
Zhao, Q., Keele, L. J., & Small, D. S. (2019) · Statistical Science · doi:10.1214/18-sts680 · Cited by 7
Kolar, A., & Steiner, P. M. (2021) · Evaluation Review · doi:10.1177/0193841x211053937 · Cited by 7
“The same view is shared by some within-study-comparisons (Pohl et al., 2009; Shadish et al., 2008) where causal effect estimates from a randomised experiment are compared to the one obtained from a corresponding non-randomised study design, that is, an observational study design.”
BackgroundResultSelf-citationEnhancing External Control Arm Analyses through Data Calibration and Hybrid Designs
Schneeweiß, S. (2024) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.3364 · Cited by 5
An Assessment of the Development and Outcomes of Determinate Sentencing in Florida
Bales, W. D., Gaes, G., Blomberg, T. G., & Pate, K. N. (2010) · Justice Research and Policy · doi:10.3818/jrp.12.1.2010.41 · Cited by 7
D’Attoma, I., Camillo, F., & Clark, M. H. (2017) · The Journal of Experimental Education · doi:10.1080/00220973.2017.1391161 · Cited by 5
“Using the traditional PS subclassification approach, Shadish et al. (2008) were able to reduce selection bias by 71% and by 96% by including additional covariates in the PS model, thereby reducing the difference between the randomized and adjusted nonrandomized effects to nearly zero.”
BackgroundMethodResultInfluentialSelf-citationComment: Strengthening Empirical Evaluation of Causal Inference Methods
Jensen, D. D. (2019) · Statistical Science · doi:10.1214/18-sts690 · Cited by 5
On Measuring and Reducing Selection Bias With a Quasi‐Doubly Randomized Preference Trial
Joyce, T., Remler, D. K., Jaeger, D. A., Altındağ, O., O'Connell, S. D., & Crockett, S. (2017) · Journal of Policy Analysis and Management · doi:10.1002/pam.21976 · Cited by 6
A Sustainable Alternative to the Gold Standard EBP: Validating Existing Programs
Meadowcroft, P., Townsend, M. Z., & Maxwell, A. (2018) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-018-9599-6 · Cited by 6
“, within-group comparisons) can yield useful results that reasonably link program services to client outcomes without the costly and often decades-long process required for RCTs.(9,10) By their very nature, RCTs limit generalization to new settings and varied populations.”
BackgroundDetermination of the Optimal Number of Strata for Bias Reduction in Propensity Score Matching.
Akers, A. (2010) · University of North Texas Digital Library (University of North Texas) · doi:10.12794/metadc28380 · Cited by 7
Adjustment when Covariates are Fallible
Pohl, S., Sengewald, M.-A. D.-P., & Steyer, R. (2016) · Wiley series in probability and statistics · doi:10.1002/9781118947074.ch15 · Cited by 6
Alvarez-Bartolo, D., & MacKinnon, D. P. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251355586 · Cited by 6
“…methods are available for adjusting for measured confounders, including analysis of covariance and weighting approaches (for an explanation of other statistical techniques to incorporate measures of confounding variables, see MacKinnon & Pirlott, 2015; Shadish et al., 2008; Valente et al., 2017).”
Li, J., & Ding, D. (2025) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2025.103018 · Cited by 10
The District-Wide Effectiveness of the Achieve3000 Program: A Quasi-Experimental Study
Borman, G. D., Yang, H., Audrain, R. L., & Park, S. J. (2023) · Journal of Education for Students Placed at Risk (JESPAR) · doi:10.1080/10824669.2023.2218093 · Cited by 4
“As past evidence suggests, matching only on conventional covariates of convenience tends to reduce bias less than matching on a broader array of predictors (Shadish et al., 2008).”
Santiago, A. M., & Leroux, J. (2021) · Journal of Urban Affairs · doi:10.1080/07352166.2021.1944170 · Cited by 6
“…commonly used in program evaluation and has been shown to replicate experimental outcome measures (e.g., Cook & Steiner, 2010; Geyer et al., 2017; Shadish et al., 2008), propensity score matching cannot guarantee the elimination of unmeasured attributes of non-HOP homebuyers that differ…”
MethodUNIVERZA V LJUBLJANI FAKULTETA ZA DRUBENE VEDE
Logar, P., Oblak, T., & Pinter, A. (2009) · Cited by 4
What Can We Learn From A Doubly Randomized Preference Trial?—An Instrumental Variables Perspective
Wing, C., & Clark, M. H. (2016) · Journal of Policy Analysis and Management · doi:10.1002/pam.21965 · Cited by 5
Self-citationAssessing the Impact of College on Students: A Four-Decade Quest to Get It Approximately Right
Pascarella, E. T. (2019) · Higher education · doi:10.1007/978-3-030-03457-3_1 · Cited by 6
Doubly weighted M-estimation for nonrandom assignment and missing outcomes
Negi, A. (2024) · Journal of Causal Inference · doi:10.1515/jci-2023-0016 · Cited by 12
“…teacher impact on student achievement has reported similar findings with pre-test scores [Chetty et al. (2014), Kane and Staiger (2008), and Shadish et al. (2008)], indicating the plausibility of unconfoundedness in 1This is a widely used assumption in the treatment effects literature and…”
ResultWolniak, G. C., & Muskens, M. (2021) · Research in Higher Education · doi:10.1007/s11162-020-09620-w · Cited by 5
Koniewski, M., & Hawrot, A. (2021) · Research Papers in Education · doi:10.1080/02671522.2021.1886318 · Cited by 9
“It can effectively approximate findings from randomised experiments (Pohl et al. 2009; Shadish, Clark, and Steiner 2008), provided that a set of covariates that are well measured and plausibly related to both the selection process and the outcome measures are included, with prior student…”
BackgroundVoorhees, N., Ortagus, J. C., & Marti, E. (2022) · Research in Higher Education · doi:10.1007/s11162-022-09694-8 · Cited by 5
“(+) marks standardized mean difference or variance ratios that exceed acceptable ranges (Rubin, 2001; Shadish et al., 2008).”
BackgroundThompson, J., White, S. J., & Chapman, S. (2020) · Pharmacy · doi:10.3390/pharmacy8030138 · Cited by 4
“This requires data to be collected from two groups at pre-test, followed by an intervention and a second data collection at post-test [28].”
MethodAnalysis of Subgroup Data of Clinical Trials
Tsai, K.-T., & Peace, K. E. (2013) · Journal of Causal Inference · doi:10.1515/jci-2012-0008 · Cited by 4
Telep, C. W. (2013) · doi:10.13021/mars/6469 · Cited by 5
Do Students Know Best? Choice, Classroom Time, and Academic Performance
Joyce, T., Crockett, S., Jaeger, D. A., Altındağ, O., O'Connell, S. D., & Remler, D. K. (2015) · National Bureau of Economic Research · doi:10.3386/w21656 · Cited by 6
Haskins, A. R. (2013) · RePEc: Research Papers in Economics · Cited by 5
Svoronos, T. (2016) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 5
Evaluation in Primary Prevention and Health Promotion
Tebes, J. K., Kaufman, J. S., Connell, C. M., Crusto, C. A., & Thai, N. D. (2014) · Springer eBooks · doi:10.1007/978-1-4614-5999-6_95 · Cited by 5
Grammar learning in a flipped classroom: Measuring achievement and students’ perceptions
Gebregziabher, H. A., Filate, A. Y., & Bishaw, K. S. (2025) · Discover Education · doi:10.1007/s44217-025-00641-0 · Cited by 6
“Quasi-experimental designs are meant to approximate as closely as possible the advantages of true experimental designs [78] by including an intervention and manipulation, but they lack one of the key features of experimental studies, which is randomization [79].”
Time for evidence-based methodology in epidemiology
Goulden, R. (2025) · International Journal of Epidemiology · doi:10.1093/ije/dyaf052 · Cited by 3
Huang, D., Goldschmidt, P., & Matrundola, D. L. T. (2014) · IJREE – International Journal for Research on Extended Education · doi:10.3224/ijree.v2i1.19537 · Cited by 3
“As stated by Shadish, Clark, and Steiner (2008), a rich set of variables and regression-based analyses related to quasi-experimental studies.”
A two-step approach to evaluating sustainability-related consumer education interventions
Gaus, H., & Mueller, C. E. (2016) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2016.10.003 · Cited by 7
(2016) · Cited by 3
Design and Analytic Features for Reducing Biases in Skill-Building Intervention Impact Forecasts
Alvarez-Vargas, D., Wan, S., Fuchs, L. S., Klein, A., & Bailey, D. H. (2022) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2022.2093298 · Cited by 3
“We follow Shadish et al. (2008) in measuring absolute bias as the absolute difference between each forecast (Figure 2, Panel B) and the experimental benchmark (Figure 2, Panel A).”
BackgroundXun, X., Qin, X., Layden, A. J., Yin, Q., Swan, S. H., Barrett, E. S., Bush, N. R., Sathyanarayana, S., & Adibi, J. J. (2022) · Frontiers in Reproductive Health · doi:10.3389/frph.2022.994436 · Cited by 3
“URMC, New York 180 (34) 90 (36) 90 (33)”
Parental Migration in Childhood and Individual Wellbeing in Adulthood
Guo, L. (2023) · Applied Research in Quality of Life · doi:10.1007/s11482-023-10185-w · Cited by 3
“…this study uses the propensity score matching (PSM) approach (Abadie & Imbens, 2016; Heckman et al., 1998; Rosenbaum & Rubin, 1983) to further estimate the linkage between early parental absence and later outcomes, which could help reduce the bias by 58 to 96 per cent (Shadish et al., 2008).”
MethodAcademic achievement of children with autistic symptoms compared to typically developing children
Sari, N. P., Luijk, M. P. C. M., Jansen, P. W., Prinzie, P., & van IJzendoorn, M. H. (2023) · European Journal of Psychology of Education · doi:10.1007/s10212-023-00758-6 · Cited by 8
“In this study, the covariates chosen for propensity matching included both child and maternal characteristics, as covariates of convenience (e.g., sex, age) are usually not sufficient (Shadish et al., 2008) for proper estimation of the propensity score.”
Broda, M., Ekholm, E., Schneider, B., & Hutton, A. C. (2018) · SAGE Open · doi:10.1177/2158244018817397 · Cited by 3
Lizano, F., & Stage, J. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07233-3_20 · Cited by 4
Information Technology Career Preparation for Rural Areas
Hahs‐Vaughn, D. L., Swan, B., & Clark, M. H. (2019) · Community College Journal of Research and Practice · doi:10.1080/10668926.2019.1626302 · Cited by 3
“…bias using propensity score matching (PSM) varies depending on the matching method, common support between the groups before matching, and specific outcomes; PS matching can reduce as much as 95% of selection bias in quasi-experiments (Austin, 2009; Bai, 2011; Shadish, Clark, & Steiner, 2008).”
MethodSelf-citationCommentary on the 2015 SPR Standards of Evidence
Biglan, A., Flay, B. R., & Wagenaar, A. C. (2015) · Prevention Science · doi:10.1007/s11121-015-0582-7 · Cited by 3
Kho, A. (2018) · Cited by 3
Change in Crop Management Strategies Could Double the Maize Yield in AfricaPreprint
Rezaei, E. E., & Gaiser, T. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3005416 · Cited by 3
“The effect of change in management practices such as fertilization management, cultivar selection and watering regime have been tested in field experiments (Shadish et al., 2008).”
BackgroundClark, M. H. (2015) · Elsevier eBooks · doi:10.1016/b978-0-08-097086-8.10557-4 · Cited by 5
Self-citationMüller, C. E. (2012) · Social Science Open Access Repository (GESIS – Leibniz Institute for the Social Sciences) · Cited by 11
Xu, R., Bautista, A., & Yang, W. (2025) · Asia Pacific Journal of Education · doi:10.1080/02188791.2025.2488923 · Cited by 2
“This study employed a quasi-experimental intervention design, which has been a popular choice among educational researchers to examine the effectiveness of interventions (Shadish et al., 2008).”
Hansen, W. B., Derzon, J. H., & Reese, E. L. (2012) · Evaluation & the Health Professions · doi:10.1177/0163278712464772 · Cited by 2
An optimal cluster-based approach for Subgroup Analysis using Information Complexity Criterion
D’Attoma, I., Attoma, N. A., & Liberati, C. (2011) · International Journal of Business Intelligence and Data Mining · doi:10.1504/ijbidm.2011.044978 · Cited by 2
“, 2010), measuring and testing balance (Ho et al., 2007; Iacus et al., 2011; Imai et al., 2008; Shadish et al., 2008; D’Attoma and Camillo, 2011) and PS estimation (King and Zeng, 2006; Ho et al.”
BackgroundReaney, M. D. (2014) · Journal of Diabetes · doi:10.1111/1753-0407.12187 · Cited by 2
Complexity, usefulness, and optimality: A response to Foster (2010).
Widaman, K. F., Dogan, S. J., Stockdale, G. D., & Conger, R. D. (2010) · Developmental Psychology · doi:10.1037/a0021293 · Cited by 2
“This method of bias reduction is similar to the covariate adjustment methods used by Shadish et al. (2008) and is commonly used in developmental psychology.”
Gender differences in hospitalised patients’ perceived reasons for their depression
Gaudiano, B. A., Nowlan, K. M., Hughes, J. A., & Miller, I. W. (2013) · Journal of Psychiatric Intensive Care · doi:10.1017/s1742646413000186 · Cited by 4
Causal Inference Using Semi-parametric Imputation
Piesse, A., Alvarez-Rojas, L., Judkins, D., & Shadish, W. R. (2010) · Cited by 2
Self-citationMatching Strategies for Causal Inference with Observational Data in Education
Kim, Y., Lubanski, S. A., & Steiner, P. M. (2018) · Springer eBooks · doi:10.1007/978-3-319-93539-3_9 · Cited by 3
Self-citationAltındağ, O., Joyce, T., & Reeder, J. A. (2019) · Evaluation Review · doi:10.1177/0193841x19865963 · Cited by 3
“A rich set of covariates may mitigate this source of bias as demonstrated by Shadish et al. (2008).”
BackgroundEdelhoch, M., FLYNN, C., & Liu, Q. (2020) · Journal of Social Policy · doi:10.1017/s0047279420000677 · Cited by 2
What Can the Millions of Random Treatments in Nonexperimental Data Reveal About Causes?
Ribeiro, A. F., Neffke, F. M. H., & Hausmann, R. (2022) · SN Computer Science · doi:10.1007/s42979-022-01319-2 · Cited by 2
“Observational methods generally fail to recover the experimental 1 The assumption that X includes all factors associated both with treatment assignment and outcomes [17, 18].”
Metodology of quantitative research application in context of branch didactics
SAFRANKO, C., & Hašková, A. (2021) · Journal of Technology and Information · doi:10.5507/jtie.2021.012 · Cited by 2
“However, according to Shadish et al. (2008) if well-designed, both aforementioned types of design produce comparable results even from the aspect of internal or external reliability.”
BackgroundComment: Causal Inference Competitions: Where Should We Aim?
Karavani, E., El‐Hay, T., Shimoni, Y., & Yanover, C. (2019) · Statistical Science · doi:10.1214/18-sts679 · Cited by 2
Capture-recapture Estimation for Conflict Data and Hierarchical Models for Program Impact Evaluation
Mitchell, S. (2014) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 4
A Response to the New England Journal of Medicine Article
Keudell, A. G. V., & Sadoghi, P. (2014) · Journal of Spinal Disorders & Techniques · doi:10.1097/bsd.0000000000000214 · Cited by 2
Design‐ and Model‐Based Analysis of Propensity Score Designs
Steiner, P. M. (2016) · Wiley series in probability and statistics · doi:10.1002/9781118947074.ch14 · Cited by 3
Self-citationIdentification of Causal Mechanisms Based on Between-Subject Double Randomization DesignsPreprint
Wunsch, C., & Strobl, R. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3209723 · Cited by 2
Harris, H. D. (2018) · JMU Scholoraly Commons (James Madison University) · Cited by 2
Analysis of covariance: univariate and multivariate approaches
Keller, B., & Marchev, D. (2022) · Elsevier eBooks · doi:10.1016/b978-0-12-818630-5.10066-1 · Cited by 2
Listening to Quackery: Reading John Wesley’s Primitive Physic in an Age of Health Care Reform
Skinner, D., & Schneider, A. J. (2016) · Journal of Medical Humanities · doi:10.1007/s10912-016-9418-0 · Cited by 1
Zhang, H., & Leite, W. L. (2024) · The Journal of Experimental Education · doi:10.1080/00220973.2024.2352766 · Cited by 1
“The study’s population model included 23 covariates obtained from real data from Shadish et al. (2008), while the manipulated conditions contained five levels of reliability (i.e., from 0.5 to 0.9) and three PSA methods (i.e., weighting, stratification, and ordinary least squares regression with…”
InfluentialUthman, M. A., Ede, E. O., & Hassan, A. M. (2024) · Advanced Journal of STEM Education · doi:10.31098/ajosed.v2i1.1451 · Cited by 1
“All the TC II Metalwork students in the four Technical Colleges constitute the sample for the study because a quasi-experimental study according to Shadish et al. (2008) says that the researcher needs a random sample of participants to the test as fair and unbiased as possible.”
BackgroundObesity in Italy: An Empirical Analysis of Healthcare Consumption, Quality of Life and Comorbidities
Brenna, E., & Jommi, C. (2025) · Medicina · doi:10.3390/medicina61061061 · Cited by 1
“The p values of the chi square test before and after matching show that a perfect balance is reached after matching and this evidence is corroborated by the Rubin’s R and B values after matching [42].”
A new four-arm within-study comparison: Design, implementation, and data
Keller, B., Wong, V. C., Park, S., Zhang, J., Sheehan, P., & Steiner, P. M. (2025) · Observational Studies · doi:10.1353/obs.2025.a963647 · Cited by 3
“This feature, suggested by Hill (2008), allows us, under some mild assumptions, to produce RCT benchmark estimates for estimands identified by random assignment.”
Self-citationOn regression-adjusted imputation estimators of average treatment effects
Lin, Z., & Han, F. (2025) · Journal of Econometrics · doi:10.1016/j.jeconom.2025.106080 · Cited by 19
Bowman, N. A., Tennessen, N. F., Katsumoto, S., & Irwin, L. N. (2025) · Research in Higher Education · doi:10.1007/s11162-025-09871-5 · Cited by 3
“That said, previous research has indicated that the quality of covariates is far more important for reducing bias than the choice of analysis (e.g., Pascarella et al., 2013 ; Pohl et al., 2009; Shadish et al., 2008).”
Evaluating multiple candidates simultaneously reduces racial disparities in promotion and tenure
Masters-Waage, T. C., Madera, J. M., Edema-Sillo, E., Aubin, A. S., Lindner, P., Gaytan, M., Yu, H. C., & Spitzmueller, C. (2026) · Nature Communications · doi:10.1038/s41467-026-69937-5 · Cited by 2
Zhu, C., Lu, Y., Tao, M., Yin, T., Li, J., Thompson, S. W., & Gu, N. (2026) · Frontiers in Physiology · doi:10.3389/fphys.2026.1750821 · Cited by 1
Tao, M., Yin, M., Zhong, Y., Xu, K., Zhu, C., Deng, J., Song, Y., Lyu, M., Chen, Z., Nassis, G. P., et al. (2026) · International Journal of Sports Science & Coaching · doi:10.1177/17479541261428779 · Cited by 1
“122 Furthermore, our study found that under higher frequency RT, training protocol, participant characteristics, and study design all had signi fi cant moderating effects on muscle strength.”
Evaluating the Effects of Standardized Patient Care Pathways on Clinical Outcomes
Romanova, A. V. (2015) · Cited by 1
Conditions for Effective Application of Analysis of Symmetrically-Predicted Endogenous Subgroups
Peck, L. R. (2015) · American Journal of Evaluation · doi:10.1177/1098214015600514 · Cited by 1
“Any evaluation with this set of variables could consider the ASPES approach for exploring the relative effectiveness of the program on endogenously defined subgroups, but, as Shadish et al. (2008) note, additional rich variables can be quite valuable.”
Method
Metadata
- Authors
- Shadish, W. R., Clark, M. H., & Steiner, P. M.
- Affiliations
- Southern Illinois University Carbondale (US)
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2008-12-01
- Volume
- 103
- Issue
- 484
- Pages
- 1334-1344
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2068144165
- Cited by
- 479
- References
- 48
- Citations by year
- 2026: 6 · 2025: 22 · 2024: 9 · 2023: 19 · 2022: 24 · 2021: 29 · 2020: 31 · 2019: 35 · 2018: 31 · 2017: 31 · 2016: 32 · 2015: 33 · 2014: 51 · 2013: 35 · 2012: 29
- FWCI
- 19.93
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- randomized experiments, assignment method, bias mitigation, propensity score adjustment, covariate adjustment, stratification, weighting, mathematical training, vocabulary training, confounding, convenience sampling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.