The DOI matches its Crossref record; 54 of the 55 references it cites were found in a registry or an index.
The importance of covariate selection in controlling for selection bias in observational studies.
Abstract
The assumption of strongly ignorable treatment assignment is required for eliminating selection bias in observational studies. To meet this assumption, researchers often rely on a strategy of selecting covariates that they think will control for selection bias. Theory indicates that the most important covariates are those highly correlated with both the real selection process and the potential outcomes. However, when planning a study, it is rarely possible to identify such covariates with certainty. In this article, we report on an extensive reanalysis of a within-study comparison that contrasts a randomized experiment and a quasi-experiment. Various covariate sets were used to adjust for initial group differences in the quasi-experiment that was characterized by self-selection into treatment. The adjusted effect sizes were then compared with the experimental ones to identify which individual covariates, and which conceptually grouped sets of covariates, were responsible for the high degree of bias reduction achieved in the adjusted quasi-experiment. Such results provide strong clues about preferred strategies for identifying the covariates most likely to reduce bias when planning a study and when the true selection process is not known.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Empirical examples exist where controlling for pre-treatment variables demonstrably increases rather than decreases bias.”
From Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable (2014) · cites this paper as Steiner et al. 2010
In the citing paper
“For example, Steiner et al. (2010) report a within-study comparison between an observational study and an experimental benchmark of the effect of an educational intervention on student test scores. In the observational study, the bias in the estimated effect of treatment on math scores increased by about 30% when controlling only for pre-treatment psychological disposition or only for pre”
From Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable (2014) · cites this paper as Steiner et al. 2010
In the citing paper
“Furthermore, they found that although controlling for additional pre-treatment variables generally reduced bias, sometimes including additional controls increased bias.”
From Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable (2014) · cites this paper as Steiner et al. 2010
In the citing paper
“If the pre-treatment collider is also a confounder, then conditioning on it will generally, but not always, remove more bias than it induces (Greenland 2003), especially if the analysis further conditions on many other pre-treatment variables (Steiner et al. 2010).”
From Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable (2014) · cites this paper as Steiner et al. 2010
Citations
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
(2002) · Cited by 13,211
Self-citationCausality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
Missing data: our view of the state of the art
Schafer, J. L., & Graham, J. W. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.2.147 · Cited by 11,638
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436
The development of markers for the Big-Five factor structure
Goldberg, L. R. (1992) · Psychological Assessment · doi:10.1037/1040-3590.4.1.26 · Cited by 5,530
Statistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293
Reducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082
Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs
Dehejia, R. H., & Wahba, S. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473858 · Cited by 2,395
Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020363010465 · Cited by 2,228 · cited 1×
“After PS-adjustment, standardized mean differences d should be close to zero, and variance ratios v should be close to one (Rubin, 2001).”
BackgroundDoubly robust estimation in missing data and causal inference models
Bang, H., & Robins, J. M. (2005) · Biometrics · doi:10.1111/j.1541-0420.2005.00377.x · Cited by 1,998
Screening Depressed Patients in Family Practice
Beck, A. T., & Beck, R. W. (1972) · Postgraduate Medicine · doi:10.1080/00325481.1972.11713319 · Cited by 1,836
Lunceford, J. K., & Davidian, M. (2004) · Statistics in Medicine · doi:10.1002/sim.1903 · Cited by 1,676
Matching using estimated propensity scores: relating theory to practice.
Rubin, D., & Thomas, N. (1996) · Biometrics · doi:10.1017/cbo9780511810725.024 · Cited by 1,445
(2007) · Cited by 1,428
(2018) · Sage eBooks · doi:10.4135/9781506326139.n223 · Cited by 1,386
Rubin, D. B. (2007) · Statistics in Medicine · doi:10.1002/sim.2739 · Cited by 1,047
Hirano, K., & Imbens, G. W. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020371312283 · Cited by 1,039
The Effectiveness of Adjustment by Subclassification in Removing Bias in Observational Studies
Cochran, W. G. (1968) · Biometrics · doi:10.2307/2528036 · Cited by 961
Semiparametric Efficiency in Multivariate Regression Models with Missing Data
Robins, J. M., & Rotnitzky, A. G. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476494 · Cited by 917
For objective causal inference, design trumps analysis
Rubin, D. B. (2008) · The Annals of Applied Statistics · doi:10.1214/08-aoas187 · Cited by 831
Using matching, instrumental variables and control functions to estimate economic choice models
Heckman, J., & Navarro-Lozano, S. (2004) · Review of Economics and Statistics · doi:10.1162/003465304323023660 · Cited by 768
Stürmer, T., Joshi, M., Glynn, R. J., Avorn, J., Rothman, K. J., & Schneeweiß, S. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.07.004 · Cited by 668
Cook, T. D., Shadish, W. R., & Wong, V. C. (2008) · Journal of Policy Analysis and Management · doi:10.1002/pam.20375 · Cited by 604
Self-citationAverage causal effects from nonrandomized studies: A practical guide and simulated example.
Schafer, J. L., & Kang, J. (2008) · Psychological Methods · doi:10.1037/a0014268 · Cited by 550
Shah, B. R., Laupacis, A., Hux, J. E., & Austin, P. C. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.016 · Cited by 493
Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008) · Journal of the American Statistical Association · doi:10.1198/016214508000000733 · Cited by 479
Self-citationKang, J., & Schafer, J. L. (2007) · Statistical Science · Cited by 368
Nonexperimental Versus Experimental Estimates of Earnings Impacts
Glazerman, S., Levy, D. M., & Myers, D. (2003) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/0002716203254879 · Cited by 352
Evaluating Kindergarten Retention Policy
Hong, G., & Raudenbush, S. W. (2006) · Journal of the American Statistical Association · doi:10.1198/016214506000000447 · Cited by 348
Cook, T. D. (2007) · Journal of Econometrics · doi:10.1016/j.jeconom.2007.05.002 · Cited by 303
Self-citationMatched Sampling for Causal Effects: Contributor Acknowledgments
Rubin, D. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810725 · Cited by 287
Rosenbaum, P. R. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10477060 · Cited by 279
Mathematics Anxiety Effects in Simple and Complex Addition
Faust, M. W. (1996) · Mathematical Cognition · doi:10.1080/135467996387534 · Cited by 211
Diaz, J. J., & Handa, S. (2006) · Journal of Human Resources · doi:10.3368/jhr.xli.2.319 · Cited by 196
Are Experiments the Only Option? A Look at Dropout Prevention Programs
Agodini, R., & Dynarski, M. (2004) · The Review of Economics and Statistics · doi:10.1162/003465304323023741 · Cited by 162
Steiner, P. M., Cook, T. D., & Shadish, W. R. (2011) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998610375835 · Cited by 152
Self-citationApplied Bayesian Modeling And Causal Inference From Incomplete-Data Perspectives
Gelman, A., & Xiaoli, M. (2014) · Cited by 152
Dropping out of High School in the United States: An Observational Study
Rosenbaum, P. R. (1986) · Journal of Educational Statistics · doi:10.3102/10769986011003207 · Cited by 141
Selection bias in evaluating treatment effects: Some formal illustrations
Goldberger, A. S. (2008) · Advances in econometrics · doi:10.1016/s0731-9053(07)00001-1 · Cited by 139
An Evaluation of the Performance of Regression Discontinuity Design on PROGRESA
Buddelmeyer, H., & Skoufias, E. (2004) · World Bank, Washington, DC eBooks · doi:10.1596/1813-9450-3386 · Cited by 131
Comparison of a Randomized and Two Quasi-Experimental Designs in a Single Outcome Evaluation
Aiken, L. S., West, S. G., Schwalm, D. E., Carroll, J. L., & Hsiung, S. (1998) · Evaluation Review · doi:10.1177/0193841x9802200203 · Cited by 105
A Comparison of Experimental and Observational Data Analyses
Hill, J., Reiter, J. P., & Zanutto, E. L. (2004) · Wiley series in probability and statistics · doi:10.1002/0470090456.ch5 · Cited by 96
Klaassen, C. A. J. (2001) · UvA-DARE (University of Amsterdam) · Cited by 93
Cook, T. D., Steiner, P. M., & Pohl, S. (2009) · Multivariate Behavioral Research · doi:10.1080/00273170903333673 · Cited by 81
Self-citationConcordance Between ACT Assessment and Recentered SAT I Sum Scores.
Dorans, N. J., Lyu, C. F., Pommerich, M., & Houston, W. M. (1997) · College and university · Cited by 80
Bloom, H. S., Michalopoulos, C., Hill, C. J., & Lei, Y. (2002) · Cited by 72
Black, D. A., Galdo, J. C., & Smith, J. A. (2007) · American Economic Review · doi:10.1257/aer.97.2.104 · Cited by 66
Bloom, H. S., Michalopoulos, C., Hill, C. J., & Lei, Y. (2002) · Cited by 64
Unbiased Causal Inference From an Observational Study: Results of a Within-Study Comparison
Pohl, S., Steiner, P. M., Eisermann, J., Soellner, R., & Cook, T. D. (2009) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373709343964 · Cited by 64
Self-citationAnalyzing Individual and Average Causal Effects via Structural Equation Models
Steyer, R. (2005) · Methodology · doi:10.1027/1614-1881.1.1.39 · Cited by 63
Hong, G., & Hong, Y. (2008) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373708328259 · Cited by 55
Predicting the Efficacy of Future Training Programs Using Past Experiences
Hotz, V. J., Imbens, G. W., & Mortimer, J. H. (1999) · National Bureau of Economic Research · doi:10.3386/t0238 · Cited by 54
Kaplan, D. M. (1999) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3404_4 · Cited by 25
Cited by
Sorted by citations
Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable
Elwert, F., & Winship, C. (2014) · Annual Review of Sociology · doi:10.1146/annurev-soc-071913-043455 · Cited by 1,168
Predictors of early growth in academic achievement: the head-toes-knees-shoulders task
McClelland, M. M., Cameron, C. E., Duncan, R., Bowles, R. P., Acock, A. C., Miao, A., & Pratt, M. E. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00599 · Cited by 597
Barker, T. H., Stone, J. C., Sears, K., Klugar, M., Leonardi-Bee, J., Tufanaru, C., Aromataris, E., & Munn, Z. (2023) · JBI Evidence Synthesis · doi:10.11124/jbies-22-00125 · Cited by 471
Frank, K. A., Maroulis, S. J., Duong, M. Q., & Kelcey, B. M. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373713493129 · Cited by 406
Using the Propensity Score Method to Estimate Causal Effects
Li, M. (2012) · Organizational Research Methods · doi:10.1177/1094428112447816 · Cited by 382
Metadata
- Authors
- Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H.
- Affiliations
- Northwestern University (US)
- Venue
- Psychological Methods
- Published
- 2010
- Volume
- 15
- Issue
- 3
- Pages
- 250-267
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2168206775
- Cited by
- 490
- References
- 55
- Citations by year
- 2026: 23 · 2025: 28 · 2024: 34 · 2023: 26 · 2022: 26 · 2021: 53 · 2020: 32 · 2019: 20 · 2018: 37 · 2017: 36 · 2016: 38 · 2015: 28 · 2014: 31 · 2013: 32 · 2012: 19
- FWCI
- 18.95
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- Funders
- US Department of Education (R305U070003)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- covariate selection, selection bias, observational studies, quasi-experiments, self-selection, randomized experiments, bias mitigation
- MeSH terms
- Data Interpretation, Statistical, Epidemiologic Methods, Humans, Mathematics, Random Allocation, Models, Statistical, Selection Bias, Causality, Patient Selection, Behavioral Research, Propensity Score
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
