CoChat
Sign Up
Cited by 490Advanced Causal Inference TechniquesMeta-analysis and systematic reviewsStatistical Methods and Bayesian Inference

The importance of covariate selection in controlling for selection bias in observational studies.

Steiner, Peter M., Cook, Thomas D., Shadish, William R., Clark, M. H.·2010·Psychological Methods
DOI 10.1037/a0018719
Open this paper in CoChat — free account

Abstract

The assumption of strongly ignorable treatment assignment is required for eliminating selection bias in observational studies. To meet this assumption, researchers often rely on a strategy of selecting covariates that they think will control for selection bias. Theory indicates that the most important covariates are those highly correlated with both the real selection process and the potential outcomes. However, when planning a study, it is rarely possible to identify such covariates with certainty. In this article, we report on an extensive reanalysis of a within-study comparison that contrasts a randomized experiment and a quasi-experiment. Various covariate sets were used to adjust for initial group differences in the quasi-experiment that was characterized by self-selection into treatment. The adjusted effect sizes were then compared with the experimental ones to identify which individual covariates, and which conceptually grouped sets of covariates, were responsible for the high degree of bias reduction achieved in the adjusted quasi-experiment. Such results provide strong clues about preferred strategies for identifying the covariates most likely to reduce bias when planning a study and when the true selection process is not known.

Integrity signals

Registry

Crossref · 10.1037/a0018719

Cites

54 of 54 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Empirical examples exist where controlling for pre-treatment variables demonstrably increases rather than decreases bias.

    “Empirical examples exist where controlling for pre-treatment variables demonstrably increases rather than decreases bias.”

    From Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable (2014) · cites this paper as Steiner et al. 2010

  • Steiner et al. found that in an observational study of an educational intervention, bias in the estimated effect on math scores increased by about 30% when controlling only for pre-treatment psychological disposition or only for pre-treatment vocabulary test scores.

    “For example, Steiner et al. (2010) report a within-study comparison between an observational study and an experimental benchmark of the effect of an educational intervention on student test scores. In the observational study, the bias in the estimated effect of treatment on math scores increased by about 30% when controlling only for pre-treatment psychological disposition or only for pre”

    From Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable (2014) · cites this paper as Steiner et al. 2010

  • Steiner et al. found that controlling for additional pre-treatment variables generally reduced bias but sometimes increased bias.

    “Furthermore, they found that although controlling for additional pre-treatment variables generally reduced bias, sometimes including additional controls increased bias.”

    From Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable (2014) · cites this paper as Steiner et al. 2010

  • If a pre-treatment collider is also a confounder, conditioning on it generally but not always removes more bias than it induces, especially if the analysis conditions on many other pre-treatment variables.

    “If the pre-treatment collider is also a confounder, then conditioning on it will generally, but not always, remove more bias than it induces (Greenland 2003), especially if the analysis further conditions on many other pre-treatment variables (Steiner et al. 2010).”

    From Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable (2014) · cites this paper as Steiner et al. 2010

Cites

Flagged first, then by citations

1 of the 55 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H.
Affiliations
Northwestern University (US)
ORCID
Venue
Psychological Methods
Publisher
American Psychological Association (APA)
Published
2010
Volume
15
Issue
3
Pages
250-267
ISSN
  • 1082-989X (ISSN-L)
  • 1939-1463
DOI
10.1037/a0018719
OpenAlex
W2168206775
Cited by
490
References
55
Citations by year
2026: 23 · 2025: 28 · 2024: 34 · 2023: 26 · 2022: 26 · 2021: 53 · 2020: 32 · 2019: 20 · 2018: 37 · 2017: 36 · 2016: 38 · 2015: 28 · 2014: 31 · 2013: 32 · 2012: 19
FWCI
18.95
Type
journal article
Language
en
Open access
closed
Locations
  • Psychological Methods (publishedVersion)
  • PubMed (publishedVersion)
Funders
US Department of Education (R305U070003)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
Keywords
covariate selection, selection bias, observational studies, quasi-experiments, self-selection, randomized experiments, bias mitigation
MeSH terms
Data Interpretation, Statistical, Epidemiologic Methods, Humans, Mathematics, Random Allocation, Models, Statistical, Selection Bias, Causality, Patient Selection, Behavioral Research, Propensity Score

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review