Unbiased Causal Inference From an Observational Study: Results of a Within-Study Comparison
The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Adjustment methods such as propensity scores and analysis of covariance are often used for estimating treatment effects in nonexperimental data. Shadish, Clark, and Steiner used a within-study comparison to test how well these adjustments work in practice. They randomly assigned participating students to a randomized or nonrandomized experiment. Treatment effects were then estimated in the experiment and compared to the adjusted nonexperimental estimates. Most of the selection bias in the nonexperiment was reduced. The present study replicates the study of Shadish et al. despite some differences in design and in the size and direction of the initial bias. The results show that the selection of covariates matters considerably for bias reduction in nonexperiments but that the choice of analysis matters less.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · cited 1×
“However, in practice, we have to deal with the fundamental problem of causal inference (Holland, 1986; Rubin, 1978); that is, we never observe both potential outcomes simultaneously.”
BackgroundIdentification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230
Reducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082
Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020363010465 · Cited by 2,228 · cited 3×
“We estimated PSs using logistic regression and selected the PS model according to two balance metrics—the standardized mean difference and the variance ratio between treatment groups (Rubin, 2001b).”
BackgroundMethodHirano, K., & Imbens, G. W. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020371312283 · Cited by 1,039 · cited 1×
“Furthermore, we combined PS techniques and covariance adjustment within the regression framework, as suggested by Hirano and Imbens (2001).”
MethodThe Effectiveness of Adjustment by Subclassification in Removing Bias in Observational Studies
Cochran, W. G. (1968) · Biometrics · doi:10.2307/2528036 · Cited by 961
For objective causal inference, design trumps analysis
Rubin, D. B. (2008) · The Annals of Applied Statistics · doi:10.1214/08-aoas187 · Cited by 831 · cited 5×
“This was done by estimating the PS for the randomized experiment using the estimated PS model of the nonrandomized experiment (Rubin, 2008a) and then discarding all cases of the random experiment scoring 0.1 standard deviations below the minimum or above the maximum of the observed nonexperimental…”
BackgroundMethodInfluentialMatching to remove bias in observational studies
Rubin, D. B. (1973) · Biometrics · doi:10.2307/2529684 · Cited by 737
Cook, T. D., Shadish, W. R., & Wong, V. C. (2008) · Journal of Policy Analysis and Management · doi:10.1002/pam.20375 · Cited by 604 · cited 11×
“465 later three-arm designs than with earlier ones (Cook et al., 2008). But the possibility of such confounding is still a problem. Shadish, Clark, and Steiner (2008) solved the problem by moving to a four-arm design. In their example, respondents (undergraduate students) were first randomly assigned to a randomized e…”
BackgroundResultInfluentialSelf-citationThe importance of covariate selection in controlling for selection bias in observational studies.
Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H. (2010) · Psychological Methods · doi:10.1037/a0018719 · Cited by 490 · cited 1×
“In total, we constructed 25 covariates from 86 questionnaire items tapping into the five domains that Steiner et al. (2009) created from the Shadish et al. items (2008): demographic variables, proxy pretests, prior academic achievement, topic preference, and psychological variables.”
MethodSelf-citation
Cited by
The importance of covariate selection in controlling for selection bias in observational studies.
Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H. (2010) · Psychological Methods · doi:10.1037/a0018719 · Cited by 490
Self-citationCook, T. D., Steiner, P. M., & Pohl, S. (2009) · Multivariate Behavioral Research · doi:10.1080/00273170903333673 · Cited by 81
Self-citationKirchmann, H., Steyer, R., Mayer, A., Joraschky, P., Schreiber-Willnow, K., & Strauss, B. (2012) · Psychotherapy Research · doi:10.1080/10503307.2011.626807 · Cited by 44
Harding, D., Sanbonmatsu, L., Duncan, G., Gennetian, L., Katz, L., Kessler, R., Kling, J., Sciandra, M., & Ludwig, J. (2021) · doi:10.3386/w28454 · Cited by 7
Metadata
- Authors
- Pohl, S., Steiner, P. M., Eisermann, J., Soellner, R., & Cook, T. D.
- Venue
- Educational Evaluation and Policy Analysis
- Published
- 2009-12
- Volume
- 31
- Issue
- 4
- Pages
- 463-479
- ISSN
- 0162-3737 (ISSN-L)
- 1935-1062
- OpenAlex
- W2121014669
- Cited by
- 62
- References
- 13
- FWCI
- 3.55
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Unbiased Causal Inference From an Observational Study: Results of a Within-Study Comparison. https://app.cochat.ai/papers/10.3102/0162373709343964
- The paper
- Pohl, S., Steiner, P. M., Eisermann, J., Soellner, R., & Cook, T. D. (2009). Unbiased Causal Inference From an Observational Study: Results of a Within-Study Comparison. Educational Evaluation and Policy Analysis. https://doi.org/10.3102/0162373709343964
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.