How Bias Reduction Is Affected by Covariate Choice, Unreliability, and Mode of Data Analysis: Results From Two Types of Within-Study Comparisons
The DOI matches its Crossref record; 45 of the 48 references it cites were found in a registry or an index.
Abstract
This study uses within-study comparisons to assess the relative importance of covariate choice, unreliability in the measurement of these covariates, and whether regression or various forms of propensity score analysis are used to analyze the outcome data. Two of the within-study comparisons are of the four-arm type, and many more are of the three-arm type. To examine unreliability, simulations of differences in reliability are deliberately introduced into the 2 four-arm studies. Results are similar across the samples of studies reviewed with their wide range of non-experimental designs and topic areas. Covariate choice counts most, unreliability next most, and the mode of data analysis hardly matters at all. Unreliability has larger effects the more important a covariate is for bias reduction, but even so the very best covariates measured with a reliability of only .60 still do better than substantively poor covariates that are measured perfectly. Why regression methods do as well as propensity score methods used in several different ways is a mystery still because, in theory, propensity scores would seem to have a distinct advantage in many practical applications, especially those where functional forms are in doubt.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“More leverage for reducing selection bias comes from (a) preintervention theoretical analysis of the selection process into treatment—or even direct observation of this process—so as to know which covariates to choose, (b) selecting local comparison groups that maximize group overlap before any covariate choice, and (c) using a heterogeneous and extensive collection of covariates that, at a min”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Cook et al. 2009
Citations
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
(1979) · Medical Entomology and Zoology · Cited by 11,252
Self-citationIdentification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230
Regression discontinuity designs: A guide to practice
Imbens, G. W., & Lemieux, T. (2008) · Journal of Econometrics · doi:10.1016/j.jeconom.2007.05.001 · Cited by 3,694
Identification and Estimation of Treatment Effects with a Regression-Discontinuity Design
Hahn, J., Todd, P., & Klaauw, W. (2001) · Econometrica · doi:10.1111/1468-0262.00183 · Cited by 2,679
Rosenbaum, P. R. (2002) · Springer Series in Statistics · doi:10.1007/978-1-4757-3692-2 · Cited by 2,394
(2013) · American Economic Review · doi:10.1257/aer.103.3.i · Cited by 1,916
Lunceford, J. K., & Davidian, M. (2004) · Statistics in Medicine · doi:10.1002/sim.1903 · Cited by 1,674
(2007) · Cited by 1,428
Counterfactuals and Causal Inference: Methods and Principles for Social Research
Morgan, S. L., & Winship, C. (2007) · doi:10.1017/cbo9780511804564 · Cited by 1,146
Research works printed but not linked: 5
- Campbell D. T., 1970, The disadvantaged child: Vol. 3. Compensatory education: A national debate, 185
- Cook T. D., 2008, Impacts of school improvement status on students with disabilities: Technical work group materials
- LaLonde R., 1986, American Economic Review, 76, 604
- 10.1080/01621459.1960.10482065 · doi:10.1080/01621459.1960.10482065 · Paper page
- Pohl S., Educational Evaluation and Policy Analysis.
Cited by
Gottfredson, D. C., Cook, T. D., Gardner, F. E. M., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015) · Prevention Science · doi:10.1007/s11121-015-0555-x · Cited by 627
Self-citationA Systematic Review of Propensity Score Methods in the Social Sciences
Thoemmes, F. J., & Kim, E. S. (2011) · Multivariate Behavioral Research · doi:10.1080/00273171.2011.540475 · Cited by 577
The importance of covariate selection in controlling for selection bias in observational studies.
Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H. (2010) · Psychological Methods · doi:10.1037/a0018719 · Cited by 490
Self-citationDavis, L., Bozick, R., Steele, J., Saunders, J., & Miles, J. (2013) · doi:10.7249/rr266 · Cited by 388
Quasi-experimental study designs series—paper 4: uses and value
Bärnighausen, T., Tugwell, P., Røttingen, J.-A., Shemilt, I., Rockers, P., Geldsetzer, P., Lavis, J., Grimshaw, J., Daniels, K., Brown, A., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.03.012 · Cited by 240
Metadata
- Authors
- Cook, T. D., Steiner, P. M., & Pohl, S.
- Affiliations
- Northwestern University (US)
- Friedrich Schiller University Jena (DE)
- Venue
- Multivariate Behavioral Research
- Publisher
- Informa UK Limited
- Published
- 2009-11-30
- Volume
- 44
- Issue
- 6
- Pages
- 828-847
- ISSN
- 0027-3171 (ISSN-L)
- 1532-7906
- OpenAlex
- W2060881560
- Cited by
- 81
- References
- 48
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 3 · 2023: 3 · 2022: 2 · 2021: 12 · 2020: 3 · 2019: 5 · 2018: 4 · 2017: 7 · 2016: 7 · 2015: 6 · 2014: 11 · 2013: 5 · 2012: 5
- FWCI
- 2.47
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Multivariate Behavioral Research (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Spencer Foundation
- University of California, Santa Cruz
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- bias mitigation, propensity score analysis, regression analysis, non-experimental design, confounding adjustment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How Bias Reduction Is Affected by Covariate Choice, Unreliability, and Mode of Data Analysis: Results From Two Types of Within-Study Comparisons. https://app.cochat.ai/papers/10.1080/00273170903333673
- The paper
- Cook, T. D., Steiner, P. M., & Pohl, S. (2009). How Bias Reduction Is Affected by Covariate Choice, Unreliability, and Mode of Data Analysis: Results From Two Types of Within-Study Comparisons. Multivariate Behavioral Research. https://doi.org/10.1080/00273170903333673
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.