For objective causal inference, design trumps analysis
Open accessAbstract
For obtaining causal inferences that are objective, and therefore have the best chance of revealing scientific truths, carefully designed and executed randomized experiments are generally considered to be the gold standard. Observational studies, in contrast, are generally fraught with problems that compromise any claim for objectivity of the resulting causal inferences. The thesis here is that observational studies have to be carefully designed to approximate randomized experiments, in particular, without examining any final outcome data. Often a candidate data set will have to be rejected as inadequate because of lack of data on key covariates, or because of lack of overlap in the distributions of key covariates between treatment and control groups, often revealed by careful propensity score analyses. Sometimes the template for the approximating randomized experiment will have to be altered, and the use of principal stratification can be helpful in doing this. These issues are discussed and illustrated using the framework of potential outcomes to define causal effects, which greatly clarifies critical issues.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Although in statistical terms the use of propensity scores may produce results that differ little from those obtained through traditional regression adjustment, they encourage clearer thinking about study design and particularly the assignment mechanism.
“Although in statistical terms the use of propensity scores may produce results that differ little from those obtained through traditional regression adjustment (70), they encourage clearer thinking about study design and particularly the assignment mechanism (66).”
From Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research (2017) · cites this paper as [66]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241 · see citing works
Self-citation(1979) · Medical Entomology and Zoology · Cited by 11,252 · see citing works
Statistical Methods for Research Workers
Fisher, R. A. (1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,883 · see citing works
RUBIN, D. B. (1976) · Biometrika · doi:10.1093/biomet/63.3.581 · Cited by 9,925 · see citing works
Self-citationEstimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · see citing works
Self-citationStatistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · see citing works
Rosenbaum, P. R., & Rubin, D. B. (1985) · The American Statistician · doi:10.1080/00031305.1985.10479383 · Cited by 4,415 · see citing works
Self-citationIdentification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230 · see citing works
Self-citationEstimating Causal Effects from Large Data Sets Using Propensity Scores
Rubin, D. V. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-8_part_2-199710151-00064 · Cited by 3,104 · see citing works
Self-citationSOME THOUGHTS ON THE DISTRIBUTION OF EARNINGS 1
Roy, A. D. (1951) · Oxford Economic Papers · doi:10.1093/oxfordjournals.oep.a041827 · Cited by 2,931 · see citing works
Bayesian inference for causal effects: The role of randomization
Rubin, D. B. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344064 · Cited by 2,503 · see citing works
Self-citationUsing Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020363010465 · Cited by 2,228 · see citing works
Self-citationCausal Inference Using Potential Outcomes: Design, Modeling, Decisions
Rubin, D. B. (2005) · Journal of the American Statistical Association · doi:10.1198/016214504000001880 · Cited by 2,173 · see citing works
Self-citationMartuza, V. R., & Kenny, D. A. (1981) · Educational Researcher · doi:10.2307/1175080 · Cited by 2,034 · see citing works
On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9
Neyman, J., Dabrowska, D. M., & Speed, T. P. (1990) · Statistical Science · doi:10.1214/ss/1177012031 · Cited by 1,616 · see citing works
Conditional Independence in Statistical Theory
Dawid, A. P. (1979) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1979.tb01052.x · Cited by 1,507 · see citing works
Principal Stratification in Causal Inference
Frangakis, C., & Rubin, D. B. (2002) · Biometrics · doi:10.1111/j.0006-341x.2002.00021.x · Cited by 1,496 · see citing works
Self-citationDesign and Analysis of Experiments
Ryan, T. P. (2007) · doi:10.1002/9780470187548.ch12 · Cited by 1,436 · see citing works
(2007) · Cited by 1,428 · see citing works
Randomization Analysis of Experimental Data: The Fisher Randomization Test Comment
Rubin, D. B. (1980) · Journal of the American Statistical Association · doi:10.2307/2287653 · Cited by 1,368 · see citing works
Self-citationThe Probability Approach in Econometrics
Haavelmo, T. (1944) · Econometrica · doi:10.2307/1906935 · Cited by 1,271 · see citing works
Assignment to Treatment Group on the Basis of a Covariate
Rubin, D. B. (1977) · Journal of Educational Statistics · doi:10.3102/10769986002001001 · Cited by 1,191 · see citing works
Self-citation(1977) · Annals of Internal Medicine · doi:10.7326/0003-4819-86-6-835_3 · Cited by 1,160 · see citing works
Counterfactuals and Causal Inference: Methods and Principles for Social Research
Morgan, S. L., & Winship, C. (2007) · doi:10.1017/cbo9780511804564 · Cited by 1,146 · see citing works
Rubin, D. B. (2007) · Statistics in Medicine · doi:10.1002/sim.2739 · Cited by 1,047 · see citing works
Self-citationThe Effectiveness of Adjustment by Subclassification in Removing Bias in Observational Studies
Cochran, W. G. (1968) · Biometrics · doi:10.2307/2528036 · Cited by 961 · see citing works
Rubin, D. B. (1979) · Journal of the American Statistical Association · doi:10.1080/01621459.1979.10482513 · Cited by 852 · see citing works
Self-citationCombining Propensity Score Matching with Additional Adjustments for Prognostic Covariates
Rubin, D., & Thomas, N. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10474233 · Cited by 796 · see citing works
Self-citationRubin, D. B. (1990) · Statistical Science · doi:10.1214/ss/1177012032 · Cited by 665 · see citing works
Self-citationThe Planning of Observational Studies of Human Populations
Cochran, W. G., & Chambers, S. (1965) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344179 · Cited by 572 · see citing works
The Design and Analysis of Experiments
Kempthorne, O. (1952) · Soil Science · doi:10.1097/00010694-195205000-00012 · Cited by 522 · see citing works
Combining Propensity Score Matching with Additional Adjustments for Prognostic Covariates
Thomas, N. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810725.025 · Cited by 508 · see citing works
Formal mode of statistical inference for causal effects
Rubin, D. B. (1990) · Journal of Statistical Planning and Inference · doi:10.1016/0378-3758(90)90077-8 · Cited by 483 · see citing works
Self-citationAhmed, A., Husain, A., Love, T. E., Gambassi, G., Dell’Italia, L. J., Francis, G. S., Gheorghiade, M., Allman, R. M., Meleth, S., & Bourge, R. C. (2006) · European Heart Journal · doi:10.1093/eurheartj/ehi890 · Cited by 469 · see citing works
Bayesian inference for causal effects in randomized experiments with noncompliance
Imbens, G. W., & Rubin, D. B. (1997) · The Annals of Statistics · doi:10.1214/aos/1034276631 · Cited by 382 · see citing works
Self-citationEstimating Treatment Effects Using Observational Data
D’Agostino, R. B. (2007) · JAMA · doi:10.1001/jama.297.3.314 · Cited by 352 · see citing works
Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810725.030 · Cited by 344 · see citing works
Self-citationCausal Inference, Path Analysis, and Recursive Structural Equations Models
Holland, P. W. (1988) · Sociological Methodology · doi:10.2307/271055 · Cited by 307 · see citing works
Cochran, W. G., & Cox, G. M. (1950) · Soil Science · doi:10.1097/00010694-195008000-00014 · Cited by 297 · see citing works
Matched Sampling for Causal Effects: Contributor Acknowledgments
Rubin, D. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810725 · Cited by 287 · see citing works
Self-citationPrincipal Stratification Approach to Broken Randomized Experiments
Barnard, J., Frangakis, C., Hill, J., & Rubin, D. B. (2003) · Journal of the American Statistical Association · doi:10.1198/016214503000071 · Cited by 268 · see citing works
Self-citationIn utero exposure to phenobarbital and intelligence deficits in adult men
Reinisch, J. M. (1995) · JAMA · doi:10.1001/jama.274.19.1518 · Cited by 239 · see citing works
Rubin, D. B., & Thomas, N. (1992) · Biometrika · doi:10.1093/biomet/79.4.797 · Cited by 188 · see citing works
Self-citationPlanning and Analysis of Observational Studies
Cochran, W. G. (1983) · Wiley series in probability and statistics · doi:10.1002/9780470316542 · Cited by 168 · see citing works
PRODAN, M. (1968) · Elsevier eBooks · doi:10.1016/b978-0-08-012441-4.50022-5 · Cited by 164 · see citing works
Principal Stratification for Causal Inference With Extended Partial Compliance
Jin, H., & Rubin, D. B. (2008) · Journal of the American Statistical Association · doi:10.1198/016214507000000347 · Cited by 149 · see citing works
Self-citationRubin, D. B. (1976) · Biometrics · doi:10.2307/2529343 · Cited by 124 · see citing works
Self-citationRandomization Analysis of Experimental Data: The Fisher Randomization Test
Basu, D. (1980) · Journal of the American Statistical Association · doi:10.1080/01621459.1980.10477512 · Cited by 115 · see citing works
Holland, P. W., & Rubin, D. B. (1982) · ETS Research Report Series · doi:10.1002/j.2333-8504.1982.tb01321.x · Cited by 97 · see citing works
Self-citationEstimating the Causal Effects of Marketing Interventions Using Propensity Score Methodology
Rubin, D., & Waterman, R. P. (2006) · Statistical Science · doi:10.1214/088342306000000259 · Cited by 94 · see citing works
Self-citationRandomization Analysis of Experimental Data: The Fisher Randomization Test Rejoinder
Basu, D. (1980) · Journal of the American Statistical Association · doi:10.2307/2287654 · Cited by 81 · see citing works
Estimating the causal effects of marketing interventions using propensity score methodologyPreprint
Rubin, D. B., & Waterman, R. P. (2006) · Cited by 69 · see citing works
Self-citationPhilosophy of Inferences from Retrospective Studies
Dorn, H. F. (1953) · American Journal of Public Health and the Nations Health · doi:10.2105/ajph.43.6_pt_1.677 · Cited by 63 · see citing works
Rubin, D. (2007) · Handbook of statistics · doi:10.1016/s0169-7161(07)27002-6 · Cited by 45 · see citing works
Self-citationWilliam G. Cochran's Contributions to the Design, Analysis, and Evaluation of Observational Studies
Rubin, D. B. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810725.004 · Cited by 18 · see citing works
Self-citationImbens, G. W., & Rubin, D. B. (2008) · The New Palgrave Dictionary of Economics · doi:10.1057/978-1-349-95121-5_2469-1 · Cited by 10 · see citing works
Self-citationRubin, D. B., Wang, X., Yin, L., et al. (2018) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780198703174.013.24 · Cited by 9 · see citing works
Self-citationOutcome-free Design of Observational Studies: Peer Influence on Smoking
LANGENSKIÖLD, & RUBIN (2008) · Annales d Économie et de Statistique · doi:10.2307/27917241 · Cited by 9 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- BLALOCK, H. (1964). Causal Inference in Nonexperimental Research. Univ. North Carolina Press, Chapel Hill.
- CAMPBELL, D. and STANLEY, J. (1963). Experimental and quasi-experimental designs for research and teaching. In Handbook of Research on Teaching (N. L. Gage, ed.). Rand McNally, Chicago.
- IMBENS, G. and RUBIN, D. (2008b). Causal Inference in Statistics, and in the Social and Biomedical Sciences. Cambridge Univ. Press, New York. To appear.
- KENNY, D. A. (1979). Correlation and Causation. Wiley, New York.
- MADDALA, G. (1977). Econometrics. McGraw-Hill, New York.
- RUBIN, D. (2006). Matched Sampling for Causal Effects. Cambridge Univ. Press, New York.
- ZELL, E., KUWANDA, M. RUBIN, D., CUTLAND, C., PATEL, R., VELAPHI S., MADHI, S. and SCHRAG, S. (2007). Conducting and analyzing a single-blind clinical trial in a developing coun- try: Prevention of perinatal sepsis, soweto, South Africa. In Proceedings of the International Statistical Institute (CD-ROM).
Cited by
Sorted by citations
Difference-in-Differences with multiple time periods
Callaway, B., & Sant’Anna, P. H. C. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.12.001 · Cited by 9,126 · see citing works
On making causal claims: A review and recommendations
Antonakis, J., Bendahan, S., Jacquart, P., & Lalive, R. (2010) · The Leadership Quarterly · doi:10.1016/j.leaqua.2010.10.010 · Cited by 2,441 · see citing works
Why Propensity Scores Should Not Be Used for Matching
King, G., & Nielsen, R. (2019) · Political Analysis · doi:10.1017/pan.2019.11 · Cited by 1,706 · see citing works
Leadership behavior and employee well-being: An integrated review and a future research agenda
Inceoglu, I., Thomas, G., Chu, C., Plans, D., & Gerbasi, A. (2018) · The Leadership Quarterly · doi:10.1016/j.leaqua.2017.12.006 · Cited by 670 · see citing works
Craig, P., Katikireddi, S. V., Leyland, A., & Popham, F. (2017) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-031816-044327 · Cited by 570 · see citing works
The Use of Propensity Scores to Assess the Generalizability of Results from Randomized Trials
Stuart, E. A., Cole, S. R., Bradshaw, C. P., & Leaf, P. J. (2011) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2010.00673.x · Cited by 524 · see citing works
Causal Inference and Observational Research: The Utility of Twins
McGue, M., Osler, M., & Christensen, K. (2010) · Perspectives on Psychological Science · doi:10.1177/1745691610383511 · Cited by 529 · see citing works
Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008) · Journal of the American Statistical Association · doi:10.1198/016214508000000733 · Cited by 479 · see citing works
The importance of covariate selection in controlling for selection bias in observational studies.
Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H. (2010) · Psychological Methods · doi:10.1037/a0018719 · Cited by 490 · see citing works
Stable weights that balance covariates for estimation with incomplete outcome data
Zubizarreta, J. R. (2015) · Journal of the American Statistical Association · doi:10.1080/01621459.2015.1023805 · Cited by 369 · see citing works
Causal Inference About the Effects of Interventions From Observational Studies in Medical Journals
Dahabreh, I. J., & Bibbins-Domingo, K. (2024) · JAMA · doi:10.1001/jama.2024.7741 · Cited by 242 · see citing works
The Role of Parallel Trends in Event Study Settings: An Application to Environmental Economics
Marcus, M., & Sant’Anna, P. H. C. (2021) · Journal of the Association of Environmental and Resource Economists · doi:10.1086/711509 · Cited by 192 · see citing works
Triangulating Evidence through the Inclusion of Genetically Informed Designs
Munafò, M. R., Higgins, J. P. T., & Smith, G. D. (2021) · Cold Spring Harbor Perspectives in Medicine · doi:10.1101/cshperspect.a040659 · Cited by 125 · see citing works
Pack, A. I., Magalang, U. J., Singh, B., Kuna, S. T., Keenan, B. T., & Maislin, G. (2021) · Sleep · doi:10.1093/sleep/zsaa229 · Cited by 114 · see citing works
Wing, C., & Cook, T. D. (2013) · Journal of Policy Analysis and Management · doi:10.1002/pam.21721 · Cited by 87 · see citing works
Rubin, D. B. (2010) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2009.08.024 · Cited by 88 · see citing works
Self-citationCook, T. D., Steiner, P. M., & Pohl, S. (2009) · Multivariate Behavioral Research · doi:10.1080/00273170903333673 · Cited by 81 · see citing works
Han, L., Goulet, J. L., Skanderson, M., Bathulapalli, H., Luther, S. L., Kerns, R. D., & Brandt, C. A. (2019) · Pain Medicine · doi:10.1093/pm/pny027 · Cited by 18 · see citing works
Bottigliengo, D., Lorenzoni, G., Ocagli, H., Martinato, M., Berchialla, P., & Gregori, D. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18136694 · Cited by 13 · see citing works
- Authors
- Rubin, D. B.
- Affiliations
- Harvard University (US)
- Corresponding author
- Rubin, Donald B.
- ORCID
- Rubin, Donald B.
- Venue
- The Annals of Applied Statistics
- Publisher
- Institute of Mathematical Statistics
- Published
- 2008-09-01
- Volume
- 2
- Issue
- 3
- Pages
- 808-840
- ISSN
- 1932-6157 (ISSN-L)
- 1941-7330
- OpenAlex
- W2010505816
- Cited by
- 831
- References
- 58
- Citations by year
- 2026: 34 · 2025: 50 · 2024: 41 · 2023: 59 · 2022: 52 · 2021: 63 · 2020: 81 · 2019: 65 · 2018: 55 · 2017: 55 · 2016: 63 · 2015: 43 · 2014: 38 · 2013: 39 · 2012: 28
- FWCI
- 20.37
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- The Annals of Applied Statistics (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Project Euclid (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (0550887)
- National Institutes of Health
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- causal inference, observational studies, randomized experiments, design-based inference, objectivity, propensity score matching, principal stratification, potential outcomes, covariate overlap, covariate selection, causal effect estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
