The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Misunderstandings Between Experimentalists and Observationalists about Causal Inference
Open accessAbstract
Summary: We attempt to clarify, and suggest how to avoid, several serious misunderstandings about and fallacies of causal inference. These issues concern some of the most fundamental advantages and disadvantages of each basic research design. Problems include improper use of hypothesis tests for covariate balance between the treated and control groups, and the consequences of using randomization, blocking before randomization and matching after assignment of treatment to achieve covariate balance. Applied researchers in a wide range of scientific disciplines seem to fall prey to one or more of these fallacies and as a result make suboptimal design or analysis choices. To clarify these points, we derive a new four-part decomposition of the key estimation errors in making causal inferences. We then show how this decomposition can help scholars from different experimental and observational research traditions to understand better each other’s inferential problems and attempted solutions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference
Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2007) · Political Analysis · doi:10.1093/pan/mpl013 · Cited by 4,205
Self-citationDoes matching overcome lalonde’s critique of nonexperimental estimators?
A. Smith, J., & E. Todd, P. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.011 · Cited by 3,176
Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review
Imbens, G. W. (2004) · Review of Economics and Statistics · doi:10.1162/003465304323023651 · Cited by 2,994
Estimating Causal Effects from Large Data Sets Using Propensity Scores
Rubin, D. B. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-8_part_2-199710151-00064 · Cited by 2,877
Rosenbaum, P. R. (2002) · Springer Series in Statistics · doi:10.1007/978-1-4757-3692-2 · Cited by 2,394
Characterizing Selection Bias Using Experimental Data
Heckman, J., Ichimura, H., Smith, J., & Todd, P. (1998) · Econometrica · doi:10.2307/2999630 · Cited by 2,286
Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020363010465 · Cited by 2,228 · cited 1×
“Other options include higher order moments than the mean, non-parametric density plots and propensity score summary statistics (e.g. Austin and Mamdani (2006) and Rubin (2001)).”
BackgroundThe Effects of Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment
Gerber, A. S., & Green, D. P. (2000) · American Political Science Review · doi:10.2307/2585837 · Cited by 1,380
The Risk Associated with Aprotinin in Cardiac Surgery
Mangano, D. T., Tudor, I. C., & Dietzel, C. (2006) · New England Journal of Medicine · doi:10.1056/nejmoa051379 · Cited by 1,099
Pocock, S. J., Assmann, S. E., Enos, L. E., & Kasten, L. E. (2002) · Statistics in Medicine · doi:10.1002/sim.1296 · Cited by 1,077
Matching Using Estimated Propensity Scores: Relating Theory to Practice
Rubin, D. B., & Thomas, N. (1996) · Biometrics · doi:10.2307/2533160 · Cited by 1,074
The Effectiveness of Adjustment by Subclassification in Removing Bias in Observational Studies
Cochran, W. G. (1968) · Biometrics · doi:10.2307/2528036 · Cited by 961
Matching to remove bias in observational studies
Rubin, D. B. (1973) · Biometrics · doi:10.2307/2529684 · Cited by 737
Matched Sampling for Causal Effects
Rubin, D. B. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810725 · Cited by 676
The Planning of Observational Studies of Human Populations
Cochran, W. G., & Chambers, S. P. (1965) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2344179 · Cited by 564
Austin, P. C., & Mamdani, M. M. (2006) · Statistics in Medicine · doi:10.1002/sim.2328 · Cited by 560
The Dangers of Extreme Counterfactuals
King, G., & Zeng, L. (2006) · Political Analysis · doi:10.1093/pan/mpj004 · Cited by 540
Self-citationTesting for baseline balance in clinical trials
Senn, S. (1994) · Statistics in Medicine · doi:10.1002/sim.4780131703 · Cited by 503 · cited 3×
“Senn (1994), page 1716, explained that this ‘common procedure’ is ‘philosophically unsound, of no practical value, and potentially misleading’.”
BackgroundResultLichter, A. S., Lippman, M. E., Danforth, D. N., d'Angelo, T., Steinberg, S. M., deMoss, E., MacDonald, H. D., Reichert, C. M., Merino, M., & Swain, S. M. (1992) · Journal of Clinical Oncology · doi:10.1200/jco.1992.10.6.976 · Cited by 347
Matching With Doses in an Observational Study of a Media Campaign Against Drug Abuse
Lu, B., Zanutto, E., Hornik, R., & Rosenbaum, P. R. (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753381896 · Cited by 236
Crosnoe, R. (2005) · American Educational Research Journal · doi:10.3102/00028312042002269 · Cited by 234
Retail Tobacco Outlet Density and Youth Cigarette Smoking: A Propensity-Modeling Approach
Novak, S. P., Reardon, S. F., Raudenbush, S. W., & Buka, S. L. (2006) · American Journal of Public Health · doi:10.2105/ajph.2004.061622 · Cited by 226
IMAI, K. (2005) · American Political Science Review · doi:10.1017/s0003055405051658 · Cited by 215
Self-citationOptimal multivariate matching before randomization
Greevy, R. (2004) · Biostatistics · doi:10.1093/biostatistics/5.2.263 · Cited by 205
Causal Inferences with Group Based Trajectory Models
Haviland, A. M., & Nagin, D. S. (2005) · Psychometrika · doi:10.1007/s11336-004-1261-y · Cited by 119
The effect of matching on the power of randomized community intervention studies
Martin, D. C., Diehr, P. K., Perrin, E. B., & Koepsell, T. D. (1993) · Statistics in Medicine · doi:10.1002/sim.4780120315 · Cited by 106
Rubin, D. B., & Stuart, E. A. (2006) · The Annals of Statistics · doi:10.1214/009053606000000407 · Cited by 80
Self-citationFamily formation among women in the U.S. military: Evidence from the nLSY
Lundquist, J. H., & Smith, H. L. (2005) · Journal of Marriage and the Family · doi:10.1111/j.0022-2445.2005.00001.x · Cited by 42
Cited by
Austin, P. C. (2011) · Multivariate Behavioral Research · doi:10.1080/00273171.2011.568786 · Cited by 12,242
“…authors have frequently used statistical significance testing to compare the mean of continuous covariates or the distribution of categorical variables between treated and untreated subjects in propensity score matched samples authors for two reasons (Imai et al., 2008; Austin, 2008b, 2009b).”
MethodAustin, P. C. (2009) · Statistics in Medicine · doi:10.1002/sim.3697 · Cited by 6,758
“Comparing the distributions of the estimated propensity score between treated and untreated subjects Some authors have proposed that the distribution of the estimated propensity score be compared between treated and untreated subjects in the propensity-score matched sample [12, 29].”
BackgroundMethodInfluentialHainmueller, J. (2012) · Political Analysis · doi:10.1093/pan/mpr025 · Cited by 6,026
“Imai, King, and Stuart(2008) criticize the use oft-tests and stopping rules and argue that all balance measures should be maximized without limit.”
Matching Methods for Causal Inference: A Review and a Look Forward
Stuart, E. A. (2010) · Statistical Science · doi:10.1214/09-sts313 · Cited by 5,478
Self-citationAustin, P. C., & Stuart, E. A. (2015) · Statistics in Medicine · doi:10.1002/sim.6607 · Cited by 4,495
Self-citation
Metadata
- Authors
- Imai, K., King, G., & Stuart, E. A.
- Venue
- Journal of the Royal Statistical Society Series A: Statistics in Society
- Publisher
- Oxford University Press (OUP)
- Published
- 2008-03-10
- Volume
- 171
- Issue
- 2
- Pages
- 481-502
- ISSN
- 0964-1998 (ISSN-L)
- 1467-985X
- OpenAlex
- W1513261862
- Cited by
- 906
- References
- 29
- FWCI
- 33.53
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.