Sensitivity of propensity score methods to the specifications
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Propensity score matching estimators have two advantages. One is that they overcome the curse of dimensionality of covariate matching, and the other is that they are nonparametric. However, the propensity score is usually unknown and needs to be estimated. If we estimate it nonparametrically, we are incurring the curse-of-dimensionality problem we are trying to avoid. If we estimate it parametrically, how sensitive the estimated treatment effects are to the specifications of the propensity score becomes an important question. In this paper, we study this issue. First, we use a Monte Carlo experimental method to investigate the sensitivity issue under the unconfoundedness assumption. We find that the estimates are not sensitive to the specifications. Next, we provide some theoretical justifications, using the insight from Rosenbaum and Rubin (1983) that any score finer than the propensity score is a balancing score. Then, we reconcile our finding with the finding in Smith and Todd (2005) that, if the unconfoundedness assumption fails, the matching results can be sensitive. However, failure of the unconfoundedness assumption will not necessarily result in sensitive estimates. Matching estimators can be speciously robust in the sense that the treatment effects are consistently overestimated or underestimated. Sensitivity checks applied in empirical studies are helpful in eliminating sensitive cases, but in general, it cannot help to solve the fundamental problem that the matching assumptions are inherently untestable. Last, our results suggest that including irrelevant variables in the propensity score will not bias the results, but overspecifying it (e.g., adding unnecessary nonlinear terms) probably will.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme
Heckman, J. J., Ichimura, H., & Todd, P. E. (1997) · The Review of Economic Studies · doi:10.2307/2971733 · Cited by 5,486
Does matching overcome lalonde’s critique of nonexperimental estimators?
A. Smith, J., & E. Todd, P. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.011 · Cited by 3,176
Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review
Imbens, G. W. (2004) · Review of Economics and Statistics · doi:10.1162/003465304323023651 · Cited by 2,994
Characterizing Selection Bias Using Experimental Data
Heckman, J., Ichimura, H., Smith, J., & Todd, P. (1998) · Econometrica · doi:10.2307/2999630 · Cited by 2,286
Structural Equations, Treatment Effects, and Econometric Policy Evaluation1
Heckman, J. J., & Vytlacil, E. (2005) · Econometrica · doi:10.1111/j.1468-0262.2005.00594.x · Cited by 1,220
Using matching, instrumental variables and control functions to estimate economic choice models
Heckman, J., & Navarro-Lozano, S. (2004) · Review of Economics and Statistics · doi:10.1162/003465304323023660 · Cited by 768
Comparison of Multivariate Matching Methods: Structures, Distances, and Algorithms
Gu, X. S., & Rosenbaum, P. R. (1993) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.1993.10474623 · Cited by 702
Zhao, Z. (2004) · Review of Economics and Statistics · doi:10.1162/003465304323023705 · Cited by 389
Self-citation
Cited by
Why Propensity Scores Should Not Be Used for Matching
King, G., & Nielsen, R. (2019) · Political Analysis · doi:10.1017/pan.2019.11 · Cited by 1,706
The performance of estimators based on the propensity score
Huber, M., Lechner, M., & Wunsch, C. (2013) · Journal of Econometrics · doi:10.1016/j.jeconom.2012.11.006 · Cited by 311
Propensity score weighting with multilevel data
Li, F., Zaslavsky, A. M., & Landrum, M. B. (2013) · Statistics in Medicine · doi:10.1002/sim.5786 · Cited by 256
Propensity scores: From naïve enthusiasm to intuitive understanding
Williamson, E., Morley, R., Lucas, A., & Carpenter, J. (2012) · Statistical Methods in Medical Research · doi:10.1177/0962280210394483 · Cited by 254
Additionality in U.S. Agricultural Conservation Programs
Claassen, R., Duquette, E. N., & Smith, D. J. (2018) · Land Economics · doi:10.3368/le.94.1.19 · Cited by 73
Maternal breastfeeding and children’s cognitive development
Koh, K. (2017) · Social Science & Medicine · doi:10.1016/j.socscimed.2017.06.012 · Cited by 52
Metadata
- Authors
- Zhao, Z.
- Venue
- Economics Letters
- Publisher
- Elsevier BV
- Published
- 2008-03
- Volume
- 98
- Issue
- 3
- Pages
- 309-319
- ISSN
- 0165-1765 (ISSN-L)
- OpenAlex
- W3122628863
- Cited by
- 129
- References
- 8
- FWCI
- 8.71
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.