Improving propensity score estimators’ robustness to model misspecification using super learner.
Open accessAbstract
The consistency of propensity score (PS) estimators relies on correct specification of the PS model. The PS is frequently estimated using main-effects logistic regression. However, the underlying model assumptions may not hold. Machine learning methods provide an alternative nonparametric approach to PS estimation. In this simulation study, we evaluated the benefit of using Super Learner (SL) for PS estimation. We created 1,000 simulated data sets (n = 500) under 4 different scenarios characterized by various degrees of deviance from the usual main-term logistic regression model for the true PS. We estimated the average treatment effect using PS matching and inverse probability of treatment weighting. The estimators' performance was evaluated in terms of PS prediction accuracy, covariate balance achieved, bias, standard error, coverage, and mean squared error. All methods exhibited adequate overall balancing properties, but in the case of model misspecification, SL performed better for highly unbalanced variables. The SL-based estimators were associated with the smallest bias in cases of severe model misspecification. Our results suggest that use of SL to estimate the PS can improve covariate balance and reduce bias in a meaningful manner in cases of serious model misspecification for treatment assignment.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Super Learner ranks candidate algorithms by performance and applies an optimal weight to each, creating ensemble algorithms that can accurately predict transfusion requirements and mortality among trauma patients.
“To capitalize on this phenomenon, Super Learner ranks a set of candidate algorithms20,28,38,39 by their performance and applies an optimal weight to each, creating ensemble algorithms that can accurately predict transfusion requirements and mortality among trauma patients.20,28,38,39”
From Artificial Intelligence and Surgical Decision-making. (2020) · cites this paper as [39]
The Super Learner section describes comparing candidate algorithms by cross-validated risk and combining them with weights chosen to minimize estimated risk. The paper studies propensity-score estimation in simulated data; it does not establish accurate prediction of transfusion requirements or mortality among trauma patients. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421 · see citing works
Cortes, C., & Vapnik, V. N. (1995) · Machine Learning · doi:10.1023/a:1022627411411 · Cited by 34,565 · see citing works
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241 · see citing works
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 21,028 · see citing works
(2005) · Springer eBooks · doi:10.1007/0-387-27255-0_9 · Cited by 20,333 · see citing works
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927 · see citing works
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325 · see citing works
Modern Applied Statistics with S
Venables, W., & Ripley, B. (2010) · Statisctics and computing/Statistics and computing · doi:10.1007/978-0-387-21706-2 · Cited by 16,188 · see citing works
Building Predictive Models in R Using the caret Package
Kühn, M. (2008) · Journal of Statistical Software · doi:10.18637/jss.v028.i05 · Cited by 9,986 · see citing works
Multivariate Adaptive Regression Splines
Friedman, J. H. (1991) · The Annals of Statistics · doi:10.1214/aos/1176347963 · Cited by 8,108 · see citing works
Rosenbaum, P., & Rubin, D. (1985) · The American Statistician · doi:10.2307/2683903 · Cited by 6,854 · see citing works
Austin, P. C. (2009) · Statistics in Medicine · doi:10.1002/sim.3697 · Cited by 6,758 · see citing works
A working guide to boosted regression trees
Elith, J., Leathwick, J. R., & Hastie, T. (2008) · Journal of Animal Ecology · doi:10.1111/j.1365-2656.2008.01390.x · Cited by 6,484 · see citing works
Pattern Recognition and Neural Networks
Ripley, B. D. (1996) · Cambridge University Press eBooks · doi:10.1017/cbo9780511812651 · Cited by 6,446 · see citing works
Rosenbaum, P. R., & Rubin, D. B. (1985) · The American Statistician · doi:10.1080/00031305.1985.10479383 · Cited by 4,415 · see citing works
Does matching overcome lalonde’s critique of nonexperimental estimators?
A. Smith, J., & E. Todd, P. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.011 · Cited by 3,176 · see citing works
Reducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082 · see citing works
Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review
Imbens, G. W. (2004) · Review of Economics and Statistics · doi:10.1162/003465304323023651 · Cited by 2,994 · see citing works
Large sample properties of matching estimators for average treatment effects
Abadie, A., & Imbens, G. W. (2006) · Econometrica · doi:10.1111/j.1468-0262.2006.00655.x · Cited by 2,908 · see citing works
Variable Selection for Propensity Score Models
Brookhart, M. A., Schneeweiss, S., Rothman, K. J., Glynn, R. J., Avorn, J., & Stürmer, T. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj149 · Cited by 2,336 · see citing works
A weakly informative default prior distribution for logistic and other regression models
Gelman, A., Jakulin, A., Pittau, M. G., & Su, Y.-S. (2008) · The Annals of Applied Statistics · doi:10.1214/08-aoas191 · Cited by 2,115 · see citing works
Hernán, M. Á., Brumback, B., & Robins, J. M. (2000) · Epidemiology · doi:10.1097/00001648-200009000-00012 · Cited by 1,772 · see citing works
Lunceford, J. K., & Davidian, M. (2004) · Statistics in Medicine · doi:10.1002/sim.1903 · Cited by 1,676 · see citing works
Sekhon, J. S. (2011) · Journal of Statistical Software · doi:10.18637/jss.v042.i07 · Cited by 1,521 · see citing works
van der Laan, M. J., Polley, E., & Hubbard, A. (2007) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1309 · Cited by 1,472 · see citing works
A WEAKLY INFORMATIVE DEFAULT PRIOR DISTRIBUTION FOR LOGISTIC AND OTHER REGRESSION MODELS
Gelman, A., Jakulin, A., & Grazia, M. (2009) · Cited by 1,352 · see citing works
McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.4.403 · Cited by 1,317 · see citing works
Kang, J. D. Y., & Schafer, J. L. (2007) · Statistical Science · doi:10.1214/07-sts227 · Cited by 1,259 · see citing works
Austin, P. C., Grootendorst, P. V., & Anderson, G. M. (2006) · Statistics in Medicine · doi:10.1002/sim.2580 · Cited by 1,138 · see citing works
Schneeweiss, S., Rassen, J. A., Glynn, R. J., Avorn, J., Mogun, H., & Brookhart, M. A. (2009) · Epidemiology · doi:10.1097/ede.0b013e3181a663cc · Cited by 1,128 · see citing works
A comparison of 12 algorithms for matching on the propensity score
Austin, P. C. (2014) · Statistics in Medicine · doi:10.1002/sim.6004 · Cited by 1,085 · see citing works
The performance of different propensity score methods for estimating marginal hazard ratios
Austin, P. C. (2012) · Statistics in Medicine · doi:10.1002/sim.5705 · Cited by 999 · see citing works
Improving propensity score weighting using machine learning
Lee, B. K., Lessler, J., & Stuart, E. A. (2009) · Statistics in Medicine · doi:10.1002/sim.3782 · Cited by 888 · see citing works
Xu, S., Ross, C. A., Raebel, M. A., Shetterly, S. M., Blanchette, C., & Smith, D. H. (2009) · Value in Health · doi:10.1111/j.1524-4733.2009.00671.x · Cited by 800 · see citing works
Stürmer, T., Joshi, M., Glynn, R., Avorn, J., Rothman, K., & Schneeweiss, S. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.07.004 · Cited by 729 · see citing works
Genetic Optimization Using Derivatives: ThergenoudPackage forR
Mebane, W. R., & Sekhon, J. S. (2011) · Journal of Statistical Software · doi:10.18637/jss.v042.i11 · Cited by 639 · see citing works
A Note on the Efficiency of Sandwich Covariance Matrix Estimation
Kauermann, G., & Carroll, R. J. (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753382309 · Cited by 630 · see citing works
An introduction to multivariate adaptive regression splines
Friedman, J. H., & Roosen, C. B. (1995) · Statistical Methods in Medical Research · doi:10.1177/096228029500400303 · Cited by 590 · see citing works
gbm: Generalized Boosted Regression Models
Ridgeway, G. K., & Developers, G. (2003) · doi:10.32614/cran.package.gbm · Cited by 580 · see citing works
Diagnosing and responding to violations in the positivity assumption
Petersen, M. L., Porter, K. E., Gruber, S., Wang, Y., & van der Laan, M. J. (2012) · Statistical Methods in Medical Research · doi:10.1177/0962280210386207 · Cited by 548 · see citing works
Self-citationIndications for propensity scores and review of their use in pharmacoepidemiology
Glynn, R. J., Schneeweiss, S., & Stürmer, T. (2006) · Basic & Clinical Pharmacology & Toxicology · doi:10.1111/j.1742-7843.2006.pto_293.x · Cited by 541 · see citing works
Westreich, D. J., Lessler, J., & Funk, M. J. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.11.020 · Cited by 505 · see citing works
Kang, J., & Schafer, J. L. (2007) · Statistical Science · Cited by 368 · see citing works
Austin, P. C. (2010) · Statistics in Medicine · doi:10.1002/sim.3854 · Cited by 358 · see citing works
Effects of Misspecification of the Propensity Score on Estimators of Treatment Effect
Drake, C. (1993) · Biometrics · doi:10.2307/2532266 · Cited by 347 · see citing works
Luellen, J., Shadish, W., & Clark, M. H. (2005) · Evaluation review · doi:10.1177/0193841x05275596 · Cited by 340 · see citing works
On principles for modeling propensity scores in medical research
Rubin, D. B. (2004) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.968 · Cited by 288 · see citing works
Evaluating uses of data mining techniques in propensity score estimation: a simulation study
Setoguchi, S., Schneeweiß, S., Brookhart, M. A., Glynn, R. J., & Cook, E. F. (2008) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.1555 · Cited by 273 · see citing works
The performance of different propensity score methods for estimating marginal odds ratios
Austin, P. C. (2006) · Statistics in Medicine · doi:10.1002/sim.2781 · Cited by 249 · see citing works
Super Learning: An Application to the Prediction of HIV-1 Drug Resistance
Sinisi, S. E., Polley, E. C., Petersen, M. L., Rhee, S., & van der Laan, M. J. (2007) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1240 · Cited by 238 · see citing works
Self-citationThe performance of different propensity-score methods for estimating relative risks
Austin, P. C. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.07.011 · Cited by 214 · see citing works
Rassen, J. A., Glynn, R. J., Brookhart, M. A., & Schneeweiß, S. (2011) · American Journal of Epidemiology · doi:10.1093/aje/kwr001 · Cited by 179 · see citing works
Collaborative Double Robust Targeted Maximum Likelihood Estimation
van der Laan, M. J., & Gruber, S. (2010) · The International Journal of Biostatistics · doi:10.2202/1557-4679.1181 · Cited by 179 · see citing works
The role of the c‐statistic in variable selection for propensity score models
Westreich, D. J., Cole, S. R., Funk, M. J., Brookhart, M. A., & Stürmer, T. (2010) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.2074 · Cited by 165 · see citing works
Asymptotics of cross-validated risk estimation in estimator selection and performance assessment
Dudoit, S., & van der Laan, M. J. (2005) · Statistical Methodology · doi:10.1016/j.stamet.2005.02.003 · Cited by 164 · see citing works
Propensity scores in intensive care and anaesthesiology literature: a systematic review
Gayat, É., Pirracchio, R., Resche‐Rigon, M., Mebazaa, A., Mary, J., & Porcher, R. (2010) · Intensive Care Medicine · doi:10.1007/s00134-010-1991-5 · Cited by 128 · see citing works
Self-citationThe cross-validated adaptive epsilon-net estimator
van der Laan, M. J., Dudoit, S., & van der Vaart, A. (2006) · Statistics & Risk Modeling · doi:10.1524/stnd.2006.24.3.373 · Cited by 126 · see citing works
Rassen, J. A., & Schneeweiß, S. (2012) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.2328 · Cited by 98 · see citing works
Weitzen, S. H., Lapane, K. L., Toledano, A. Y., Hume, A. L., & Mor, V. (2004) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.986 · Cited by 93 · see citing works
Marijuana use and depression among adults: testing for causal associations
Harder, V. S., Morral, A. R., & Arkes, J. A. (2006) · Addiction · doi:10.1111/j.1360-0443.2006.01545.x · Cited by 84 · see citing works
Do Instrumental Variables Belong in Propensity Scores?
Bhattacharya, J., & Vogt, W. (2007) · doi:10.3386/t0343 · Cited by 81 · see citing works
Targeted Estimation of Nuisance Parameters to Obtain Valid Statistical Inference
van der Laan, M. J. (2014) · The International Journal of Biostatistics · doi:10.1515/ijb-2012-0038 · Cited by 79 · see citing works
twang: Toolkit for Weighting and Analysis of Nonequivalent Groups
Cefalu, M., Ridgeway, G., McCaffrey, D., et al. (2006) · doi:10.32614/cran.package.twang · Cited by 77 · see citing works
Harder, V. S., Stuart, E. A., & Anthony, J. C. (2008) · American Journal of Epidemiology · doi:10.1093/aje/kwn184 · Cited by 76 · see citing works
Impact of mis‐specification of the treatment model on estimates from a marginal structural model
Lefebvre, G., Delaney, J. A., & Platt, R. W. (2008) · Statistics in Medicine · doi:10.1002/sim.3200 · Cited by 66 · see citing works
D’Agostino (1998) · doi:10.1002/(sici)1097-0258(19981015)17:19
Flagged first, then by citations
Cited by
Sorted by citations
Artificial Intelligence and Surgical Decision-making.
Loftus, T. J., Tighe, P. J., Filiberto, A. C., Efron, P. A., Brakenridge, S. C., Mohr, A. M., Rashidi, P., Upchurch, G. R., & Bihorac, A. (2020) · JAMA Surgery · doi:10.1001/jamasurg.2019.4917 · Cited by 611 · see citing works
Reflection on modern methods: when worlds collide—prediction, machine learning and causal inference
Blakely, T., Lynch, J., Simons, K., Bentley, R., & Rose, S. (2021) · International Journal of Epidemiology · doi:10.1093/ije/dyz132 · Cited by 98 · see citing works
Ferri-García, R., & Rueda, M. D. M. (2020) · PLOS ONE · doi:10.1371/journal.pone.0231500 · Cited by 67 · see citing works
Machine Learning in Policy Evaluation: New Tools for Causal Inference
Kreif, N., & DiazOrdaz, K. (2019) · Oxford Research Encyclopedia of Economics and Finance · doi:10.1093/acrefore/9780190625979.013.256 · Cited by 57 · see citing works
- Authors
- Pirracchio, R., Petersen, M. L., & van der Laan, M.
- Affiliations
- Hôpital Européen Georges-Pompidou (FR)
- Corresponding author
- Pirracchio, Romain
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2014-12-16
- Volume
- 181
- Issue
- 2
- Pages
- 108-119
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- PMC
- PMC4351345
- OpenAlex
- W2129092711
- Cited by
- 186
- References
- 67
- Citations by year
- 2026: 10 · 2025: 15 · 2024: 23 · 2023: 18 · 2022: 16 · 2021: 19 · 2020: 22 · 2019: 17 · 2018: 21 · 2017: 10 · 2016: 12 · 2015: 2 · 2014: 1
- FWCI
- 9.15
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- American Journal of Epidemiology (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- Institut National de la Santé et de la Recherche Médicale
- Université Paris Descartes
- Université Paris Diderot
- Assistance publique-Hôpitaux de Paris
- National Institutes of Health
- National Institute of Allergy and Infectious Diseases (R01AI074345)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- propensity score estimation, super learner, model misspecification, covariate balance, inverse probability of treatment weighting, propensity score matching, average treatment effect, machine learning, logistic regression, bias mitigation
- MeSH terms
- Artificial Intelligence, Computer Simulation, Data Interpretation, Statistical, Epidemiologic Methods, Humans, Bias, Propensity Score
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
