A tutorial on propensity score estimation for multiple treatments using generalized boosted models
Open accessAbstract
The use of propensity scores to control for pretreatment imbalances on observed variables in non‐randomized or observational studies examining the causal effects of treatments or interventions has become widespread over the past decade. For settings with two conditions of interest such as a treatment and a control, inverse probability of treatment weighted estimation with propensity scores estimated via boosted models has been shown in simulation studies to yield causal effect estimates with desirable properties. There are tools (e.g., the twang package in R) and guidance for implementing this method with two treatments. However, there is not such guidance for analyses of three or more treatments. The goals of this paper are twofold: (1) to provide step‐by‐step guidance for researchers who want to implement propensity score weighting for multiple treatments and (2) to propose the use of generalized boosted models (GBM) for estimation of the necessary propensity score weights. We define the causal quantities that may be of interest to studies of multiple treatments and derive weighted estimators of those quantities. We present a detailed plan for using GBM to estimate propensity scores and using those scores to estimate weights and causal effects. We also provide tools for assessing balance and overlap of pretreatment variables among treatment groups in the context of multiple treatments. A case study examining the effects of three treatment programs for adolescent substance abuse demonstrates the methods. Copyright © 2013 John Wiley & Sons, Ltd.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The early stopping concept can be extended to C > 2 treatment categories and K risk factors by using the ASAM criterion in equation (A.4).
“We can extend this concept straightforwardly to the case of 𝐶 > 2 treatment categories and 𝐾 risk factors by considering”
From Customer price sensitivities in competitive insurance markets (2022) · cites this paper as McCaffrey et al. (2013)
Section 4.1 describes early stopping by minimizing a balance criterion, including mean absolute standardized bias across pretreatment covariates, and extends the balance metric to more than two treatments. Section 5.2 applies a mean standardized-bias stopping rule to three treatments. Our copy does not identify an ASAM criterion or equation (A.4), so that specific attribution is unsupported. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,588 · see citing works
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241 · see citing works
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · see citing works
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,834 · see citing works
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · see citing works
Friedman, J. H. (2002) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(01)00065-2 · Cited by 7,067 · see citing works
Marginal Structural Models and Causal Inference in Epidemiology
Robins, J. M., Hernán, M. Á., & Brumback, B. (2000) · Epidemiology · doi:10.1097/00001648-200009000-00011 · Cited by 5,581 · see citing works
Matching Methods for Causal Inference: A Review and a Look Forward
Stuart, E. A. (2010) · Statistical Science · doi:10.1214/09-sts313 · Cited by 5,478 · see citing works
Statistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · see citing works
Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference
Ho, D. E., Imai, K., King, G., & Stuart, E. (2007) · Political Analysis · doi:10.1093/pan/mpl013 · Cited by 4,411 · see citing works
Reducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082 · see citing works
Rosenbaum, P. R. (2002) · Springer series in statistics · doi:10.1007/978-1-4757-3692-2 · Cited by 2,414 · see citing works
Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs
Dehejia, R. H., & Wahba, S. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473858 · Cited by 2,395 · see citing works
Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020363010465 · Cited by 2,228 · see citing works
The role of the propensity score in estimating dose-response functions
Imbens, G. W. (2000) · Biometrika · doi:10.1093/biomet/87.3.706 · Cited by 1,747 · see citing works
McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.4.403 · Cited by 1,317 · see citing works
Self-citationKang, J. D. Y., & Schafer, J. L. (2007) · Statistical Science · doi:10.1214/07-sts227 · Cited by 1,259 · see citing works
Hirano, K., & Imbens, G. W. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020371312283 · Cited by 1,039 · see citing works
Causal Inference With General Treatment Regimes
Imai, K., & van Dyk, D. V. (2004) · Journal of the American Statistical Association · doi:10.1198/016214504000001187 · Cited by 1,022 · see citing works
The Effectiveness of Adjustment by Subclassification in Removing Bias in Observational Studies
Cochran, W. G. (1968) · Biometrics · doi:10.2307/2528036 · Cited by 961 · see citing works
8. Misunderstandings Between Experimentalists and Observationalists About Causal Inference
Imai, K., King, G., & Stuart, E. A. (2008) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2007.00527.x · Cited by 895 · see citing works
Improving propensity score weighting using machine learning
Lee, B. K., Lessler, J., & Stuart, E. A. (2009) · Statistics in Medicine · doi:10.1002/sim.3782 · Cited by 888 · see citing works
Predictive Margins with Survey Data
Graubard, B. I., & Korn, E. L. (1999) · Biometrics · doi:10.1111/j.0006-341x.1999.00652.x · Cited by 875 · see citing works
Harder, V. S., Stuart, E. A., & Anthony, J. C. (2010) · Psychological Methods · doi:10.1037/a0019623 · Cited by 802 · see citing works
The Cannabis Youth Treatment (CYT) Study: main findings from two randomized trials.
Dennis, M. L., Godley, S., Diamond, G., Tims, F., Babor, T., Donaldson, J., Liddle, H., Titus, J. C., Kaminer, Y., Webb, C., et al. (2004) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2003.09.005 · Cited by 783 · see citing works
Lechner, M. (2001) · ZEW economic studies · doi:10.1007/978-3-642-57615-7_3 · Cited by 728 · see citing works
Kane, L. T., Fang, T., Galetta, M. S., Goyal, D. K. C., Nicholson, K. J., Kepler, C. K., Vaccaro, A. R., & Schroeder, G. D. (2020) · Clinical Spine Surgery A Spine Publication · doi:10.1097/bsd.0000000000000932 · Cited by 610 · see citing works
Weight Trimming and Propensity Score Weighting
Lee, B. K., Lessler, J., & Stuart, E. A. (2011) · PLoS ONE · doi:10.1371/journal.pone.0018174 · Cited by 507 · see citing works
Westreich, D. J., Lessler, J., & Funk, M. J. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.11.020 · Cited by 505 · see citing works
The Consistency Statement in Causal Inference
Cole, S. R., & Frangakis, C. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818ef366 · Cited by 413 · see citing works
Kang, J., & Schafer, J. L. (2007) · Statistical Science · Cited by 368 · see citing works
Effects of Adjusting for Instrumental Variables on Bias and Precision of Effect Estimates
Myers, J. A., Rassen, J. A., Gagne, J. J., Huybrechts, K. F., Schneeweiß, S., Rothman, K. J., Joffe, M. M., & Glynn, R. J. (2011) · American Journal of Epidemiology · doi:10.1093/aje/kwr364 · Cited by 286 · see citing works
The Multiple Propensity Score as Control for Bias in the Comparison of More Than Two Treatment Arms
Spreeuwenberg, M. D., Bartak, A., Croon, M. A., Hagenaars, J. A. P., Busschbach, J. J. V., Andrea, H., Twisk, J. W. R., & Stijnen, T. (2010) · Medical Care · doi:10.1097/mlr.0b013e3181c1328f · Cited by 226 · see citing works
Econometric Evaluation of Labour Market Policies
Lechner, M., & Pfeiffer, F. (2001) · ZEW economic studies · doi:10.1007/978-3-642-57615-7 · Cited by 205 · see citing works
Generalized propensity score for estimating the average treatment effect of multiple treatments
Feng, P., Zhou, X., Zou, Q., Fan, M., & Li, X. (2011) · Statistics in Medicine · doi:10.1002/sim.4168 · Cited by 180 · see citing works
Foster, E. M. (2003) · Medical Care · doi:10.1097/01.mlr.0000089629.62884.22 · Cited by 146 · see citing works
Programme Evaluation with Multiple Treatments
Frölich, M. (2004) · Journal of Economic Surveys · doi:10.1111/j.0950-0804.2004.00001.x · Cited by 139 · see citing works
Why prefer double robust estimators in causal inference?
Neugebauer, R. S., & van der Laan, M. J. (2004) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2004.06.060 · Cited by 125 · see citing works
Waxmonsky, J. G. (2004) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-200403000-00022 · Cited by 114 · see citing works
Zanutto, E. L., Lü, B., & Hornik, R. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030001059 · Cited by 108 · see citing works
Adolescent Substance Abuse Treatment in the United States
Segal, B., Morral, A. R., & Stevens, S. J. (2014) · Routledge eBooks · doi:10.4324/9781315821344 · Cited by 91 · see citing works
Huang, I., Frangakis, C., Dominici, F., Diette, G. B., & Wu, A. W. (2005) · Health Services Research · doi:10.1111/j.1475-6773.2005.00352.x · Cited by 89 · see citing works
Ramchand, R., Griffin, B. A., Suttorp, M. J., Harris, K. M., & Morral, A. R. (2011) · Journal of Studies on Alcohol and Drugs · doi:10.15288/jsad.2011.72.380 · Cited by 34 · see citing works
Self-citationDoes Weighting for Nonresponse Increase the Variance of Survey Means? (Conference Paper)Preprint
Little, R. J. A., & Vartivarian, S. (2005) · RePEc: Research Papers in Economics · Cited by 32 · see citing works
Dennis, M. L., Ives, M. L., White, M. K., & Muck, R. D. (2008) · Journal of Psychoactive Drugs · doi:10.1080/02791072.2008.10399757 · Cited by 23 · see citing works
Implementation of MET/CBT 5 for Adolescents
Riley, K. J., Rieckmann, T. R., & McCarty, D. W. (2008) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-008-9111-9 · Cited by 22 · see citing works
Hunter, S. B., Ramchand, R., Griffin, B. A., Suttorp, M. J., McCaffrey, D. F., & Morral, A. R. (2011) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2011.11.003 · Cited by 22 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- [27]Wooldridge J Econometric of Analysis of Cross Section and Panel Data MIT Press Cambridge, MA 2002
Cited by
Sorted by citations
High-dimensional characterization of post-acute sequelae of COVID-19
Al-Aly, Z., Xie, Y., & Bowe, B. (2021) · Nature · doi:10.1038/s41586-021-03553-9 · Cited by 1,518 · see citing works
Long-term neurologic outcomes of COVID-19
Xu, E., Xie, Y., & Al-Aly, Z. (2022) · Nature Medicine · doi:10.1038/s41591-022-02001-z · Cited by 530 · see citing works
George, M. D., Baker, J. F., Winthrop, K., Hsu, J. Y., Wu, Q., Chen, L., Xie, F., Yun, H., & Curtis, J. R. (2020) · Annals of Internal Medicine · doi:10.7326/m20-1594 · Cited by 198 · see citing works
Kemp, A. H., Brunoni, A. R., Santos, I. S., Nunes, M. A., Dantas, E. M., Carvalho de Figueiredo, R., Pereira, A. C., Ribeiro, A. L. P., Mill, J. G., Andreão, R. V., et al. (2014) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2014.13121605 · Cited by 189 · see citing works
Jones, B. E., Ying, J., Stevens, V., Haroldsen, C., He, T., Nevers, M., Christensen, M. A., Nelson, R. E., Stoddard, G. J., Sauer, B. C., et al. (2020) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2019.7495 · Cited by 122 · see citing works
Kemp, A. H., Koenig, J., & Thayer, J. F. (2017) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.09.006 · Cited by 114 · see citing works
Zhang, X., Wong, G. L., Yip, T. C., Tse, Y., Liang, L. Y., Hui, V. W., Lin, H., Li, G., Lai, J. C., Chan, H. L., et al. (2022) · Hepatology · doi:10.1002/hep.32294 · Cited by 93 · see citing works
Naylor, M. D., Hirschman, K. B., Hanlon, A. L., Bowles, K. H., Bradway, C., McCauley, K. M., & Pauly, M. V. (2014) · Journal of Comparative Effectiveness Research · doi:10.2217/cer.14.14 · Cited by 91 · see citing works
Ferri-García, R., & Rueda, M. D. M. (2020) · PLOS ONE · doi:10.1371/journal.pone.0231500 · Cited by 67 · see citing works
Shen, E., Koyama, S. Y., Huynh, D. N., Watson, H. L., Mittman, B., Kanter, M. H., & Nguyen, H. Q. (2017) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.7061 · Cited by 64 · see citing works
Bentley, R., Baker, E., Simons, K., Simpson, J. A., & Blakely, T. (2018) · International Journal of Epidemiology · doi:10.1093/ije/dyy116 · Cited by 52 · see citing works
MacNeil Vroomen, J., Bosmans, J. E., Eekhout, I., Joling, K. J., van Mierlo, L. D., Meiland, F. J. M., van Hout, H. P. J., & de Rooij, S. E. (2016) · PLOS ONE · doi:10.1371/journal.pone.0160908 · Cited by 48 · see citing works
Reputation Burning: Analyzing the Impact of Brand Sponsorship on Social Influencers
Cheng, M. (., & Zhang, S. (2025) · Management Science · doi:10.1287/mnsc.2023.00193 · Cited by 43 · see citing works
Graham, A. L., Papandonatos, G. D., Erar, B., & Stanton, C. A. (2015) · Health Psychology · doi:10.1037/hea0000278 · Cited by 40 · see citing works
Effects of prenatal bacterial infection on cognitive performance in early childhood
Lee, Y. H., Papandonatos, G. D., Savitz, D. A., Heindel, W. C., & Buka, S. L. (2020) · Paediatric and Perinatal Epidemiology · doi:10.1111/ppe.12603 · Cited by 21 · see citing works
Ramchand, R., Griffin, B. A., Slaughter, M. E., Almirall, D., & McCaffrey, D. F. (2014) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2014.06.012 · Cited by 18 · see citing works
Self-citationImproving Adherence to Smoking Cessation Treatment: Smoking Outcomes in a Web-based Randomized Trial
Graham, A. L., Papandonatos, G. D., Cha, S., Erar, B., & Amato, M. S. (2018) · Annals of Behavioral Medicine · doi:10.1093/abm/kax023 · Cited by 18 · see citing works
Svensson, A., Toll, A., Lebrec, J., Miftaraj, M., Franzén, S., & Eliasson, B. (2021) · Diabetes, Obesity and Metabolism · doi:10.1111/dom.14276 · Cited by 17 · see citing works
Customer price sensitivities in competitive insurance markets
Verschuren, R. M. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117133 · Cited by 13 · see citing works
Bottigliengo, D., Lorenzoni, G., Ocagli, H., Martinato, M., Berchialla, P., & Gregori, D. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18136694 · Cited by 13 · see citing works
- Authors
- McCaffrey, D. F., Griffin, B. A., Almirall, D., Slaughter, M. E., Ramchand, R., & Burgette, L. F.
- Affiliations
- RAND Corporation (US)
- University of Michigan (US)
- Corresponding author
- McCaffrey, Daniel F.
- ORCID
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2013-03-18
- Volume
- 32
- Issue
- 19
- Pages
- 3388-3414
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- PMC
- PMC3710547
- OpenAlex
- W2048470090
- Cited by
- 1,450
- References
- 47
- Citations by year
- 2026: 121 · 2025: 141 · 2024: 163 · 2023: 161 · 2022: 179 · 2021: 168 · 2020: 162 · 2019: 107 · 2018: 74 · 2017: 62 · 2016: 61 · 2015: 23 · 2014: 23 · 2013: 2 · 2012: 1
- FWCI
- 32.07
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Statistics in Medicine (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Deep Blue (University of Michigan) (submittedVersion)
- Funders
- Substance Abuse and Mental Health Services Administration (270-07-0191)
- Adolescent Treatment Model (270-98-7047, 270-97-7011, 277-00-6500, 270-2003-00006)
- Effective Adolescent Treatment (270-2003-00006)
- Strengthening Communities-Youth (277-00-6500, 270-2003-00006)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- propensity score estimation, multiple treatments, generalized boosted models, inverse probability of treatment weighting, causal effect estimation, observational studies, balance assessment
- MeSH terms
- Adolescent, Clinical Trials as Topic, Humans, Models, Statistical, Treatment Outcome, Substance-Related Disorders, Propensity Score
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
