The Use of Propensity Scores to Assess the Generalizability of Results from Randomized Trials
Open accessAbstract
Summary: Randomized trials remain the most accepted design for estimating the effects of interventions, but they do not necessarily answer a question of primary interest: will the programme be effective in a target population in which it may be implemented? In other words, are the results generalizable? There has been very little statistical research on how to assess the generalizability, or ‘external validity’, of randomized trials. We propose the use of propensity-score-based metrics to quantify the similarity of the participants in a randomized trial and a target population. In this setting the propensity score model predicts participation in the randomized trial, given a set of covariates. The resulting propensity scores are used first to quantify the difference between the trial participants and the target population, and then to match, subclassify or weight the control group outcomes to the population, assessing how well the propensity-score-adjusted outcomes track the outcomes that are actually observed in the population. These metrics can serve as a first step in assessing the generalizability of results from randomized trials to target populations. The paper lays out these ideas, discusses the assumptions underlying the approach and illustrates the metrics by using data on the evaluation of a schoolwide prevention programme called ‘Positive behavioral interventions and supports’.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Controlling confounding in observational studies, by assigning propensity scores to participants with respect to their individual characteristics is regarded as best practice.
“Controlling confounding in observational studies, by assigning propensity scores to participants with respect to their individual characteristics is regarded as best practice.50,51”
From Mindfulness-based interventions for psychological wellbeing and quality of life in men with prostate cancer: A systematic review and meta-analysis (2021) · cites this paper as [51]
Section 2.1 (page unavailable) says propensity-score matching in observational studies focuses on reducing treatment-selection bias associated with observed characteristics. That supports the confounding-control aspect, but the paper does not characterize assigning propensity scores as “best practice”; its own application concerns the generalizability of randomized trials. (LLM, conf=0.85)
Experimental — accuracy not yet measuredPropensity scores based on site-level covariates are useful for controlling some degree of assignment bias.
“Propensity scores, based on site-level covariates, are useful in controlling for some degree of assignment bias (98, 104).”
From An Overview of Research and Evaluation Designs for Dissemination and Implementation (2017) · cites this paper as [104]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241 · see citing works
(2002) · Cited by 13,211 · see citing works
Wasserman, S. S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,347 · see citing works
RUBIN, D. B. (1976) · Biometrika · doi:10.1093/biomet/63.3.581 · Cited by 9,925 · see citing works
Matching Methods for Causal Inference: A Review and a Look Forward
Stuart, E. A. (2010) · Statistical Science · doi:10.1214/09-sts313 · Cited by 5,478 · see citing works
Self-citationStatistical Methods for Meta-Analysis.
Freeman, P. R., Hedges, L. V., & Olkin, I. (1986) · Biometrics · doi:10.2307/2531069 · Cited by 4,832 · see citing works
Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference
Ho, D. E., Imai, K., King, G., & Stuart, E. (2007) · Political Analysis · doi:10.1093/pan/mpl013 · Cited by 4,411 · see citing works
Self-citationA generalization of sampling without replacement from a finite universe
Horvitz, D. G., & Thompson, D. J. (1952) · Journal of the American Statistical Association · doi:10.1080/01621459.1952.10483446 · Cited by 4,293 · see citing works
A Generalization of Sampling Without Replacement From a Finite Universe
Horvitz, D. G., & Thompson, D. J. (1952) · Journal of the American Statistical Association · doi:10.2307/2280784 · Cited by 3,011 · see citing works
Constructing Inverse Probability Weights for Marginal Structural Models
Cole, S. R., & Hernan, M. A. (2008) · American Journal of Epidemiology · doi:10.1093/aje/kwn164 · Cited by 2,796 · see citing works
Self-citationExternal validity of randomised controlled trials: “To whom do the results of this trial apply?”
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(04)17670-8 · Cited by 2,544 · see citing works
Rosenbaum, P. R. (2002) · Springer series in statistics · doi:10.1007/978-1-4757-3692-2 · Cited by 2,414 · see citing works
Variable Selection for Propensity Score Models
Brookhart, M. A., Schneeweiss, S., Rothman, K. J., Glynn, R. J., Avorn, J., & Stürmer, T. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj149 · Cited by 2,336 · see citing works
Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020363010465 · Cited by 2,228 · see citing works
Lunceford, J. K., & Davidian, M. (2004) · Statistics in Medicine · doi:10.1002/sim.1903 · Cited by 1,676 · see citing works
Statistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials
Wang, R., Lagakos, S. W., Ware, J. H., Hunter, D. J., & Drazen, J. M. (2007) · New England Journal of Medicine · doi:10.1056/nejmsr077003 · Cited by 1,445 · see citing works
Standards of Evidence: Criteria for Efficacy, Effectiveness and Dissemination
Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., Mościcki, E. K., Schinke, S., Valentine, J. C., & Ji, P. (2005) · Prevention Science · doi:10.1007/s11121-005-5553-y · Cited by 1,338 · see citing works
Rosenbaum, P., & Rubin, D. (1983) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1983.tb01242.x · Cited by 1,264 · see citing works
Assignment to Treatment Group on the Basis of a Covariate
Rubin, D. B. (1977) · Journal of Educational Statistics · doi:10.3102/10769986002001001 · Cited by 1,191 · see citing works
Factors relevant to the validity of experiments in social settings.
Campbell, D. T. (1957) · Psychological Bulletin · doi:10.1037/h0040950 · Cited by 1,143 · see citing works
Controlling Bias in Observational Studies: A Review
Cochran, W. G. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810725.005 · Cited by 986 · see citing works
A Promising Approach for Expanding and Sustaining School-Wide Positive Behavior Support
Sugai, G. M., & Horner, R. R. (2006) · School Psychology Review · doi:10.1080/02796015.2006.12087989 · Cited by 954 · see citing works
Subgroup analysis in randomised controlled trials: importance, indications, and interpretation
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(05)17709-5 · Cited by 942 · see citing works
8. Misunderstandings Between Experimentalists and Observationalists About Causal Inference
Imai, K., King, G., & Stuart, E. A. (2008) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2007.00527.x · Cited by 895 · see citing works
Self-citationBradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010) · Journal of Positive Behavior Interventions · doi:10.1177/1098300709334798 · Cited by 868 · see citing works
Self-citationFor objective causal inference, design trumps analysis
Rubin, D. B. (2008) · The Annals of Applied Statistics · doi:10.1214/08-aoas187 · Cited by 831 · see citing works
Horner, R. H., Sugai, G. M., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., & Esperanza, J. (2009) · Journal of Positive Behavior Interventions · doi:10.1177/1098300709332067 · Cited by 826 · see citing works
Observational Studies Analyzed Like Randomized Experiments
Hernán, M. Á., Alonso, Á., Logan, R. W., Grodstein, F., Michels, K. B., Willett, W. C., Manson, J. E., & Robins, J. M. (2008) · Epidemiology · doi:10.1097/ede.0b013e3181875e61 · Cited by 822 · see citing works
Combining Propensity Score Matching with Additional Adjustments for Prognostic Covariates
Rubin, D., & Thomas, N. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10474233 · Cited by 796 · see citing works
Flay, B. R. (1986) · Preventive Medicine · doi:10.1016/0091-7435(86)90024-1 · Cited by 723 · see citing works
Reader's guide to critical appraisal of cohort studies: 2. Assessing potential for confounding
Mamdani, M., Sykora, K., Li, P., Normand, S. T., Streiner, D. L., Austin, P. C., Rochon, P. A., & Anderson, G. (2005) · BMJ · doi:10.1136/bmj.330.7497.960 · Cited by 651 · see citing works
Full Matching in an Observational Study of Coaching for the SAT
Hansen, B. B. (2004) · Journal of the American Statistical Association · doi:10.1198/016214504000000647 · Cited by 648 · see citing works
Recent developments in meta‐analysis
Sutton, A. J., & Higgins, J. P. T. (2007) · Statistics in Medicine · doi:10.1002/sim.2934 · Cited by 583 · see citing works
Web Survey Methods: Introduction
Couper, M. P., & Miller, P. G. (2008) · Public Opinion Quarterly · doi:10.1093/poq/nfn066 · Cited by 567 · see citing works
Improving the Quality of Health Care for Mental and Substance-Use Conditions
Institute of Medicine (2006) · National Academies Press eBooks · doi:10.17226/11470 · Cited by 559 · see citing works
Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial.
Cole, S., & Stuart, E. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq084 · Cited by 535 · see citing works
Self-citationHead Start Impact Study. Final Report.
Puma, M. J., Bell, S., Cook, R., Heid, C. A., Shapiro, G., Broene, P., Jenkins, F. J., Fletcher, P., Quinn, L., Friedman, J., et al. (2010) · Cited by 519 · see citing works
Examining the Role of Implementation Quality in School-Based Prevention Using the PATHS Curriculum
Kam, C.-M., Greenberg, M. T., & Walls, C. T. (2003) · Prevention Science · doi:10.1023/a:1021786811186 · Cited by 516 · see citing works
Combining Propensity Score Matching with Additional Adjustments for Prognostic Covariates
Thomas, N. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810725.025 · Cited by 508 · see citing works
The Use of Matched Sampling and Regression Adjustment to Remove Bias in Observational Studies
Rubin, D. B. (1973) · Biometrics · doi:10.2307/2529685 · Cited by 456 · see citing works
The Use of Matched Sampling and Regression Adjustment to Remove Bias in Observational Studies
Rubin, D. B. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810725.008 · Cited by 451 · see citing works
Bradshaw, C. P., Koth, C. W., Thornton, L. A., & Leaf, P. J. (2009) · Prevention Science · doi:10.1007/s11121-008-0114-9 · Cited by 448 · see citing works
Self-citationSite Selection Bias in Program Evaluation *
Allcott, H. (2015) · The Quarterly Journal of Economics · doi:10.1093/qje/qjv015 · Cited by 426 · see citing works
The prognostic analogue of the propensity score
Hansen, B. B. (2008) · Biometrika · doi:10.1093/biomet/asn004 · Cited by 379 · see citing works
Kang, J., & Schafer, J. L. (2007) · Statistical Science · Cited by 368 · see citing works
Large-scale randomized evidence: Large, simple trials and overviews of trials
Peto, R., Collins, R., & Gray, R. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)00150-o · Cited by 365 · see citing works
Evaluating Kindergarten Retention Policy
Hong, G., & Raudenbush, S. W. (2006) · Journal of the American Statistical Association · doi:10.1198/016214506000000447 · Cited by 348 · see citing works
A Characterization of Optimal Designs for Observational Studies
Rosenbaum, P. R. (1991) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1991.tb01848.x · Cited by 331 · see citing works
Understanding the Divergent Data on Postmenopausal Hormone Therapy
Grodstein, F., Clarkson, T. B., & Manson, J. E. (2003) · New England Journal of Medicine · doi:10.1056/nejmsb022365 · Cited by 328 · see citing works
Stuart, E. A., & Green, K. M. (2008) · Developmental Psychology · doi:10.1037/0012-1649.44.2.395 · Cited by 315 · see citing works
Self-citationHead Start Impact Study: First Year Findings.
Puma, M. J., Bell, S., Cook, R., Heid, C. A., & López, M. (2005) · Cited by 278 · see citing works
Mechanism Experiments and Policy Evaluations
Ludwig, J., Kling, J. R., & Mullainathan, S. (2011) · The Journal of Economic Perspectives · doi:10.1257/jep.25.3.17 · Cited by 238 · see citing works
Rubin, D. B., & Thomas, N. (1992) · Biometrika · doi:10.1093/biomet/79.4.797 · Cited by 188 · see citing works
Maryland Statewide PBIS Initiative
Barrett, S. B., Bradshaw, C. P., & Lewis-Palmer, T. (2008) · Journal of Positive Behavior Interventions · doi:10.1177/1098300707312541 · Cited by 188 · see citing works
Self-citationBrown, C. H., Wang, W., Kellam, S. G., Muthén, B. O., Petras, H., Toyinbo, P., Poduska, J., Ialongo, N., Wyman, P. A., Chamberlain, P., et al. (2008) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2007.11.013 · Cited by 177 · see citing works
Using RE-AIM Metrics to Evaluate Diabetes Self-Management Support Interventions
Glasgow, R. E., Nelson, C. C., Strycker, L. A., & King, D. K. (2006) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2005.08.037 · Cited by 173 · see citing works
Teacher Observation of Classroom Adaptation—Checklist: Development and Factor Structure
Koth, C., Bradshaw, C. P., & Leaf, P. (2009) · Measurement and Evaluation in Counseling and Development · doi:10.1177/0748175609333560 · Cited by 152 · see citing works
Self-citationMeWAnalysis by the Confidence Profile Method: The Statistical Synthesis of Evidence
Lenth, R. V. (1993) · Technometrics · doi:10.1080/00401706.1993.10485067 · Cited by 121 · see citing works
Haneuse, S., Schildcrout, J. S., Crane, P. K., Sonnen, J. A., Breitner, J. C. S., & Larson, E. B. (2009) · Neuroepidemiology · doi:10.1159/000197389 · Cited by 106 · see citing works
Beyond Efficacy: The STAR*D Trial
Insel, T. R. (2006) · American Journal of Psychiatry · doi:10.1176/appi.ajp.163.1.5 · Cited by 105 · see citing works
Meta-Analysis by the Confidence Profile Method: The Statistical Synthesis of Evidence
Lenth, R. V., Eddy, D. M., Hasselblad, V., & Shachter, R. D. (1993) · Technometrics · doi:10.2307/1269690 · Cited by 80 · see citing works
Weisberg, H. I., Hayden, V., & Pontes, V. P. (2009) · Clinical Trials · doi:10.1177/1740774509102563 · Cited by 73 · see citing works
Greenhouse, J. B., Kaizar, E. E., Kelleher, K. J., Seltman, H. J., & Gardner, W. P. (2008) · Statistics in Medicine · doi:10.1002/sim.3218 · Cited by 64 · see citing works
The calibration of treatment effects from clinical trials to target populations
Frangakis, C. (2009) · Clinical Trials · doi:10.1177/1740774509103868 · Cited by 52 · see citing works
Seftor, N. S., Mamun, A. A., & Schirm, A. L. (2009) · US Department of Education · Cited by 18 · see citing works
Evaluating bias correction in weighted proportional hazards regression
Pan, Q., & Schaubel, D. E. (2008) · Lifetime Data Analysis · doi:10.1007/s10985-008-9102-4 · Cited by 15 · see citing works
Hahn, S., Williamson, P. R., Hutton, J. L., Garner, P., & Flynn, E. V. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001230)19:24
Flagged first, then by citations
Research works printed but not linked: 1
- Cook, T. D. (2007) Evidence-based practice: where do we stand? In Proc. 20th A. Res. Conf. System of Care for Children's Mental Health: Expanding the Research Base. Tampa: University of South Florida. (Available from http://rtckids.fmhi.usf.edu/rtcconference/20thconference/iding.cfm.)
Cited by
Sorted by citations
Understanding and misunderstanding randomized controlled trials
Deaton, A., & Cartwright, N. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2017.12.005 · Cited by 1,841 · see citing works
Kennedy-Martin, T., Curtis, S., Faries, D., Robinson, S., & Johnston, J. (2015) · Trials · doi:10.1186/s13063-015-1023-4 · Cited by 878 · see citing works
An Overview of Research and Evaluation Designs for Dissemination and Implementation
Brown, C. H., Curran, G., Palinkas, L. A., Aarons, G. A., Wells, K. B., Jones, L., Collins, L. M., Duan, N., Mittman, B. S., Wallace, A., et al. (2017) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-031816-044215 · Cited by 509 · see citing works
Frank, K. A., Maroulis, S. J., Duong, M. Q., & Kelcey, B. M. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373713493129 · Cited by 406 · see citing works
Stable weights that balance covariates for estimation with incomplete outcome data
Zubizarreta, J. R. (2015) · Journal of the American Statistical Association · doi:10.1080/01621459.2015.1023805 · Cited by 369 · see citing works
Get real in individual participant data (IPD) meta‐analysis: a review of the methodology
Debray, T. P. A., Moons, K. G. M., van Valkenhoef, G., Efthimiou, O., Hummel, N., Groenwold, R. H. H., Reitsma, J. B., & on behalf of the GetReal methods review group (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1160 · Cited by 348 · see citing works
Inference for Nonprobability Samples
Elliott, M. R., & Valliant, R. (2017) · Statistical Science · doi:10.1214/16-sts598 · Cited by 331 · see citing works
Assessing the generalizability of randomized trial results to target populations
Stuart, E. A., Bradshaw, C. P., & Leaf, P. J. (2015) · Prevention Science · doi:10.1007/s11121-014-0513-z · Cited by 296 · see citing works
Self-citationGeneralizing Study Results: A Potential Outcomes Perspective
Lesko, C. R., Buchanan, A. L., Westreich, D., Edwards, J. K., Hudgens, M. G., & Cole, S. R. (2017) · Epidemiology · doi:10.1097/ede.0000000000000664 · Cited by 269 · see citing works
Self-citationWaasdorp, T. E. (2012) · Archives of Pediatrics & Adolescent Medicine · doi:10.1001/archpediatrics.2011.755 · Cited by 263 · see citing works
Yang, S., Kim, J. K., & Song, R. (2020) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/rssb.12354 · Cited by 73 · see citing works
Quantifying representativeness in randomized clinical trials using machine learning fairness metrics
Qi, M., Cahan, O., Foreman, M. A., Gruen, D. M., Das, A. K., & Bennett, K. P. (2021) · JAMIA Open · doi:10.1093/jamiaopen/ooab077 · Cited by 27 · see citing works
Nnate, D. A., Anyachukwu, C. C., Igwe, S. E., & Abaraogu, U. O. (2021) · Psycho-Oncology · doi:10.1002/pon.5749 · Cited by 24 · see citing works
“Such approach has also been recommended for a more generalisable randomised trials where certain covariates exist by chance.(52,53)”
Background
- Authors
- Stuart, E. A., Cole, S. R., Bradshaw, C. P., & Leaf, P. J.
- Affiliations
- Johns Hopkins University (US)
- University of North Carolina at Chapel Hill (US)
- Corresponding author
- Stuart, Elizabeth A.
- Venue
- Journal of the Royal Statistical Society Series A: Statistics in Society
- Publisher
- Oxford University Press (OUP)
- Published
- 2010-11-30
- Volume
- 174
- Issue
- 2
- Pages
- 369-386
- ISSN
- 0964-1998 (ISSN-L)
- 1467-985X
- PMC
- PMC4051511
- OpenAlex
- W1591419506
- Cited by
- 524
- References
- 67
- Citations by year
- 2026: 20 · 2025: 54 · 2024: 44 · 2023: 43 · 2022: 39 · 2021: 53 · 2020: 47 · 2019: 33 · 2018: 39 · 2017: 39 · 2016: 44 · 2015: 21 · 2014: 21 · 2013: 12 · 2012: 9
- FWCI
- 7.04
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the Royal Statistical Society Series A (Statistics in Society) (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Carolina Digital Repository (University of North Carolina at Chapel Hill) (submittedVersion)
- Collection of Biostatistics Research Archive (submittedVersion)
- UNC Libraries
- Funders
- National Institute of Mental Health (K25 MH083846, 1 R01 MH67948-1A1)
- Centers for Disease Control (R49/CCR318627)
- Institute of Education Sciences (R305A090307)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- propensity score, generalizability, target population, covariate balance, matching, subclassification, weighting, Positive Behavioral Interventions and Supports
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
