Generalizing Study Results: A Potential Outcomes Perspective
Open accessAbstract
Great care is taken in epidemiologic studies to ensure the internal validity of causal effect estimates; however, external validity has received considerably less attention. When the study sample is not a random sample of the target population, the sample average treatment effect, even if internally valid, cannot usually be expected to equal the average treatment effect in the target population. The utility of an effect estimate for planning purposes and decision making will depend on the degree of departure from the true causal effect in the target population due to problems with both internal and external validity. Herein, we review concepts from recent literature on generalizability, one facet of external validity, using the potential outcomes framework. Identification conditions sufficient for external validity closely parallel identification conditions for internal validity, namely conditional exchangeability; positivity; the same distributions of the versions of treatment; no interference; and no measurement error. We also require correct model specification. Under these conditions, we discuss how a version of direct standardization (the g-formula, adjustment formula, or transport formula) or inverse probability weighting can be used to generalize a causal effect from a study sample to a well-defined target population, and demonstrate their application in an illustrative example.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
External validity may be threatened by the limited generalizability of study populations to relevant populations or differences between the study environment and real-world interventions.
“External validity may be threatened by the limited generalizability of study populations to relevant populations or differences between the study environment and real-world interventions [18, 19].”
From Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review (2018) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Sample Selection Bias as a Specification Error
Heckman, J. J. (1979) · Econometrica · doi:10.2307/1912352 · Cited by 28,779 · see citing works
Marginal Structural Models and Causal Inference in Epidemiology
Robins, J. M., Hernán, M. Á., & Brumback, B. (2000) · Epidemiology · doi:10.1097/00001648-200009000-00011 · Cited by 5,581 · see citing works
Statistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · see citing works
Causal Diagrams for Epidemiologic Research
Greenland, S., Pearl, J., & Robins, J. M. (1999) · Epidemiology · doi:10.1097/00001648-199901000-00008 · Cited by 3,728 · see citing works
Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available: Table 1.
Hernán, M. A., & Robins, J. M. (2016) · American Journal of Epidemiology · doi:10.1093/aje/kwv254 · Cited by 3,115 · see citing works
Air Pollution and Hospital Admissions for Heart Disease in Eight U.S. Counties
Schwartz, J. D. (1999) · Epidemiology · doi:10.1097/00001648-199901000-00005 · Cited by 3,044 · see citing works
A structural approach to selection bias
Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004) · Epidemiology · doi:10.1097/01.ede.0000135174.63482.43 · Cited by 2,844 · see citing works
External validity of randomised controlled trials: “To whom do the results of this trial apply?”
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(04)17670-8 · Cited by 2,544 · see citing works
Robins, J. (1986) · Mathematical Modelling · doi:10.1016/0270-0255(86)90088-6 · Cited by 2,451 · see citing works
Maternal Pesticide Exposure from Multiple Sources and Selected Congenital Anomalies
Shaw, G. M., Wasserman, C. R., O’Malley, C. D., Nelson, V. E., & Jackson, R. J. (1999) · Epidemiology · doi:10.1097/00001648-199901000-00011 · Cited by 1,783 · see citing works
Hernan, M. A. (2002) · American Journal of Epidemiology · doi:10.1093/aje/155.2.176 · Cited by 1,445 · see citing works
Estimating causal effects from epidemiological data
Hernán, M. Á., & Robins, J. M. (2006) · Journal of Epidemiology & Community Health · doi:10.1136/jech.2004.029496 · Cited by 1,029 · see citing works
Why Representativeness Should Be Avoided
Rothman, K. J., Gallacher, J. E., & Hatch, E. E. (2013) · International Journal of Epidemiology · doi:10.1093/ije/dys223 · Cited by 861 · see citing works
Observational Studies Analyzed Like Randomized Experiments
Hernán, M. Á., Alonso, Á., Logan, R. W., Grodstein, F., Michels, K. B., Willett, W. C., Manson, J. E., & Robins, J. M. (2008) · Epidemiology · doi:10.1097/ede.0b013e3181875e61 · Cited by 822 · see citing works
Toward Causal Inference With Interference
Hudgens, M. G., & Halloran, M. E. (2008) · Journal of the American Statistical Association · doi:10.1198/016214508000000292 · Cited by 768 · see citing works
Self-citationCausal inference and the data-fusion problem
Bareinboim, E., & Pearl, J. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1510507113 · Cited by 621 · see citing works
Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial.
Cole, S., & Stuart, E. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq084 · Cited by 535 · see citing works
Self-citationThe Use of Propensity Scores to Assess the Generalizability of Results from Randomized Trials
Stuart, E. A., Cole, S. R., Bradshaw, C. P., & Leaf, P. J. (2011) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2010.00673.x · Cited by 524 · see citing works
Self-citationExternal Validity: From Do-Calculus to Transportability Across Populations
Pearl, J., & Bareinboim, E. (2014) · Statistical Science · doi:10.1214/14-sts486 · Cited by 455 · see citing works
A New Criterion for Confounder Selection
VanderWeele, T. J., & Shpitser, I. (2011) · Biometrics · doi:10.1111/j.1541-0420.2011.01619.x · Cited by 434 · see citing works
On causal inference in the presence of interference
Tchetgen, E. J. T., & VanderWeele, T. J. (2010) · Statistical Methods in Medical Research · doi:10.1177/0962280210386779 · Cited by 420 · see citing works
The Consistency Statement in Causal Inference
Cole, S. R., & Frangakis, C. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818ef366 · Cited by 413 · see citing works
Self-citationExternal validity: From do-calculus to transportability across populations
Pearl, J., & Bareinboim, E. (2013) · Cited by 398 · see citing works
Maldonado, G., & Greenland, S. (2002) · International Journal of Epidemiology · doi:10.1093/intjepid/31.2.422 · Cited by 396 · see citing works
Does obesity shorten life? The importance of well-defined interventions to answer causal questions
Hernán, M. A., & Taubman, S. L. (2008) · International Journal of Obesity · doi:10.1038/ijo.2008.82 · Cited by 384 · cited 2× · see citing works
“28 (4) One person’s exposure does not affect another person’s outcome (no interference).”
Concerning the Consistency Assumption in Causal Inference
VanderWeele, T. J. (2009) · Epidemiology · doi:10.1097/ede.0b013e3181bd5638 · Cited by 368 · see citing works
Robins, J. M., Sued, M., Lei-Gomez, Q., Rotnitzky, A. G., Ciencias, F., & Naturales, E. (2010) · Cited by 346 · see citing works
Robins, J. M., Sued, M., Lei-Gomez, Q., & Rotnitzky, A. G. (2007) · Statistical Science · doi:10.1214/07-sts227d · Cited by 314 · see citing works
Population-based Cohort Studies
Szklo, M. (1998) · Epidemiologic Reviews · doi:10.1093/oxfordjournals.epirev.a017974 · Cited by 299 · see citing works
Compound Treatments and Transportability of Causal Inference
Hernán, M. A., & VanderWeele, T. J. (2011) · Epidemiology · doi:10.1097/ede.0b013e3182109296 · Cited by 256 · see citing works
How to assess the external validity of therapeutic trials: a conceptual approach
Dekkers, O. M., Elm, E. V., Algra, A., Romijn, J. A., & Vandenbroucke, J. P. (2010) · International Journal of Epidemiology · doi:10.1093/ije/dyp174 · Cited by 219 · see citing works
A General Algorithm for Deciding Transportability of Experimental Results
Bareinboim, E., & Pearl, J. (2013) · Journal of Causal Inference · doi:10.1515/jci-2012-0004 · Cited by 209 · see citing works
Perils and potentials of self‐selected entry to epidemiological studies and surveys
Keiding, N., & Louis, T. A. (2016) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12136 · Cited by 199 · see citing works
Predicting the efficacy of future training programs using past experiences at other locations
Joseph Hotz, V., Imbens, G. W., & Mortimer, J. H. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.009 · Cited by 190 · see citing works
Moore, K. L., & van der Laan, M. J. (2008) · Statistics in Medicine · doi:10.1002/sim.3445 · Cited by 185 · see citing works
Improving Generalizations From Experiments Using Propensity Score Subclassification
Tipton, E. (2012) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998612441947 · Cited by 170 · see citing works
Using causal diagrams to guide analysis in missing data problems
Daniel, R. M., Kenward, M. G., Cousens, S. N., & De Stavola, B. L. (2012) · Statistical Methods in Medical Research · doi:10.1177/0962280210394469 · Cited by 155 · cited 1× · see citing works
“17,35 This example highlights the importance of balancing study design decisions to maximize both internal and external validity; internal and external bias both exist on a continuous scales (as degrees rather than as dichotomies) and relatively minor violations of internal validity may be…”
On the Consistency Rule in Causal Inference
Pearl, J. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181f5d3fd · Cited by 152 · see citing works
"Toward a Clearer Definition of Confounding" Revisited With Directed Acyclic Graphs
Howards, P. P., Schisterman, E. F., Poole, C., Kaufman, J. S., & Weinberg, C. R. (2012) · American Journal of Epidemiology · doi:10.1093/aje/kws127 · Cited by 143 · see citing works
Gandhi, M., Ameli, N., Bacchetti, P., Sharp, G. B., French, A. L., Young, M., Gange, S. J., Anastos, K., Holman, S., Levine, A. M., et al. (2005) · AIDS · doi:10.1097/01.aids.0000189866.67182.f7 · Cited by 94 · see citing works
Leveraging prognostic baseline variables to gain precision in randomized trials
Colantuoni, E., & Rosenblum, M. (2015) · Statistics in Medicine · doi:10.1002/sim.6507 · Cited by 89 · see citing works
Conditional independence in sample selection models
Angrist, J. D. (1997) · Economics Letters · doi:10.1016/s0165-1765(97)00022-0 · Cited by 74 · see citing works
Edwards, J. K., Cole, S. R., & Westreich, D. J. (2015) · International Journal of Epidemiology · doi:10.1093/ije/dyu272 · Cited by 71 · see citing works
Self-citationGeneralizing Experimental Findings
Pearl, J. (2015) · Journal of Causal Inference · doi:10.1515/jci-2015-0025 · Cited by 70 · see citing works
Greenhouse, J. B., Kaizar, E. E., Kelleher, K. J., Seltman, H. J., & Gardner, W. P. (2008) · Statistics in Medicine · doi:10.1002/sim.3218 · Cited by 64 · see citing works
Commentary: External validity of results of randomized trials: disentangling a complex concept
Rothwell, P. M. (2009) · International Journal of Epidemiology · doi:10.1093/ije/dyp305 · Cited by 64 · see citing works
Compound Treatments, Transportability, and the Structural Causal Model
Petersen, M. L. (2011) · Epidemiology · doi:10.1097/ede.0b013e3182126127 · Cited by 45 · see citing works
Invited Commentary: Every Good Randomization Deserves Observation
Westreich, D. J., & Edwards, J. K. (2015) · American Journal of Epidemiology · doi:10.1093/aje/kwv200 · Cited by 18 · see citing works
Self-citationRepresentativeness is not helpful in studying heterogeneity of effects across subgroups
Rothman, K. J., Hatch, E. E., & Gallacher, J. (2014) · International Journal of Epidemiology · doi:10.1093/ije/dyt265 · Cited by 15 · see citing works
Hoggatt, K. J., & Greenland, S. (2013) · Epidemiology · doi:10.1097/ede.0000000000000023 · Cited by 11 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [3]Bareinboim E Pearl J Transportability of causal effects: Completeness results AAAI Conference on Artificial Intelligence North America 2012
- [4]Bareinboim E Tian J Pearl J Recovering from Selection Bias in Causal and Statistical Inference AAAI Citeseer 2014 2410 2416
- [14]Pearl J Causality Cambridge university press 2009
Cited by
Sorted by citations
Stamatakis, E., Owen, K. B., Shepherd, L., Drayton, B., Hamer, M., & Bauman, A. E. (2021) · Epidemiology · doi:10.1097/ede.0000000000001316 · Cited by 192 · see citing works
UK Biobank, big data, and the consequences of non-representativeness
Keyes, K. M., & Westreich, D. (2019) · The Lancet · doi:10.1016/s0140-6736(18)33067-8 · Cited by 187 · see citing works
Self-citationTarget Validity and the Hierarchy of Study Designs
Westreich, D., Edwards, J. K., Lesko, C. R., Cole, S. R., & Stuart, E. A. (2019) · American Journal of Epidemiology · doi:10.1093/aje/kwy228 · Cited by 165 · see citing works
Self-citationHaber, N., Smith, E. R., Moscoe, E., Andrews, K., Audy, R., Bell, W., Brennan, A. T., Breskin, A., Kane, J. C., Karra, M., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0196346 · Cited by 97 · see citing works
Hughes, R. A., Davies, N. M., Davey Smith, G., & Tilling, K. (2019) · Epidemiology · doi:10.1097/ede.0000000000000972 · Cited by 89 · see citing works
Approaches to addressing missing values, measurement error, and confounding in epidemiologic studies
van Smeden, M., Penning de Vries, B. B. L., Nab, L., & Groenwold, R. H. H. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.11.006 · Cited by 32 · see citing works
- Authors
- Lesko, C. R., Buchanan, A. L., Westreich, D., Edwards, J. K., Hudgens, M. G., & Cole, S. R.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- Corresponding author
- Lesko, Catherine R.
- ORCID
- Venue
- Epidemiology
- Published
- 2017-07
- Volume
- 28
- Issue
- 4
- Pages
- 553-561
- ISSN
- 1044-3983 (ISSN-L)
- PMC
- PMC5466356
- OpenAlex
- W2598995014
- Cited by
- 269
- References
- 50
- Citations by year
- 2026: 19 · 2025: 34 · 2024: 24 · 2023: 30 · 2022: 35 · 2021: 38 · 2020: 29 · 2019: 32 · 2018: 20 · 2017: 6 · 2016: 1
- FWCI
- 4.38
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Journal of Media Literacy Education (submittedVersion, cc-by)
- Funders
- National Institutes of Health (1dp2hd084070-01)
- National Institute on Alcohol Abuse and Alcoholism (U01AA020793)
- National Heart, Lung, and Blood Institute (U01HL121812, U01HL146194)
- National Institute of Allergy and Infectious Diseases (R01AI100654, K01AI125087, U01AI103390, R24AI067039, P30AI050410)
- Eunice Kennedy Shriver National Institute of Child Health and Human Development (DP2-HD-08-4070)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- generalizability, internal validity, target population, average treatment effect, potential outcomes framework, positivity, model specification, direct standardization, g-formula, inverse probability weighting, causal effect estimation
- MeSH terms
- Epidemiologic Methods, Female, Humans, Male, Predictive Value of Tests, Statistics as Topic, United States, Reproducibility of Results, Bias, Causality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
