The DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index.
Target Validity and the Hierarchy of Study Designs
Open accessAbstract
In recent years, increasing attention has been paid to problems of external validity, specifically to methodological approaches for both quantitative generalizability and transportability of study results. However, most approaches to these issues have considered external validity separately from internal validity. Here we argue that considering either internal or external validity in isolation may be problematic. Further, we argue that a joint measure of the validity of an effect estimate with respect to a specific population of interest may be more useful: We call this proposed measure target validity. In this work, we introduce and formally define target bias as the total difference between the true causal effect in the target population and the estimated causal effect in the study sample, and target validity as target bias = 0. We illustrate this measure with a series of examples and show how this measure may help us to think more clearly about comparisons between experimental and nonexperimental research results. Specifically, we show that even perfect internal validity does not ensure that a causal effect will be unbiased in a specific target population.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“after discovering a causal mechanism, researchers still need to identify conditions that modify its effects. This requires testing for differences in effects between people, calling for more diverse samples than those initially tested [3,7].”
From Hasty generalizations and generics in medical research: A systematic review (2024) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2002) · Cited by 13,211
Causal Diagrams for Epidemiologic Research
Greenland, S., Pearl, J., & Robins, J. M. (1999) · Epidemiology · doi:10.1097/00001648-199901000-00008 · Cited by 3,728
Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs
Concato, J., Shah, N., & Horwitz, R. I. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422507 · Cited by 3,563
Air Pollution and Hospital Admissions for Heart Disease in Eight U.S. Counties
Schwartz, J. D. (1999) · Epidemiology · doi:10.1097/00001648-199901000-00005 · Cited by 3,045
Understanding and misunderstanding randomized controlled trials
Deaton, A., & Cartwright, N. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2017.12.005 · Cited by 1,841
Maternal Pesticide Exposure from Multiple Sources and Selected Congenital Anomalies
Shaw, G. M., Wasserman, C. R., O’Malley, C. D., Nelson, V. E., & Jackson, R. J. (1999) · Epidemiology · doi:10.1097/00001648-199901000-00011 · Cited by 1,783
8. Misunderstandings Between Experimentalists and Observationalists About Causal Inference
Imai, K., King, G., & Stuart, E. A. (2008) · Yale University Press eBooks · doi:10.1111/j.1467-985x.2007.00527.x · Cited by 925
Self-citationWhy Representativeness Should Be Avoided
Rothman, K. J., Gallacher, J. E., & Hatch, E. E. (2013) · International Journal of Epidemiology · doi:10.1093/ije/dys223 · Cited by 861
Holm, J., Nowicka, P., Farpour‐Lambert, N. J., O’Malley, G., Hassapidou, M. N., Weiss, R., & Baker, J. L. (2014) · Obesity Facts · doi:10.1159/000365773 · Cited by 596
Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial.
Cole, S., & Stuart, E. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq084 · Cited by 535
Self-citation
Flagged first, then by citations
1 of the 34 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Bareinboim, 2014, Advances in Neural Information Processing Systems 26., 136
- Bareinboim, 2015, Advances in Neural Information Processing Systems 27., 280
- Pearl, 2011, 2011 JSM Proceedings: Papers Presented at the Joint Statistical Meetings, Miami Beach, Florida, July 30–August 4, 2011, and Other ASA-Sponsored Conferences, 157
- Bareinboim, 2014, Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence and the Twenty-Sixth Innovative Applications of Artificial Intelligence Conference., 2410
Cited by
Sorted by citations
Items 101–165 of 165
Eshun‐Wilson, I., Ford, N. P., Mody, A., Beres, L. K., Schwartz, S., Baral, S., & Geng, E. (2023) · PLoS Medicine · doi:10.1371/journal.pmed.1004168 · Cited by 4
Cao, Z., Cho, Y., & Li, F. (2024) · Pharmaceutical Statistics · doi:10.1002/pst.2354 · Cited by 3
Lijing Lin, A., Poppe, K., Wood, A., Martin, G. P., Peek, N. B., & Sperrin, M. (2024) · Frontiers in Epidemiology · doi:10.3389/fepid.2024.1326306 · Cited by 3
On the Comparative Analysis of Average Treatment Effects Estimation via Data Combination
Wu, P., Luo, S., & Geng, Z. (2024) · Journal of the American Statistical Association · doi:10.1080/01621459.2024.2435656 · Cited by 10
Recruitment Challenges for Low‐Risk Health System Intervention Trials in Older Adults: A Case Study
Anzuoni, K., Field, T. S., Mazor, K. M., Zhou, Y., Garber, L., Kapoor, A., & Gurwitz, J. H. (2020) · Journal of the American Geriatrics Society · doi:10.1111/jgs.16696 · Cited by 3
A neural network approach to identifying adolescent adjustment.
Nair, J., Nair, S. S., Kashani, J. H., Reid, J. C., & Rao, V. G. (2001) · PubMed · Cited by 3
Saltzman, R., Zucker, I. J., Campbell, K., Gandhi, D. A., Otiono, K., Weber, A., Masterson, T. A., & Ramasamy, R. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4150356 · Cited by 3
Methods in Causal Inference Part 1: Causal Diagrams and ConfoundingPreprint
Bulbulia, J. A. (2023) · doi:10.31234/osf.io/b23k7 · Cited by 3
Ensuring target trials and target estimands are on target for intended use populations
Bell, K., Medcalf, E., & Stanaway, F. (2025) · International Journal of Epidemiology · doi:10.1093/ije/dyaf138 · Cited by 3
When data generate populations
Chioléro, A., & Carmeli, C. (2023) · International Journal of Epidemiology · doi:10.1093/ije/dyad166 · Cited by 2
Forsgren, E., Ekberg, S., Smedby, K. E. E., Nylund, P., Sjöblom, T., Flogegård, M., Sjöström, S., Hultdin, M., Hallén, K., Hellström, M., et al. (2024) · British Journal of Haematology · doi:10.1111/bjh.19732 · Cited by 2
About Research: Clinical Versus Statistical Significance
Wouk, K., Kenny, K. S., & Chetwynd, E. M. (2025) · Journal of Human Lactation · doi:10.1177/08903344251320587 · Cited by 2
Keyes, K. M., Gimbrone, C., Rutherford, C. G., Zhang, Y., Choi, K. W., Smith, L. H., Greenland, P., Smoller, J. W., & Argos, M. (2026) · Psychological Medicine · doi:10.1017/s0033291725102420 · Cited by 2
Calibrating Chromatography: How Tswett Broke the Experimenters’ Regress
Livengood, J., & Edwards, A. (2019) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/axz037 · Cited by 2
Akentarah, R., Amankwa, I. A., & Awuma, W. (2024) · International Journal of Accounting and Economics Studies · doi:10.14419/q2hwnq26 · Cited by 2
Causal Inference, Causal Effect Estimation, and Systematic Error
Westreich, D. J. (2019) · doi:10.1093/oso/9780190665760.003.0004 · Cited by 2
Self-citationThe Promise, and Challenges, of Methods to Enhance the External Validity of Randomized Trial Results
Stuart, E. A., & Lesko, C. R. (2020) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.1992 · Cited by 2
Self-citationA Clinician's Guide to Conducting Research on Causal Effects
Lyons, V. H., Robinson, J. R., Mills, B. M., Killien, E. Y., & Mooney, S. J. (2022) · Journal of Surgical Research · doi:10.1016/j.jss.2022.04.059 · Cited by 2
Brunner, M., Preckel, F., Goetz, T., Lüdtke, O., & Keller, L. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/6pmhr · Cited by 2
A selection of challenges in addressing selection bias
Howards, P. P., & Johnson, C. Y. (2024) · Paediatric and Perinatal Epidemiology · doi:10.1111/ppe.13102 · Cited by 2
Unmeasured confounding with and without randomization
Htoo, P., & Stürmer, T. (2021) · Pragmatic Randomized Clinical Trials · doi:10.1016/b978-0-12-817663-4.00016-7
Strategic Change Management and Performance of National Hospital Insurance Fund in Kenya
Issack, S. A., & Rugami, J. M. (2022) · International Journal of Business Management Entrepreneurship and Innovation · doi:10.35942/jbmed.v4i3.295 · Cited by 1
Gonçalves, B. P., & Suzuki, E. (2025) · Journal of Epidemiology · doi:10.2188/jea.je20250409 · Cited by 1
Terefe, B., Mengistu, A. K., Gedefaw, A. E., Taye, E. A., Kebede, F. B., Sani, J., Baykemagn, N. D., Yehuala, T. Z., & Worku, A. (2025) · BMC Infectious Diseases · doi:10.1186/s12879-025-12032-9 · Cited by 1
Moving From Representativeness Toward Transportability in an Era of Digital and Big Data
Chioléro, A. (2026) · International Journal of Public Health · doi:10.3389/ijph.2026.1609306 · Cited by 1
Leahy, T. P., Keenan, C., Robertshaw, E., McStravick, M., Barbier, S., Duffield, S. J., & Kent, S. (2026) · Value in Health Regional Issues · doi:10.1016/j.vhri.2026.101651 · Cited by 1
Christiaens, A., Wang, A., McAllister, D., McManus, R. J., & Sheppard, J. P. (2026) · Age and Ageing · doi:10.1093/ageing/afag192 · Cited by 1
Lee, H., & Lamb, S. E. (2021) · Journal of physiotherapy · doi:10.1016/j.jphys.2021.08.007 · Cited by 1
Woźniak-Jęchorek, B. (2023) · Ekonomista · doi:10.52335/ekon/161838 · Cited by 1
Who Is Represented in Neurology Data Sources?
Mayeda, E. R., & Hayes‐Larson, E. (2023) · Neurology · doi:10.1212/wnl.0000000000207652 · Cited by 1
Lu, G., & Phawitpiriyakliti, C. (2023) · ICCMETS Conference · doi:10.62788/rr879ln · Cited by 1
The Authors Respond to “Issues With the Consecutive-Pregnancies Approach”
Yu, Y., Bodnar, L. M., Brooks, M. M., Himes, K. P., & Naimi, A. I. (2019) · American Journal of Epidemiology · doi:10.1093/aje/kwz080
Westreich, D. J. (2019) · doi:10.1093/oso/9780190665760.003.0007
Self-citationWestreich, D. J. (2019) · Oxford University Press eBooks · doi:10.1093/oso/9780190665760.003.0010
Self-citationApproaches to mitigate bias in the design and analysis of pRCTs
Raman, S. R., Landi, S. N., Lund, J. L., & Funk, M. J. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-817663-4.00021-0
Westreich, D. J. (2019) · doi:10.1093/oso/9780190665760.003.0006
Self-citationLijing Lin, A., Poppe, K., Wood, A., Martin, G. P., Peek, N. B., & Sperrin, M. (2022) · Research Square · doi:10.21203/rs.3.rs-1824359/v1
Lijing Lin, A., Poppe, K., Wood, A., Martin, G. P., Peek, N. B., & Sperrin, M. (2022) · Research Square · doi:10.21203/rs.3.rs-1824359/v2
Lesko, C. R., Fox, M. P., & Edwards, J. K. (2023) · American Journal of Epidemiology · doi:10.1093/aje/kwad008
Self-citationWarniasih, K., Wijayanti, P. S., Syahruzah, J. K., & Rianto (2023) · PROCEEDING OF THE 7TH INTERNATIONAL CONFERENCE OF SCIENCE, TECHNOLOGY, AND INTERDISCIPLINARY RESEARCH (IC-STAR 2021) · doi:10.1063/5.0105514
Integrating and Generalizing Causal Estimates
Dayal, V., & Murugesan, A. (2023) · Springer eBooks · doi:10.1007/978-981-99-3905-3_12
Banegas, J. R., Laguna, J. M., Caballero, F. F., Guallar‐Castillón, P., López‐García, E., Graciani, A., Cabanas‐Sánchez, V., Damián, J., Ortolá, R., Royo‐Bordonada, M. Á., et al. (2024) · Epidemiologic Methods · doi:10.1515/em-2023-0015
Clinical research reporting paradigms may incompletely describe participant identities
Enache, O. M., Rosas, L. G., & Rose, S. (2024) · American Journal of Epidemiology · doi:10.1093/aje/kwae291
Diakoniatyöntekijöiden näkemyksiä diakoniatyön vaikuttavuudesta
Pajunen, K., Soininen, J., & Mättö, T. (2024) · Hallinnon Tutkimus · doi:10.37450/ht.143888
Effects of Injury Registry Data on Policymaking, Hospitalizations, and Mortality: Systematic Review
Medeiros-de-Souza, A. C., Lopes, L. E. S., Bandeira, T. F. G. D. S., Correia, L. R., de Sá, N. N. B., & de Oliveira, B. Z. (2025) · JMIR Public Health and Surveillance · doi:10.2196/67115
Pain, Attitudes, and (In)Action: Divergent Legacies of Herbicidal Warfare in VietnamPreprint
Ito, G., Tran, D., Sawada, H., Khan, G. D., & Yoshida, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5397061
Brunner, M., Preckel, F., Goetz, T., Lüdtke, O., & Keller, L. (2025) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/6pmhr_v2
Nilsson, A., Bonander, C., Strömberg, U., & Björk, J. (2020) · Figshare · doi:10.6084/m9.figshare.13467449
Arredondo-Nontol, M. R., Arredondo-Nontol, R., Reto, N., & Carrasco, A. G. M. (2026) · PLoS ONE · doi:10.1371/journal.pone.0343815
Causal Inference from Heterogeneous Data with Missing Data : Application to Critical Care Management
Mayer, I. (2021) · theses.fr (ABES) · doi:10.70675/2f432d6cz11f8z4d0bza530zbc10fbe668bc
Tinnitus : psychoacoustics and development of a bitherapy
Daoud, E. (2022) · theses.fr (ABES) · doi:10.70675/e0421484zc98cz40d8z9e79z289e82803dbb
Xia, Q., Manandhar-Sasaki, P., Bertolino, D., Torian, L. V., Edelstein, Z. R., & Zheng, Y. (2026) · Social Science & Medicine · doi:10.1016/j.socscimed.2026.119341
Sunzeri, F. (2026) · Springer eBooks · doi:10.1007/978-3-032-14465-2_26
Midlife hypertension and late‐life cognition: weighting the LifeAfter90 study
Colbeth, H., Roscoe, J. N., Corrada, M. M., Meunier, C. C., George, K. M., Glymour, M. M., Gilsanz, P., & Whitmer, R. A. (2026) · Alzheimer s & Dementia · doi:10.1002/alz.71544
Revisiting fracture risk with pregabalin: selection bias and the challenge of causal inference
Peng, X.-Y., & Chen, H.-L. (2026) · Pain · doi:10.1097/j.pain.0000000000003982
Li, L., Liu, Y., Wu, Z., Liao, Y., Li, S., Chow, S., 廖彬池, Zhang, D., Xie, Z., & Wu, Y. (2026) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.70172
Wilson, E., Jong, F. D., & Peters, L. (2025) · International Journal Of Pharmaceutical Research And Allied Sciences · doi:10.51847/pz7auemasj
Vera-Ponce, V. J., & Ballena-Caicedo, J. (2026) · Epidemiologia · doi:10.3390/epidemiologia7050119
Wu, Z., Zhou, Y., Li, L., Liu, Y., Liao, Y., Chen, K., Jia, H., Meng, W., Deng, J., 廖彬池, et al. (2026) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.70194
Tiansong, Z. H. A. O., Zhang, W., Zhang, H., Ding, B., & Zhang, H. (2026) · International Journal of Pavement Engineering · doi:10.1080/10298436.2026.2745033
Gillen, D. L. (2026) · Learning Health Systems · doi:10.1002/lrh2.70128
DeMessie, B., Essibayi, M. A., Salim, H. A., Karandish, A., Azzam, A. Y., Khatri, D., Haranhalli, N., Baker, A., Zampolin, R. L., Brook, A., et al. (2026) · Neurosurgery · doi:10.1227/neu.0000000000004203
Metadata
- Authors
- Westreich, D., Edwards, J. K., Lesko, C. R., Cole, S. R., & Stuart, E. A.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- Johns Hopkins University (US)
- Corresponding author
- Westreich, Daniel
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-10-09
- Volume
- 188
- Issue
- 2
- Pages
- 438-443
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- PMC
- PMC6357801
- OpenAlex
- W2900846215
- Cited by
- 165
- References
- 34
- Citations by year
- 2026: 15 · 2025: 15 · 2024: 28 · 2023: 26 · 2022: 24 · 2021: 21 · 2020: 23 · 2019: 12
- FWCI
- 11.62
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- American Journal of Epidemiology (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Carolina Digital Repository (University of North Carolina at Chapel Hill) (submittedVersion)
- UNC Libraries
- Funders
- Eunice Kennedy Shriver National Institute of Child Health and Human Development
- National Institutes of Health (DP2-HD084070)
- National Institute of Allergy and Infectious Diseases (R01 AI100654)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- target bias, generalizability, internal validity, causal effect estimation, target population, transportability, causal inference
- MeSH terms
- Epidemiologic Methods, Humans, Reproducibility of Results, Bias, Causality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review