Clinical Trial Generalizability Assessment in the Big Data Era: A Review
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Clinical studies, especially randomized, controlled trials, are essential for generating evidence for clinical practice. However, generalizability is a long‐standing concern when applying trial results to real‐world patients. Generalizability assessment is thus important, nevertheless, not consistently practiced. We performed a systematic review to understand the practice of generalizability assessment. We identified 187 relevant articles and systematically organized these studies in a taxonomy with three dimensions: (i) data availability (i.e., before or after trial (a priori vs. a posteriori generalizability)); (ii) result outputs (i.e., score vs. nonscore); and (iii) populations of interest. We further reported disease areas, underrepresented subgroups, and types of data used to profile target populations. We observed an increasing trend of generalizability assessments, but < 30% of studies reported positive generalizability results. As a priori generalizability can be assessed using only study design information (primarily eligibility criteria), it gives investigators a golden opportunity to adjust the study design before the trial starts. Nevertheless, < 40% of the studies in our review assessed a priori generalizability. With the wide adoption of electronic health records systems, rich real‐world patient databases are increasingly available for generalizability assessment; however, informatics tools are lacking to support the adoption of generalizability assessment practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kennedy-Martin, T., Curtis, S., Faries, D., Robinson, S., & Johnston, J. (2015) · Trials · doi:10.1186/s13063-015-1023-4 · Cited by 878
Representation of the Elderly, Women, and Minorities in Heart Failure Clinical Trials
Heiat, A., Gross, C. P., & Krumholz, H. M. (2002) · Archives of Internal Medicine · doi:10.1001/archinte.162.15.1682 · Cited by 723 · cited 2×
“For example, Heiat et al.(52) analyzed the enrolled patients in 59 heart failure clinical trials and found that older adults and female and nonwhite patients were underrepresented in these trials.”
BackgroundAssessing the generalizability of randomized trial results to target populations
Stuart, E. A., Bradshaw, C. P., & Leaf, P. J. (2015) · Prevention Science · doi:10.1007/s11121-014-0513-z · Cited by 296
Finding What Works in Health Care: Standards for Systematic Reviews
Committee on Standards for Systematic Reviews of Comparative Effectiveness Research, Board on Health Care Services, & Institute of Medicine (2011) · doi:10.17226/13059 · Cited by 289
Wisniewski, S. R., Rush, A. J., Nierenberg, A. A., Gaynes, B. N., Warden, D., Luther, J. F., McGrath, P. J., Lavori, P. W., Thase, M. E., Fava, M., et al. (2009) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2008.08071027 · Cited by 223 · cited 2×
“In general, we excluded studies that merely compared the patients in different arms of a single trial; nevertheless, this study was included as it used broad inclusion and minimal exclusion criteria to evaluate whether phase III clinical trials can recruit representative depressed outpatients.(23) Table 3 shows the nu…”
ResultCriteria2Query: a natural language interface to clinical databases for cohort definition
Yuan, C., Ryan, P. B., Ta, C., Guo, Y., Li, Z., Hardin, J., Makadia, R., Jin, P., Shang, N., Kang, T., et al. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy178 · Cited by 170
Self-citationHoertel, N., Le Strat, Y., Blanco, C., Lavaud, P., & Dubertret, C. (2012) · Depression and Anxiety · doi:10.1002/da.21937 · Cited by 61
Costa, L. J., Hari, P. N., & Kumar, S. K. (2016) · Leukemia & Lymphoma · doi:10.3109/10428194.2016.1170828 · Cited by 57
Research works printed but not linked: 58
- 10.1136/bmj.e1004 · DOI
- 10.1056/NEJMp1615879 · DOI · Paper page
- 10.1097/AOG.0b013e3182a9ca67 · DOI · Paper page
- 10.1016/j.jpsychores.2013.12.007 · DOI
- 10.1377/hlthaff.2014.0122 · DOI
- 10.1136/bmj.333.7563.346 · DOI · Paper page
- 10.4088/JCP.12m07935 · DOI · Paper page
- 10.1177/1740774515571917 · DOI
- 10.1186/1745-6215-13-237 · DOI
- 10.1002/mpr.1609 · DOI
- 10.1016/j.jad.2017.01.008 · DOI
- 10.1136/bmj.b2535 · DOI · Paper page
- 10.11613/BM.2012.031 · DOI · Paper page
- 10.1093/aje/kwx164 · DOI · Paper page
- 10.1093/aje/kwq084 · DOI · Paper page
- 10.1093/aje/kwq084 · DOI · Paper page
- 10.1177/0193841X16660663 · DOI
- 10.1016/j.tips.2015.08.007 · DOI
- 10.4338/ACI-2013-12-RA-0105 · DOI
- 10.1016/j.jbi.2016.01.007 · DOI
- 10.1016/j.jbi.2016.09.003 · DOI
- 10.1111/nyas.13195 · DOI
- 10.1016/j.jacc.2015.10.037 · DOI · Paper page
- 10.1093/cid/civ1222 · DOI
- 10.1111/ajad.12714 · DOI
- 10.1046/j.1468-1293.2000.00019.x · DOI
- 10.1016/j.ijmedinf.2016.12.008 · DOI
- 10.1111/j.1467-985X.2010.00673.x · DOI · Paper page
- 10.1111/add.13327 · DOI
- 10.1007/s12032-018-1167-7 · DOI
- 10.1016/j.cardfail.2017.04.006 · DOI
- 10.1007/s00586-016-4850-4 · DOI
- Wittbrodt E.T., 2018, Generalizability of glucagon‐like peptide‐1 receptor agonist cardiovascular outcome trials enrollment criteria to the US type 2 diabetes population, Am. J. Manag. Care, 24, S146
- 10.1016/j.jclinepi.2017.10.016 · DOI
- 10.1186/1471-2482-12-2 · DOI
- 10.1016/S0165-5876(01)00504-3 · DOI
- 10.1186/1745-6215-15-310 · DOI
- 10.3390/jcm7080206 · DOI
- 10.1111/bjd.17849 · DOI
- 10.1016/S0304-3959(99)00074-3 · DOI · Paper page
- 10.1007/s00381-016-3328-3 · doi:10.1007/s00381-016-3328-3
- 10.1093/aje/kwx287 · DOI
- 10.1001/archinte.167.1.68 · DOI · Paper page
- 10.1016/j.eururo.2018.10.054 · doi:10.1016/j.eururo.2018.10.054
- Botsis T., 2010, Secondary use of EHR: data quality issues and informatics opportunities, Summit Transl. Bioinform., 2010, 1
- 10.1016/j.jbi.2014.01.009 · DOI · Paper page
- 10.1001/jama.297.11.1233 · DOI · Paper page
- 10.1200/JCO.2007.15.3726 · DOI · Paper page
- 10.2217/cer.15.27 · DOI
- 10.1001/jama.2019.17016 · doi:10.1001/jama.2019.17016
- 10.1002/pds.4718 · DOI
- Zhang H., 2018, Computable eligibility criteria through ontology‐driven data access: a case study of hepatitis C virus trials, AMIA Ann. Symp. Proc., 2018, 1601
- 10.1038/nrd.2017.25 · DOI
- Lane K., 2018, African American screening and enrollment in (clot lysis: evaluating accelerated resolution of intraventricular hemorrhage III) CLEAR III, Clin. Res. (Alex.), 32
- 10.1016/j.drugalcdep.2017.02.024 · DOI
- 10.1177/2047487318815967 · doi:10.1177/2047487318815967
- 10.18553/jmcp.2017.16367 · DOI
- 10.1370/afm.516 · DOI · Paper page
Cited by
Marques, L., Costa, B., Pereira, M., Silva, A., Santos, J., Saldanha, L., Silva, I., Magalhães, P., Schmidt, S., & Vale, N. (2024) · Pharmaceutics · doi:10.3390/pharmaceutics16030332 · Cited by 390
A scoping review of fair machine learning techniques when using real-world data
Huang, Y., Guo, J., Chen, W.-H., Lin, H.-Y., Tang, H., Wang, F., Xu, H., & Bian, J. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104622 · Cited by 61
Self-citationQuantifying representativeness in randomized clinical trials using machine learning fairness metrics
Qi, M., Cahan, O., Foreman, M. A., Gruen, D. M., Das, A. K., & Bennett, K. P. (2021) · JAMIA Open · doi:10.1093/jamiaopen/ooab077 · Cited by 27
Hasty generalizations and generics in medical research: A systematic review
Peters, U., Sherling, H. R., & Chin-Yee, B. (2024) · PLOS ONE · doi:10.1371/journal.pone.0306749 · Cited by 11
Metadata
- Authors
- He, Z., Tang, X., Yang, X., Guo, Y., George, T. J., Charness, N., Quan Hem, K. B., Hogan, W., & Bian, J.
- Venue
- Clinical and Translational Science
- Publisher
- Wiley
- Published
- 2020-04-10
- Volume
- 13
- Issue
- 4
- Pages
- 675-684
- ISSN
- 1752-8054 (ISSN-L)
- 1752-8062
- OpenAlex
- W3006162251
- Cited by
- 130
- References
- 8
- FWCI
- 8.81
- Type
- journal article
- Language
- en
- Open access
- gold
- Funders
- National Institute on Aging
- National Institutes of Health (R21AG061431;, UL1TR001427)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.