Quantifying representativeness in randomized clinical trials using machine learning fairness metrics
Open accessPublished version of a preprint
The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Abstract
Objective: We help identify subpopulations underrepresented in randomized clinical trials (RCTs) cohorts with respect to national, community-based or health system target populations by formulating population representativeness of RCTs as a machine learning (ML) fairness problem, deriving new representation metrics, and deploying them in easy-to-understand interactive visualization tools. Materials and Methods: We represent RCT cohort enrollment as random binary classification fairness problems, and then show how ML fairness metrics based on enrollment fraction can be efficiently calculated using easily computed rates of subpopulations in RCT cohorts and target populations. We propose standardized versions of these metrics and deploy them in an interactive tool to analyze 3 RCTs with respect to type 2 diabetes and hypertension target populations in the National Health and Nutrition Examination Survey. Results: We demonstrate how the proposed metrics and associated statistics enable users to rapidly examine representativeness of all subpopulations in the RCT defined by a set of categorical traits (eg, gender, race, ethnicity, smoking status, and blood pressure) with respect to target populations. Discussion: The normalized metrics provide an intuitive standardized scale for evaluating representation across subgroups, which may have vastly different enrollment fractions and rates in RCT study cohorts. The metrics are beneficial complements to other approaches (eg, enrollment fractions) used to identify generalizability and health equity of RCTs. Conclusion: By quantifying the gaps between RCT and target populations, the proposed methods can support generalizability evaluation of existing RCT cohorts. The interactive visualization tool can be readily applied to identified underrepresented subgroups with respect to any desired source or target populations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Ahmed, A., Jorna, T., & Bhandari, S. (2016) · Nephron · doi:10.1159/000447068 · Cited by 11,717
A Randomized Trial of Intensive versus Standard Blood-Pressure Control
The SPRINT Research Group (2015) · New England Journal of Medicine · doi:10.1056/nejmoa1511939 · Cited by 6,325
The ALLHAT Officers and Coordinators for the ALLHAT Collaborative Research Group (2002) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.288.23.2981 · Cited by 5,664
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
Analysis of Complex Survey Samples
Lumley, T. (2004) · Journal of Statistical Software · doi:10.18637/jss.v009.i08 · Cited by 2,425
Participation in Cancer Clinical Trials: Race-, Sex-, and Age-Based Disparities
Murthy, V. H., Krumholz, H. M., & Gross, C. P. (2004) · JAMA · doi:10.1001/jama.291.22.2720 · Cited by 2,078
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581
U.S. Disparities in Health: Descriptions, Causes, and Mechanisms
Adler, N. E., & Rehkopf, D. H. (2008) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.29.020907.090852 · Cited by 1,212 · cited 2×
“Our contributions The contributions discussed in this paper are (1) formulating the problem of representativeness evaluation in RCTs as a comparison between a truly random sampling function in a target population and the actual sampling observed in the clinical trial cohort; (2) deriving new metrics for representative…”
BackgroundMethod(2017) · PubMed · Cited by 961
Cited by
Artificial intelligence applied to clinical trials: opportunities and challenges
Askin, S., Burkhalter, D., Calado, G., & El Dakrouni, S. (2023) · Health and Technology · doi:10.1007/s12553-023-00738-2 · Cited by 285
The Problem of Fairness in Synthetic Healthcare Data
Bhanot, K., Qi, M., Erickson, J. S., Guyon, I., & Bennett, K. P. (2021) · Entropy · doi:10.3390/e23091165 · Cited by 91
Self-citationHomeyer, A., Geißler, C., Schwen, L. O., Zakrzewski, F., Evans, T., Strohmenger, K., Westphal, M., Bülow, R. D., Kargl, M., Karjauv, A., et al. (2022) · Modern Pathology · doi:10.1038/s41379-022-01147-y · Cited by 95
An Ontology for Fairness Metrics
Franklin, J. S., Bhanot, K., Ghalwash, M. F., Bennett, K. P., McCusker, J. P., & McGuinness, D. L. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534137 · Cited by 55
Self-citationQi, M., Santos, H., da Silva, P. P., McGuinness, D. L., & Bennett, K. P. (2023) · PLoS ONE · doi:10.1371/journal.pone.0290692 · Cited by 51
Self-citation
Metadata
- Authors
- Qi, M., Cahan, O., Foreman, M. A., Gruen, D. M., Das, A. K., & Bennett, K. P.
- Affiliations
- Rensselaer Polytechnic Institute (US)
- IBM Research - Cambridge (United States) (US)
- Corresponding author
- Bennett, Kristin P
- Venue
- JAMIA Open
- Publisher
- Oxford University Press (OUP)
- Published
- 2021-07-01
- Volume
- 4
- Issue
- 3
- Pages
- ooab077-ooab077
- ISSN
- 2574-2531 (ISSN-L)
- PMC
- PMC8460438
- OpenAlex
- W3202137438
- Cited by
- 27
- References
- 57
- Citations by year
- 2026: 4 · 2025: 2 · 2024: 7 · 2023: 8 · 2022: 5 · 2021: 1
- FWCI
- 4.55
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- JAMIA Open (publishedVersion, cc-by-nc)
- medRxiv (acceptedVersion)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- IBM Research AI Horizons Network
- United Health Foundation
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- representativeness, randomized controlled trials, generalizability, health equity, interactive visualization, type 2 diabetes, hypertension, NHANES, target population
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Quantifying representativeness in randomized clinical trials using machine learning fairness metrics. https://app.cochat.ai/papers/10.1093/jamiaopen/ooab077
- The paper
- Qi, M., Cahan, O., Foreman, M. A., Gruen, D. M., Das, A. K., & Bennett, K. P. (2021). Quantifying representativeness in randomized clinical trials using machine learning fairness metrics. JAMIA Open. https://doi.org/10.1093/jamiaopen/ooab077
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.