CoChat

Quantifying representativeness in randomized clinical trials using machine learning fairness metrics

Open access

Published version of a preprint

The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.

Abstract

Objective: We help identify subpopulations underrepresented in randomized clinical trials (RCTs) cohorts with respect to national, community-based or health system target populations by formulating population representativeness of RCTs as a machine learning (ML) fairness problem, deriving new representation metrics, and deploying them in easy-to-understand interactive visualization tools. Materials and Methods: We represent RCT cohort enrollment as random binary classification fairness problems, and then show how ML fairness metrics based on enrollment fraction can be efficiently calculated using easily computed rates of subpopulations in RCT cohorts and target populations. We propose standardized versions of these metrics and deploy them in an interactive tool to analyze 3 RCTs with respect to type 2 diabetes and hypertension target populations in the National Health and Nutrition Examination Survey. Results: We demonstrate how the proposed metrics and associated statistics enable users to rapidly examine representativeness of all subpopulations in the RCT defined by a set of categorical traits (eg, gender, race, ethnicity, smoking status, and blood pressure) with respect to target populations. Discussion: The normalized metrics provide an intuitive standardized scale for evaluating representation across subgroups, which may have vastly different enrollment fractions and rates in RCT study cohorts. The metrics are beneficial complements to other approaches (eg, enrollment fractions) used to identify generalizability and health equity of RCTs. Conclusion: By quantifying the gaps between RCT and target populations, the proposed methods can support generalizability evaluation of existing RCT cohorts. The interactive visualization tool can be readily applied to identified underrepresented subgroups with respect to any desired source or target populations.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamiaopen/ooab077

Cites

58 of 58 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. The causal models of truly randomized clinical trials and biased randomized clinical trials. X represents the subject covariates; Y is the sampling of a subject to a trial; T indicates the treatment; and Z is the outcome. The black arrows represent causal dependencies between variables. A. In the causal model for truly randomized clinical trials, no dependency should exist between X and Y. Thus, the observed probability of outcome Z given the treatment is a good estimate of whether the treatment causes the outcome. B. In the causal model for biased randomized clinical trials, an arrow exists between X and Y′, which indicates the dependence. Thus, invalid causal inferences may be estimated for treatment efficacy among some subpopulations and result in unfair and avoidable population health disparities.
Figure 1. The causal models of truly randomized clinical trials and biased randomized clinical trials. X represents the subject covariates; Y is the sampling of a subject to a trial; T indicates the treatment; and Z is the outcome. The black arrows represent causal dependencies between variables. A. In the causal model for truly randomized clinical trials, no dependency should exist between X and Y. Thus, the observed probability of outcome Z given the treatment is a good estimate of whether the treatment causes the outcome. B. In the causal model for biased randomized clinical trials, an arrow exists between X and Y′, which indicates the dependence. Thus, invalid causal inferences may be estimated for treatment efficacy among some subpopulations and result in unfair and avoidable population health disparities.CC BY-NC · PubMed Central
Figure 2. The shift of representativeness distribution of Log Disparity metric for different patient subgroups with type 2 diabetes in Action to Control Cardiovascular Risk in Diabetes. The green line corresponds to the ideal rate for the subgroup determined from National Health and Nutrition Examination Survey. The brown line indicates the rate actually observed. A. Log Disparity as function of observed rate for female subgroup. B. Log Disparity as function of observed rate for female non-Hispanic black subgroup.
Figure 2. The shift of representativeness distribution of Log Disparity metric for different patient subgroups with type 2 diabetes in Action to Control Cardiovascular Risk in Diabetes. The green line corresponds to the ideal rate for the subgroup determined from National Health and Nutrition Examination Survey. The brown line indicates the rate actually observed. A. Log Disparity as function of observed rate for female subgroup. B. Log Disparity as function of observed rate for female non-Hispanic black subgroup.CC BY-NC · PubMed Central
Figure 3. Representativeness of subgroups defined by a single protected attribute using Log Disparity for 3 real-world randomized clinical trials (RCTs). Subgroups are defined by demographic characteristics. Teal cells with a star indicate that no statistically significant difference between subgroups from the RCT and target population. Ages are in years. Abbreviations: C: cohort; TP: target population.
Figure 3. Representativeness of subgroups defined by a single protected attribute using Log Disparity for 3 real-world randomized clinical trials (RCTs). Subgroups are defined by demographic characteristics. Teal cells with a star indicate that no statistically significant difference between subgroups from the RCT and target population. Ages are in years. Abbreviations: C: cohort; TP: target population.CC BY-NC · PubMed Central
Figure 4. Representativeness of subgroups defined by a single protected attribute using Log Disparity for 3 real-world randomized clinical trials. Subgroups are defined by clinical characteristics. Systolic blood pressure unit = mm Hg; Fasting glucose unit = mmol/L.
Figure 4. Representativeness of subgroups defined by a single protected attribute using Log Disparity for 3 real-world randomized clinical trials. Subgroups are defined by clinical characteristics. Systolic blood pressure unit = mm Hg; Fasting glucose unit = mmol/L.CC BY-NC · PubMed Central
Figure 5. Representativeness results measured by Log Disparity. A. Color code of representativeness levels. B. Representativeness of Action to Control Cardiovascular Risk in Diabetes randomized clinical trial (RCT) subgroups in sunburst plot with inner to outer rings defined by demographic characteristics gender, age, race/ethnicity, and education level, respectively. C. Representativeness of Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial RCT subgroups in sunburst plot with inner to outer rings defined by risk factors systolic blood pressure, body mass index, and smoking status, respectively. D. Representativeness of Systolic Blood Pressure Intervention Trial RCT subgroups in sunburst plot with inner to outer rings defined by lab results total cholesterol and fasting glucose, respectively.
Figure 5. Representativeness results measured by Log Disparity. A. Color code of representativeness levels. B. Representativeness of Action to Control Cardiovascular Risk in Diabetes randomized clinical trial (RCT) subgroups in sunburst plot with inner to outer rings defined by demographic characteristics gender, age, race/ethnicity, and education level, respectively. C. Representativeness of Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial RCT subgroups in sunburst plot with inner to outer rings defined by risk factors systolic blood pressure, body mass index, and smoking status, respectively. D. Representativeness of Systolic Blood Pressure Intervention Trial RCT subgroups in sunburst plot with inner to outer rings defined by lab results total cholesterol and fasting glucose, respectively.CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

See all 34

Metadata

Authors
Qi, M., Cahan, O., Foreman, M. A., Gruen, D. M., Das, A. K., & Bennett, K. P.
Affiliations
  • Rensselaer Polytechnic Institute (US)
  • IBM Research - Cambridge (United States) (US)
Corresponding author
Bennett, Kristin P
Venue
JAMIA Open
Published
2021-07-01
Volume
4
Issue
3
Pages
ooab077-ooab077
ISSN
2574-2531 (ISSN-L)
OpenAlex
W3202137438
Cited by
27
References
57
Citations by year
2026: 4 · 2025: 2 · 2024: 7 · 2023: 8 · 2022: 5 · 2021: 1
FWCI
4.55
Type
journal article
Language
en
Open access
Locations
  • JAMIA Open (publishedVersion, cc-by-nc)
  • medRxiv (acceptedVersion)
  • Europe PMC (PubMed Central) (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
Funders
  • IBM Research AI Horizons Network
  • United Health Foundation
Topics
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords
representativeness, randomized controlled trials, generalizability, health equity, interactive visualization, type 2 diabetes, hypertension, NHANES, target population

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.