CoChat
Sign Up
Cited by 137Advanced Causal Inference TechniquesMeta-analysis and systematic reviewsStatistical Methods in Clinical Trials

The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.

Translating evidence into practice: eligibility criteria fail to eliminate clinically significant differences between real-world and study populations

Open access
Averitt, Amelia J., Weng, Chunhua, Ryan, Patrick, Perotte, Adler·2020·npj Digital Medicine
DOI 10.1038/s41746-020-0277-8PMC PMC7214444
Open this paper in CoChat — free accountOpen PDF

Abstract

Randomized controlled trials (RCTs) are regarded as the most reputable source of evidence. In some studies, factors beyond the intervention itself may contribute to the measured effect, an occurrence known as heterogeneity of treatment effect (HTE). If the RCT population differs from the real-world population on factors that induce HTE, the trials effect will not replicate. The RCTs eligibility criteria should identify the sub-population in which its evidence will replicate. However, the extent to which the eligibility criteria identify the appropriate population is unknown, which raises concerns for generalizability. We compared reported data from RCTs with real-world data from the electronic health records of a large, academic medical center that was curated according to RCT eligibility criteria. Our results show fundamental differences between the RCT population and our observational cohorts, which suggests that eligibility criteria may be insufficient for identifying the applicable real-world population in which RCT evidence will replicate.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41746-020-0277-8

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Summary of ∆RCT for baseline characteristics of Indication Only vs RCT and ∆RCT Indication + Eligibilty Criteria vs. RCT. a ACCOMPLISH trial b NCT01189890 trial (sitagliptin vs. glimepiride), c PROVE-IT trial d RENAAL trial. The shape of the marker corresponds to the data type. Circles (●) denote the standardized difference in the mean of continuous data. Pluses (+) denote the difference in percentage points of discrete data.
Fig. 1 Summary of ∆RCT for baseline characteristics of Indication Only vs RCT and ∆RCT Indication + Eligibilty Criteria vs. RCT. a ACCOMPLISH trial b NCT01189890 trial (sitagliptin vs. glimepiride), c PROVE-IT trial d RENAAL trial. The shape of the marker corresponds to the data type. Circles (●) denote the standardized difference in the mean of continuous data. Pluses (+) denote the difference in percentage points of discrete data.CC BY · PubMed Central
Fig. 2 Pipeline to operationalize eligibility criteria using OHDSI tools. The process begins by identifying the resources (e.g., an RCT protocol) that detail the eligibility criteria of a trial. Each criterion is then extracted and mapped to codified concepts in a controlled vocabulary. The concept is then mapped to the OHDSI common data model (CDM), which aggregates the same concepts from different vocabularies, into a single standardized concept. This concept is then refined to best define the eligibility criterion.
Fig. 2 Pipeline to operationalize eligibility criteria using OHDSI tools. The process begins by identifying the resources (e.g., an RCT protocol) that detail the eligibility criteria of a trial. Each criterion is then extracted and mapped to codified concepts in a controlled vocabulary. The concept is then mapped to the OHDSI common data model (CDM), which aggregates the same concepts from different vocabularies, into a single standardized concept. This concept is then refined to best define the eligibility criterion.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 41 references

Research works printed but not linked: 3

  1. Djulbegovic, B. & Guyatt G. in Users Guide to Medical Literature. 3rd edn. (McGraw-Hill Education, 1976).
  2. Velasco, E. in Encyclopedia of Research Design (ed. Salkind, N) (SAGE, Thousand Oaks Publications, 2010).
  3. Karanis, Y. B., Canta, F. A. B., Mitrofan, L., Mistry, H. & Anger, C. ‘Research’ vs ‘real world’ patients: the representativeness of clinical trial participants. Ann. Oncol. (2016) https://doi.org/10.1093/annonc/mdw392.51/2800468/Research-vs-real-world-patients-the · DOI

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Averitt, A. J., Weng, C., Ryan, P., & Perotte, A.
Affiliations
Columbia University (US)
Corresponding author
Perotte, Adler
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2020-05-11
Volume
3
Issue
1
Pages
67-67
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-020-0277-8
PMC
PMC7214444
OpenAlex
W3021472140
Cited by
137
References
41
Citations by year
2026: 14 · 2025: 20 · 2024: 24 · 2023: 31 · 2022: 21 · 2021: 23 · 2020: 4
FWCI
13.93
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine (R01LM009886-10, T15LM007079)
  • Janssen Pharmaceuticals (CU15-2317)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords
eligibility criteria, treatment effect heterogeneity, generalizability, real-world data, electronic health records, randomized controlled trials, observational cohort, evidence-to-practice translation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review