CoChat
Sign Up
Cited by 878Statistical Methods in Clinical TrialsMeta-analysis and systematic reviewsEthics in Clinical Research

A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results

Open access
Kennedy-Martin, Tessa, Curtis, Sarah, Faries, Douglas, Robinson, Susan, Johnston, Joseph·2015·Trials
DOI 10.1186/s13063-015-1023-4PMC PMC4632358
Open this paper in CoChat — free accountOpen PDF

Abstract

Randomized controlled trials (RCTs) are conducted under idealized and rigorously controlled conditions that may compromise their external validity. A literature review was conducted of published English language articles that reported the findings of studies assessing external validity by a comparison of the patient sample included in RCTs reporting on pharmaceutical interventions with patients from everyday clinical practice. The review focused on publications in the fields of cardiology, mental health, and oncology. A range of databases were interrogated (MEDLINE; EMBASE; Science Citation Index; Cochrane Methodology Register). Double-abstract review and data extraction were performed as per protocol specifications. Out of 5,456 de-duplicated abstracts, 52 studies met the inclusion criteria (cardiology, n = 20; mental health, n = 17; oncology, n = 15). Studies either performed an analysis of the baseline characteristics (demographic, socioeconomic, and clinical parameters) of RCT-enrolled patients compared with a real-world population, or assessed the proportion of real-world patients who would have been eligible for RCT inclusion following the application of RCT inclusion/exclusion criteria. Many of the included studies concluded that RCT samples are highly selected and have a lower risk profile than real-world populations, with the frequent exclusion of elderly patients and patients with co-morbidities. Calculation of ineligibility rates in individual studies showed that a high proportion of the general disease population was often excluded from trials. The majority of studies (n = 37 [71.2 %]) explicitly concluded that RCT samples were not broadly representative of real-world patients and that this may limit the external validity of the RCT. Authors made a number of recommendations to improve external validity. Findings from this review indicate that there is a need to improve the external validity of RCTs such that physicians treating patients in real-world settings have the appropriate evidence on which to base their clinical decisions. This goal could be achieved by trial design modification to include a more representative patient sample and by supplementing RCT evidence with data generated from observational studies. In general, a thoughtful approach to clinical evidence generation is required in which the trade-offs between internal and external validity are considered in a holistic and balanced manner.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13063-015-1023-4

Cites

78 of 78 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The absence of strict exclusion criteria and the location of clinics in large cities across the Netherlands enhance the external validity of the results.

    “No strict exclusion criteria are used and clinics are in large cities across the Netherlands, which enhances the external validity of our results [119, 120].”

    From Eye movement desensitization and reprocessing (EMDR) therapy or supportive counseling prior to exposure therapy in patients with panic disorder: study protocol for a multicenter randomized controlled trial (IMPROVE) (2023) · cites this paper as [120]

    The Discussion says restrictive inclusion and exclusion criteria can limit a trial’s external validity, supporting the rationale for avoiding strict criteria. The review does not assess the citing study’s criteria or show that clinics in large Dutch cities enhance external validity; its Results discuss study settings generally, not that claim. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • High risk of suicide, psychotic symptoms, or treatment-resistant depression are factors that have traditionally been used to exclude patients from clinical trials, leaving a gap in the knowledge of acute management in clinical settings.

    “In depression research, high risk of suicide, psychotic symptoms, or treatment-resistant depression are factors that have traditionally been used to exclude patients from clinical trials, leaving a gap in the knowledge of acute management in clinical settings (52, 53).”

    From Total Sleep Deprivation Followed by Bright Light Therapy as Rapid Relief for Depression: A Pragmatic Randomized Controlled Trial (2021) · cites this paper as [52]

    The Results (Mental health; page unavailable) identify suicide risk as a common reason for excluding patients from mental-health RCTs. The Results (Explicit factors; page unavailable) also report substantial ineligibility among patients with depression and raise concerns about applying trial results in clinical practice. Our copy does not identify psychotic symptoms or treatment-resistant depression as exclusion factors, or specifically discuss a gap in knowledge about acute management. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The emphasis in trials of pharmaceutical products is on internal validity and reducing bias, and extensive work has confirmed the importance of this.

    “The emphasis in trials of pharmaceutical products is on internal validity and reducing bias, and extensive work has confirmed the importance of this. 54”

    From Evaluating Digital Health Interventions (2016) · cites this paper as [54]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. [68]Velasco E Salkind NJ Inclusion criteria Encyclopedia of research, volume 1 2010 Thousand Oaks SAGE Publications, Inc 589 91
  2. [69]Fletcher R Fletcher SW Fletcher GS Fletcher R Fletcher SW Fletcher GS Chapter 9, Treatment Clinical epidemiology: the essentials 2014 5 Baltimore Wolters Kluwer 132 52

Cited by

Sorted by citations

Related

Authors
Kennedy-Martin, T., Curtis, S., Faries, D., Robinson, S., & Johnston, J.
Affiliations
  • Eli Lilly (United States) (US)
  • Kennedy Martin Health Outcomes (United Kingdom) (GB)
ORCID
Venue
Trials
Publisher
Springer Science and Business Media LLC
Published
2015-11-03
Volume
16
Issue
1
Pages
495-495
ISSN
1745-6215 (ISSN-L)
DOI
10.1186/s13063-015-1023-4
PMC
PMC4632358
OpenAlex
W2120729896
Cited by
878
References
78
Citations by year
2026: 75 · 2025: 106 · 2024: 102 · 2023: 114 · 2022: 97 · 2021: 118 · 2020: 103 · 2019: 61 · 2018: 38 · 2017: 45 · 2016: 18 · 2015: 1
FWCI
30.25
Type
journal article
Language
en
Open access
Licence
Locations
  • Trials (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
Eli Lilly and Company
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
Keywords
generalizability, sample representativeness, randomized controlled trials, inclusion and exclusion criteria, real-world patients, patient selection bias, comorbidities, internal validity, mental health, pharmacological interventions
MeSH terms
Cardiology, Data Interpretation, Statistical, Humans, Medical Oncology, Mental Health Services, Research Design, Reproducibility of Results, Randomized Controlled Trials as Topic, Treatment Outcome, Evidence-Based Medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review