CoChat
Sign Up
Cited by 919Meta-analysis and systematic reviewsAdvanced Causal Inference TechniquesStatistical Methods in Clinical Trials

Comparison of Evidence of Treatment Effects in Randomized and Nonrandomized Studies

Ioannidis, John P. A.·2001·JAMA
DOI 10.1001/jama.286.7.821
Open this paper in CoChat — free account

Abstract

CONTEXT: There is substantial debate about whether the results of nonrandomized studies are consistent with the results of randomized controlled trials on the same topic. OBJECTIVES: To compare results of randomized and nonrandomized studies that evaluated medical interventions and to examine characteristics that may explain discrepancies between randomized and nonrandomized studies. DATA SOURCES: MEDLINE (1966-March 2000), the Cochrane Library (Issue 3, 2000), and major journals were searched. STUDY SELECTION: Forty-five diverse topics were identified for which both randomized trials (n = 240) and nonrandomized studies (n = 168) had been performed and had been considered in meta-analyses of binary outcomes. DATA EXTRACTION: Data on events per patient in each study arm and design and characteristics of each study considered in each meta-analysis were extracted and synthesized separately for randomized and nonrandomized studies. DATA SYNTHESIS: Very good correlation was observed between the summary odds ratios of randomized and nonrandomized studies (r = 0.75; P<.001); however, nonrandomized studies tended to show larger treatment effects (28 vs 11; P =.009). Between-study heterogeneity was frequent among randomized trials alone (23%) and very frequent among nonrandomized studies alone (41%). The summary results of the 2 types of designs differed beyond chance in 7 cases (16%). Discrepancies beyond chance were less common when only prospective studies were considered (8%). Occasional differences in sample size and timing of publication were also noted between discrepant randomized and nonrandomized studies. In 28 cases (62%), the natural logarithm of the odds ratio differed by at least 50%, and in 15 cases (33%), the odds ratio varied at least 2-fold between nonrandomized studies and randomized trials. CONCLUSIONS: Despite good correlation between randomized trials and nonrandomized studies-in particular, prospective studies-discrepancies beyond chance do occur and differences in estimated magnitude of treatment effect are very common.

Integrity signals

Registry

Crossref · 10.1001/jama.286.7.821

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The healthy subject effect is a bias in which subjects with generally healthier lifestyles adopt a particular treatment in a nonrandomized study, resulting in unobserved confounding.

    “The healthy subject effect is a particularly pernicious form of bias in which subjects with generally healthier lifestyles adopt a particular treatment in a nonrandomized study, resulting in unobserved confounding [268].”

    From Personality and Longevity: Knowns, Unknowns, and Implications for Public Health and Personalized Medicine (2011) · cites this paper as [268]

Cites

Flagged first, then by citations

1 of the 50 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Ioannidis, J. P. A.
Affiliations
Tufts Medical Center (US)
Corresponding author
Ioannidis, John P. A.
ORCID
Ioannidis, John P. A.
Venue
JAMA
Publisher
American Medical Association (AMA)
Published
2001-08-15
Volume
286
Issue
7
Pages
821
ISSN
  • 0002-9955 (ISSN-L)
  • 0098-7484
DOI
10.1001/jama.286.7.821
OpenAlex
W2084917404
Cited by
919
References
50
Citations by year
2026: 11 · 2025: 18 · 2024: 39 · 2023: 24 · 2022: 46 · 2021: 59 · 2020: 53 · 2019: 39 · 2018: 31 · 2017: 41 · 2016: 39 · 2015: 40 · 2014: 42 · 2013: 45 · 2012: 39
FWCI
58.25
Type
journal article
Language
en
Open access
closed
Locations
  • JAMA (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords
randomized controlled trials, non-randomized studies, observational studies, treatment effect estimates, odds ratio, study heterogeneity, prospective study, medical interventions, publication timing
MeSH terms
Clinical Trials as Topic, Data Interpretation, Statistical, Humans, Meta-Analysis as Topic, Randomized Controlled Trials as Topic, Evidence-Based Medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review