The DOI matches its Crossref record; 49 of the 50 references it cites were found in a registry or an index.
Comparison of Evidence of Treatment Effects in Randomized and Nonrandomized Studies
Abstract
CONTEXT: There is substantial debate about whether the results of nonrandomized studies are consistent with the results of randomized controlled trials on the same topic. OBJECTIVES: To compare results of randomized and nonrandomized studies that evaluated medical interventions and to examine characteristics that may explain discrepancies between randomized and nonrandomized studies. DATA SOURCES: MEDLINE (1966-March 2000), the Cochrane Library (Issue 3, 2000), and major journals were searched. STUDY SELECTION: Forty-five diverse topics were identified for which both randomized trials (n = 240) and nonrandomized studies (n = 168) had been performed and had been considered in meta-analyses of binary outcomes. DATA EXTRACTION: Data on events per patient in each study arm and design and characteristics of each study considered in each meta-analysis were extracted and synthesized separately for randomized and nonrandomized studies. DATA SYNTHESIS: Very good correlation was observed between the summary odds ratios of randomized and nonrandomized studies (r = 0.75; P<.001); however, nonrandomized studies tended to show larger treatment effects (28 vs 11; P =.009). Between-study heterogeneity was frequent among randomized trials alone (23%) and very frequent among nonrandomized studies alone (41%). The summary results of the 2 types of designs differed beyond chance in 7 cases (16%). Discrepancies beyond chance were less common when only prospective studies were considered (8%). Occasional differences in sample size and timing of publication were also noted between discrepant randomized and nonrandomized studies. In 28 cases (62%), the natural logarithm of the odds ratio differed by at least 50%, and in 15 cases (33%), the odds ratio varied at least 2-fold between nonrandomized studies and randomized trials. CONCLUSIONS: Despite good correlation between randomized trials and nonrandomized studies-in particular, prospective studies-discrepancies beyond chance do occur and differences in estimated magnitude of treatment effect are very common.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The healthy subject effect is a particularly pernicious form of bias in which subjects with generally healthier lifestyles adopt a particular treatment in a nonrandomized study, resulting in unobserved confounding [268].”
From Personality and Longevity: Knowns, Unknowns, and Implications for Public Health and Personalized Medicine (2011) · cites this paper as [268]
Citations
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Statistical Aspects of the Analysis of Data From Retrospective Studies of Disease
(1959) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/22.4.719 · Cited by 14,689
Hulley, S. (1998) · JAMA · doi:10.1001/jama.280.7.605 · Cited by 6,216
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634
(1994) · New England Journal of Medicine · doi:10.1056/nejm199404143301501 · Cited by 4,737
Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs
Concato, J., Shah, N., & Horwitz, R. I. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422507 · Cited by 3,563
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 1×
“Publication bias and a time lag for negative studies may also be operating, regardless of study design.(56,57) Quality is also important to consider, and in this regard, sometimes nonrandomized or randomized studies,(58) or both may have important quality defects.”
BackgroundClinical Trials: A Practical Approach.
Hill, I. D., & Pocock, S. (1984) · Biometrics · doi:10.2307/2531186 · Cited by 2,918
A comparison of observational studies and randomized, controlled trials.
Benson, K., & Hartz, A. J. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422506 · Cited by 2,347
Efficacy of BCG Vaccine in the Prevention of Tuberculosis
Colditz, G. A. (1994) · JAMA · doi:10.1001/jama.1994.03510330076038 · Cited by 2,021
Cited by
Sorted by citations
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Grading quality of evidence and strength of recommendations
(2004) · BMJ · doi:10.1136/bmj.328.7454.1490 · Cited by 8,477
Siegel, J. D., Rhinehart, E., Jackson, M. M., & Chiarello, L. A. (2007) · American Journal of Infection Control · doi:10.1016/j.ajic.2007.10.007 · Cited by 3,506
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-2 · Cited by 2,755
Metadata
- Authors
- Ioannidis, J. P. A.
- Affiliations
- Tufts Medical Center (US)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- JAMA
- Publisher
- American Medical Association (AMA)
- Published
- 2001-08-15
- Volume
- 286
- Issue
- 7
- Pages
- 821
- ISSN
- 0002-9955 (ISSN-L)
- 0098-7484
- OpenAlex
- W2084917404
- Cited by
- 919
- References
- 50
- Citations by year
- 2026: 11 · 2025: 18 · 2024: 39 · 2023: 24 · 2022: 46 · 2021: 59 · 2020: 53 · 2019: 39 · 2018: 31 · 2017: 41 · 2016: 39 · 2015: 40 · 2014: 42 · 2013: 45 · 2012: 39
- FWCI
- 58.25
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- JAMA (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- randomized controlled trials, non-randomized studies, observational studies, treatment effect estimates, odds ratio, study heterogeneity, prospective study, medical interventions, publication timing
- MeSH terms
- Clinical Trials as Topic, Data Interpretation, Statistical, Humans, Meta-Analysis as Topic, Randomized Controlled Trials as Topic, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
