The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials
Open accessAbstract
OBJECTIVE: To summarise comparisons of randomised clinical trials and non-randomised clinical trials, trials with adequately concealed random allocation versus inadequately concealed random allocation, and high quality trials versus low quality trials where the effect of randomisation could not be separated from the effects of other methodological manoeuvres. DESIGN: Systematic review. SELECTION CRITERIA: Cohorts or meta-analyses of clinical trials that included an empirical assessment of the relation between randomisation and estimates of effect. DATA SOURCES: Cochrane Review Methodology Database, Medline, SciSearch, bibliographies, hand searching of journals, personal communication with methodologists, and the reference lists of relevant articles. MAIN OUTCOME MEASURES: Relation between randomisation and estimates of effect. RESULTS: Eleven studies that compared randomised controlled trials with non-randomised controlled trials (eight for evaluations of the same intervention and three across different interventions), two studies that compared trials with adequately concealed random allocation and inadequately concealed random allocation, and five studies that assessed the relation between quality scores and estimates of treatment effects, were identified. Failure to use random allocation and concealment of allocation were associated with relative increases in estimates of effects of 150% or more, relative decreases of up to 90%, inversion of the estimated effect and, in some cases, no difference. On average, failure to use randomisation or adequate concealment of allocation resulted in larger estimates of effect due to a poorer prognosis in non-randomly selected control groups compared with randomly selected control groups. CONCLUSIONS: Failure to use adequately concealed random allocation can distort the apparent effects of care in either direction, causing the effects to seem either larger or smaller than they really are. The size of these distortions can be as large as or larger than the size of the effects that are to be detected.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Shadish and colleagues did not find a significant correlation between the degree of clinical representativeness and the size of effects reported in studies of psychotherapy.
“Shadish and colleagues 89 did not find a significant correlation between the degree of clinical representativeness (i.e., RCTs vs. naturalistic studies) and the size of effects reported in studies of psychotherapy.”
From The Effectiveness of Long-Term Psychoanalytic Therapy: A Systematic Review of Empirical Studies (2009) · cites this paper as [89]
The Results and Tables 2 and 4 compare randomized with non-randomized trials and report some correlations between study quality and treatment effects, but not the claimed finding about psychotherapy. Searches for Shadish, clinical representativeness, psychotherapy, and naturalistic studies found no passage addressing that finding. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 45 · Cited by 770
Research works printed but not linked: 32
- Committee for Evaluating Medical Technologies in Clinical Use. Assessing medical technologies. Washington DC: National Academy Press, 1985:76-8.
- Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ 1996;312:1215-8.
- Weiss CH. Evaluation. Methods for studying programs and policies. 2nd ed. Upper Saddle River: Prentice Hall, 1998:229-33.
- Clarke M, Carling C, Oxman AD, eds. Cochrane Review Methodology Database. The Cochrane Library. Oxford: Update Software, 1998. Issue 3.
- Sacks H, Chalmers TC, Smith H Jr. Randomized versus historical controls for clinical trials. Am J Med 1982;72:233-40.
- Diehl LF, Perry DJ. A comparison of randomized concurrent control groups with matched historical control groups: are historical controls valid? J Clin Oncol 1986;4:1114-20.
- Reimold SC, Chalmers TC, Berlin JA, Antman EM. Assessment of the efficacy and safety of antiarrhythmic therapy for chronic atrial fibrillation: observations on the role of trial design and implications of drug related mortality. Am Heart J 1992;124:924-32.
- Recurrent Miscarriage Immunotherapy Trialists Group. Worldwide collaborative observational study and meta analysis on allogenic leukocyte immunotherapy for recurrent spontaneous abortion. Am J Reprod Immunol 1994;32:55-72.
- Watson A, Vandekerckhove P, Lilford R, Vail A, Brosens I, Hughes E. A meta-analysis of the therapeutic role of oil soluble contrast media at hys- terosalpingography: a surprising result? Fertil Steril 1994;61:470-7.
- Pyorala S, Huttunen NP, Uhari M. A review and meta-analysis of hormo- nal treatment of cryptorchidism. J Clin Endocrinol Metab 1995;80:2795-9.
- Carroll D, Tramer M, McQuay H, Nye B, Moore A. Randomization is important in studies with pain outcomes: systematic review of transcuta- neous electrical nerve stimulation in acute postoperative pain. Br J Anaesth 1996;77:798-803.
- Colditz GA, Miller JN, Mosteller F. How study design affects outcomes in comparisons of therapy. I: medical. Stat Med 1989;8:441-54.
- Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparisons of therapy. II: surgical. Stat Med 1989;8:455-66.
- Ottenbacher K. Impact of random assignment on study outcome: an empirical examination. Control Clin Trials 1992;13:50-61.
- Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treat- ment effects in controlled trials. JAMA 1995;273:408-12.
- Emerson JD, Burdick E, Hoaglin DC, Mosteller F, Chalmers TC. An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. Control Clin Trials 1990;11:339-52.
- Imperiale TF, McCullough AJ. Do corticosteroids reduce mortality from alcoholic hepatitis? A meta analysis of the randomized trials. Ann Intern Med 1990;113:299-307.
- Nurmohamed MT, Rosendaal FR, Buller HR, Dekker E, Hommes DW, Vandenbroucke JP, et al. Low molecular weight heparin versus standard heparin in general and orthopaedic surgery: a meta-analysis. Lancet 1992;340:152-6.
- Khan KS, Daya S, Jadad A. The importance of quality of primary studies in producing unbiased systematic reviews. Arch Intern Med 1996;156:661-6.
- Ortiz Z, Shea B, Suarez Almazor ME, Moher D, Wells GA, Tugwell P. The efficacy of folic acid and folinic acid in reducing methotrexate gastrointestinal toxicity in rheumatoid arthritis. A meta-analysis of rand- omized controlled trials. J Rheumatol 1998;25:36-43.
- Chalmers I. Assembling comparison groups to assess the effects of health care. J R Soc Med 1997;90:379-86.
- NHS Centre for Reviews and Dissemination. Database of abstracts of reviews of effectiveness. The Cochrane Library. Oxford: Update Software, 1998. Issue 3. 27 Cochrane Empirical Methodological Studies Methods Group. The Cochrane Library. Oxford: Update Software, 1998. Issue 3.
- Forgie MA, Wells PS, Laupacis A, Fergusson D. Preoperative autologous donation decreases allogeneic transfusion but increases exposure to all red blood cell transfusion: results of a meta-analysis. Arch Intern Med 1998;158:610-6.
- Colditz GA, Brewer TF, Berkey CS, Wilson ME, Burdick E, Fineberg HV, et al. Efficacy of BCG vaccine in the prevention of tuberculosis. Meta analysis of the published literature. JAMA 1994;271:698-702.
- Stieb D, Frayha HH, Oxman AD, Shannon HS, Hutchison BG, Crombie F. The effectiveness and usefulness of Haemophilus influenzae type b vaccines: a systematic overview (meta-analysis). Can Med Assoc J 1990;142:719-32.
- Stern JM, Simes RJ. Publication bias: evidence of delayed publication of clinical research projects. BMJ 1997;315:640-5.
- Counsell CE, Clarke MJ, Slattery J, Sandercock PAG. The miracle of DICE therapy for acute stroke: fact or fictional product of subgroup analysis? BMJ 1994;309:1677-81.
- Egger M, Davey SG, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997;315:629-34.
- Guyatt GH, Sackett DL, Cook DJ, for the Evidence-Based Working Group. Users' guides to the medical literature, II: how to use an article about therapy or prevention, A: are the results of the study valid? JAMA 1993;270:2598-601.
- Dans AL, Dans LF, Guyatt GH, Richardson S. Users' guides to the medi- cal literature: XIV. How to decide on the applicability of clinical trial results to your patient. JAMA 1998;279:545-9.
- Cochrane Methods Working Group on Applicability and Recommenda- tions. The Cochrane Library. Oxford: Update Software, 1998. Issue 3.
- Guyatt GH, Sackett DL, Cook DJ, for the Evidence-Based Working Group. Users' guides to the medical literature, II: how to use an article about therapy or prevention, B: what were the results and will they help me in caring for my patients? JAMA 1994;270:59-63.
- Authors
- Kunz, R., & Oxman, A. D.
- Affiliations
- Stiftung Charité (DE)
- Norwegian Institute of Public Health (NO)
- Corresponding author
- Oxman, A. D
- ORCID
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 1998-10-31
- Volume
- 317
- Issue
- 7167
- Pages
- 1185-1190
- ISSN
- 0007-1447 (ISSN-L)
- 0959-8138
- 1468-5833
- PMC
- PMC28700
- OpenAlex
- W1982489161
- Cited by
- 770
- References
- 45
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 14 · 2023: 14 · 2022: 6 · 2021: 9 · 2020: 14 · 2019: 16 · 2018: 18 · 2017: 12 · 2016: 26 · 2015: 18 · 2014: 35 · 2013: 25 · 2012: 33
- FWCI
- 46.77
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- BMJ (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- randomization, allocation concealment, randomized controlled trials, trial quality, bias in treatment effect estimates, control group selection, selection bias
- MeSH terms
- Clinical Trials as Topic, Data Collection, Quality Control, Random Allocation, Selection Bias, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
