CoChat
Sign Up
PreprintCited by 35Meta-analysis and systematic reviewsStatistical Methods in Clinical TrialsAdvanced Causal Inference Techniques

The DOI matches its Crossref record; the references it cites are not checked yet.

The impact of blinding on trial results: A systematic review and meta-analysis

Open access
Pitre, Tyler, Kirsh, Sarah, Jassal, Tanvir, Anderson, Mason, Padoan, Adelia, Xiang, Alexander, Mah, Jasmine, Zeraatkar, Dena·2023·Cochrane Evidence Synthesis and Methods
DOI 10.1101/2023.03.05.23286821PMC PMC11795910

Preprint — published as 10.1002/cesm.12015

Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Blinding—the concealment of the arm to which participants have been randomized—is an important consideration for assessing risk of bias of randomized trials. A growing body of evidence has, however, yielded inconsistent results on whether trials without blinding produce biased findings. Objective: To conduct a systematic review and meta-analysis of the evidence addressing whether trials with and without blinding produce different results. Methods: We searched MEDLINE, EMBASE, Cochrane Reviews, JBI EBP, and Web of Science, from inception to May 2022, for studies comparing the results of trials with and without blinding. Pairs of reviewers, working independently and in duplicate, reviewed search results for eligible studies and extracted data. We pooled the results of studies comparing trials with and without blinding of patients, healthcare providers/investigators, and outcome assessors/adjudicators using frequentist random-effects meta-analyses. We coded study results such that a ratio of odds ratio (ROR) < 1 and difference in standardized mean difference (dSMD) < 0 indicate that trials without blinding overestimate treatment effects. Results: We identified 47 eligible studies. For dichotomous outcomes, we found low certainty evidence that trials without blinding of patients and healthcare providers, outcome assessors/adjudicators, and patients may slightly overestimate treatment effects. For continuous outcomes, we found low certainty evidence that trials without blinding of outcome assessors/adjudicators and patients may slightly overestimate treatment effects. Conclusion: Our systematic review and meta-analysis suggests that blinding may influence trial results in select situations—albeit the findings are of low certainty and the magnitude of effect is modest. In the absence of high certainty evidence suggesting that trials with and without blinding produce similar results, investigators should be cautious about interpreting the results of trials without blinding.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2023.03.05.23286821

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists haven’t been fetched yet

References 78 · Cited by 35

Research works printed but not linked: 82

  1. 10.1016/S0140-6736(02)07816-9 · DOI · Paper page
  2. 10.1136/bmj.321.7259.504 · DOI
  3. 10.1136/bmj.d5928 · DOI · Paper page
  4. 2010, Practical tips for surgical research: blinding: who, what, when, why, how?, Can J Surg, 53, 345
  5. 10.1037/1089-2680.2.2.175 · DOI · Paper page
  6. 1963, On the social psychology of the psychological experiment: the experimenter’s hypothesis as unintended determinant of experimental results, Am Sci, 51, 268
  7. 2018, The Clinical Trials Transformation Initiative: Methodology supporting the mission, Clin Trials, 15, 13, 10.1177/1740774518755054 · DOI
  8. 2020, Increasing operational and scientific efficiency in clinical trials, Br J Cancer, 123, 1207, 10.1038/s41416-020-0990-8 · DOI
  9. 10.1136/bmj.l6802 · DOI · Paper page
  10. 2018, No firm evidence that lack of blinding affects estimates of mortality in randomized clinical trials of intensive care interventions: a systematic review and meta-analysis, J Clin Epidemiol, 100, 71, 10.1016/j.jclinepi.2018.04.016 · DOI
  11. 2018, The impact of non-blinding in critical care medicine trials, J Crit Care, 48, 414, 10.1016/j.jcrc.2018.09.031 · DOI
  12. 2021, Association Between Lack of Blinding and Mortality Results in Critical Care Randomized Controlled Trials: A Meta-Epidemiological Study, Crit Care Med, 49, 1800, 10.1097/CCM.0000000000005065 · DOI
  13. 10.1093/aje/kwx344 · DOI · Paper page
  14. 10.1371/journal.pone.0159267 · DOI · Paper page
  15. 10.1136/ebmed-2017-110713 · DOI · Paper page
  16. Zeraatkar D , Pitre T , Diaz-Martinez JP , Chu D , Rochwerg B , Lamontagne F , et al. Effects of allocation concealment and blinding in trials addressing treatments for COVID-19: A methods study. medRxiv. 2022:2022.08.03.22278348. · DOI
  17. 10.1016/j.jclinepi.2016.04.005 · DOI · Paper page
  18. 10.1001/jama.285.15.2000 · DOI · Paper page
  19. 10.1002/14651858.Ed000142 · DOI · Paper page
  20. 10.1136/bmj.315.7109.629 · DOI · Paper page
  21. 10.1136/bmj.l6890 · DOI · Paper page
  22. 10.1136/bmj.39489.470347.AD · DOI · Paper page
  23. 10.1016/j.jclinepi.2017.05.006 · DOI · Paper page
  24. Zeng L , Brignardello-Petersen R , Hultcrantz M , Mustafa RA , Murad MH , Iorio A , et al. GRADE Guidance 34: update on rating imprecision using a minimally contextualized approach. J Clin Epidemiol. 2022. · DOI · Paper page
  25. 10.1016/j.jclinepi.2019.10.014 · DOI · Paper page
  26. 10.1016/S0197-2456(97)00097-4 · DOI
  27. 10.7326/0003-4819-135-11-200112040-00010 · DOI
  28. 2011, Do unblinded assessors bias muscle strength outcomes in randomized controlled trials of progressive resistance strength training in older adults?, American journal of physical medicine & rehabilitation / Association of Academic Physiatrists, 90, 190, 10.1097/PHM.0b013e31820174b3 · DOI
  29. 2009, The importance of allocation concealment and patient blinding in osteoarthritis trials: A meta-epidemiologic study, Arthritis Care and Research, 61, 1633, 10.1002/art.24894 · DOI
  30. 10.1001/jama.1995.03520290060030 · DOI
  31. 2021, A meta-epidemiological study of bias in randomized clinical trials of open and laparoscopic surgery, The British journal of surgery, 108, 477, 10.1093/bjs/znab035 · DOI
  32. 10.1097/PHM.0000000000000521 · DOI
  33. 2018, Some Cochrane risk-of-bias items are not important in osteoarthritis trials: a meta-epidemiological study based on Cochrane reviews, Journal of clinical epidemiology, 95, 128, 10.1016/j.jclinepi.2017.11.026 · DOI
  34. 2015, The effects of blinding on the outcomes of psychotherapy and pharmacotherapy for adult depression: A meta-analysis, European psychiatry : the journal of the Association of European Psychiatrists, 30, 685, 10.1016/j.eurpsy.2015.06.005 · DOI · Paper page
  35. 2021, Local Investigators Significantly Overestimate Overall Response Rates Compared to Blinded Independent Central Reviews in Phase 2 Oncology Trials, Journal of Clinical Pharmacology, 61, 810, 10.1002/jcph.1790 · DOI
  36. 2014, Bias due to lack of patient blinding in clinical trials, A systematic review of trials randomizing patients to blind and nonblind sub-studies. International Journal of Epidemiology, 43, 1272
  37. 10.1093/ejo/cjv049 · DOI
  38. 2021, Association Between Lack of Blinding and Mortality Results in Critical Care Randomized Controlled Trials: A Meta-Epidemiological Study, Critical care medicine, 49, 1800, 10.1097/CCM.0000000000005065 · DOI
  39. 2020, Impact of blinding on estimated treatment effects in randomised clinical trials: Meta-epidemiological study, The BMJ, 368, l6802
  40. 10.2106/JBJS.F.00683 · DOI
  41. 10.1007/s00423-019-01761-6 · DOI
  42. 2018, Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions, BMC medical research methodology, 18, 42, 10.1186/s12874-018-0491-0 · DOI
  43. 2018, No firm evidence that lack of blinding affects estimates of mortality in randomized clinical trials of intensive care interventions: a systematic review and meta-analysis, JOURNAL OF CLINICAL EPIDEMIOLOGY, 100, 71, 10.1016/j.jclinepi.2018.04.016 · DOI
  44. 2018, The impact of nonblinding in critical care medicine trials, Journal of Critical Care, 48, 414, 10.1016/j.jcrc.2018.09.031 · DOI
  45. 2018, Effectiveness and adequacy of blinding in the moderation of pain outcomes: Systematic review and meta-analyses of dry needling trials, PeerJ, 2018, e5318
  46. 2014, Do randomized clinical trials with inadequate blinding report enhanced placebo effects for intervention groups and nocebo effects for placebo groups?, Systematic Reviews, 3, 14, 10.1186/2046-4053-3-14 · DOI
  47. 10.1161/CIRCOUTCOMES.115.002533 · DOI
  48. 10.1093/ije/dyt270 · DOI
  49. 10.1136/bmj.e1119 · DOI
  50. 2013, Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors, CMAJ : Canadian Medical Association journal = journal de l’Association medicale canadienne, 185, E201, 10.1503/cmaj.120744 · DOI · Paper page
  51. 2016, Comparison of central adjudication of outcomes and onsite outcome assessment on treatment effect estimates, Cochrane Database of Systematic Reviews, 2016, MR000043
  52. Trone JC , Chapelle C , Ollier E , Bertoletti L , Cucherat M , Mismetti P , et al. Statistical controversies in clinical research: limitations of open-label studies assessing antiangiogenic therapies with regard to evaluation of vascular adverse drug events-a meta-analysis. Annals of Oncology. 2018;29(4). · DOI
  53. 10.1001/jama.287.22.2973 · DOI · Paper page
  54. 1999, Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses, Health Technol Assess, 3, i
  55. 10.1001/jama.282.11.1054 · DOI · Paper page
  56. 2012, Double blind vs, open design on treatment effect of new oral anticoagulants in atrial fibrillation: A meta-analysis. Fundamental and Clinical Pharmacology, 26, 17
  57. 10.7326/0003-4819-157-6-201209180-00537 · DOI · Paper page
  58. 2013, Single-center trials tend to provide larger treatment effects than multicenter trials: a systematic review, J Clin Epidemiol, 66, 1271, 10.1016/j.jclinepi.2013.05.016 · DOI
  59. 2014, Quantifying bias in randomized controlled trials in child health: a meta-epidemiological study, PLoS One, 9, e88008, 10.1371/journal.pone.0088008 · DOI
  60. 10.1093/ije/dyt074 · DOI
  61. 2014, A meta-epidemiological study to examine the association between bias and treatment effects in neonatal trials, Evid Based Child Health, 9, 1052, 10.1002/ebch.1985 · DOI
  62. 10.1093/ije/dym087 · DOI · Paper page
  63. 10.1176/appi.ajp.162.3.578 · DOI
  64. 2003, How important are comprehensive literature searches and the assessment of trial quality in systematic reviews?, Empirical study. Health Technol Assess, 7, 1, 10.3310/hta7010 · DOI · Paper page
  65. 10.1002/sim.2752 · DOI
  66. 10.1136/bmj.39465.451748.AD · DOI · Paper page
  67. 2017, Association Between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: The ROBES Meta-Epidemiologic Study, American Journal of Epidemiology, 187, 1113
  68. 10.1001/jama.2014.8166 · DOI · Paper page
  69. 2009, Empirical evidence of an association between internal validity and effect size in randomized controlled trials of low-back pain, Spine (Phila Pa 1976), 34, 1685, 10.1097/BRS.0b013e3181ab6a78 · DOI
  70. 10.1111/j.1600-051X.2008.01291.x · DOI
  71. 10.1111/j.1467-985X.2008.00548.x · DOI
  72. 10.1002/sim.1184 · DOI · Paper page
  73. 10.1136/bmj.l4898 · DOI · Paper page
  74. 10.1016/S0895-4356(02)00446-8 · DOI
  75. 10.1016/j.jclinepi.2011.04.015 · DOI · Paper page
  76. 10.1177/1740774506069153 · DOI
  77. 10.1001/jama.2009.1679 · DOI
  78. 10.1001/jama.2012.13444 · DOI · Paper page
  79. 10.1186/s12916-021-02020-6 · DOI
  80. 2014, Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials, Cochrane Database Syst Rev, 2014, Mr000034
  81. 10.1056/NEJM200006223422506 · DOI · Paper page
  82. 10.1186/s13063-018-2535-5 · DOI

Metadata

Authors
Pitre, T., Kirsh, S., Jassal, T., Anderson, M., Padoan, A., Xiang, A., Mah, J., & Zeraatkar, D.
Affiliations
  • McMaster University (CA)
  • Dalhousie University (CA)
Corresponding author
Zeraatkar, Dena
ORCID
Venue
Cochrane Evidence Synthesis and Methods
Publisher
openRxiv
Published
2023-03-06
Volume
1
Issue
4
Pages
e12015-e12015
ISSN
2832-9023 (ISSN-L)
DOI
10.1101/2023.03.05.23286821
PMC
PMC11795910
OpenAlex
W4381663760
Cited by
35
References
78
Citations by year
2026: 14 · 2025: 14 · 2024: 7
FWCI
2.48
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Cochrane Evidence Synthesis and Methods (publishedVersion, cc-by)
  • medRxiv (acceptedVersion)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Keywords
blinding, risk of bias, overestimation of treatment effects

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review