The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors
Open accessAbstract
OBJECTIVE: To evaluate the impact of non-blinded outcome assessment on estimated treatment effects in randomised clinical trials with binary outcomes. DESIGN: Systematic review of trials with both blinded and non-blinded assessment of the same binary outcome. For each trial we calculated the ratio of the odds ratios--the odds ratio from non-blinded assessments relative to the corresponding odds ratio from blinded assessments. A ratio of odds ratios <1 indicated that non-blinded assessors generated more optimistic effect estimates than blinded assessors. We pooled the individual ratios of odds ratios with inverse variance random effects meta-analysis and explored reasons for variation in ratios of odds ratios with meta-regression. We also analysed rates of agreement between blinded and non-blinded assessors and calculated the number of patients needed to be reclassified to neutralise any bias. DATA SOURCES: PubMed, Embase, PsycINFO, CINAHL, Cochrane Central Register of Controlled Trials, HighWire Press, and Google Scholar. ELIGIBILITY CRITERIA FOR SELECTING STUDIES: Randomised clinical trials with blinded and non-blinded assessment of the same binary outcome. RESULTS: We included 21 trials in the main analysis (with 4391 patients); eight trials provided individual patient data. Outcomes in most trials were subjective--for example, qualitative assessment of the patient's function. The ratio of the odds ratios ranged from 0.02 to 14.4. The pooled ratio of odds ratios was 0.64 (95% confidence interval 0.43 to 0.96), indicating an average exaggeration of the non-blinded odds ratio by 36%. We found no significant association between low ratios of odds ratios and scores for outcome subjectivity (P=0.27); non-blinded assessor's overall involvement in the trial (P=0.60); or outcome vulnerability to non-blinded patients (P=0.52). Blinded and non-blinded assessors agreed in a median of 78% of assessments (interquartile range 64-90%) in the 12 trials with available data. The exaggeration of treatment effects associated with non-blinded assessors was induced by the misclassification of a median of 3% of the assessed patients per trial (1-7%). CONCLUSIONS: On average, non-blinded assessors of subjective binary outcomes generated substantially biased effect estimates in randomised clinical trials, exaggerating odds ratios by 36%. This bias was compatible with a high rate of agreement between blinded and non-blinded outcome assessors and driven by the misclassification of few patients.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Blinded outcome assessment is however a substantial threat to validity (Hróbjartsson 2012).”
From Honey as a topical treatment for wounds (2015) · cites this paper as Hróbjartsson 2012
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Wood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538
Brandstrup, B., Tønnesen, H., Beier-Holgersen, R., Hjortsø, E., Ørding, H., Lindorff-Larsen, K., Rasmussen, M. S., Lanng, C., Wallin, L., Iversen, L. H., et al. (2003) · Annals of Surgery · doi:10.1097/01.sla.0000094387.50865.23 · Cited by 1,585
Meltzer, H. Y. (2003) · Archives of General Psychiatry · doi:10.1001/archpsyc.60.1.82 · Cited by 1,393
GOVENDER, S., CSIMMA, C., GENANT, H. K., & VALENTIN-OPRAN, A. (2002) · The Journal of Bone and Joint Surgery-American Volume · doi:10.2106/00004623-200212000-00001 · Cited by 1,308
Experimenter Effects in Behavioral Research.
Titchener, J. L. (1967) · Archives of Internal Medicine · doi:10.1001/archinte.1967.00300050109027 · Cited by 1,194
Publication bias: evidence of delayed publication in a cohort study of clinical research projects
Stern, J. M., & Simes, R. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.640 · Cited by 818
Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials
Pildal, J., Hrobjartsson, A., Jorgensen, K., Hilden, J., Altman, D., & Gotzsche, P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym087 · Cited by 472
Self-citationJONES, A. L., BUCHOLZ, R. W., BOSSE, M. J., MIRZA, S. K., LYON, T. R., WEBB, L. X., POLLAK, A. N., GOLDEN, J. D., & VALENTIN-OPRAN, A. (2006) · The Journal of Bone and Joint Surgery-American Volume · doi:10.2106/00004623-200607000-00002 · Cited by 444
Cited by
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials
Chan, A.-W., Tetzlaff, J. M., Gotzsche, P. C., Altman, D. G., Mann, H., Berlin, J. A., Dickersin, K., Hrobjartsson, A., Schulz, K. F., Parulekar, W. R., et al. (2013) · BMJ · doi:10.1136/bmj.e7586 · Cited by 6,251
Jakobsen, J. C., Gluud, C., Wetterslev, J., & Winkel, P. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0442-1 · Cited by 2,747
Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000411 · Cited by 2,677
Fogel, D. B. (2018) · Contemporary Clinical Trials Communications · doi:10.1016/j.conctc.2018.08.001 · Cited by 1,035
Experimental design and analysis and their reporting: new guidance for publication in BJP
Curtis, M. J., Bond, R. A., Spina, D., Ahluwalia, A., Alexander, S. P. A., Giembycz, M. A., Gilchrist, A., Hoyer, D., Insel, P. A., Izzo, A. A., et al. (2015) · British Journal of Pharmacology · doi:10.1111/bph.12856 · Cited by 1,033
PRISMA for Abstracts: Reporting Systematic Reviews in Journal and Conference Abstracts
Beller, E. M., Glasziou, P. P., Altman, D. G., Hopewell, S., Bastian, H., Chalmers, I., Gøtzsche, P. C., Lasserson, T., Tovey, D., & for the PRISMA for Abstracts Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001419 · Cited by 706
Honey as a topical treatment for wounds
Jull, A. B., Cullum, N., Dumville, J. C., Westby, M. J., Deshpande, S., & Walker, N. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005083.pub4 · Cited by 653
Project AIM: Autism intervention meta-analysis for studies of young children.
Sandbank, M., Bottema-Beutel, K., Crowley, S., Cassidy, M., Dunham, K., Feldman, J. I., Crank, J., Albarran, S. A., Raj, S., Mahbub, P., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000215 · Cited by 606
Schwenk, E. S., Viscusi, E. R., Buvanendran, A., Hurley, R. W., Wasan, A. D., Narouze, S., Bhatia, A., Davis, F. N., Hooten, W. M., & Cohen, S. P. (2018) · Regional Anesthesia and Pain Medicine · doi:10.1097/aap.0000000000000806 · Cited by 584
Reporting animal research: Explanation and Elaboration for the ARRIVE guidelines 2019
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/703355 · Cited by 559
Lee, H., Cashin, A. G., Lamb, S. E., Hopewell, S., Vansteelandt, S., VanderWeele, T. J., MacKinnon, D. P., Mansell, G., Collins, G. S., Golub, R. M., et al. (2021) · JAMA · doi:10.1001/jama.2021.14075 · Cited by 548
Hróbjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Hilden, J., Boutron, I., Ravaud, P., & Brorson, S. (2013) · Canadian Medical Association Journal · doi:10.1503/cmaj.120744 · Cited by 527
“8 It may be tempting to use measures for inter-observer agreement (e.g., weighted κ , intraclass correlation coefficients) as surrogate markers for risk of observer bias.”
BackgroundMethodSelf-citationJauhar, S., McKenna, P. J., Radua, J., Fung, E., Salvador, R., & Laws, K. R. (2014) · British Journal of Psychiatry · doi:10.1192/bjp.bp.112.116285 · Cited by 517
Gordon, B. R., McDowell, C. P., Hallgren, M., Meyer, J. D., Lyons, M., & Herring, M. P. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.0572 · Cited by 498
Hróbjartsson, A., Emanuelsson, F., Skou Thomsen, A. S., Hilden, J., & Brorson, S. (2014) · International Journal of Epidemiology · doi:10.1093/ije/dyu115 · Cited by 462
Self-citationJefferson, T., Jones, M., Doshi, P., Spencer, E. A., Onakpoya, I., & Heneghan, C. J. (2014) · BMJ · doi:10.1136/bmj.g2545 · Cited by 447
Why clinical trial outcomes fail to translate into benefits for patients
Heneghan, C., Goldacre, B., & Mahtani, K. R. (2017) · Trials · doi:10.1186/s13063-017-1870-2 · Cited by 404
Jakobsen, J. C., Katakam, K. K., Schou, A., Hellmuth, S. G., Stallknecht, S. E., Leth-Møller, K., Iversen, M., Banke, M. B., Petersen, I. J., Klingenberg, S. L., et al. (2017) · BMC Psychiatry · doi:10.1186/s12888-016-1173-2 · Cited by 366
Cohen, S. P., Bhatia, A., Buvanendran, A., Schwenk, E. S., Wasan, A. D., Hurley, R. W., Viscusi, E. R., Narouze, S., Davis, F. N., Ritchie, E. C., et al. (2018) · Regional Anesthesia and Pain Medicine · doi:10.1097/aap.0000000000000808 · Cited by 350
Harris, L., Hamilton, S., Azevedo, L. B., Olajide, J., De Brún, C., Waller, G., Whittaker, V., Sharp, T., Lean, M., Hankey, C., et al. (2018) · JBI Database of Systematic Reviews and Implementation Reports · doi:10.11124/jbisrir-2016-003248 · Cited by 319
Page, M. J., Higgins, J. P. T., Clayton, G., Sterne, J. A. C., Hróbjartsson, A., & Savović, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0159267 · Cited by 284
Self-citationMoustgaard, H., Clayton, G. L., Jones, H. E., Boutron, I., Jørgensen, L., Laursen, D. R. T., Olsen, M. F., Paludan-Müller, A., Ravaud, P., Savović, J., et al. (2020) · BMJ · doi:10.1136/bmj.l6802 · Cited by 300
“17 Similarly, a systematic review of 12 trials randomising patients to blinded and non-blinded substudies reported a pronounced bias due to lack of patient blinding in complementary/ alternative medicine trials with patient reported outcomes, exaggerating effect sizes by 0.56 standard deviations.”
InfluentialSelf-citationTuyttens, F. A. M., de Graaf, S., Heerkens, J. L. T., Jacobs, L., Nalon, E., Ott, S., Stadig, L., Van Laer, E., & Ampe, B. (2014) · Animal Behaviour · doi:10.1016/j.anbehav.2014.02.007 · Cited by 274
Evidence of experimental bias in the life sciences: why we need blind data recording
Holman, L., Head, M. L., Lanfear, R., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002190 · Cited by 263
Aday, J. S., Heifets, B. D., Pratscher, S. D., Bradley, E., Rosen, R., & Woolley, J. D. (2022) · Psychopharmacology · doi:10.1007/s00213-022-06123-7 · Cited by 237
CONSORT 2025 explanation and elaboration: updated guideline for reporting randomised trials
Hopewell, S., Chan, A.-W., Collins, G. S., Hróbjartsson, A., Moher, D., Schulz, K. F., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081124 · Cited by 205
Self-citationOrhurhu, V., Orhurhu, M. S., Bhatia, A., & Cohen, S. P. (2019) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000004185 · Cited by 179
Age Related Multiple Sclerosis Severity Score: Disability ranked by age
Manouchehrinia, A., Westerlind, H., Kingwell, E., Zhu, F., Carruthers, R., Ramanujam, R., Ban, M., Glaser, A., Sawcer, S., Tremlett, H., et al. (2017) · Multiple Sclerosis Journal · doi:10.1177/1352458517690618 · Cited by 174
Negative pressure wound therapy for treating foot wounds in people with diabetes mellitus.
Liu, Z., Dumville, J. C., Hinchliffe, R. J., Cullum, N., Game, F., Stubbs, N., Sweeting, M., & Peinemann, F. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010318.pub3 · Cited by 173
Algorithmic fairness in computational medicine
Xu, J., Xiao, Y., Wang, W. H., Ning, Y., Shenkman, E. A., Bian, J., & Wang, F. (2022) · eBioMedicine · doi:10.1016/j.ebiom.2022.104250 · Cited by 170
Should antidepressants be used for major depressive disorder?
Jakobsen, J. C., Gluud, C., & Kirsch, I. (2020) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2019-111238 · Cited by 162
SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials
Hróbjartsson, A., Boutron, I., Hopewell, S., Moher, D., Schulz, K. F., Collins, G. S., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081660 · Cited by 171
Self-citationHistory repeating: guidelines to address common problems in psychedelic science
van Elk, M., & Fried, E. I. (2023) · Therapeutic Advances in Psychopharmacology · doi:10.1177/20451253231198466 · Cited by 137
Juul, S., Gluud, C., Simonsen, S., Frandsen, F. W., Kirsch, I., & Jakobsen, J. C. (2021) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2020-111407 · Cited by 135
Kahan, B. C., Cro, S., Doré, C. J., Bratton, D. J., Rehal, S., Maskell, N. A., Rahman, N., & Jairath, V. (2014) · Trials · doi:10.1186/1745-6215-15-456 · Cited by 123
Empirical evaluation of which trial characteristics are associated with treatment effect estimates
Dechartres, A., Trinquart, L., Faber, T., & Ravaud, P. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.005 · Cited by 85
Self-citationDigital training for building resilience: systematic review, meta-analysis, and meta-regression
Ang, W. H. D., Chew, H. S. J., Dong, J., Yi, H., Mahendren, R., & Lau, Y. (2022) · Stress and Health · doi:10.1002/smi.3154 · Cited by 83
“For studies that largely use subjective measures, a blinded assessor can reduce the risk of exaggerated reporting (Hróbjartsson et al., 2012).”
BackgroundYoung, I., Greig, J., Wilhelm, B. J., & Waddell, L. A. (2019) · Journal of Food Protection · doi:10.4315/0362-028x.jfp-19-108 · Cited by 68
Fool’s gold? Why blinded trials are not always best
Anand, Norrie, Bradley, McAuley, & Clarke (2020) · BMJ · doi:10.1136/bmj.l6228 · Cited by 58
Wang, Y., Parpia, S., Couban, R., Wang, Q., Armijo-Olivo, S., Bassler, D., Briel, M., Brignardello-Petersen, R., Gluud, L. L., Keitz, S. A., et al. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.11.001 · Cited by 58
Self-citationThe impact of blinding on trial results: A systematic review and meta‐analysis
(2023) · doi:10.1002/cesm.12015 · Cited by 35
Bello, S., Moustgaard, H., & Hróbjartsson, A. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.08.002 · Cited by 33
Vogel, M. M. E., Combs, S. E., & Kessel, K. A. (2017) · Frontiers in Oncology · doi:10.3389/fonc.2017.00037 · Cited by 24
van der Ende, N. A., Roozenbeek, B., Broderick, J. P., Khatri, P., Lingsma, H. F., & Dippel, D. W. (2023) · International Journal of Stroke · doi:10.1177/17474930221131706 · Cited by 14
Simpson, A., El-Refaie, A., Stephenson, C., Chen, Y.-P. P., Deng, D., Erickson, S., Tay, D., Morris, M. E., Doube, W., & Caelli, T. (2015) · Deafness & Education International · doi:10.1179/1557069x14y.0000000046 · Cited by 13
Nielsen, S. S., Skou, S. T., Larsen, A. E., Polianskis, R., Pawlak, W. Z., Vægter, H. B., Søndergaard, J., & Christensen, J. R. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-060920 · Cited by 12
O’Keeffe, S. (2026) · Trials · doi:10.1186/s13063-026-09698-0
“Given exaggerated treatment effects when outcome assessors are unblinded [6, 7], it is crucial that sufficient detail on blinding methods is reported in RCTs so that conclusions drawn from these studies can be made with confidence [14].”
Metadata
- Authors
- Hrobjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Hilden, J., Boutron, I., Ravaud, P., & Brorson, S.
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2012-02-27
- Volume
- 344
- Issue
- feb27 2
- Pages
- e1119-e1119
- ISSN
- 0007-1447 (ISSN-L)
- 0959-8138
- 1468-5833
- OpenAlex
- W2102355184
- Cited by
- 483
- References
- 45
- FWCI
- 9.32
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.