The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors
Open accessAbstract
OBJECTIVE: To evaluate the impact of non-blinded outcome assessment on estimated treatment effects in randomised clinical trials with binary outcomes. DESIGN: Systematic review of trials with both blinded and non-blinded assessment of the same binary outcome. For each trial we calculated the ratio of the odds ratios--the odds ratio from non-blinded assessments relative to the corresponding odds ratio from blinded assessments. A ratio of odds ratios <1 indicated that non-blinded assessors generated more optimistic effect estimates than blinded assessors. We pooled the individual ratios of odds ratios with inverse variance random effects meta-analysis and explored reasons for variation in ratios of odds ratios with meta-regression. We also analysed rates of agreement between blinded and non-blinded assessors and calculated the number of patients needed to be reclassified to neutralise any bias. DATA SOURCES: PubMed, Embase, PsycINFO, CINAHL, Cochrane Central Register of Controlled Trials, HighWire Press, and Google Scholar. ELIGIBILITY CRITERIA FOR SELECTING STUDIES: Randomised clinical trials with blinded and non-blinded assessment of the same binary outcome. RESULTS: We included 21 trials in the main analysis (with 4391 patients); eight trials provided individual patient data. Outcomes in most trials were subjective--for example, qualitative assessment of the patient's function. The ratio of the odds ratios ranged from 0.02 to 14.4. The pooled ratio of odds ratios was 0.64 (95% confidence interval 0.43 to 0.96), indicating an average exaggeration of the non-blinded odds ratio by 36%. We found no significant association between low ratios of odds ratios and scores for outcome subjectivity (P=0.27); non-blinded assessor's overall involvement in the trial (P=0.60); or outcome vulnerability to non-blinded patients (P=0.52). Blinded and non-blinded assessors agreed in a median of 78% of assessments (interquartile range 64-90%) in the 12 trials with available data. The exaggeration of treatment effects associated with non-blinded assessors was induced by the misclassification of a median of 3% of the assessed patients per trial (1-7%). CONCLUSIONS: On average, non-blinded assessors of subjective binary outcomes generated substantially biased effect estimates in randomised clinical trials, exaggerating odds ratios by 36%. This bias was compatible with a high rate of agreement between blinded and non-blinded outcome assessors and driven by the misclassification of few patients.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Wood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538
Brandstrup, B., Tønnesen, H., Beier-Holgersen, R., Hjortsø, E., Ørding, H., Lindorff-Larsen, K., Rasmussen, M. S., Lanng, C., Wallin, L., Iversen, L. H., et al. (2003) · Annals of Surgery · doi:10.1097/01.sla.0000094387.50865.23 · Cited by 1,585
Meltzer, H. Y. (2003) · Archives of General Psychiatry · doi:10.1001/archpsyc.60.1.82 · Cited by 1,393
GOVENDER, S., CSIMMA, C., GENANT, H. K., & VALENTIN-OPRAN, A. (2002) · The Journal of Bone and Joint Surgery-American Volume · doi:10.2106/00004623-200212000-00001 · Cited by 1,308
Experimenter Effects in Behavioral Research.
Titchener, J. L. (1967) · Archives of Internal Medicine · doi:10.1001/archinte.1967.00300050109027 · Cited by 1,194
Publication bias: evidence of delayed publication in a cohort study of clinical research projects
Stern, J. M., & Simes, R. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.640 · Cited by 818
Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials
Pildal, J., Hrobjartsson, A., Jorgensen, K., Hilden, J., Altman, D., & Gotzsche, P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym087 · Cited by 472
Self-citationJONES, A. L., BUCHOLZ, R. W., BOSSE, M. J., MIRZA, S. K., LYON, T. R., WEBB, L. X., POLLAK, A. N., GOLDEN, J. D., & VALENTIN-OPRAN, A. (2006) · The Journal of Bone and Joint Surgery-American Volume · doi:10.2106/00004623-200607000-00002 · Cited by 444
A Controlled Trial of Valganciclovir as Induction Therapy for Cytomegalovirus Retinitis
Martin, D. F., Sierra-Madero, J., Walmsley, S., Wolitz, R. A., Macey, K., Georgiou, P., Robinson, C. A., & Stempien, M. J. (2002) · New England Journal of Medicine · doi:10.1056/nejmoa011759 · Cited by 394
Reporting methods of blinding in randomized trials assessing nonpharmacological treatments
Boutron, I., Guittet, L., Estellat, C., Moher, D., Hróbjartsson, A., & Ravaud, P. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040061 · Cited by 389
Self-citationNoseworthy, J. H., Ebers, G. C., Vandervoort, M. K., Farquhar, R. E., Yetisir, E., & Roberts, R. (1994) · Neurology · doi:10.1212/wnl.44.1.16 · Cited by 341
Burkhoff, D., Schmidt, S., Schulman, S. P., Myers, J., Resar, J., Becker, L. C., Weiss, J., & Jones, J. W. (1999) · The Lancet · doi:10.1016/s0140-6736(99)08113-1 · Cited by 312
Devereaux, P. J. (2001) · JAMA · doi:10.1001/jama.285.15.2000 · Cited by 258
The use of a collagen sponge/living cell composite material to treat donor sites in burn patients
Still, J., Glat, P., Silverstein, P., Griswold, J., & Mozingo, D. (2003) · Burns · doi:10.1016/s0305-4179(03)00164-5 · Cited by 251
Kadish, A., Nademanee, K., Volosin, K., Krueger, S., Neelagaru, S., Raval, N., Obel, O., Weiner, S., Wish, M., Carson, P., et al. (2011) · American Heart Journal · doi:10.1016/j.ahj.2010.10.025 · Cited by 231
SWIONTKOWSKI, M. F., ARO, H. T., DONELL, S., ESTERHAI, J. L., GOULET, J., JONES, A., KREGOR, P. J., NORDSLETTEN, L., PAIEMENT, G., & PATEL, A. (2006) · The Journal of Bone and Joint Surgery-American Volume · doi:10.2106/00004623-200606000-00013 · Cited by 204
Aro, H. T., Govender, S., Patel, A. D., Hernigou, P., Perera de Gregorio, A., Popescu, G. I., Golden, J. D., Christensen, J., & Valentin, A. (2011) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.i.01763 · Cited by 180
Meta‐analytically Quantifying the Reliability and Biasability of Forensic Experts
Dror, I., & Rosenthal, R. (2008) · Journal of Forensic Sciences · doi:10.1111/j.1556-4029.2008.00762.x · Cited by 176
Randomized clinical trial of honey-impregnated dressings for venous leg ulcers
Jull, A., Walker, N., Parag, V, Molan, P., & Rodgers, A. (2008) · Journal of British Surgery · doi:10.1002/bjs.6059 · Cited by 175
Percutaneous transmyocardial laser revascularisation for severe angina: the PACIFIC randomised trial
Oesterle, S. N., Sanborn, T. A., Ali, N., Resar, J., Ramee, S. R., Heuser, R., Dean, L., Knopf, W., Schofield, P., Schaer, G. L., et al. (2000) · The Lancet · doi:10.1016/s0140-6736(00)03203-7 · Cited by 172
Vaginal Mesh for Prolapse: A Randomized Controlled Trial
Iglesia, C. B., Sokol, A. I., Sokol, E. R., Kudish, B. I., Gutman, R. E., Peterson, J. L., & Shott, S. (2010) · Obstetrics & Gynecology · doi:10.1097/aog.0b013e3181e7d7f8 · Cited by 158
Nüesch, E., Reichenbach, S., Trelle, S., Rutjes, A. W., Liewald, K., Sterchi, R., Altman, D. G., & Jüni, P. (2009) · Arthritis Care & Research · doi:10.1002/art.24894 · Cited by 154
Who is blinded in randomized clinical trials? A study of 200 trials and a survey of authors
Haahr, M. T., & Hróbjartsson, A. (2006) · Clinical Trials · doi:10.1177/1740774506069153 · Cited by 152
Baumann, L., Shamban, A. T., Lupo, M. P., Monheit, G. D., Thomas, J. A., Murphy, D. K., Walker, P., & FOR THE JUVÉDERM VS. ZYPLAST NASOLABIAL FOLD STUDY GROUP (2007) · Dermatologic Surgery · doi:10.1111/j.1524-4725.2007.33352.x · Cited by 150
Miller, R. S., Steward, D. L., Tami, T. A., Sillars, M. J., Seiden, A. M., Shete, M. M., Paskowski, C., & Welge, J. A. (2003) · Otolaryngology · doi:10.1016/s0194-5998(03)00460-1 · Cited by 147
Reporting of Outcomes in Orthopaedic Randomized Trials: Does Blinding of Outcome Assessors Matter?
Poolman, R. W., Struijs, P. A. A., Krips, R., Sierevelt, I. N., Marti, R. K., Farrokhyar, F., & Bhandari, M. (2007) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.f.00683 · Cited by 140
How often are our numbers wrong?
Rosenthal, R. (1978) · American Psychologist · doi:10.1037/0003-066x.33.11.1005 · Cited by 115
Evaluating the benefit of event adjudication of cardiovascular outcomes in large simple RCTs
Pogue, J. M., Walter, S. D., & Yusuf, S. (2009) · Clinical Trials · doi:10.1177/1740774509105223 · Cited by 114
Treatment of Mild, Moderate, and Severe Onychomycosis Using 870- and 930-nm Light Exposure
Landsman, A. S., Robbins, A. H., Angelini, P. F., Wu, C. C., Cook, J. J., Oster, M., & Bornstein, E. (2010) · Journal of the American Podiatric Medical Association · doi:10.7547/1000166 · Cited by 104
Valldeoriola, F., Regidor, I., Mínguez‐Castellanos, A., Lezcano, E., Ruiz, P., Rojo, A., Salvador, A., Castro, A., Grandas, F., Kulisevsky, J., et al. (2009) · Journal of Neurology Neurosurgery & Psychiatry · doi:10.1136/jnnp.2009.174342 · Cited by 73
Karanicolas, P. J., Bhandari, M., Taromi, B., Akl, E. A., Bassler, D., Alonso‐Coello, P., Comas, D. R., Bryant, D., Smith, S. E., Walter, S. D., et al. (2008) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.g.00963 · Cited by 72
Seeing What We Want to See: Confirmation Bias in Animal Behavior Research
Marsh, D. M., & Hanlon, T. J. (2007) · Ethology · doi:10.1111/j.1439-0310.2007.01406.x · Cited by 53
Treatment of Mild, Moderate, and Severe Onychomycosis Using 870- and 930-nm Light Exposure
Landsman, A. S., & Robbins, A. H. (2012) · Journal of the American Podiatric Medical Association · doi:10.7547/1020169 · Cited by 53
Cyclophosphamide and plasma exchange in multiple sclerosis
Noseworthy, J. H., Vandervoort, M. K., Mf, P., Ebers, G. C., Shumak, K. H., Seland, T. P., Roberts, R. A., Yetisir, E., Gent, M., & Taylor, D. W. (1991) · The Lancet · doi:10.1016/0140-6736(91)93226-y · Cited by 47
Mason, P., Back, S., & Fields, H. (1992) · Journal of Neuroscience · doi:10.1523/jneurosci.12-10-04023.1992 · Cited by 25
Karanicolas, P. J., Bhandari, M., Walter, S. D., Heels‐Ansdell, D., & Guyatt, G. H. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.05.006 · Cited by 24
Cyclophosphamide and plasma exchange in multiple sclerosis
Khatri, B. O., McQuillen, M. P., Hoffmann, R. G., Weiner, H., Hauser, S., Dawson, D., Hafler, D. A., Mackin, G. A., Orav, E. J., Currier, R., et al. (1991) · The Lancet · doi:10.1016/0140-6736(91)92688-x · Cited by 24
Observer bias in scoring boys' and girls' aggression
Lyons, J., & Serbin, L. A. (1986) · Sex Roles · doi:10.1007/bf00287581 · Cited by 20
Letter: The Combined Use of Needle with Hair Transplanter for Hair Recipient Sites
LEE, D.-Y., CHOI, Y.-L., KIM, M.-G., KIM, J.-A., LEE, K.-J., PARK, J.-H., Cho, H.-J., Yang, J., & Sim, W. (2007) · Dermatologic Surgery · doi:10.1111/j.1524-4725.2007.33026.x · Cited by 20
Vaginal Mesh for Prolapse: A Randomized Controlled Trial
Iglesia, C. B., Sokol, A. I., Gutman, R. E., Sokol, E. R., Kudish, B., & Shott, S. (2010) · Obstetrics and Gynecology · doi:10.1097/aog.0b013e3181ff98b1 · Cited by 18
Topic of Histopathology Blinding in Nonclinical Safety Biomarker Qualification Studies
Burkhardt, J. E., Ennulat, D., Pandher, K., Solter, P. F., Troth, S. P., Boyce, R. W., & Zabka, T. S. (2010) · Toxicologic Pathology · doi:10.1177/0192623310371221 · Cited by 15
Vaginal Mesh for Prolapse: A Randomized Controlled Trial
Iglesia, C. B., Sokol, A. I., Gutman, R. E., Sokol, E. R., Kudish, B., & Shott, S. (2010) · Obstetrics and Gynecology · doi:10.1097/aog.0b013e3181ff78e1 · Cited by 11
Waibel, K. H., Golding, H., Manischewitz, J. F., King, L. R., Tuchscherer, M., Topolski, R. L., & Walsh, D. S. (2006) · Clinical Infectious Diseases · doi:10.1086/499957 · Cited by 7
Cited by
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials
Chan, A.-W., Tetzlaff, J. M., Gotzsche, P. C., Altman, D. G., Mann, H., Berlin, J. A., Dickersin, K., Hrobjartsson, A., Schulz, K. F., Parulekar, W. R., et al. (2013) · BMJ · doi:10.1136/bmj.e7586 · Cited by 6,251
Jakobsen, J. C., Gluud, C., Wetterslev, J., & Winkel, P. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0442-1 · Cited by 2,747
Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000411 · Cited by 2,677
Fogel, D. B. (2018) · Contemporary Clinical Trials Communications · doi:10.1016/j.conctc.2018.08.001 · Cited by 1,035
Metadata
- Authors
- Hrobjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Hilden, J., Boutron, I., Ravaud, P., & Brorson, S.
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2012-02-27
- Volume
- 344
- Issue
- feb27 2
- Pages
- e1119-e1119
- ISSN
- 0007-1447 (ISSN-L)
- 0959-8138
- 1468-5833
- OpenAlex
- W2102355184
- Cited by
- 483
- References
- 45
- FWCI
- 9.32
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.