Cited by 527Meta-analysis and systematic reviewsStatistical Methods in Clinical TrialsReliability and Agreement in Measurement

The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.

Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors

Open access

Hróbjartsson, Asbjørn, Thomsen, Ann Sofia Skou, Emanuelsson, Frida, Tendal, Britta, Hilden, Jørgen, Boutron, Isabelle, Ravaud, Philippe, Brorson, Stig·2013·Canadian Medical Association Journal

DOI 10.1503/cmaj.120744PMC PMC3589328

Open this paper in CoChat — free account Open PDF

Abstract

BACKGROUND: Clinical trials are commonly done without blinded outcome assessors despite the risk of bias. We wanted to evaluate the effect of nonblinded outcome assessment on estimated effects in randomized clinical trials with outcomes that involved subjective measurement scales. METHODS: We conducted a systematic review of randomized clinical trials with both blinded and nonblinded assessment of the same measurement scale outcome. We searched PubMed, EMBASE, PsycINFO, CINAHL, Cochrane Central Register of Controlled Trials, HighWire Press and Google Scholar for relevant studies. Two investigators agreed on the inclusion of trials and the outcome scale. For each trial, we calculated the difference in effect size (i.e., standardized mean difference between nonblinded and blinded assessments). A difference in effect size of less than 0 suggested that nonblinded assessors generated more optimistic estimates of effect. We pooled the differences in effect size using inverse variance random-effects meta-analysis and used metaregression to identify potential reasons for variation. RESULTS: We included 24 trials in our review. The main meta-analysis included 16 trials (involving 2854 patients) with subjective outcomes. The estimated treatment effect was more beneficial when based on nonblinded assessors (pooled difference in effect size -0.23 [95% confidence interval (CI) -0.40 to -0.06]). In relative terms, nonblinded assessors exaggerated the pooled effect size by 68% (95% CI 14% to 230%). Heterogeneity was moderate (I(2) = 46%, p = 0.02) and unexplained by metaregression. INTERPRETATION: We provide empirical evidence for observer bias in randomized clinical trials with subjective measurement scale outcomes. A failure to blind assessors of outcomes in such trials results in a high risk of substantial bias.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1503/cmaj.120744

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Cochrane Handbook for Systematic Reviews of Interventions
Higgins, J. P. T., Green, S., & Ben Van Den, A. (2020) · International Coaching Psychology Review · doi:10.53841/bpsicpr.2020.15.2.123 · Cited by 5,670
Bilateral Deep Brain Stimulation vs Best Medical Therapy for Patients With Advanced Parkinson Disease A Randomized Controlled Trial
Weaver, F. M. (2009) · JAMA · doi:10.1001/jama.2008.929 · Cited by 1,535
Clozapine Treatment for Suicidality in Schizophrenia<subtitle>International Suicide Prevention Trial (InterSePT)</subtitle>
Meltzer, H. Y. (2003) · Archives of General Psychiatry · doi:10.1001/archpsyc.60.1.82 · Cited by 1,393
Progressive resistance strength training for improving physical function in older adults
Liu, C.-J., & Latham, N. K. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002759.pub2 · Cited by 1,258
Some Problems in Interval Estimation
Fieller, E. C. (1954) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1954.tb00159.x · Cited by 1,159
Influence of Reported Study Design Characteristics on Intervention Effect Estimates From Randomized, Controlled Trials
Savović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-6-201209180-00537 · Cited by 1,049

Flagged first, then by citations

Show all 52 references

Research works printed but not linked: 4

[2]2. Haahr MT, Hróbjartsson A. Who is blind in randomised clinical trials? An analysis of 200 trials and a survey of authors. Clin Tri- als 2006;3:360-5.
[21]21. Meltzer HY, Alphs L, Green AI, et al. Clozapine treatment for suicidality in schizophrenia: International Suicide Prevention Trial (InterSePT). Arch Gen Psychiatry 2003;60:82-91.
[34]34. Realmuto GM, Erickson WD, Yellin AM, et al. Clinical compar- ison of thiothixene and thioridazine in schizophrenic adoles- cents. Am J Psychiatry 1984;141:440-2.
[41]41. Dodd DC. Blind slide reading or the uninformed versus the informed pathologist. Comments Toxicol 1988;2:88-91.

Sorted by citations

Comparative efficacy and acceptability of 21 antidepressant drugs for the acute treatment of adults with major depressive disorder: a systematic review and network meta-analysis
Cipriani, A., Furukawa, T. A., Salanti, G., Chaimani, A., Atkinson, L. Z., Ogawa, Y., Leucht, S., Ruhe, H. G., Turner, E. H., Higgins, J. P. T., et al. (2018) · The Lancet · doi:10.1016/s0140-6736(17)32802-7 · Cited by 3,541
When and how should multiple imputation be used for handling missing data in randomised clinical trials – a practical guide with flowcharts
Jakobsen, J. C., Gluud, C., Wetterslev, J., & Winkel, P. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0442-1 · Cited by 2,747
Efficacy of treatments for anxiety disorders: a meta-analysis
Bandelow, B., Reitt, M., Röver, C., Michaelis, S., Görlich, Y., & Wedekind, D. (2015) · International Clinical Psychopharmacology · doi:10.1097/yic.0000000000000078 · Cited by 555
Diet, physical activity and behavioural interventions for the treatment of overweight or obese children from the age of 6 to 11 years
Mead, E., Brown, T. J., Rees, K., Azevedo, L. B., Whittaker, V., Jones, D., Olajide, J., Mainardi, G. M., Corpeleijn, E., O’Malley, C. L., et al. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012651 · Cited by 550
Compression stockings to prevent post-thrombotic syndrome: a randomised placebo-controlled trial
Kahn, S. R., Shapiro, S., Wells, P. S., Rodger, M. A., Kovacs, M. J., Anderson, D. R., Tagalakis, V., Houweling, A. H., Ducruet, T., Holcroft, C., et al. (2013) · The Lancet · doi:10.1016/s0140-6736(13)61902-9 · Cited by 521
Bias due to lack of patient blinding in clinical trials. A systematic review of trials randomizing patients to blind and nonblind sub-studies
Hróbjartsson, A., Emanuelsson, F., Skou Thomsen, A. S., Hilden, J., & Brorson, S. (2014) · International Journal of Epidemiology · doi:10.1093/ije/dyu115 · Cited by 462
Self-citation
Diet, physical activity and behavioural interventions for the treatment of overweight or obese adolescents aged 12 to 17 years
Al-Khudairy, L., Loveman, E., Colquitt, J. L., Mead, E., Johnson, R. E., Fraser, H., Olajide, J., Murphy, M., Velho, R., O’Malley, C. L., et al. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012691 · Cited by 472
Selective serotonin reuptake inhibitors versus placebo in patients with major depressive disorder. A systematic review with meta-analysis and trial sequential analysis
Jakobsen, J. C., Katakam, K. K., Schou, A., Hellmuth, S. G., Stallknecht, S. E., Leth-Møller, K., Iversen, M., Banke, M. B., Petersen, I. J., Klingenberg, S. L., et al. (2017) · BMC Psychiatry · doi:10.1186/s12888-016-1173-2 · Cited by 366
Comparative Efficacy and Acceptability of 21 Antidepressant Drugs for the Acute Treatment of Adults With Major Depressive Disorder: A Systematic Review and Network Meta-Analysis
Cipriani, A., Furukawa, T. A., Salanti, G., Chaimani, A., Atkinson, L. Z., Ogawa, Y., Leucht, S., Ruhe, H. G., Turner, E. H., Higgins, J. P. T., et al. (2018) · Focus · doi:10.1176/appi.focus.16407 · Cited by 328
Diet, physical activity or both for prevention or delay of type 2 diabetes mellitus and its associated complications in people at increased risk of developing type 2 diabetes mellitus
Hemmingsen, B., Gimenez-Perez, G., Mauricio, D., Roqué i Figuls, M., Metzendorf, M.-I., & Richter, B. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003054.pub4 · Cited by 316
Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study
Moustgaard, H., Clayton, G. L., Jones, H. E., Boutron, I., Jørgensen, L., Laursen, D. R. T., Olsen, M. F., Paludan-Müller, A., Ravaud, P., Savović, J., et al. (2020) · BMJ · doi:10.1136/bmj.l6802 · Cited by 300
Self-citation
Intensive glucose control versus conventional glucose control for type 1 diabetes mellitus
Fullerton, B., Jeitler, K., Seitz, M., Horvath, K., Berghold, A., & Siebenhofer, A. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009122.pub2 · Cited by 299
Empirical Evidence of Study Design Biases in Randomized Trials: Systematic Review of Meta-Epidemiological Studies
Page, M. J., Higgins, J. P. T., Clayton, G., Sterne, J. A. C., Hróbjartsson, A., & Savović, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0159267 · Cited by 284
Self-citation
Observer bias in animal behaviour research: can we believe what we score, if we score what we believe?
Tuyttens, F., de Graaf, S., Heerkens, J., Jacobs, L., Nalon, E., Ott, S., Stadig, L. M., laer, E. V., & Ampe, B. (2014) · Animal Behaviour · doi:10.1016/j.anbehav.2014.02.007 · Cited by 275
Evidence of experimental bias in the life sciences: why we need blind data recording
Holman, L., Head, M. L., Lanfear, R., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002190 · Cited by 263
Interventions to Promote Fundamental Movement Skills in Childcare and Kindergarten: A Systematic Review and Meta-Analysis
Wick, K., Leeger‐Aschmann, C. S., Monn, N. D., Radtke, T., Ott, L. V., Rebholz, C. E., Cruz, S. S., Gerber, N., Schmutz, E. A., Puder, J. J., et al. (2017) · Sports Medicine · doi:10.1007/s40279-017-0723-1 · Cited by 312
The effects of parent-implemented language interventions on child linguistic outcomes: A meta-analysis
Heidlage, J. K., Cunningham, J. E., Kaiser, A. P., Trivette, C. M., Barton, E. E., Frey, J. R., & Roberts, M. Y. (2019) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2018.12.006 · Cited by 294
Diet, physical activity, and behavioural interventions for the treatment of overweight or obesity in preschool children up to the age of 6 years
Colquitt, J. L., Loveman, E., O’Malley, C. L., Azevedo, L. B., Mead, E., Al-Khudairy, L., Ells, L. J., Metzendorf, M., & Rees, K. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012105 · Cited by 239
Anxious brain networks: A coordinate-based activation likelihood estimation meta-analysis of resting-state functional connectivity studies in anxiety
Xu, J., Van Dam, N. T., Feng, C., Luo, Y., Ai, H., Gu, R., & Xu, P. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.11.005 · Cited by 212
CONSORT 2025 explanation and elaboration: updated guideline for reporting randomised trials
Hopewell, S., Chan, A.-W., Collins, G. S., Hróbjartsson, A., Moher, D., Schulz, K. F., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081124 · Cited by 205
Self-citation
A web-based psychoeducational program for informal caregivers of patients with Alzheimer’s disease: a pilot randomized controlled trial
Cristancho-Lacroix, V., Wrobel, J., Cantegreil-Kallen, I., Dub, T., Rouquette, A., & Rigaud, A.-S. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.3717 · Cited by 205
Patient-reported outcome measures (PROMs) as proof of treatment efficacy
Kluzek, S., Dean, B. J. F., & Wartolowska, K. A. (2021) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2020-111573 · Cited by 184
Culturally appropriate health education for people in ethnic minority groups with type 2 diabetes mellitus
Attridge, M., Creamer, J., Ramsden, M., Cannings-John, R. L., & Hawthorne, K. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006424.pub3 · Cited by 201
Placebo response and effect in randomized clinical trials: meta-research with focus on contextual effects
Hafliðadóttir, S. H., Juhl, C. B., Nielsen, S. M., Henriksen, M., Harris, I. A., Bliddal, H., & Christensen, R. (2021) · Trials · doi:10.1186/s13063-021-05454-8 · Cited by 183
Algorithmic fairness in computational medicine
Xu, J., Xiao, Y., Wang, W. H., Ning, Y., Shenkman, E. A., Bian, J., & Wang, F. (2022) · eBioMedicine · doi:10.1016/j.ebiom.2022.104250 · Cited by 170
Should antidepressants be used for major depressive disorder?
Jakobsen, J. C., Gluud, C., & Kirsch, I. (2020) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2019-111238 · Cited by 162
Drug interventions for the treatment of obesity in children and adolescents
Axon, E., Atkinson, G., Richter, B., Metzendorf, M., Baur, L. A., Finer, N., Corpeleijn, E., O’Malley, C. L., & Ells, L. J. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012436 · Cited by 164
World Federation of Societies of Biological Psychiatry (WFSBP) guidelines for treatment of anxiety, obsessive-compulsive and posttraumatic stress disorders – Version 3. Part I: Anxiety disorders
Bandelow, B., Allgulander, C., Baldwin, D. S., Costa, D. L. D. C., Denys, D., Dilbaz, N., Domschke, K., Eriksson, E., Fineberg, N. A., Hättenschwiler, J., et al. (2023) · The World Journal of Biological Psychiatry · doi:10.1080/15622975.2022.2086295 · Cited by 158
Intraoperative neuromonitoring versus visual nerve identification for prevention of recurrent laryngeal nerve injury in adults undergoing thyroid surgery
Cirocchi, R., Arezzo, A., D’Andrea, V., Abraha, I., Popivanov, G., Avenia, N., Gerardi, C., Henry, B. M., Randolph, J., & Barczyński, M. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012483.pub2 · Cited by 159
Mesenchymal stem cells improve locomotor recovery in traumatic spinal cord injury: Systematic review with meta-analyses of rat models
Oliveri, R. S., Bello, S., & Biering‐Sørensen, F. (2013) · Neurobiology of Disease · doi:10.1016/j.nbd.2013.10.014 · Cited by 156
Ultrasound-guided arterial cannulation for paediatrics
Aouad-Maroun, M., Raphael, C. K., Sayyid, S. K., Farah, F., & Akl, E. A. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011364.pub2 · Cited by 147
Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors
Hróbjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Rasmussen, J. V., Hilden, J., Boutron, I., Ravaud, P., & Brorson, S. (2014) · International Journal of Epidemiology · doi:10.1093/ije/dyt270 · Cited by 142
Self-citation
Comparative efficacy and tolerability of pharmacological treatments for the treatment of acute bipolar depression: a systematic review and network meta-analysis
Bahji, A., Ermacora, D., Stephenson, C., Hawken, E. R., & Vazquez, G. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.03.030 · Cited by 139
Psychological interventions for diabetes-related distress in adults with type 2 diabetes mellitus
Chew, B. H., Vos, R. C., Metzendorf, M.-I., Scholten, R. J., & Rutten, G. E. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011469.pub2 · Cited by 138
Blinding in Clinical Trials: Seeing the Big Picture
Monaghan, T. F., Agudelo, C. W., Rahman, S. N., Wein, A. J., Lazar, J., Everaert, K., & Dmochowski, R. R. (2021) · Medicina · doi:10.3390/medicina57070647 · Cited by 154
Parent-only interventions for childhood overweight or obesity in children aged 5 to 11 years
Loveman, E., Al-Khudairy, L., Johnson, R. E., Robertson, W., Colquitt, J. L., Mead, E., Ells, L. J., Metzendorf, M., & Rees, K. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012008 · Cited by 155
Blinding in randomised clinical trials of psychological interventions: a retrospective study of published trial reports
Juul, S., Gluud, C., Simonsen, S., Frandsen, F. W., Kirsch, I., & Jakobsen, J. C. (2021) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2020-111407 · Cited by 135
Methods for assessment of keel bone damage in poultry
Casey-Trott, T. M., Heerkens, J., Petrik, M. T., Regmi, P., Schrader, L., Toscano, M. J., & Widowski, T. M. (2015) · Poultry Science · doi:10.3382/ps/pev223 · Cited by 127
Oral vitamin B 12 versus intramuscular vitamin B 12 for vitamin B 12 deficiency
Wang, H., Li, L., Qin, L., Song, Y., Vidal‐Alaball, J., & Liu, T. H. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004655.pub3 · Cited by 126
Reducing bias in open-label trials where blinded outcome assessment is not feasible: strategies from two randomised trials
Kahan, B. C., Cro, S., Doré, C. J., Bratton, D. J., Rehal, S., Maskell, N. A., Rahman, N., & Jairath, V. (2014) · Trials · doi:10.1186/1745-6215-15-456 · Cited by 123
Nutrition support in hospitalised adults at nutritional risk
Feinberg, J. B., Nielsen, E. E., Korang, S. K., Engell, K. H., Nielsen, M. S., Zhang, K., Didriksen, M., Lund, L., Lindahl, N., Hallum, S., et al. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011598.pub2 · Cited by 136
Catalogue of bias: observer bias
Mahtani, K. R., Spencer, E. A., Brassey, J., & Heneghan, C. J. (2018) · BMJ Evidence-Based Medicine · doi:10.1136/ebmed-2017-110884 · Cited by 118
Long-term effects of weight-reducing drugs in people with hypertension
Siebenhofer, A., Jeitler, K., Horvath, K., Berghold, A., Posch, N. A. S., Meschik, J., & Semlitsch, T. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007654.pub4 · Cited by 118
CogSMART Compensatory Cognitive Training for Traumatic Brain Injury
Twamley, E. W., Thomas, K. R., Gregory, A. M., Jak, A. J., Bondi, M. W., Delis, D. C., & Lohr, J. B. (2014) · Journal of Head Trauma Rehabilitation · doi:10.1097/htr.0000000000000076 · Cited by 117
Intraoperative neuromonitoring versus visual nerve identification for prevention of recurrent laryngeal nerve injury in adults undergoing thyroid surgery
Cirocchi, R., D’Andrea, V., Arezzo, A., Abraha, I., Passera, R., Avenia, N., Randolph, J., & Barczyński, M. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012483 · Cited by 109
IMPRoving Outcomes for children exposed to domestic ViolencE (IMPROVE): an evidence synthesis
Howarth, E., Moore, T. H., Welton, N. J., Lewis, N. V., Stanley, N., MacMillan, H. L., Shaw, A. R. G., Hester, M., Bryden, P. A., & Feder, G. S. (2016) · Public Health Research · doi:10.3310/phr04100 · Cited by 112
Predictors and Causes of Long-Term Mortality in Elderly Patients with Acute Venous Thromboembolism: A Prospective Cohort Study
Faller, N., Limacher, A., Méan, M., Righini, M., Aschwanden, M., Beer, J. H., Frauchiger, B., Osterwalder, J., Kucher, N., Lämmle, B., et al. (2016) · The American Journal of Medicine · doi:10.1016/j.amjmed.2016.09.008 · Cited by 104
Bias in Before–After Studies: Narrative Overview for Anesthesiologists
Ho, A. M. ‐., Phelan, R., Mizubuti, G. B., Murdoch, J. A. C., Wickett, S., Ho, A. K., Shyam, V., & Gilron, I. (2017) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000002705 · Cited by 109
A smartphone application supporting patients with psoriasis improves adherence to topical treatment: a randomized controlled trial
Svendsen, M. T., Andersen, F., Andersen, K. H., Pottegård, A., Johannessen, H., Möller, S., August, B., Feldman, S. R., & Andersen, K. E. (2018) · British Journal of Dermatology · doi:10.1111/bjd.16667 · Cited by 99
Transperitoneal versus retroperitoneal laparoscopic adrenalectomy for adrenal tumours in adults
Arezzo, A., Bullano, A., Cochetti, G. G., Cirocchi, R., Randolph, J., Mearini, E. E., Evangelista, A., Ciccone, G., Bonjer, H. J., & Morino, M. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011668.pub2 · Cited by 96
Statistically Significant Antidepressant-Placebo Differences on Subjective Symptom-Rating Scales Do Not Prove That the Drugs Work: Effect Size and Method Bias Matter!
Hengartner, M. P., & Plöderl, M. (2018) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2018.00517 · Cited by 99
Short-acting insulin analogues versus regular human insulin for adults with type 1 diabetes mellitus
Fullerton, B., Siebenhofer, A., Jeitler, K., Horvath, K., Semlitsch, T., Berghold, A., Plank, J., Pieber, T. R., & Gerlach, F. M. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012161 · Cited by 94
Screening for type 2 diabetes mellitus
Peer, N., Balakrishna, Y., & Durão, S. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005266.pub2 · Cited by 101
Lack of blinding of outcome assessors in animal model experiments implies risk of observer bias
Bello, S., Krogsbøll, L. T., Gruber, J., Zhao, Z. J., Fischer, D., & Hróbjartsson, A. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.04.008 · Cited by 91
Self-citation
Empirical evaluation of which trial characteristics are associated with treatment effect estimates
Dechartres, A., Trinquart, L., Faber, T., & Ravaud, P. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.005 · Cited by 85
Self-citation
(Ultra-)long-acting insulin analogues versus NPH insulin (human isophane insulin) for adults with type 2 diabetes mellitus
Semlitsch, T., Engler, J., Siebenhofer, A., Jeitler, K., Berghold, A., & Horvath, K. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005613.pub4 · Cited by 80
Cognitive behavioural therapy halves the risk of repeated suicide attempts: systematic review.
Gøtzsche, P. C., & Gøtzsche, P. K. (2017) · Journal of the Royal Society of Medicine · doi:10.1177/0141076817731904 · Cited by 79
Thyroid surgery for Graves’ disease and Graves’ ophthalmopathy
Liu, Z. W., Masterson, L. M., Fish, B. M., Jani, P., & Chatterjee, K. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010576.pub2 · Cited by 83
Concomitant atrial fibrillation surgery for people undergoing cardiac surgery
Huffman, M. D., Karmali, K. N., Berendsen, M. A., Andrei, A., Kruse, J. T., McCarthy, P. M., & Malaisrie, S. C. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011814.pub2 · Cited by 82
Non-nutritive sweeteners for diabetes mellitus
Lohner, S., de Gaudry, D. K., Toews, I., Ferenci, T., & Meerpohl, J. J. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012885.pub2 · Cited by 76
Resveratrol for adults with type 2 diabetes mellitus
Jeyaraman, M. M., Al‐Yousif, N., Mann, A. S., Dolinsky, V. W., Rabbani, R., Zarychanski, R., & Abou-Setta, A. M. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011919.pub2 · Cited by 79
Subcutaneous rapid-acting insulin analogues for diabetic ketoacidosis
Andrade-Castellanos, C. A., Colunga‐Lozano, L. E., Delgado-Figueroa, N., & González‐Padilla, D. A. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011281.pub2 · Cited by 74
Total or near-total thyroidectomy versus subtotal thyroidectomy for multinodular non-toxic goitre in adults
Cirocchi, R., Trastulli, S., Randolph, J., Guarino, S., Rocco, G. D., Arezzo, A., D’Andrea, V., Santoro, A., Barczyński, M., & Avenia, N. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010370.pub2 · Cited by 74
Mapping ongoing nutrition intervention trials in muscle, sarcopenia, and cachexia: a scoping review of future research
Orsso, C. E., Montes‐Ibarra, M., Findlay, M., van der Meij, B. S., de van der Schueren, M. A. E., Landi, F., Laviano, A., & Prado, C. M. (2022) · Journal of Cachexia, Sarcopenia and Muscle · doi:10.1002/jcsm.12954 · Cited by 73
Feasibility of surgical randomised controlled trials with a placebo arm: a systematic review
Wartolowska, K. A., Collins, P. G. S., Hopewell, S., Judge, A., Dean, B. J. F., Rombach, I., Beard, D. J., & Carr, A. J. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010194 · Cited by 72
Paroxetine versus other anti-depressive agents for depression
Purgato, M., Papola, D., Gastaldon, C., Trespidi, C., Magni, L. R., Rizzo, C., Furukawa, T. A., Watanabe, N., Cipriani, A., & Barbui, C. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006531.pub2 · Cited by 73
Metformin for prevention or delay of type 2 diabetes mellitus and its associated complications in persons at increased risk for the development of type 2 diabetes mellitus
Madsen, K. S., Chi, Y., Metzendorf, M.-I., Richter, B., & Hemmingsen, B. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008558.pub2 · Cited by 70
Long-term effects of weight-reducing drugs in people with hypertension
Siebenhofer, A., Winterholer, S., Jeitler, K., Horvath, K., Berghold, A., Krenn, C., & Semlitsch, T. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007654.pub5 · Cited by 70
Surgery for the treatment of obesity in children and adolescents
Ells, L. J., Mead, E., Atkinson, G., Corpeleijn, E., Roberts, K. E., Viner, R., Baur, L. A., Metzendorf, M., & Richter, B. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011740 · Cited by 75
Subjective and objective outcomes in randomized clinical trials: definitions differed in methods publications and were often absent from trial reports
Moustgaard, H., Bello, S., Miller, F. G., & Hróbjartsson, A. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.06.020 · Cited by 69
Self-citation
Self management interventions for type 2 diabetes in adult people with severe mental illness
McBain, H. B., Mulligan, K., Haddad, M., Flood, C., Jones, J., & Simpson, A. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011361.pub2 · Cited by 70
Zinc supplementation for the prevention of type 2 diabetes mellitus in adults with insulin resistance
Dib, R. E., Gameiro, O. L., Ogata, M. S. P., Módolo, N. S. P., Braz, L. G., Jorge, E. C., do Nascimento, P., & Beletate, V. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005525.pub3 · Cited by 71
Cerebral near-infrared spectroscopy monitoring versus treatment as usual for extremely preterm infants: a protocol for the SafeBoosC randomised clinical phase III trial
Hansen, M. L., Pellicer, A., Gluud, C. N., Dempsey, E. M., Mintzer, J. P., Hyttel-Sørensen, S., Heuchan, A. M., Hagmann, C. F., Ergenekon, E., Dimitriou, G., et al. (2019) · Trials · doi:10.1186/s13063-019-3955-6 · Cited by 68
Ultrasound‐Guided Knee Injections Are More Accurate Than Blind Injections: A Systematic Review of Randomized Controlled Trials
FANG, W. H., Chen, X. T., & Vangsness, C. T. (2021) · Arthroscopy Sports Medicine and Rehabilitation · doi:10.1016/j.asmr.2021.01.028 · Cited by 67
Systematic review of interventions for Indigenous adults with mental and substance use disorders in Australia, Canada, New Zealand and the United States
Leske, S., Harris, M. G., Charlson, F. J., Ferrari, A. J., Baxter, A. J., Logan, J. M., Toombs, M., & Whiteford, H. (2016) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867416662150 · Cited by 65
Lee Silverman Voice Treatment for dysarthria in patients with Parkinson’s disease: a systematic review and meta‐analysis
Yuan, F., Guo, X., Wei, X., Xie, F. X., Zheng, J., Huang, Y., Huang, Z.-H., Chang, Z., Li, H.-X., Guo, Y., et al. (2020) · European Journal of Neurology · doi:10.1111/ene.14399 · Cited by 65
Short-acting insulin analogues versus regular human insulin for adult, non-pregnant persons with type 2 diabetes mellitus
Fullerton, B., Siebenhofer, A., Jeitler, K., Horvath, K., Semlitsch, T., Berghold, A., & Gerlach, F. M. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013228 · Cited by 67
Dipeptidyl-peptidase (DPP)-4 inhibitors and glucagon-like peptide (GLP)-1 analogues for prevention or delay of type 2 diabetes mellitus and its associated complications in people at increased risk for the development of type 2 diabetes mellitus
Hemmingsen, B., Sonne, D. P., Metzendorf, M.-I., & Richter, B. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012204.pub2 · Cited by 63
Metformin monotherapy for adults with type 2 diabetes mellitus
Gnesin, F., Thuesen, A. C. B., Kähler, L. K. A., Madsbad, S., & Hemmingsen, B. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012906.pub2 · Cited by 63
The evolution of assessing bias in Cochrane systematic reviews of interventions: celebrating methodological contributions of the Cochrane Collaboration
Turner, L., Boutron, I., Hróbjartsson, A., Altman, D. G., & Moher, D. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-79 · Cited by 66
Self-citation
Optimal dosing of antibiotics in critically ill patients by using continuous/extended infusions: a systematic review and meta-analysis
Chant, C. A., Leung, A., & Friedrich, J. O. (2013) · Critical Care · doi:10.1186/cc13134 · Cited by 61
Expectancy Effects, Failure of Blinding Integrity, and Placebo Response in Trials of Treatments for Psychiatric Disorders
Huneke, N. T. M., Veronesi, G. F., Garner, M. J., Baldwin, D. S., & Cortese, S. (2025) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2025.0085 · Cited by 68
Insulin secretagogues for prevention or delay of type 2 diabetes mellitus and its associated complications in persons at increased risk for the development of type 2 diabetes mellitus
Hemmingsen, B., Sonne, D. P., Metzendorf, M., & Richter, B. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012151.pub2 · Cited by 76
Hypofractionation for clinically localized prostate cancer
Hickey, B. E., James, M. L., Daly, T., Soh, F.-Y., & Jeffery, M. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011462.pub2 · Cited by 63
Interventions for treatment of COVID-19: Second edition of a living systematic review with meta-analyses and trial sequential analyses (The LIVING Project)
Juul, S., Nielsen, E. E., Feinberg, J. B., Siddiqui, F., Kamp, C. B., Barot, E., Holgersson, J., Nielsen, N., Bentzer, P., Veroniki, A. A., et al. (2021) · PLoS ONE · doi:10.1371/journal.pone.0248132 · Cited by 57
Levothyroxine or minimally invasive therapies for benign thyroid nodules
Bandeira-Echtler, E., Bergerhoff, K., & Richter, B. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004098.pub2 · Cited by 60
Minimally invasive parathyroidectomy guided by intraoperative parathyroid hormone monitoring (IOPTH) and preoperative imaging versus bilateral neck exploration for primary hyperparathyroidism in adults
Ahmadieh, H., Kreidieh, O., Akl, E. A., & Fuleihan, G. E. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010787.pub2 · Cited by 61
Fool’s gold? Why blinded trials are not always best
Anand, Norrie, Bradley, McAuley, & Clarke (2020) · BMJ · doi:10.1136/bmj.l6228 · Cited by 58
EULAR recommendations for the reporting of ultrasound studies in rheumatic and musculoskeletal diseases (RMDs)
Costantino, F., Carmona, L., Boers, M., Backhaus, M., Balint, P. V., Bruyn, G. A. W., Christensen, R., Conaghan, P. G., Ferreira, R. J. O., Garrido-Castro, J. L., et al. (2021) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2020-219816 · Cited by 58
Compelling evidence from meta-epidemiological studies demonstrates overestimation of effects in randomized trials that fail to optimize randomization and blind patients and outcome assessors
Wang, Y., Parpia, S., Couban, R., Wang, Q., Armijo-Olivo, S., Bassler, D., Briel, M., Brignardello-Petersen, R., Gluud, L. L., Keitz, S. A., et al. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.11.001 · Cited by 58
Self-citation
Alpha-glucosidase inhibitors for prevention or delay of type 2 diabetes mellitus and its associated complications in people at increased risk of developing type 2 diabetes mellitus
Moelands, S. V., Lucassen, P. L., Akkermans, R. P., De Grauw, W. J., & Van de Laar, F. A. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005061.pub3 · Cited by 57
The effect of perioperative anesthetics for prevention of postoperative delirium on general anesthesia: A network meta-analysis
Cui, Y., Li, G., Cao, R., Luan, L., & Kla, K. M. (2019) · Journal of Clinical Anesthesia · doi:10.1016/j.jclinane.2019.06.028 · Cited by 59
A systematic review of the effects of acupuncture on xerostomia and hyposalivation
Assy, Z., & Brand, H. S. (2018) · BMC Complementary and Alternative Medicine · doi:10.1186/s12906-018-2124-x · Cited by 55
Predicting cognitive behavioral therapy outcome in the outpatient sector based on clinical routine data: A machine learning approach
Hilbert, K., Kunas, S. L., Lueken, U., Kathmann, N., Fydrich, T., & Fehm, L. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2019.103530 · Cited by 54
Qualitative evaluation of artificial intelligence-generated weight management diet plans
Kim, D. W., Park, J. S., Sharma, K., Velazquez, A., Li, L., Ostrominski, J. W., Tran, T., Peréz, R. H. S., & Shin, J. (2024) · Frontiers in Nutrition · doi:10.3389/fnut.2024.1374834 · Cited by 54
Raising Awareness for the Replication Crisis in Clinical Psychology by Focusing on Inconsistencies in Psychotherapy Research: How Much Can We Rely on Published Findings from Efficacy Trials?
Hengartner, M. P. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00256 · Cited by 57
The Effect of Mandala Coloring on State Anxiety: A Systematic Review and Meta-Analysis
Støre, S. J., & Jakobsson, N. (2022) · Art Therapy · doi:10.1080/07421656.2021.2003144 · Cited by 54
Methodological Flaws, Conflicts of Interest, and Scientific Fallacies: Implications for the Evaluation of Antidepressants’ Efficacy and Harm
Hengartner, M. P. (2017) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2017.00275 · Cited by 53
Insulin monotherapy compared with the addition of oral glucose-lowering agents to insulin for people with type 2 diabetes already on insulin therapy and inadequate glycaemic control
Vos, R. C., van Avendonk, M. J., Jansen, H., Goudswaard, A. N. N., van den Donk, M., Gorter, K. J., Kerssen, A., & Rutten, G. E. H. M. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006992.pub2 · Cited by 54
Towards evidence-based medicine for paediatricians
Phillips, B. (2018) · Archives of Disease in Childhood · doi:10.1136/archdischild-2018-315014 · Cited by 50

Authors: Hróbjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Hilden, J., Boutron, I., Ravaud, P., & Brorson, S.
Affiliations: University of Copenhagen (DK)
ORCID: Hróbjartsson, Asbjørn
Thomsen, Ann Sofia Skou
Emanuelsson, Frida
Tendal, Britta
Hilden, Jørgen
Boutron, Isabelle
Ravaud, Philippe
Brorson, Stig
Venue: Canadian Medical Association Journal
Publisher: CMA Impact Inc.
Published: 2013-01-28
Volume: 185
Issue: 4
Pages: E201-E211
ISSN: 0820-3946 (ISSN-L)
1488-2329
DOI: 10.1503/cmaj.120744
PMC: PMC3589328
OpenAlex: W2006175454
Cited by: 527
References: 52
Citations by year: 2026: 30 · 2025: 26 · 2024: 23 · 2023: 54 · 2022: 46 · 2021: 43 · 2020: 35 · 2019: 45 · 2018: 51 · 2017: 31 · 2016: 54 · 2015: 37 · 2014: 29 · 2013: 21
FWCI: 9.63
Type: journal article
Language: en
Open access: gold
doi.org
Locations: Canadian Medical Association Journal (publishedVersion)
Europe PMC (PubMed Central) (publishedVersion)
PubMed Central (publishedVersion)
PubMed (publishedVersion)
Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
Funders: U.S. Department of Veterans Affairs
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Keywords: observer bias, subjective outcomes, risk of bias, effect size estimation, bias in clinical trials
MeSH terms: Double-Blind Method, Observer Variation, Randomized Controlled Trials as Topic, Outcome Assessment, Health Care

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks