The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors
Open accessAbstract
BACKGROUND: Clinical trials are commonly done without blinded outcome assessors despite the risk of bias. We wanted to evaluate the effect of nonblinded outcome assessment on estimated effects in randomized clinical trials with outcomes that involved subjective measurement scales. METHODS: We conducted a systematic review of randomized clinical trials with both blinded and nonblinded assessment of the same measurement scale outcome. We searched PubMed, EMBASE, PsycINFO, CINAHL, Cochrane Central Register of Controlled Trials, HighWire Press and Google Scholar for relevant studies. Two investigators agreed on the inclusion of trials and the outcome scale. For each trial, we calculated the difference in effect size (i.e., standardized mean difference between nonblinded and blinded assessments). A difference in effect size of less than 0 suggested that nonblinded assessors generated more optimistic estimates of effect. We pooled the differences in effect size using inverse variance random-effects meta-analysis and used metaregression to identify potential reasons for variation. RESULTS: We included 24 trials in our review. The main meta-analysis included 16 trials (involving 2854 patients) with subjective outcomes. The estimated treatment effect was more beneficial when based on nonblinded assessors (pooled difference in effect size -0.23 [95% confidence interval (CI) -0.40 to -0.06]). In relative terms, nonblinded assessors exaggerated the pooled effect size by 68% (95% CI 14% to 230%). Heterogeneity was moderate (I(2) = 46%, p = 0.02) and unexplained by metaregression. INTERPRETATION: We provide empirical evidence for observer bias in randomized clinical trials with subjective measurement scale outcomes. A failure to blind assessors of outcomes in such trials results in a high risk of substantial bias.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We considered only double-blind trials because we included placebo in the network meta-analysis, and because this study design increases methodological rigour by minimising performance and ascertainment biases.7”
From Comparative efficacy and acceptability of 21 antidepressant drugs for the acute treatment of adults with major depressive disorder: a systematic review and network meta-analysis (2018) · cites this paper as [7]
Experimental — accuracy not yet measuredIn the citing paper
“Trials at high or unclear risk of bias tend to overestimate benefits and underestimate harms.”
From Should antidepressants be used for major depressive disorder? (2020) · cites this paper as [38]
In the citing paper
“We evaluated the risk of performance bias separately for each outcome (Hróbjartsson 2013).”
From Diet, physical activity or both for prevention or delay of type 2 diabetes mellitus and its associated complications in people at increased risk of developing type 2 diabetes mellitus (2017) · cites this paper as Hróbjartsson 2013
In the citing paper
“We evaluated the risk of detection bias separately for each outcome (Hróbjartsson 2013).”
From Diet, physical activity or both for prevention or delay of type 2 diabetes mellitus and its associated complications in people at increased risk of developing type 2 diabetes mellitus (2017) · cites this paper as Hróbjartsson 2013
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Cochrane Handbook for Systematic Reviews of Interventions
Higgins, J. P. T., Green, S., & Ben Van Den, A. (2020) · International Coaching Psychology Review · doi:10.53841/bpsicpr.2020.15.2.123 · Cited by 5,670
Weaver, F. M. (2009) · JAMA · doi:10.1001/jama.2008.929 · Cited by 1,535
Meltzer, H. Y. (2003) · Archives of General Psychiatry · doi:10.1001/archpsyc.60.1.82 · Cited by 1,393
Progressive resistance strength training for improving physical function in older adults
Liu, C.-J., & Latham, N. K. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002759.pub2 · Cited by 1,258
Some Problems in Interval Estimation
Fieller, E. C. (1954) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1954.tb00159.x · Cited by 1,159
Savović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-6-201209180-00537 · Cited by 1,049
Flagged first, then by citations
Research works printed but not linked: 4
- [2]2. Haahr MT, Hróbjartsson A. Who is blind in randomised clinical trials? An analysis of 200 trials and a survey of authors. Clin Tri- als 2006;3:360-5.
- [21]21. Meltzer HY, Alphs L, Green AI, et al. Clozapine treatment for suicidality in schizophrenia: International Suicide Prevention Trial (InterSePT). Arch Gen Psychiatry 2003;60:82-91.
- [34]34. Realmuto GM, Erickson WD, Yellin AM, et al. Clinical compar- ison of thiothixene and thioridazine in schizophrenic adoles- cents. Am J Psychiatry 1984;141:440-2.
- [41]41. Dodd DC. Blind slide reading or the uninformed versus the informed pathologist. Comments Toxicol 1988;2:88-91.
Cited by
Sorted by citations
Cipriani, A., Furukawa, T. A., Salanti, G., Chaimani, A., Atkinson, L. Z., Ogawa, Y., Leucht, S., Ruhe, H. G., Turner, E. H., Higgins, J. P. T., et al. (2018) · The Lancet · doi:10.1016/s0140-6736(17)32802-7 · Cited by 3,541
Jakobsen, J. C., Gluud, C., Wetterslev, J., & Winkel, P. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0442-1 · Cited by 2,747
Efficacy of treatments for anxiety disorders: a meta-analysis
Bandelow, B., Reitt, M., Röver, C., Michaelis, S., Görlich, Y., & Wedekind, D. (2015) · International Clinical Psychopharmacology · doi:10.1097/yic.0000000000000078 · Cited by 555
Mead, E., Brown, T. J., Rees, K., Azevedo, L. B., Whittaker, V., Jones, D., Olajide, J., Mainardi, G. M., Corpeleijn, E., O’Malley, C. L., et al. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012651 · Cited by 550
Compression stockings to prevent post-thrombotic syndrome: a randomised placebo-controlled trial
Kahn, S. R., Shapiro, S., Wells, P. S., Rodger, M. A., Kovacs, M. J., Anderson, D. R., Tagalakis, V., Houweling, A. H., Ducruet, T., Holcroft, C., et al. (2013) · The Lancet · doi:10.1016/s0140-6736(13)61902-9 · Cited by 521
Metadata
- Authors
- Hróbjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Hilden, J., Boutron, I., Ravaud, P., & Brorson, S.
- Affiliations
- University of Copenhagen (DK)
- ORCID
- Venue
- Canadian Medical Association Journal
- Publisher
- CMA Impact Inc.
- Published
- 2013-01-28
- Volume
- 185
- Issue
- 4
- Pages
- E201-E211
- ISSN
- 0820-3946 (ISSN-L)
- 1488-2329
- PMC
- PMC3589328
- OpenAlex
- W2006175454
- Cited by
- 527
- References
- 52
- Citations by year
- 2026: 30 · 2025: 26 · 2024: 23 · 2023: 54 · 2022: 46 · 2021: 43 · 2020: 35 · 2019: 45 · 2018: 51 · 2017: 31 · 2016: 54 · 2015: 37 · 2014: 29 · 2013: 21
- FWCI
- 9.63
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Canadian Medical Association Journal (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
- Funders
- U.S. Department of Veterans Affairs
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- observer bias, subjective outcomes, risk of bias, effect size estimation, bias in clinical trials
- MeSH terms
- Double-Blind Method, Observer Variation, Randomized Controlled Trials as Topic, Outcome Assessment, Health Care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
