CoChat

Disagreements in risk of bias assessment for randomised controlled trials included in more than one Cochrane systematic reviews: a research on research study using cross-sectional design

Open access
Bertizzolo, Lorenzo, Bossuyt, Patrick, Atal, Ignacio, Ravaud, Philippe, Dechartres, Agnes · 2019 · BMJ Open
Cited by 35

The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.

Abstract

Objectives: Assess the frequency and reasons for disagreements in risk of bias assessments for randomised controlled trials (RCTs) included in more than one Cochrane review. Design: Research on research study, using cross-sectional design. Data sources: 2796 Cochrane reviews published between March 2011 and September 2014. Data selection: RCTs included in more than one review. Data extraction: Risk of bias assessment and support for judgement for five key risk of bias items. Data synthesis: For each item, we compared risk of bias assessment made in each review and calculated proportion of agreement. Two reviewers independently analysed 50% of all disagreements by comparing support for each judgement with information from study report to evaluate whether disagreements were related to a difference in information (eg, contact the study author) or a difference in interpretation (same support for judgement but different interpretation). They also identified main reasons for different interpretation. Results: 1604 RCTs were included in more than one review. Proportion of agreement ranged from 57% (770/1348 trials) for incomplete outcome data to 81% for random sequence generation (1193/1466). Most common source of disagreement was difference in interpretation of the same information, ranging from 65% (88/136) for random sequence generation to 90% (56/62) for blinding of participants and personnel. Access to different information explained 32/136 (24%) disagreements for random sequence generation and 38/205 (19%) for allocation concealment. Disagreements related to difference in interpretation were frequently related to incomplete or unclear reporting in the study report (83% of disagreements related to different interpretation for random sequence generation). Conclusions: Risk of bias judgements of RCTs included in more than one Cochrane review differed substantially. Most disagreements were related to a difference in interpretation of an incomplete or unclear description in the study report. A clearer guidance on common causes of incomplete information may improve agreement.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmjopen-2018-028382

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 12

  1. Hrobjartsson, 2013, Assessing risk of bias in randomised clinical trials included in Cochrane Reviews: the why is easy, the how is a challenge, Cochrane Database Syst Rev, 4, ED000058
  2. 10.1136/bmj.d5928 · DOI · Paper page
  3. 10.1001/jama.1995.03520290060030 · DOI
  4. 10.1016/S0140-6736(98)01085-X · DOI · Paper page
  5. Higgins JPT , Altman DG , Sterne JAC . Chapter 8: Assessing risk of bias in included studies. In: Higgins JP , Green S , eds. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 (updated March 2011) ed: The Cochrane Collaboration, 2011. www.handbook.cochrane.org.
  6. Hartling, 2011, Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma, PLoS One, 6, 10.1371/journal.pone.0017242 · DOI · Paper page
  7. Jordan, 2017, There were large discrepancies in risk of bias tool judgments when a randomized controlled trial appeared in more than one systematic review, J Clin Epidemiol, 81, 72, 10.1016/j.jclinepi.2016.08.012 · DOI
  8. Wilkins, 2017, Risk of bias in assessing Risk of Bias, Ophthalmic Physiol Opt, 37, 107, 10.1111/opo.12333 · DOI
  9. Review Manager (RevMan) [Computer program] [program]. Version 5.3 version. Copenhagen: The Nordic Cochrane Centre, The Cochrane Collaboration, 2014.
  10. deMelo VV . Conference: Advances in Logic Based Intelligent Systems, 2005.
  11. Propadalo I , Tranfic M , Vuka I , et al . In Cochrane reviews, risk of bias assessments for allocation concealment were frequently not in line with Cochrane’s Handbook guidance. J Clin Epidemiol 2019;106.doi:10.1016/j.jclinepi.2018.10.002 · DOI · Paper page
  12. 10.1002/sim.5402 · DOI · Paper page

Cited by

See all 35

Metadata

Authors
Bertizzolo, L., Bossuyt, P., Atal, I., Ravaud, P., & Dechartres, A.
Venue
BMJ Open
Publisher
BMJ
Published
2019-04
Volume
9
Issue
4
Pages
e028382
ISSN
2044-6055 (ISSN-L)
OpenAlex
W2928947675
Cited by
35
References
16
FWCI
0.61
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.