CoChat

Evaluation of the Cochrane Collaboration’s tool for assessing the risk of bias in randomized trials: focus groups, online survey, proposed recommendations and their implementation

Open access

The DOI matches its Crossref record; 25 of the 26 references it cites were found in a registry or an index.

Abstract

BACKGROUND: In 2008, the Cochrane Collaboration introduced a tool for assessing the risk of bias in clinical trials included in Cochrane reviews. The risk of bias (RoB) tool is based on narrative descriptions of evidence-based methodological features known to increase the risk of bias in trials. METHODS: To assess the usability of this tool, we conducted an evaluation by means of focus groups, online surveys and a face-to-face meeting. We obtained feedback from a range of stakeholders within The Cochrane Collaboration regarding their experiences with, and perceptions of, the RoB tool and associated guidance materials. We then assessed this feedback in a face-to-face meeting of experts and stakeholders and made recommendations for improvements and further developments of the RoB tool. RESULTS: The survey attracted 380 responses. Respondents reported taking an average of between 10 and 60 minutes per study to complete their RoB assessments, which 83% deemed acceptable. Most respondents (87% of authors and 95% of editorial staff) thought RoB assessments were an improvement over past approaches to trial quality assessment. Most authors liked the standardized approach (81%) and the ability to provide quotes to support judgements (74%). A third of participants disliked the increased workload and found the wording describing RoB judgements confusing. The RoB domains reported to be the most difficult to assess were incomplete outcome data and selective reporting of outcomes. Authors expressed the need for more guidance on how to incorporate RoB assessments into meta-analyses and review conclusions. Based on this evaluation, recommendations were made for improvements to the RoB tool and the associated guidance. The implementation of these recommendations is currently underway. CONCLUSIONS: Overall, respondents identified positive experiences and perceptions of the RoB tool. Revisions of the tool and associated guidance made in response to this evaluation, and improved provision of training, may improve implementation.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/2046-4053-3-37

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 25 references

Research works printed but not linked: 3

  1. Review Manager (RevMan): Version 5.0. Computer Program. 2008, Copenhagen: The Nordic Cochrane Centre, The Cochrane Collaboration
  2. Cochrane Effective Practice and Organisation of Care Review Group: Data Collection Checklist. 2012, Ottawa, ON: Ottawa Hospital Research Institute
  3. Review Manager (RevMan): Version 5.1. Computer Program. 2011, Copenhagen: The Nordic Cochrane Centre, The Cochrane Collaboration

Cited by

See all 346

Metadata

Authors
Savović, J., Weeks, L., Sterne, J. A., Turner, L., Altman, D. G., Moher, D., & Higgins, J. P.
Affiliations
  • University of Bristol (GB)
  • Ottawa Hospital Research Institute (CA)
  • University of Oxford (GB)
  • University of Ottawa (CA)
Corresponding author
Savović, Jelena
Venue
Systematic Reviews
Published
2014-04-15
Volume
3
Issue
1
Pages
37-37
ISSN
2046-4053 (ISSN-L)
OpenAlex
W2102228952
Cited by
340
References
26
Citations by year
2026: 13 · 2025: 37 · 2024: 34 · 2023: 18 · 2022: 29 · 2021: 49 · 2020: 43 · 2019: 36 · 2018: 21 · 2017: 23 · 2016: 16 · 2015: 20 · 2014: 1
FWCI
19.51
Type
journal article
Language
en
Open access
Licence
  • cc-by
  • creativecommons.org/licenses/by/2.0 (tdm)
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Bristol Research (University of Bristol) (submittedVersion)
  • uO Research (University of Ottawa) (submittedVersion, public-domain)
Funders
  • Cancer Research UK (16895, C-5592)
  • National Institute for Health and Care Research (NF-SI-0611-10168)
  • Canadian Institutes of Health Research
  • Medical Research Council (MR/K025643/1, G0701659/1)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Sociology and Political Science › Delphi Technique in Research
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
Keywords
risk of bias, Cochrane risk of bias tool, trial quality assessment, usability evaluation, stakeholder perspectives, selective reporting, meta-analysis
MeSH terms
Data Collection, Humans, Bias, Randomized Controlled Trials as Topic, Focus Groups, Risk Assessment, Evidence-Based Medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Evaluation of the Cochrane Collaboration’s tool for assessing the risk of bias in randomized trials: focus groups, online survey, proposed recommendations and their implementation. https://app.cochat.ai/papers/10.1186/2046-4053-3-37
The paper
Savović, J., Weeks, L., Sterne, J. A., Turner, L., Altman, D. G., Moher, D., & Higgins, J. P. (2014). Evaluation of the Cochrane Collaboration’s tool for assessing the risk of bias in randomized trials: focus groups, online survey, proposed recommendations and their implementation. Systematic Reviews. https://doi.org/10.1186/2046-4053-3-37

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.