CoChat

Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials: a cross-sectional study

Open access

The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.

Abstract

Objective: We examined how assessments of risk of bias of primary studies are carried out and incorporated into the statistical analysis and overall findings of a systematic review. Design: A cross-sectional review. Sample: We assessed 200 systematic reviews of randomised trials published between January and March 2012; Cochrane (n=100), non-Cochrane (Database of Reviews of Effects) (n=100). Main outcomes: Our primary outcome was a descriptive analysis of how assessments of risk of bias are carried out, the methods used, and the extent to which such assessments were incorporated into the statistical analysis and overall review findings. Results: While Cochrane reviews routinely reported the method of risk of bias assessment and presented their results either in text or table format, 20% of non-Cochrane reviews failed to report the method used and 39% did not present the assessment results. Where it was possible to evaluate the individual results of the risk of bias assessment (n=154), 75% (n=116/154) of reviews had ≥1 trial at high risk of bias; the median proportion of trials per review at high risk of bias was 50% (IQR 31% to 89%). Despite this, only 56% (n=65/116) incorporated the risk of bias assessment into the interpretation of the results in the abstract and 41% (n=47/116) (49%; n=40/81 Cochrane and 20%; n=7/35 non-Cochrane) incorporated the risk of bias assessment into the interpretation of the conclusions. Of the 83% (n=166/200) systematic reviews which included a meta-analysis, only 11% (n=19/166) incorporated the risk of bias assessment into the statistical analysis. Conclusions: Cochrane reviews were more likely than non-Cochrane reviews to report how risk of bias assessments of primary studies were carried out; however, both frequently failed to take such assessments into account in the statistical analysis and conclusions of the systematic review.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmjopen-2013-003342

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Inclusion of systematic reviews (published between 1 January to 31 March 2012).
Figure 1 Inclusion of systematic reviews (published between 1 January to 31 March 2012).CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 22 references

Research works printed but not linked: 1

  1. Schulz KF Grimes DA . The Lancet hadbook of essential concepts in clincal research. London: Elsevier 2006.

Cited by

See all 76

Metadata

Authors
Hopewell, S., Boutron, I., Altman, D. G., & Ravaud, P.
Affiliations
  • Inserm (FR)
  • University of Oxford (GB)
Corresponding author
Hopewell, Sally
Venue
BMJ Open
Publisher
BMJ
Published
2013-08
Volume
3
Issue
8
Pages
e003342
ISSN
2044-6055 (ISSN-L)
OpenAlex
W2121498030
Cited by
66
References
21
Citations by year
2026: 5 · 2025: 5 · 2024: 4 · 2023: 1 · 2022: 6 · 2021: 4 · 2020: 6 · 2019: 6 · 2018: 2 · 2017: 1 · 2016: 9 · 2015: 8 · 2014: 8 · 2013: 1
FWCI
1.78
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • BMJ Open (publishedVersion, cc-by-nc)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
Keywords
risk of bias assessment, systematic reviews, randomised trials, reporting quality, meta-analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials: a cross-sectional study. https://app.cochat.ai/papers/10.1136/bmjopen-2013-003342
The paper
Hopewell, S., Boutron, I., Altman, D. G., & Ravaud, P. (2013). Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials: a cross-sectional study. BMJ Open. https://doi.org/10.1136/bmjopen-2013-003342

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.