CoChat

Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols

Open access
Farrah, Kelly, Young, Kelsey, Tunis, Matthew C., Zhao, Linlu · 2019 · Systematic Reviews

The DOI matches its Crossref record; 32 of the 34 references it cites were found in a registry or an index.

Abstract

Background: Systematic reviews of health interventions are increasingly incorporating evidence outside of randomized controlled trials (RCT). While non-randomized study (NRS) types may be more prone to bias compared to RCT, the tools used to evaluate risk of bias (RoB) in NRS are less straightforward and no gold standard tool exists. The objective of this study was to evaluate the planned use of RoB tools in systematic reviews of health interventions, specifically for reviews that planned to incorporate evidence from RCT and/or NRS. Methods: We evaluated a random sample of non-Cochrane protocols for systematic reviews of interventions registered in PROSPERO between January 1 and October 12, 2018. For each protocol, we extracted data on the types of studies to be included (RCT and/or NRS) as well as the name and number of RoB tools planned to be used according to study design. We then conducted a longitudinal analysis of the most commonly reported tools in the random sample. Using keywords and name variants for each tool, we searched PROSPERO records by year since the inception of the database (2011 to December 7, 2018), restricting the keyword search to the “Risk of bias (quality) assessment” field. Results: In total, 471 randomly sampled PROSPERO protocols from 2018 were included in the analysis. About two-thirds (63%) of these planned to include NRS, while 37% restricted study design to RCT or quasi-RCT. Over half of the protocols that planned to include NRS listed only a single RoB tool, most frequently the Cochrane RoB Tool. The Newcastle-Ottawa Scale and ROBINS-I were the most commonly reported tools for NRS (39% and 33% respectively) for systematic reviews that planned to use multiple RoB tools. Looking at trends over time, the planned use of the Cochrane RoB Tool and ROBINS-I seems to be increasing. Conclusions: While RoB tool selection for RCT was consistent, with the Cochrane RoB Tool being the most frequently reported in PROSPERO protocols, RoB tools for NRS varied widely. Results suggest a need for more education and awareness on the appropriate use of RoB tools for NRS. Given the heterogeneity of study designs comprising NRS, multiple RoB tools tailored to specific designs may be required.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13643-019-1172-8

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Trends over time for the most frequently cited RoB tools in the included 2018 PROSPERO protocols by type of tool. Percentage of total non-Cochrane systematic review protocols on interventions in PROSPERO, by year, for tools for RCT (a), tools for NRS (b), multi-design tools (c), and suites of tools (d). CASP = Critical Appraisal Skills Program; EPHPP = Effective Public Health Practice Project tool; JBI = Joanna Briggs Institute; MINORS = Methodological Index for Non-Randomized Studies; MMAT = Mixed Methods Assessment Tool; NHLBI = National Heart, Lung, and Blood Institute (National Institutes of Health); NOS = Newcastle-Ottawa Scale; NRS = non-randomized studies; PEDro = Physiotherapy Evidence Database; RCT = randomized controlled trial; RoB = risk of bias; ROBINS-I = Risk Of Bias In Non-randomized Studies - of Interventions. Limits: intervention reviews; exclude: Cochrane protocols; restrict to field: assessment of bias
Fig. 1 Trends over time for the most frequently cited RoB tools in the included 2018 PROSPERO protocols by type of tool. Percentage of total non-Cochrane systematic review protocols on interventions in PROSPERO, by year, for tools for RCT (a), tools for NRS (b), multi-design tools (c), and suites of tools (d). CASP = Critical Appraisal Skills Program; EPHPP = Effective Public Health Practice Project tool; JBI = Joanna Briggs Institute; MINORS = Methodological Index for Non-Randomized Studies; MMAT = Mixed Methods Assessment Tool; NHLBI = National Heart, Lung, and Blood Institute (National Institutes of Health); NOS = Newcastle-Ottawa Scale; NRS = non-randomized studies; PEDro = Physiotherapy Evidence Database; RCT = randomized controlled trial; RoB = risk of bias; ROBINS-I = Risk Of Bias In Non-randomized Studies - of Interventions. Limits: intervention reviews; exclude: Cochrane protocols; restrict to field: assessment of biasCC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 32 references

Research works printed but not linked: 7

  1. Jarde A, Losilla J-M, Vives J. Methodological quality assessment tools of non-experimental studies: a systematic review. Ann Psychol. 2012;28(2):617–28.
  2. Law M, Steward D, Pollock N, Letts L, Bosch J, Westmorland M. Critical review form quantitative studies. Hamilton: McMaster University; 1998. https://srs-mcmaster.ca/wp-content/uploads/2015/04/Critical-Review-Form-Quantitative-Studies-English.pdf. Accessed 8 Feb 2019
  3. McMaster Evidence Review and Synthesis Team. Effective Public Health Practice Project (EPHPP): quality assessment tool for quantitative studies. Hamilton: McMaster Evidence Review & Synthesis Centre; 2018. https://merst.ca/ephpp/. Accessed 15 Dec 2018
  4. Centre for Evidence-Based Medicine (CEBM). Oxford Centre for Evidence-based Medicine. Levels of evidence. Oxford: Nuffield Department of Primary Care Health Sciences; 2009. https://www.cebm.net/2009/06/oxford-centre-evidence-based-medicine-levels-evidence-march-2009/. Accessed 15 Oct 2018
  5. Critical Appraisal Skills Programme (CASP). CASP checklists. Oxford: CASP; 2018. https://casp-uk.net/casp-tools-checklists/. Accessed 15 Oct 2018
  6. Effective Practice and Organisation of Care (EPOC) Group. EPOC resources for review authors: Cochrane; 2017. https://epoc.cochrane.org/resources/epoc-resources-review-authors. Accessed 25 Jan 2019
  7. Public Health Ontario. MetaQAT - critical appraisal tool. Toronto: Ontario Agency for Health Protection and Promotion; 2018. https://www.publichealthontario.ca/en/ServicesAndTools/Pages/Critical-Appraisal-Tool.aspx. Accessed 15 Oct 2018

Cited by

See all 272

Metadata

Authors
Farrah, K., Young, K., Tunis, M. C., & Zhao, L.
Affiliations
Public Health Agency of Canada (CA)
Corresponding author
Zhao, Linlu
Venue
Systematic Reviews
Published
2019-11-15
Volume
8
Issue
1
Pages
280-280
ISSN
2046-4053 (ISSN-L)
OpenAlex
W2984078042
Cited by
246
References
34
Citations by year
2026: 16 · 2025: 39 · 2024: 29 · 2023: 49 · 2022: 45 · 2021: 61 · 2020: 7
FWCI
5.72
Type
journal article
Language
en
Open access
Licence
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • uO Research (University of Ottawa) (submittedVersion, public-domain)
Funders
  • Public Health Agency of Canada
  • Public Health Agency
  • Health Canada
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Sociology and Political Science › Delphi Technique in Research
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
Keywords
risk of bias, risk of bias assessment tools, non-randomized studies, Prospero, Cochrane risk of bias tool, ROBINS-I, Newcastle-Ottawa Scale, health interventions, randomized controlled trials, quality assessment
MeSH terms
Systematic Reviews as Topic, Humans, Registries, Research Design, Bias, Selection Bias, Biomedical Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols. https://app.cochat.ai/papers/10.1186/s13643-019-1172-8
The paper
Farrah, K., Young, K., Tunis, M. C., & Zhao, L. (2019). Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols. Systematic Reviews. https://doi.org/10.1186/s13643-019-1172-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.