Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols
Open accessThe DOI matches its Crossref record; 32 of the 34 references it cites were found in a registry or an index.
Abstract
Background: Systematic reviews of health interventions are increasingly incorporating evidence outside of randomized controlled trials (RCT). While non-randomized study (NRS) types may be more prone to bias compared to RCT, the tools used to evaluate risk of bias (RoB) in NRS are less straightforward and no gold standard tool exists. The objective of this study was to evaluate the planned use of RoB tools in systematic reviews of health interventions, specifically for reviews that planned to incorporate evidence from RCT and/or NRS. Methods: We evaluated a random sample of non-Cochrane protocols for systematic reviews of interventions registered in PROSPERO between January 1 and October 12, 2018. For each protocol, we extracted data on the types of studies to be included (RCT and/or NRS) as well as the name and number of RoB tools planned to be used according to study design. We then conducted a longitudinal analysis of the most commonly reported tools in the random sample. Using keywords and name variants for each tool, we searched PROSPERO records by year since the inception of the database (2011 to December 7, 2018), restricting the keyword search to the “Risk of bias (quality) assessment” field. Results: In total, 471 randomly sampled PROSPERO protocols from 2018 were included in the analysis. About two-thirds (63%) of these planned to include NRS, while 37% restricted study design to RCT or quasi-RCT. Over half of the protocols that planned to include NRS listed only a single RoB tool, most frequently the Cochrane RoB Tool. The Newcastle-Ottawa Scale and ROBINS-I were the most commonly reported tools for NRS (39% and 33% respectively) for systematic reviews that planned to use multiple RoB tools. Looking at trends over time, the planned use of the Cochrane RoB Tool and ROBINS-I seems to be increasing. Conclusions: While RoB tool selection for RCT was consistent, with the Cochrane RoB Tool being the most frequently reported in PROSPERO protocols, RoB tools for NRS varied widely. Results suggest a need for more education and awareness on the appropriate use of RoB tools for NRS. Given the heterogeneity of study designs comprising NRS, multiple RoB tools tailored to specific designs may be required.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Participants mentioned that the first versions of the reviews have a large authorship team while the subsequent versions with smaller changes required a much smaller team.”
From Extension of the PRISMA 2020 statement for living systematic reviews (PRISMA-LSR): checklist and explanation (2024) · cites this paper as [49]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“This led the author teams to question when people should come off the author list, and to request more guidance around this issue. 49”
From Extension of the PRISMA 2020 statement for living systematic reviews (PRISMA-LSR): checklist and explanation (2024) · cites this paper as [49]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“in a mixed methods evaluation with participants involved in Cochrane and non-Cochrane LSRs, participants discussed authorship issues as a complexity in the production of LSRs.”
From Extension of the PRISMA 2020 statement for living systematic reviews (PRISMA-LSR): checklist and explanation (2024) · cites this paper as [49]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
(2014) · Medical Entomology and Zoology · Cited by 17,633
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
Stang, A. (2010) · European Journal of Epidemiology · doi:10.1007/s10654-010-9491-z · Cited by 17,396
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Zhang, Q., Wang, C., Zheng, M., Li, Y., Li, J., Zhang, L., Shang, X., & Yan, C. (2015) · Cerebrovascular Diseases · doi:10.1159/000369778 · Cited by 11,533
Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., et al. (2017) · BMJ · doi:10.1136/bmj.j4008 · Cited by 10,636
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2014) · International Journal of Surgery · doi:10.1016/j.ijsu.2014.07.013 · Cited by 9,909
GRADE guidelines: 3. Rating the quality of evidence
Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., Kunz, R., Brozek, J., Vist, G. E., Falck-Ytter, Y., Meerpohl, J., & Norris, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.015 · Cited by 8,285
Research works printed but not linked: 7
- Jarde A, Losilla J-M, Vives J. Methodological quality assessment tools of non-experimental studies: a systematic review. Ann Psychol. 2012;28(2):617–28.
- Law M, Steward D, Pollock N, Letts L, Bosch J, Westmorland M. Critical review form quantitative studies. Hamilton: McMaster University; 1998. https://srs-mcmaster.ca/wp-content/uploads/2015/04/Critical-Review-Form-Quantitative-Studies-English.pdf. Accessed 8 Feb 2019
- McMaster Evidence Review and Synthesis Team. Effective Public Health Practice Project (EPHPP): quality assessment tool for quantitative studies. Hamilton: McMaster Evidence Review & Synthesis Centre; 2018. https://merst.ca/ephpp/. Accessed 15 Dec 2018
- Centre for Evidence-Based Medicine (CEBM). Oxford Centre for Evidence-based Medicine. Levels of evidence. Oxford: Nuffield Department of Primary Care Health Sciences; 2009. https://www.cebm.net/2009/06/oxford-centre-evidence-based-medicine-levels-evidence-march-2009/. Accessed 15 Oct 2018
- Critical Appraisal Skills Programme (CASP). CASP checklists. Oxford: CASP; 2018. https://casp-uk.net/casp-tools-checklists/. Accessed 15 Oct 2018
- Effective Practice and Organisation of Care (EPOC) Group. EPOC resources for review authors: Cochrane; 2017. https://epoc.cochrane.org/resources/epoc-resources-review-authors. Accessed 25 Jan 2019
- Public Health Ontario. MetaQAT - critical appraisal tool. Toronto: Ontario Agency for Health Protection and Promotion; 2018. https://www.publichealthontario.ca/en/ServicesAndTools/Pages/Critical-Appraisal-Tool.aspx. Accessed 15 Oct 2018
Cited by
COVID-19 in 7780 pediatric patients: A systematic review
Hoang, A., Chorath, K., Moreira, A., Evans, M., Burmeister-Morton, F., Burmeister, F., Naqvi, R., Petershack, M., & Moreira, A. (2020) · eClinicalMedicine · doi:10.1016/j.eclinm.2020.100433 · Cited by 533
Hickin, N., Käll, A., Shafran, R., Sutcliffe, S., Manzotti, G., & Langan, D. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2021.102066 · Cited by 271
Igelström, E., Campbell, M., Craig, P., & Katikireddi, S. V. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.08.022 · Cited by 186
Delgado, A. H. S., Sauro, S., Lima, A. F., Loguércio, A. D., Bona, Á. D., Mazzoni, A., Collares, F. M., Staxrud, F., Ferracane, J. L., Tsoi, J. K. H., et al. (2022) · Journal of Dentistry · doi:10.1016/j.jdent.2022.104350 · Cited by 163
Casolino, R., Paiella, S., Azzolina, D., Beer, P. A., Corbo, V., Lorenzoni, G., Gregori, D., Golan, T., Braconi, C., Froeling, F. E. M., et al. (2021) · Journal of Clinical Oncology · doi:10.1200/jco.20.03238 · Cited by 158
Metadata
- Authors
- Farrah, K., Young, K., Tunis, M. C., & Zhao, L.
- Affiliations
- Public Health Agency of Canada (CA)
- Corresponding author
- Zhao, Linlu
- Venue
- Systematic Reviews
- Published
- 2019-11-15
- Volume
- 8
- Issue
- 1
- Pages
- 280-280
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC6857304
- OpenAlex
- W2984078042
- Cited by
- 246
- References
- 34
- Citations by year
- 2026: 16 · 2025: 39 · 2024: 29 · 2023: 49 · 2022: 45 · 2021: 61 · 2020: 7
- FWCI
- 5.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- uO Research (University of Ottawa) (submittedVersion, public-domain)
- Funders
- Public Health Agency of Canada
- Public Health Agency
- Health Canada
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- risk of bias, risk of bias assessment tools, non-randomized studies, Prospero, Cochrane risk of bias tool, ROBINS-I, Newcastle-Ottawa Scale, health interventions, randomized controlled trials, quality assessment
- MeSH terms
- Systematic Reviews as Topic, Humans, Registries, Research Design, Bias, Selection Bias, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols. https://app.cochat.ai/papers/10.1186/s13643-019-1172-8
- The paper
- Farrah, K., Young, K., Tunis, M. C., & Zhao, L. (2019). Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols. Systematic Reviews. https://doi.org/10.1186/s13643-019-1172-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.