CoChat

Toward a comprehensive evidence map of overview of systematic review methods: paper 2—risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence

Open access

The DOI matches its Crossref record; 95 of the 96 references it cites were found in a registry or an index.

Abstract

BACKGROUND: Overviews of systematic reviews (SRs) attempt to systematically retrieve and summarise the results of multiple systematic reviews. This is the second of two papers from a study aiming to develop a comprehensive evidence map of the methods used in overviews. Our objectives were to (a) develop a framework of methods for conducting, interpreting and reporting overviews (stage I)-the Methods for Overviews of Reviews (MOoR) framework-and (b) to create an evidence map by mapping studies that have evaluated overview methods to the framework (stage II). In the first paper, we reported findings for the four initial steps of an overview (specification of purpose, objectives and scope; eligibility criteria; search methods; data extraction). In this paper, we report the remaining steps: assessing risk of bias; synthesis, presentation and summary of the findings; and assessing certainty of the evidence arising from the overview. METHODS: In stage I, we identified cross-sectional studies, guidance documents and commentaries that described methods proposed for, or used in, overviews. Based on these studies, we developed a framework of possible methods for overviews, categorised by the steps in conducting an overview. Multiple iterations of the framework were discussed and refined by all authors. In stage II, we identified studies evaluating methods and mapped these evaluations to the framework. RESULTS: Forty-two stage I studies described methods relevant to one or more of the latter steps of an overview. Six studies evaluating methods were included in stage II. These mapped to steps involving (i) the assessment of risk of bias (RoB) in SRs (two SRs and three primary studies, all reporting evaluation of RoB tools) and (ii) the synthesis, presentation and summary of the findings (one primary study evaluating methods for measuring overlap). CONCLUSION: Many methods have been described for use in the latter steps in conducting an overview; however, evaluation and guidance for applying these methods is sparse. The exception is RoB assessment, for which a multitude of tools exist-several with sufficient evaluation and guidance to recommend their use. Evaluation of other methods is required to provide a comprehensive evidence map.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13643-018-0784-8

Cites

95 of 95 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Summary of the research reported in each paper
Fig. 1 Summary of the research reported in each paperCC BY · PubMed Central
Fig. 2 Stages in the development of an evidence map of overview methods
Fig. 2 Stages in the development of an evidence map of overview methodsCC BY · PubMed Central
Fig. 3 Flowchart of the main search for stages I and II studies
Fig. 3 Flowchart of the main search for stages I and II studiesCC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 95 references

Research works printed but not linked: 18

  1. Whiting P, Davies P, Savović J, Caldwell D, Churchill R. Chapter 4. Phase 2: review of existing quality assessment tools for systematic reviews. Evidence to inform the development of ROBIS, a new tool to assess the risk of bias in systematic reviews, Available from http://www.robis-tool.info [accessed 20/11/2017]; 2013. p. 22–35.
  2. Bai A, Shukla VK, Bak G, Wells G. Chapter 4: tools selected through QAT project. In: quality assessment tools project report. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2012.
  3. Whiting P, Davies, P., Savović, J., Caldwell, D., Churchill, R.: Chapter 5. Phase 3: review of studies that have used the AMSTAR tool. Evidence to inform the development of ROBIS, a new tool to assess the risk of bias in systematic reviews, Available from http://www.robis-tool.info [accessed 20/11/2017]. 2013.
  4. Parmelli E, Banzi R, Fernandez Del Rio MDP, Minozzi S, Moja L, Pecoraro V, Liberati A: Using AMSTAR to assess the methodological quality of systematic reviews: an external validation study. Poster presentation at the 19th Cochrane Colloquium; 2011 Oct 19-22; Madrid, Spain [abstract]. In Cochrane Database Syst Rev, Supplement, vol. Suppl. pp. 139; 2011:139.
  5. Kung J, Chiappelli F, Cajulis OO, Avezova R, Kossan G, Chew L, et al. From systematic reviews to clinical recommendations for evidence-based health care: validation of revised assessment of multiple systematic reviews (R-AMSTAR) for grading of clinical relevance. Open Dent J. 2010;4:4–91.
  6. Cochrane Methods Group. About the Cochrane Methodology Register: Cochrane; 2012. http://www.cochranelibrary.com/help/the-cochrane-methodology-register-july-issue-2012.html
  7. CMIMG C: Review Type & Methodological Considerations --Background Paper for the First Part of the Paris CMIMG Discussion. 2012.
  8. Foisy MFR, Dryden DM, Hartling L. Grading the quality of evidence in existing systematic reviews: challenges and considerations. In: 22nd Cochrane Colloquium. Hyderabad: Wiley; 2014.
  9. Büchter R, Pieper D, Jerinic P. Overviews of systematic reviews often do not assess methodological quality of included reviews. Poster. In: 19th Cochrane Colloquium, vol. Suppl. Madrid: Cochrane Database Syst Rev; 2011. p. 105–6.
  10. Robinson KA, Chou R, Berkman ND, Newberry SJ, Fu R, Hartling L, Dryden D, Butler M, Foisy M, Anderson J, Motu’apuaka ML, Relevo R, Guise JM, Chang S. Integrating bodies of evidence: existing systematic reviews and primary studies. In: Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville: Agency for Healthcare Research and Quality (US); 2008.
  11. White CM, Ip S, McPheeters MC, Tim S, Chou R, Lohr KN, Robinson K, McDonald K, Whitlock EP. Using existing systematic reviews to replace de novo processes in conducting comparative effectiveness reviews. In Methods Guide for Comparative Effectiveness Reviews. Rockville: Agency for Healthcare Research and. Quality; 2009.
  12. Salanti G, Becker L, Caldwell D, Churchill R, Higgins J, Li T, Schmid C. Evolution of Cochrane Intervention Reviews and Overviews of Reviews to better accommodate comparisons among multiple interventions. In: Report from a meeting of the Cochrane Comparing Multiple Interventions Methods Groups: Cochrane Comparing Multiple Interventions Methods Groups; 2011.
  13. Wagner S, White M, Schultz I, Iverson R, Hsu V, McGuire L, Schultz W. Assessing a systematic review of systematic reviews: developing a criteria. In: Innovation in worker health and safety: Annual Conference, Canadian Association for Research on Work and Health, June 1-2, 2012. Vancouver; 2012. https://www.wwdpi.org/SiteCollectionDocuments/CIRPD-Research/CARWH2012/P3_MethodologicalCriteria.pdf .
  14. McMaster University: Health systems evidence. Available from: http://www.healthsystemsevidence.org /. 2011.
  15. Unit PHR. Critical Appraisal Skills Programme (CASP). 10 questions to help you make sense of reviews. Retrieved from: http://www.casp-uk.net /. Oxford: Public Health Resource Unit; 2006.
  16. FOCUS. FOCUS critical appraisal tool. London: The Royal College of Psychiatrists; 2001.
  17. Joanna Briggs Institute. RAPid: Rapid Appraisal protocol internet database. Adelaide: The Joanna Briggs Institute; 2006.
  18. Task Force on Systematic Review and Guidelines. Assessing the quality and applicability of systematic reviews (AQASR). Available from http://www.ktdrr.org/aqasr . Austin: National Center for the Dissemination of Disability Research; 2011.

Cited by

See all 139

Metadata

Authors
Lunny, C., Brennan, S. E., McDonald, S., & McKenzie, J. E.
Affiliations
Monash University (AU)
Corresponding author
McKenzie, Joanne E.
Venue
Systematic Reviews
Published
2018-10-12
Volume
7
Issue
1
Pages
159-159
ISSN
2046-4053 (ISSN-L)
OpenAlex
W2898047957
Cited by
139
References
96
Citations by year
2026: 7 · 2025: 16 · 2024: 13 · 2023: 29 · 2022: 27 · 2021: 22 · 2020: 15 · 2019: 10
FWCI
2.34
Type
journal article
Language
en
Open access
Licence
  • cc-by
  • creativecommons.org/licenses/by/4.0/ (tdm)
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Monash University Research Portal (Monash University) (acceptedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • Australian Postgraduate Award, International Postgraduate Research Scholarship
  • National Health and Medical Research Council (1143429)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
  • Social Sciences › Sociology and Political Science › Delphi Technique in Research
Keywords
overviews of systematic reviews, risk of bias assessment, quality of evidence, evidence map, methodological evaluation, reporting guidelines
MeSH terms
Systematic Reviews as Topic, Humans, Research Design, Review Literature as Topic, Bias, Evidence-Based Medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Toward a comprehensive evidence map of overview of systematic review methods: paper 2—risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence. https://app.cochat.ai/papers/10.1186/s13643-018-0784-8
The paper
Lunny, C., Brennan, S. E., McDonald, S., & McKenzie, J. E. (2018). Toward a comprehensive evidence map of overview of systematic review methods: paper 2—risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence. Systematic Reviews. https://doi.org/10.1186/s13643-018-0784-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.