Toward a comprehensive evidence map of overview of systematic review methods: paper 2—risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence
Open accessThe DOI matches its Crossref record; 95 of the 96 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Overviews of systematic reviews (SRs) attempt to systematically retrieve and summarise the results of multiple systematic reviews. This is the second of two papers from a study aiming to develop a comprehensive evidence map of the methods used in overviews. Our objectives were to (a) develop a framework of methods for conducting, interpreting and reporting overviews (stage I)-the Methods for Overviews of Reviews (MOoR) framework-and (b) to create an evidence map by mapping studies that have evaluated overview methods to the framework (stage II). In the first paper, we reported findings for the four initial steps of an overview (specification of purpose, objectives and scope; eligibility criteria; search methods; data extraction). In this paper, we report the remaining steps: assessing risk of bias; synthesis, presentation and summary of the findings; and assessing certainty of the evidence arising from the overview. METHODS: In stage I, we identified cross-sectional studies, guidance documents and commentaries that described methods proposed for, or used in, overviews. Based on these studies, we developed a framework of possible methods for overviews, categorised by the steps in conducting an overview. Multiple iterations of the framework were discussed and refined by all authors. In stage II, we identified studies evaluating methods and mapped these evaluations to the framework. RESULTS: Forty-two stage I studies described methods relevant to one or more of the latter steps of an overview. Six studies evaluating methods were included in stage II. These mapped to steps involving (i) the assessment of risk of bias (RoB) in SRs (two SRs and three primary studies, all reporting evaluation of RoB tools) and (ii) the synthesis, presentation and summary of the findings (one primary study evaluating methods for measuring overlap). CONCLUSION: Many methods have been described for use in the latter steps in conducting an overview; however, evaluation and guidance for applying these methods is sparse. The exception is RoB assessment, for which a multitude of tools exist-several with sufficient evaluation and guidance to recommend their use. Evaluation of other methods is required to provide a comprehensive evidence map.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The data extraction approaches outlined in the MOoR framework to manage discrepant data across systematic reviews involve retrieving either published or registry reports of the primary studies, or contacting systematic review or primary study authors, or both, for clarification regarding discrepancies [7, 16].”
From Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews (2021) · cites this paper as [16]
Experimental — accuracy not yet measuredIn the citing paper
“A systematic map of all methods used in overviews, called the MOoR framework, identified nine methods to manage the overlap used across four steps in the conduct of an overview [7, 16].”
From Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews (2021) · cites this paper as [16]
In the citing paper
“Discrepant and overlapping risk of bias assessments across systematic reviews can be resolved by: (a) extracting the risk of bias assessment of primary studies from the included systematic reviews, using data extraction approaches to manage missing, flawed assessments, or discrepant assessments of the same primary study; or (b) re-assessing all primary studies using a common risk of bias tool.”
From Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews (2021) · cites this paper as [16]
In the citing paper
“However, approaches for assessing reporting bias in overviews of reviews have been catalogued.16”
From Reporting guideline for overviews of reviews of healthcare interventions: development of the PRIOR statement. (2022) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305
Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., et al. (2017) · BMJ · doi:10.1136/bmj.j4008 · Cited by 10,636 · cited 1×
“One recent example is the publication of AMSTAR 2 [41].”
Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
Aromataris, E., Fernandez, R., Godfrey, C. M., Holly, C., Khalil, H., & Tungpunkom, P. (2015) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000055 · Cited by 2,587 · cited 3×
“…how to present the RoB assessments for SRs 1.3.1 Display assessments in table(s) (e.g. overall rating in summary of findings table, and another table with RoB items for each SR) Aromataris 2015 [39, 59]; Becker [4]; Chen 2014 [46]; Hartling 2012 [53]; Smith 2011 [77] 1.3.2 Display assessments…”
InfluentialROBIS: A new tool to assess risk of bias in systematic reviews was developed
Whiting, P., Savović, J., Higgins, J. P. T., Caldwell, D. M., Reeves, B. C., Shea, B., Davies, P., Kleijnen, J., & Churchill, R. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.06.005 · Cited by 2,272
Shea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., Henry, D. A., & Boers, M. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.10.009 · Cited by 1,755
Industry sponsorship and research outcome
Lundh, A., Lexchin, J., Mintzes, B., Schroll, J. B., & Bero, L. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub3 · Cited by 1,697
Summing Up: The Science of Reviewing Research
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,427
Systematic review finds overlapping reviews were not mentioned in every other overview
Pieper, D., Antoine, S.-L., Mathes, T., Neugebauer, E. A. M., & Eikermann, M. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.11.007 · Cited by 1,318 · cited 3×
“Of the four primary studies evaluating methods [15, 19, 34, 35]: (i) none referred to a study protocol or noted the existence of one, (ii) three used convenience samples as a method to select the sample of SRs to which the tool/measure was applied, (iii) the three studies that evaluated RoB tools either used a conveni…”
Meta-Analyses of Randomized Controlled Trials
Sacks, H. S., Berrier, J., Reitman, D., Ancona-Berk, V. A., & Chalmers, T. C. (1987) · New England Journal of Medicine · doi:10.1056/nejm198702193160806 · Cited by 1,250
Research works printed but not linked: 18
- Whiting P, Davies P, Savović J, Caldwell D, Churchill R. Chapter 4. Phase 2: review of existing quality assessment tools for systematic reviews. Evidence to inform the development of ROBIS, a new tool to assess the risk of bias in systematic reviews, Available from http://www.robis-tool.info [accessed 20/11/2017]; 2013. p. 22–35.
- Bai A, Shukla VK, Bak G, Wells G. Chapter 4: tools selected through QAT project. In: quality assessment tools project report. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2012.
- Whiting P, Davies, P., Savović, J., Caldwell, D., Churchill, R.: Chapter 5. Phase 3: review of studies that have used the AMSTAR tool. Evidence to inform the development of ROBIS, a new tool to assess the risk of bias in systematic reviews, Available from http://www.robis-tool.info [accessed 20/11/2017]. 2013.
- Parmelli E, Banzi R, Fernandez Del Rio MDP, Minozzi S, Moja L, Pecoraro V, Liberati A: Using AMSTAR to assess the methodological quality of systematic reviews: an external validation study. Poster presentation at the 19th Cochrane Colloquium; 2011 Oct 19-22; Madrid, Spain [abstract]. In Cochrane Database Syst Rev, Supplement, vol. Suppl. pp. 139; 2011:139.
- Kung J, Chiappelli F, Cajulis OO, Avezova R, Kossan G, Chew L, et al. From systematic reviews to clinical recommendations for evidence-based health care: validation of revised assessment of multiple systematic reviews (R-AMSTAR) for grading of clinical relevance. Open Dent J. 2010;4:4–91.
- Cochrane Methods Group. About the Cochrane Methodology Register: Cochrane; 2012. http://www.cochranelibrary.com/help/the-cochrane-methodology-register-july-issue-2012.html
- CMIMG C: Review Type & Methodological Considerations --Background Paper for the First Part of the Paris CMIMG Discussion. 2012.
- Foisy MFR, Dryden DM, Hartling L. Grading the quality of evidence in existing systematic reviews: challenges and considerations. In: 22nd Cochrane Colloquium. Hyderabad: Wiley; 2014.
- Büchter R, Pieper D, Jerinic P. Overviews of systematic reviews often do not assess methodological quality of included reviews. Poster. In: 19th Cochrane Colloquium, vol. Suppl. Madrid: Cochrane Database Syst Rev; 2011. p. 105–6.
- Robinson KA, Chou R, Berkman ND, Newberry SJ, Fu R, Hartling L, Dryden D, Butler M, Foisy M, Anderson J, Motu’apuaka ML, Relevo R, Guise JM, Chang S. Integrating bodies of evidence: existing systematic reviews and primary studies. In: Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville: Agency for Healthcare Research and Quality (US); 2008.
- White CM, Ip S, McPheeters MC, Tim S, Chou R, Lohr KN, Robinson K, McDonald K, Whitlock EP. Using existing systematic reviews to replace de novo processes in conducting comparative effectiveness reviews. In Methods Guide for Comparative Effectiveness Reviews. Rockville: Agency for Healthcare Research and. Quality; 2009.
- Salanti G, Becker L, Caldwell D, Churchill R, Higgins J, Li T, Schmid C. Evolution of Cochrane Intervention Reviews and Overviews of Reviews to better accommodate comparisons among multiple interventions. In: Report from a meeting of the Cochrane Comparing Multiple Interventions Methods Groups: Cochrane Comparing Multiple Interventions Methods Groups; 2011.
- Wagner S, White M, Schultz I, Iverson R, Hsu V, McGuire L, Schultz W. Assessing a systematic review of systematic reviews: developing a criteria. In: Innovation in worker health and safety: Annual Conference, Canadian Association for Research on Work and Health, June 1-2, 2012. Vancouver; 2012. https://www.wwdpi.org/SiteCollectionDocuments/CIRPD-Research/CARWH2012/P3_MethodologicalCriteria.pdf .
- McMaster University: Health systems evidence. Available from: http://www.healthsystemsevidence.org /. 2011.
- Unit PHR. Critical Appraisal Skills Programme (CASP). 10 questions to help you make sense of reviews. Retrieved from: http://www.casp-uk.net /. Oxford: Public Health Resource Unit; 2006.
- FOCUS. FOCUS critical appraisal tool. London: The Royal College of Psychiatrists; 2001.
- Joanna Briggs Institute. RAPid: Rapid Appraisal protocol internet database. Adelaide: The Joanna Briggs Institute; 2006.
- Task Force on Systematic Review and Guidelines. Assessing the quality and applicability of systematic reviews (AQASR). Available from http://www.ktdrr.org/aqasr . Austin: National Center for the Dissemination of Disability Research; 2011.
Cited by
Gates, M., Gates, A., Pieper, D., Fernandes, R. M., Tricco, A. C., Moher, D., Brennan, S. E., Li, T., Pollock, M., Lunny, C., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070849 · Cited by 856
Self-citationHuman health impacts of exposure to phthalate plasticizers: An overview of reviews
Eales, J., Bethel, A., Galloway, T., Hopkinson, P., Morrissey, K., Short, R. E., & Garside, R. (2022) · Environment International · doi:10.1016/j.envint.2021.106903 · Cited by 367
Prehabilitation in adult patients undergoing surgery: an umbrella review of systematic reviews
McIsaac, D. I., Gill, M., Boland, L., Hutton, B., Branje, K., Shaw, J., Grudzinski, A. L., Barone, N., Gillis, C., Akhtar, S., et al. (2022) · British Journal of Anaesthesia · doi:10.1016/j.bja.2021.11.014 · Cited by 312
Graphical Representation of Overlap for OVErviews: GROOVE tool
Bracchiglione, J., Meza, N., Bangdiwala, S. I., Niño de Guzmán, E., Urrútia, G., Bonfill, X., & Madrid, E. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1557 · Cited by 299
“10,17,18 Matrices of evidence are grids that link SRs with their included primary studies.”
BackgroundPollock, M., Fernandes, R. M., Pieper, D., Tricco, A. C., Gates, M., Gates, A., & Hartling, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1252-9 · Cited by 228
Metadata
- Authors
- Lunny, C., Brennan, S. E., McDonald, S., & McKenzie, J. E.
- Affiliations
- Monash University (AU)
- Corresponding author
- McKenzie, Joanne E.
- Venue
- Systematic Reviews
- Published
- 2018-10-12
- Volume
- 7
- Issue
- 1
- Pages
- 159-159
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC6186052
- OpenAlex
- W2898047957
- Cited by
- 139
- References
- 96
- Citations by year
- 2026: 7 · 2025: 16 · 2024: 13 · 2023: 29 · 2022: 27 · 2021: 22 · 2020: 15 · 2019: 10
- FWCI
- 2.34
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Monash University Research Portal (Monash University) (acceptedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Australian Postgraduate Award, International Postgraduate Research Scholarship
- National Health and Medical Research Council (1143429)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- overviews of systematic reviews, risk of bias assessment, quality of evidence, evidence map, methodological evaluation, reporting guidelines
- MeSH terms
- Systematic Reviews as Topic, Humans, Research Design, Review Literature as Topic, Bias, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Toward a comprehensive evidence map of overview of systematic review methods: paper 2—risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence. https://app.cochat.ai/papers/10.1186/s13643-018-0784-8
- The paper
- Lunny, C., Brennan, S. E., McDonald, S., & McKenzie, J. E. (2018). Toward a comprehensive evidence map of overview of systematic review methods: paper 2—risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence. Systematic Reviews. https://doi.org/10.1186/s13643-018-0784-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.


