The DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index.
Toward a comprehensive evidence map of overview of systematic review methods: paper 1—purpose, eligibility, search and data extraction
Open accessAbstract
BACKGROUND: Overviews of systematic reviews attempt to systematically retrieve and summarise the results of multiple systematic reviews. Methods for conducting, interpreting and reporting overviews are in their infancy. To date, there has been no evidence map of the methods used in overviews, thus making it difficult to determine the gaps and priorities for methods research. Our objectives were to develop and populate a comprehensive framework of methods for conducting, interpreting and reporting overviews (stage I) and to create an evidence map by mapping studies that have evaluated overview methods to the framework (stage II). METHODS: We searched methods collections (e.g. Cochrane Methodology Register, Meth4ReSyn library, AHRQ Effective Health Care Program) to identify eligible studies for both stages of this research. In stage I, cross-sectional studies, guidance documents and commentaries that described methods proposed for, or used in, overviews were used to develop and populate the framework of methods. Drafts and multiple iterations of the framework were discussed and refined by all authors. In stage II, we identified and described studies evaluating overview methods and mapped these evaluations to the framework. RESULTS: In this paper, we present results for the four initial steps of conducting an overview: (a) specification of the purpose, objectives and scope, (b) specification of the eligibility criteria, (c) search methods and (d) data extraction. Twenty-nine studies mentioned or described methods relevant to one or more of these steps. In the developed framework, identified methods and approaches were grouped according to the steps an overview author would need to undertake. Fifteen studies evaluated identified methods, all of which mapped to the search methods step. These studies either reported the development and evaluation of a new search filter to retrieve systematic reviews or compared the performance of multiple filters. CONCLUSION: Gaps in the evaluation of methods were found for the majority of steps in the framework. More empirical studies are needed to evaluate the methods outlined and provide a comprehensive evidence map. The framework is useful for planning these evaluations and for planning methods required to deal with challenges that arise when conducting an overview.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The data extraction approaches outlined in the MOoR framework to manage discrepant data across systematic reviews involve retrieving either published or registry reports of the primary studies, or contacting systematic review or primary study authors, or both, for clarification regarding discrepancies [7, 16].”
From Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews (2021) · cites this paper as [7]
Experimental — accuracy not yet measuredIn the citing paper
“A systematic map of all methods used in overviews, called the MOoR framework, identified nine methods to manage the overlap used across four steps in the conduct of an overview [7, 16].”
From Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews (2021) · cites this paper as [7]
In the citing paper
“Discrepant and overlapping risk of bias assessments across systematic reviews can be resolved by: (a) extracting the risk of bias assessment of primary studies from the included systematic reviews, using data extraction approaches to manage missing, flawed assessments, or discrepant assessments of the same primary study; or (b) re-assessing all primary studies using a common risk of bias tool.”
From Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews (2021) · cites this paper as [7]
In the citing paper
“The guideline is based on currently available evidence regarding the reporting of published overviews,30 a scoping review of currently available methodological guidance,14 and other recent evidence maps regarding the methods used in overviews of reviews.15 16”
From Reporting guideline for overviews of reviews of healthcare interventions: development of the PRIOR statement. (2022) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scoping studies: towards a methodological framework
Arksey, H., & O'Malley, L. (2005) · International Journal of Social Research Methodology · doi:10.1080/1364557032000119616 · Cited by 36,742 · cited 1×
“Evidence mapping is a systematic method used to characterise and catalogue a body of literature pertaining to evidence on a topic and is useful for identifying gaps in the literature [8, 9].”
MethodAromataris, E., Fernandez, R., Godfrey, C. M., Holly, C., Khalil, H., & Tungpunkom, P. (2015) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000055 · Cited by 2,587 · cited 10×
“5 Define the objectives using PICO elements (or equivalent) to develop an answerable question Baker 2014 [29]; Becker 2008 [1]; Cooper 2012 [32]; Hartling 2012 [35]; JBI 2015 [40, 41]; Li 2012 [44]; Ryan 2009 [53, 54]; Smith 2011 [57]; Robinson 2015 [48–52]; Thomson 2010 [58] CMIMG Comparing Multiple Interventions Met…”
BackgroundMethodInfluentialMethodology in conducting a systematic review of systematic reviews of healthcare interventions
Smith, V., Devane, D., Begley, C. M., & Clarke, M. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-15 · Cited by 1,227
The Joanna Briggs Institute reviewers' manual 2015: methodology for JBI scoping reviews
Peters, M. D. J., Godfrey, C. M., McInerney, P. A., Soares, C. B., Khalil, H., & Parker, D. (2015) · Cited by 1,135
Ioannidis, J. P. A. (2009) · Canadian Medical Association Journal · doi:10.1503/cmaj.081086 · Cited by 716
Optimal search strategies for retrieving systematic reviews from Medline: analytical survey
Montori, V. M., Wilczynski, N. L., Morgan, D., & Haynes, R. B. (2005) · BMJ · doi:10.1136/bmj.38336.804167.47 · Cited by 422
Edwards, P., Clarke, M., DiGuiseppi, C., Pratap, S., Roberts, I., & Wentz, R. (2002) · Statistics in Medicine · doi:10.1002/sim.1190 · Cited by 340
Evidence & Gap Maps: A tool for promoting evidence informed policy and strategic research agendas
Snilstveit, B., Vojtkova, M., Bhavsar, A., Stevenson, J., & Gaarder, M. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.05.015 · Cited by 323
A descriptive analysis of overviews of reviews published between 2000 and 2011.
Hartling, L., Chisholm, A., Thomson, D., & Dryden, D. M. (2012) · PLoS ONE · doi:10.1371/journal.pone.0049667 · Cited by 241 · cited 10×
“5 Define the objectives using PICO elements (or equivalent) to develop an answerable question Baker 2014 [29]; Becker 2008 [1]; Cooper 2012 [32]; Hartling 2012 [35]; JBI 2015 [40, 41]; Li 2012 [44]; Ryan 2009 [53, 54]; Smith 2011 [57]; Robinson 2015 [48–52]; Thomson 2010 [58] CMIMG Comparing Multiple Interventions Met…”
BackgroundMethodInfluentialBecker, L. A., & Oxman, A. D. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch22 · Cited by 218
Research works printed but not linked: 8
- CMIMG C: Review Type & Methodological Considerations --background paper for the first part of the Paris CMIMG discussion. 2012.
- Hartling L, Dryden D, Vandermeer B, Fernandes R. Generating empirical evidence to support methods for overviews of reviews. In: Cochrane Colloquium: 2013; Quebec City, Canada; 2013.
- Kramer S, Langendam M, Elbers R, Scholten R, Hooft L. Preparing an overview of reviews: lessons learned. Poster. In: Cochrane colloquium: 2009 Oct 11–14 2009; Singapore; 2009.
- Buchter R, Pieper D, Jerinic P. Overviews of systematic reviews often do not assess methodological quality of included reviews. Poster. In: 19th Cochrane colloquium. Madrid: Cochrane Database Syst Rev; 2011. p. 105–6.
- Robinson KA, Chou R, Berkman ND, Newberry SJ, Fu R, Hartling L, Dryden D, Butler M, Foisy M, Anderson J, et al. Integrating bodies of evidence: existing systematic reviews and primary studies. In: Methods guide for effectiveness and comparative effectiveness reviews. Rockville: Agency for Healthcare Research and Quality (US); 2008.
- White CM, Ip S, McPheeters MC, Tim S, Chou R, Lohr KN, Robinson K, McDonald K, Whitlock EP. Using existing systematic reviews to replace de novo processes in conducting comparative effectiveness reviews. In: Methods guide for comparative effectiveness reviews. Rockville: Agency for Healthcare Research and Quality; 2009.
- Ryan R, Hill S. Developing an overview of systematic reviews: a framework for synthesising the evidence on interventions to improve communication. In: Cochrane Colloquium: 2005; Melbourne, Australia; 2005.
- Büchter R, Pieper D. How do authors of Cochrane overviews deal with conflicts of interest relating to their own systematic reviews? In: Cochrane Colloquium: 2015; Vienna, Austria; 2015.
Cited by
Gates, M., Gates, A., Pieper, D., Fernandes, R. M., Tricco, A. C., Moher, D., Brennan, S. E., Li, T., Pollock, M., Lunny, C., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070849 · Cited by 856
Self-citationAdverse Outcomes of Polypharmacy in Older People: Systematic Review of Reviews
Davies, L. E., Spiers, G., Kingston, A., Todd, A., Adamson, J., & Hanratty, B. (2020) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2019.10.022 · Cited by 544
Health-related quality of life in breast cancer patients: review of reviews from 2008 to 2018
Mokhtari-Hessari, P., & Montazeri, A. (2020) · Health and Quality of Life Outcomes · doi:10.1186/s12955-020-01591-x · Cited by 437
Human health impacts of exposure to phthalate plasticizers: An overview of reviews
Eales, J., Bethel, A., Galloway, T., Hopkinson, P., Morrissey, K., Short, R. E., & Garside, R. (2022) · Environment International · doi:10.1016/j.envint.2021.106903 · Cited by 367
Hunt, H., Pollock, A., Campbell, P., Estcourt, L., & Brunton, G. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0695-8 · Cited by 314
Metadata
- Authors
- Lunny, C., Brennan, S. E., McDonald, S., & McKenzie, J. E.
- Affiliations
- Monash University (AU)
- Corresponding author
- McKenzie, Joanne E.
- Venue
- Systematic Reviews
- Published
- 2017-11-21
- Volume
- 6
- Issue
- 1
- Pages
- 231-231
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC5698938
- OpenAlex
- W2768872497
- Cited by
- 184
- References
- 52
- Citations by year
- 2026: 11 · 2025: 14 · 2024: 18 · 2023: 32 · 2022: 32 · 2021: 35 · 2020: 20 · 2019: 14 · 2018: 7
- FWCI
- 2.77
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Monash University Research Portal (Monash University) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Monash University
- Medical Research Council
- National Health and Medical Research Council (1072366)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- overviews of systematic reviews, evidence map, search filters, eligibility criteria, data extraction, literature search strategies
- MeSH terms
- Humans, Research Design, Review Literature as Topic, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Toward a comprehensive evidence map of overview of systematic review methods: paper 1—purpose, eligibility, search and data extraction. https://app.cochat.ai/papers/10.1186/s13643-017-0617-1
- The paper
- Lunny, C., Brennan, S. E., McDonald, S., & McKenzie, J. E. (2017). Toward a comprehensive evidence map of overview of systematic review methods: paper 1—purpose, eligibility, search and data extraction. Systematic Reviews. https://doi.org/10.1186/s13643-017-0617-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.