Selecting and implementing overview methods: implications from five exemplar overviews
Open accessThe DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Overviews of systematic reviews are an increasingly popular method of evidence synthesis; there is a lack of clear guidance for completing overviews and a number of methodological challenges. At the UK Cochrane Symposium 2016, methodological challenges of five overviews were explored. Using data from these five overviews, practical implications to support methodological decision making of authors writing protocols for future overviews are proposed. METHODS: Methods, and their justification, from the five exemplar overviews were tabulated and compared with areas of debate identified within current literature. Key methodological challenges and implications for development of overview protocols were generated and synthesised into a list, discussed and refined until there was consensus. RESULTS: Methodological features of three Cochrane overviews, one overview of diagnostic test accuracy and one mixed methods overview have been summarised. Methods of selection of reviews and data extraction were similar. Either the AMSTAR or ROBIS tool was used to assess quality of included reviews. The GRADE approach was most commonly used to assess quality of evidence within the reviews. Eight key methodological challenges were identified from the exemplar overviews. There was good agreement between our findings and emerging areas of debate within a recent published synthesis. Implications for development of protocols for future overviews were identified. CONCLUSIONS: Overviews are a relatively new methodological innovation, and there are currently substantial variations in the methodological approaches used within different overviews. There are considerable methodological challenges for which optimal solutions are not necessarily yet known. Lessons learnt from five exemplar overviews highlight a number of methodological decisions which may be beneficial to consider during the development of an overview protocol.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our search strategy retrieved 119 unique records, and the forward citation searching retrieved 92 citations (Fig. 2). After deduplication 199 remained, and after screening abstracts and full text, seven were included [17–19, 28–31]. One additional conference citation was found through expert knowledge on the topic [32], and one paper recently published was included after completion of the first”
From Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews (2021) · cites this paper as [18]
Experimental — accuracy not yet measuredIn the citing paper
“Since publication of the MOoR framework, methods for overlap have been published [17–19].”
From Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews (2021) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2014) · Medical Entomology and Zoology · Cited by 17,633
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
Aromataris, E., Fernandez, R., Godfrey, C. M., Holly, C., Khalil, H., & Tungpunkom, P. (2015) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000055 · Cited by 2,587 · cited 2×
“These include the QUality Of Reporting Of Meta-analyses (QUOROM) statement [31], Methodological Expectations of Cochrane Intervention Reviews guidance (MECIR) [32] and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, as well as the growing body of guidance specific to overviews […”
MethodROBIS: A new tool to assess risk of bias in systematic reviews was developed
Whiting, P., Savović, J., Higgins, J. P. T., Caldwell, D. M., Reeves, B. C., Shea, B., Davies, P., Kleijnen, J., & Churchill, R. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.06.005 · Cited by 2,272
Three techniques for integrating data in mixed methods studies
O'Cathain, A., Murphy, E., & Nicholl, J. (2010) · BMJ · doi:10.1136/bmj.c4587 · Cited by 1,584
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483
Systematic review finds overlapping reviews were not mentioned in every other overview
Pieper, D., Antoine, S.-L., Mathes, T., Neugebauer, E. A. M., & Eikermann, M. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.11.007 · Cited by 1,318
Methodology in conducting a systematic review of systematic reviews of healthcare interventions
Smith, V., Devane, D., Begley, C. M., & Clarke, M. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-15 · Cited by 1,227
Research works printed but not linked: 10
- Pollock A, Hunt H, Campbell P, Estcourt L, Brunton G. Cochrane overviews of reviews: exploring the methods and challenges. Birmingham: UK and Ireland Cochrane Symposium; 2016.
- Hunt H, Kuzma E, C H: A review of existing systematic reviews summarising the accuracy of brief cognitive assessments for identifying dementia, particularly for use in primary care. Protocol..In PROSPERO PROSPERO online; 2016.
- Brunton G, Dickson K, Khatwa M, Caird J, Oliver S, Hinds K, Thomas J. Developing evidence-informed, employer-led workplace health. London: EPPI-Centre, Social Science Research Unit, UCL Institute of Education, University College London; 2016.
- INVOLVE. Public involvement in systematic reviews: Supplement to the briefing notes for researchers. Eastleigh: INVOLVE; 2012.
- MECIR. The Methodological Expectations of Cochrane Intervention Reviews (MECIR).Standards for Cochrane new reviews of interventions and their updates. 2017.
- Becker L, Oxman A: Chapter 22: Overviews of reviews. In Cochrane Handbook for Systematic Reviews of Interventions Version 510 (JPT H, S G eds.): The Cochrane Collaboration; 2011. Available from www.handbook.cochrane.org .
- PSTF. U.S. Preventive Services Task Force. Procedure Manual. In: AHRQ publication no 08-05118-EF. Rockville: U.S. Preventive Services Task Force; 2008.
- NICE. National Institute for Health and Clinical Excellence. The Guidelines Manual. London: National Institute for Health and Clinical Excellence; 2006.
- CCN: What is a systematic review? http://consumers.cochrane.org/what-systematic-review : Cochrane consumers network (CCN); 2017.
- EPOC. Effective Practice and Organisation of Care (EPOC): reporting the effects of an intervention in EPOC reviews. Section 24. How to report the effects of an intervention. In: EPOC Resources for review authors. Oslo: Norwegian Knowledge Centre for the Health Services; 2016.
Cited by
Meeting the review family: exploring review types and associated information retrieval requirements
Sutton, A., Clowes, M., Preston, L., & Booth, A. (2019) · Health Information & Libraries Journal · doi:10.1111/hir.12276 · Cited by 570
Adverse Outcomes of Polypharmacy in Older People: Systematic Review of Reviews
Davies, L. E., Spiers, G., Kingston, A., Todd, A., Adamson, J., & Hanratty, B. (2020) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2019.10.022 · Cited by 544
Klaic, M., Kapp, S., Hudson, P., Chapman, W., Denehy, L., Story, D., & Francis, J. J. (2022) · Implementation Science · doi:10.1186/s13012-021-01171-7 · Cited by 394
Hunt, H., Pollock, A., Campbell, P., Estcourt, L., & Brunton, G. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0695-8 · Cited by 314
Self-citationLunny, C., Pieper, D., Thabet, P., & Kanji, S. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01269-y · Cited by 252
Metadata
- Authors
- Pollock, A., Campbell, P., Brunton, G., Hunt, H., & Estcourt, L.
- Affiliations
- Glasgow Caledonian University (GB)
- University College London (GB)
- University of Exeter (GB)
- John Radcliffe Hospital (GB)
- Corresponding author
- Pollock, Alex
- Venue
- Systematic Reviews
- Published
- 2017-07-18
- Volume
- 6
- Issue
- 1
- Pages
- 145-145
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC5516331
- OpenAlex
- W2738509363
- Cited by
- 134
- References
- 56
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 20 · 2023: 28 · 2022: 15 · 2021: 11 · 2020: 19 · 2019: 16 · 2018: 9 · 2017: 1
- FWCI
- 10.90
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Open Research Exeter (University of Exeter) (publishedVersion, cc-by)
- ResearchOnline (Glasgow Caledonian University) (acceptedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Scottish Government
- National Institute for Health and Care Research (13/89/09)
- Physiotherapy Research Foundation
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- overviews of systematic reviews, methodological issues, evidence synthesis, AMSTAR, ROBIS, GRADE, review quality assessment
- MeSH terms
- Data Accuracy, Decision Making, Humans, Publications, Research Design, Review Literature as Topic, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Selecting and implementing overview methods: implications from five exemplar overviews. https://app.cochat.ai/papers/10.1186/s13643-017-0534-3
- The paper
- Pollock, A., Campbell, P., Brunton, G., Hunt, H., & Estcourt, L. (2017). Selecting and implementing overview methods: implications from five exemplar overviews. Systematic Reviews. https://doi.org/10.1186/s13643-017-0534-3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.