Clarifying differences between reviews within evidence ecosystems
Open accessThe DOI matches its Crossref record; 45 of the 46 references it cites were found in a registry or an index.
Abstract
This paper builds on a 2012 paper by the same authors which argued that the types and brands of systematic review do not sufficiently differentiate between the many dimensions of different review questions and review methods (Gough et al., Syst Rev 1:28, 2012). The current paper extends this argument by considering the dynamic contexts, or 'evidence ecosystems', within which reviews are undertaken; the fact that these ecosystems are constantly changing; and the relevance of this broader context for understanding 'dimensions of difference' in the unfolding development and refinement of review methods.The concept of an evidence ecosystem is used to consider particular issues within the three key dimensions of difference outlined in the 2012 paper of (1) review aims and approach, (2) structure and components of reviews, and (3) breadth, depth, and 'work done' by reviews.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., et al. (2017) · BMJ · doi:10.1136/bmj.j4008 · Cited by 10,636
ROBIS: A new tool to assess risk of bias in systematic reviews was developed
Whiting, P., Savović, J., Higgins, J. P. T., Caldwell, D. M., Reeves, B. C., Shea, B., Davies, P., Kleijnen, J., & Churchill, R. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.06.005 · Cited by 2,272
The Many Meanings of Research Utilization
Weiss, C. H. (1979) · Public Administration Review · doi:10.2307/3109916 · Cited by 2,222
Avoidable waste in the production and reporting of research evidence
Chalmers, I., & Glasziou, P. (2009) · The Lancet · doi:10.1016/s0140-6736(09)60329-9 · Cited by 2,208
Grading quality of evidence and strength of recommendations for diagnostic tests and strategies
Schünemann, H. J., Oxman, A. D., Brozek, J., Glasziou, P., Jaeschke, R., Vist, G. E., Williams, J. W., Kunz, R., Craig, J., Montori, V. M., et al. (2008) · BMJ · doi:10.1136/bmj.39500.677199.ae · Cited by 1,782
An Enquiry concerning Human Understanding
Hume, D. (1748) · The Clarendon Edition of the Works of David Hume: An Enquiry concerning Human Understanding · doi:10.1093/oseo/instance.00032980 · Cited by 1,595
RAMESES publication standards: realist syntheses
Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-21 · Cited by 1,374
Alonso-Coello, P., Schünemann, H. J., Moberg, J., Brignardello-Petersen, R., Akl, E. A., Davoli, M., Treweek, S., Mustafa, R. A., Rada, G., Rosenbaum, S., et al. (2016) · BMJ · doi:10.1136/bmj.i2016 · Cited by 1,317
Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series
Lewin, S., Booth, A., Glenton, C., Munthe-Kaas, H., Rashidian, A., Wainwright, M., Bohren, M. A., Tunçalp, Ö., Colvin, C. J., Garside, R., et al. (2018) · Implementation Science · doi:10.1186/s13012-017-0688-3 · Cited by 1,231
Clarifying differences between review designs and methods
Gough, D., Thomas, J., & Oliver, S. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-28 · Cited by 1,063 · cited 5×
“In our original paper on ‘Clarifying differences between reviews designs and methods’ we argued that there is a lack of agreed terminology for different types of reviews, that the terms that did exist were not used consistently, and that there was also considerable overlap in approaches within different review types a…”
BackgroundMethodInfluentialSelf-citationParkhurst, J. (2016) · London School of Economics and Political Science Research Online (London School of Economics and Political Science) · doi:10.4324/9781315675008 · Cited by 756
Evidence-based policy: the promise of ‘realist synthesis’
Pawson, R. (2002) · Evaluation · doi:10.1177/135638902401462448 · Cited by 729
Living systematic review: 1. Introduction-the why, what, when, and how
Elliott, J. H., Synnot, A., Turner, T., Simmonds, M., Akl, E. A., McDonald, S., Salanti, G., Meerpohl, J., MacLehose, H., Hilton, J., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.08.010 · Cited by 655
Self-citationAvoidable Waste in the Production and Reporting of Research Evidence
Chalmers, I., & Glasziou, P. (2009) · Obstetrics & Gynecology · doi:10.1097/aog.0b013e3181c3020d · Cited by 600
Living Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap
Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins, J. P. T., Mavergames, C., & Gruen, R. L. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001603 · Cited by 596
Self-citationStewart, L. A., & Tierney, J. F. (2002) · Evaluation & the Health Professions · doi:10.1177/0163278702025001006 · Cited by 571
Underreporting Research Is Scientific Misconduct
Chalmers, L. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100121018 · Cited by 551
Systems thinking, knowledge and action: towards better models and methods
Best, A., & Holmes, B. (2010) · Evidence & Policy · doi:10.1332/174426410x502284 · Cited by 444
RAMESES publication standards: realist syntheses
Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013) · Journal of Advanced Nursing · doi:10.1111/jan.12095 · Cited by 385
Living systematic reviews: 2. Combining human and machine effort
Thomas, J., Noel-Storr, A., Marshall, I., Wallace, B., McDonald, S., Mavergames, C., Glasziou, P., Shemilt, I., Synnot, A., Turner, T., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.08.011 · Cited by 355 · cited 2×
“A review level perspective suggests that the power calculation should be based on the sample size necessary for the outcome to change the conclusions of the existing systematic review findings (often by providing a more precise estimate of effect and reducing uncertainty) [37, 38].”
BackgroundSelf-citationMichie, S., Thomas, J., Johnston, M., Aonghusa, P. M., Shawe-Taylor, J., Kelly, M. P., Deleris, L. A., Finnerty, A. N., Marques, M. M., Norris, E., et al. (2017) · Implementation Science · doi:10.1186/s13012-017-0641-5 · Cited by 338
Self-citationSynthesising quantitative evidence in systematic reviews of complex health interventions
Higgins, J. P. T., López-López, J. A., Becker, B. J., Davies, S. R., Dawson, S., Grimshaw, J. M., McGuinness, L. A., Moore, T. H. M., Rehfuess, E. A., Thomas, J., et al. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000858 · Cited by 322 · cited 1×
“In addition, there has been greater use of multicomponent mixed methods reviews to combine the results of studies that both test and generate theory, and interest in complexity science and systems theory to better understand the interactions between an intervention and the context within which it is introduced [22, 23…”
BackgroundSelf-citationWelch, V., Petticrew, M., Petkovic, J., Moher, D., Waters, E., White, H., Tugwell, P., Atun, R., Awasthi, S., Barbour, V., et al. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.09.001 · Cited by 297
The Potential of Transdisciplinarity
Gibbons, M., & Nowotny, H. (2001) · Transdisciplinarity: Joint Problem Solving among Science, Technology, and Society · doi:10.1007/978-3-0348-8419-8_7 · Cited by 250
Petticrew, M., Knai, C., Thomas, J., Rehfuess, E. A., Noyes, J., Gerhardus, A., Grimshaw, J. M., Rutter, H., & McGill, E. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000899 · Cited by 204
Self-citationRehfuess, E. A., Stratil, J. M., Scheel, I. B., Portela, A., Norris, S. L., & Baltussen, R. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000844 · Cited by 200
An introduction to systematic reviews (2nd Edition)
Gough, D., Oliver, S., & Thomas, J. (2017) · Psychology Teaching Review · doi:10.53841/bpsptr.2017.23.2.95 · Cited by 179
Self-citationThomas, J., O’Mara-Eves, A., & Brunton, G. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-67 · Cited by 177
Self-citation“Precision” Public Health — Between Novelty and Hype
Chowkwanyun, M., Bayer, R., & Galea, S. (2018) · New England Journal of Medicine · doi:10.1056/nejmp1806634 · Cited by 151
Rehfuess, E. A., Booth, A., Brereton, L., Burns, J., Gerhardus, A., Mozygemba, K., Oortwijn, W., Pfadenhauer, L. M., Tummers, M., van der Wilt, G., et al. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1254 · Cited by 148
Kneale, D., Thomas, J., & Harris, K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0142187 · Cited by 127
Self-citationGough, D., Kiwan, D., Sutcliffe, K., Simpson, D., & Houghton, N. (2003) · UCL Discovery (University College London) · Cited by 101
Self-citationCaird, J., Sutcliffe, K., Kwan, I., Dickson, K., & Thomas, J. (2015) · Evidence & Policy · doi:10.1332/174426514x13988609036850 · Cited by 81
Self-citationAn overview of conducting systematic reviews with network meta-analysis
Caldwell, D. M. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-109 · Cited by 79
Chapter 11: Undertaking network meta-analyses
Chaimani, A., Caldwell, D. M., Li, T., Higgins, J. P. T., & Salanti, G. (2019) · BORIS (University Library Bern) · Cited by 51
UK What Works Centres: Aims, methods and contexts
Gough, D., Maidment, C., & Sharples, J. (2018) · UCL Discovery (University College London) · Cited by 51
Self-citationCommonality and Diversity in Reviews
Gough, D., & Thomas, J. (2017) · SAGE Publications Ltd eBooks · doi:10.4135/9781036234942.n3 · Cited by 47
Self-citationOliver, S., & Dickson, K. (2016) · Evidence & Policy · doi:10.1332/174426415x14399963605641 · Cited by 34
Self-citationKneale, D., Thomas, J., Bangpan, M., Waddington, H., & Gough, D. (2018) · Journal of Development Effectiveness · doi:10.1080/19439342.2018.1530278 · Cited by 26
Self-citationKneale, D., Thomas, J., O’Mara-Eves, A., & Wiggins, R. D. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1320 · Cited by 20
Self-citationStakeholder Engagement for Development Impact Evaluation and Evidence Synthesis
Oliver, S. R., Roche, C., Stewart, R. J., Bangpan, M., Dickson, K., Pells, K., Cartwright, N. D., Hargreaves, J., & Gough, D. (2018) · doi:10.51744/cip3 · Cited by 19
Self-citationProducing policy relevant systematic reviews: navigating the policy-research interface
Oliver, S. R., Bangpan, M., & Dickson, K. (2017) · Evidence & Policy · doi:10.1332/174426417x14987303892442 · Cited by 19
Self-citationEffective and efficient committee work: A Systematic overview of multidisciplinary literatures
Oliver, S. R., Hollingworth, K., Briner, R. B., Swann, C., Hinds, K., & Roche, C. (2018) · Evidence Base · doi:10.21307/eb-2018-002 · Cited by 18
Self-citationUntitled work
Moher (2009) · doi:10.1371/journal.pmed1000097
Research works printed but not linked: 3
- Langer L Tripney J Gough D The science of using science: researching the use of sesearch evidence in decision-making 2016 London EPPI-Centre, Social Science Research Unit, UCL Institute of Education, University College London 2016
- Gough D Kenny C Vigurs C Stansfield C Rosen R Taylor T Social values related to the development of health and care guidance: literature review for NICE by its Research Support Unit. Report of the Research Support Unit for the National Institute for Health and Care Excellence 2014 London EPPI-Centre, Social Science Research Unit, Institute of Education, University of London
- Peersman G (1996) A descriptive mapping of health promotion studies in young people, EPPI Research Report 1996 London EPPI-Centre, Social Science Research Unit, Institute of Education, University of London
Cited by
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Self-citationThe PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01626-4 · Cited by 13,874
Self-citationThe PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · International Journal of Surgery · doi:10.1016/j.ijsu.2021.105906 · Cited by 11,748
Self-citationThe PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · PLoS Medicine · doi:10.1371/journal.pmed.1003583 · Cited by 5,148
“Systematic reviews therefore generate various types of knowledge for different users of reviews (such as patients, healthcare providers, researchers, and policy makers).[1,2]To ensure a systematic review is valuable to users, authors should prepare a transparent, complete, and accurate account of why the review was do…”
BackgroundSelf-citationThe PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/v7gm2 · Cited by 5,177
Self-citation
Metadata
- Authors
- Gough, D., Thomas, J., & Oliver, S.
- Affiliations
- University College London (GB)
- University of Johannesburg (ZA)
- Corresponding author
- Gough, David
- Venue
- Systematic Reviews
- Published
- 2019-07-15
- Volume
- 8
- Issue
- 1
- Pages
- 170-170
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC6631872
- OpenAlex
- W2961682569
- Cited by
- 115
- References
- 46
- Citations by year
- 2026: 13 · 2025: 14 · 2024: 14 · 2023: 10 · 2022: 28 · 2021: 24 · 2020: 11 · 2019: 1
- FWCI
- 10.69
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- World Health Organization
- Department of Health and Social Care
- Department for International Development
- Alliance for Health Policy and Systems Research
- Economic and Social Research Council
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- evidence ecosystem, review methodology
- MeSH terms
- Systematic Reviews as Topic, Humans, Research Design, Ecosystem
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
![Fig. 1 Evidence use ecosystem analytical framework [5]](https://figures.cochat.ai/paper-figures/0b2328b899504b538627600ec326184b/clarifying-differences-between-reviews-within-evidence-ecosystems-figure-1.jpg)
![Fig. 2 ‘Dimensions of difference’ in approaches to synthesis [2]](https://figures.cochat.ai/paper-figures/2bf7d6ea096a41d997a113182461fbdf/clarifying-differences-between-reviews-within-evidence-ecosystems-figure-2.jpg)
![Fig. 3 Map of personal development planning research: research method by country [47]](https://figures.cochat.ai/paper-figures/e4a3ffcc1cbe4d32ab2dc7113c227606/clarifying-differences-between-reviews-within-evidence-ecosystems-figure-3.jpg)