Eligibility screening in evidence synthesis of environmental management topics
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index; 4 of the 40 sources its claims cite were checked for support.
Abstract
The eligibility screening step of a systematic review or systematic map (sometimes referred to as ‘study selection’, ‘evidence selection’ or ‘inclusion screening’) determines the scope of the evidence that may answer the review or map question. Eligibility screening involves the development, testing and application of eligibility criteria (inclusion and exclusion criteria) by an evidence synthesis review team, based on methods pre-specified in the review or map protocol. Some parts of the process require judgement, meaning that consistent and transparent reporting of the eligibility criteria and the process for applying them are essential in order to reduce the risk of introducing errors or bias. The existing Collaboration for Environmental Evidence (CEE) Guidelines for Systematic Reviews in Environmental Management (version 4.2, March 2013) give relatively limited guidance on how to conduct eligibility screening. In this paper we provide more in-depth information on good practice methods for this step of evidence synthesis, based on a critical consideration of existing guidance and current practice. Our aim is to provide recommendations to support those conducting CEE systematic reviews or systematic maps for environmental management questions; however, the methods we describe are generic and should be broadly applicable across a wide range of environmental research topics.
Overview
CoChat Summary
## TL;DR
This paper is a methods/guidance article on how to conduct eligibility screening in systematic reviews and systematic maps for environmental management topics. It recommends pre-specified criteria, at least two screeners where possible, pilot testing, duplicate removal, and transparent reporting to reduce error and bias.
## Key findings
- The authors report that bibliographic searches may yield “thousands or sometimes tens of thousands” of references, making efficient screening and duplicate removal important.
- They cite one estimate that a single screener would miss about 8% of eligible studies on average, while two people working independently would miss no studies; using two reviewers also increased identified randomized studies by 9%.
- Based on 11 Environmental Evidence protocols published in January–July 2017, they note several reporting gaps: most did not specify the number of screeners; most did not say which screening step agreement assessed; none justified the sample size for agreement testing; and 45% did not report whether the agreement sample was random.
- In Box 2, they give a hypothetical example of 8,000 articles with 99.4% overall agreement and Cohen’s kappa of 0.62, showing that high overall agreement can still hide important include/exclude discrepancies.
## Methods
- The paper is a guidance/methodology article rather than an empirical study; it provides recommendations for planning, applying, and reporting eligibility screening.
- Its recommendations are based on critical consideration of the existing CEE guidelines, current practice as seen in 11 recent Environmental Evidence protocols, and relevant guidance from environmental, health, and social science evidence synthesis.
- The paper develops recommendations through narrative synthesis of prior guidance and examples, including a PICO-type case study on China’s Conversion of Cropland to Forest Programme (CCFP).
## Limitations
- The paper states that its assessment of current practice is based on 11 protocols from a single journal and a limited time window (January–July 2017), so it may not represent broader practice.
- It also states that the methods are recommendations derived from guidance and current practice, not from a new comparative empirical evaluation of screening methods.
- Evident from the text: several recommendations are qualitative and based on expert judgment; the paper does not provide new outcome data from actual screening exercises beyond the cited estimates and illustrative hypothetical examples.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 2 Not valid · 23 Not checked · 35 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, recently-published evidence syntheses and protocols indicate that the kappa statistic is increasingly being used for a different purpose: to demonstrate high reviewer agreement in support of employing only one screener to assess the majority of articles.”
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, when interpreting screener agreement it should be borne in mind that potentially important discrepancies between screeners can occur even if screener agreement statistics indicate high overall rates of agreement (Box2).”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“most of the publications did not specify the number of screeners for one or more of the screening steps;”
plainn = 11- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“none of the publications provided a justification for the size of the sample of articles to be used in assessing screener agreements;”
plainn = 11- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“nearly half (45%) of the publications did not report whether the sample would be selected randomly.”
plainn = 11- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
14 more claims
Create a free account to see all 63 claims and to verify the 36 references still open.
- Probably not a substantial claim. 37
- It may state two separate claims as one. 19
Citations
Cites
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · cited 8×
“Although there is often a single article for each study, it should never be assumed that this is the case [3].”
BackgroundInfluentialLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.1. The Cochrane Collaboration
Higgins, J. P. T., Green, S. R., & Higgins, J. P. T. (2008) · Cited by 7,171
Cochrane Handbook for Systematic Reviews of Interventions
Tarsilla, M. (2010) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v6i14.284 · Cited by 5,178
Practical Statistics for Medical Research.
Bartko, J. J., & Altman, D. G. (1992) · Journal of the American Statistical Association · doi:10.2307/2290246 · Cited by 4,797
Systematic reviews in the social sciences: a practical guide
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5664 · Cited by 4,089
Systematic Reviews in the Social Sciences: A Practical Guide
Petticrew, M., & Roberts, H. (2006) · doi:10.1002/9780470754887 · Cited by 3,586
Systematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,245
Finding What Works in Health Care: Standards for Systematic Reviews
Eden, J., Levit, L. A., Berg, A. O., & Morton, S. C. (2013) · Cited by 943
A methodology for systematic mapping in environmental sciences
James, K. L., Randall, N. P., & Haddaway, N. R. (2016) · Environmental Evidence · doi:10.1186/s13750-016-0059-6 · Cited by 769 · cited 3×
“Nevertheless, an underlying expectation of systematic maps is that the searching and eligibility screening steps should be conducted with the same rigor as for systematic reviews [5].”
BackgroundInfluential
Research works printed but not linked: 6
- CEE (Collaboration for Environmental Evidence). Guidelines for systematic review and evidence synthesis in environmental management. Version 4.2. 2013. http://www.environmentalevidence.org/wp-content/uploads/2014/06/Review-guidelines-version-4.2-final.pdf .
- DEFRA (Department for Environment Food and Rural Affairs). Emerging tools and techniques to deliver timely and cost effective evidence reviews. London: DEFRA; 2015.
- Campbell Collaboration. Systematic reviews: policies and guidelines version 1.2. Oslo: The Steering Group of the Campbell Collaboration; 2014.
- Higgins JPT, Lasserson T, Chandler J, Tovey D, Churchill R. Methodological expectations of Cochrane intervention reviews (MECIR). London: Cochrane; 2016.
- McDonagh M, Peterson K, Raina P, Chang S, Shekelle P. Avoiding bias in selecting studies—methods guide for comparative effectiveness reviews. Rockville: Agency for Healthcare Research and Quality (AHRQ); 2013.
- von Elm E, Tramer MR, Jüni P, Egger M. Does duplicate publication of trials introduce bias in systematic reviews? A systematic review [abstract]. In: 11th Cochrane Colloquium: 2003 Oct 26–31; Barcelona, Spain.
Cited by
Systematic searching for environmental evidence using multiple tools and sources
Livoreil, B., Glanville, J., Haddaway, N. R., Bayliss, H., Bethel, A., de Lachapelle, F. F., Robalino, S., Savilaakso, S., Zhou, W., Petrokofsky, G., et al. (2017) · Environmental Evidence · doi:10.1186/s13750-017-0099-6 · Cited by 156
Self-citationSavilaakso, S., Johansson, A., Häkkilä, M., Uusitalo, A., Sandgren, T., Mönkkönen, M., & Puttonen, P. (2021) · Environmental Evidence · doi:10.1186/s13750-020-00215-7 · Cited by 129
“Multiple articles from one study site (i.e. linked articles) were appraised as a group to avoid inclusion of duplicate data following Frampton et al. [37].”
BackgroundTrends and knowledge gaps in field research investigating effects of anthropogenic noise
Jerem, P., & Mathews, F. (2020) · Conservation Biology · doi:10.1111/cobi.13510 · Cited by 118
“We used a PICO framework (Frampton et al. 2017) to define our inclusion criteria.”
Method(EFSA), E. F. S. A., Devos, Y., Muñoz‐Guajardo, I., Álvarez, F., & Glanville, J. (2019) · EFSA Supporting Publications · doi:10.2903/sp.efsa.2019.en-1614 · Cited by 80
Gutierrez, G., Stephenson, C., Eadie, J., Asadpour, K., & Alavi, N. (2024) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2024.1356773 · Cited by 71
Metadata
- Authors
- Frampton, G. K., Livoreil, B., & Petrokofsky, G.
- Affiliations
- University of Southampton (GB)
- Fondation Pour la Recherche Sur la Biodiversité (FR)
- University of Oxford (GB)
- Corresponding author
- Frampton, Geoff K.
- Venue
- Environmental Evidence
- Published
- 2017-09-20
- Volume
- 6
- Issue
- 1
- ISSN
- 2047-2382 (ISSN-L)
- OpenAlex
- W2759190001
- Cited by
- 95
- References
- 36
- Citations by year
- 2026: 14 · 2025: 19 · 2024: 16 · 2023: 20 · 2022: 5 · 2021: 7 · 2020: 6 · 2019: 6 · 2017: 2
- FWCI
- 3.44
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Environmental Evidence (publishedVersion, cc-by)
- ePrints Soton (University of Southampton) (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Environmental Science › Management, Monitoring, Policy and Law › Environmental and Social Impact Assessments
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- eligibility screening, eligibility criteria, inclusion and exclusion criteria, systematic reviews, systematic map, environmental management, reporting transparency, bias mitigation, study selection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Eligibility screening in evidence synthesis of environmental management topics. https://app.cochat.ai/papers/10.1186/s13750-017-0102-2
- The paper
- Frampton, G. K., Livoreil, B., & Petrokofsky, G. (2017). Eligibility screening in evidence synthesis of environmental management topics. Environmental Evidence. https://doi.org/10.1186/s13750-017-0102-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.