Quality assessment tools for evidence from environmental science
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
Assessment of the quality of studies is a critical component of evidence syntheses such as systematic reviews (SRs) that are used to inform policy decisions. To reduce the potential for reviewer bias, and to ensure that the findings of SRs are transparent and reproducible, organisations such as the Cochrane Collaboration , the Campbell Collaboration , and the Collaboration for Environmental Evidence , recommend the use of formal quality assessment tools as opposed to informal expert judgment. However, there is a bewildering array of around 300 formal quality assessment tools that have been identified in the literature, and it has been demonstrated that the use of different tools for the assessment of the same studies can result in different estimates of quality, which can potentially reverse the conclusions of a SR. It is therefore important to consider carefully, the choice of quality assessment tool. We argue that quality assessment tools should: (1) have proven construct validity (i.e. the assessment criteria have demonstrable link with what they purport to measure), (2) facilitate inter-reviewer agreement, (3) be applicable across study designs, and (4) be quick and easy to use. Our aim was to examine current best practice for quality assessment in healthcare and investigate the extent to which these best practices could be useful for assessing the quality of environmental science studies. The feasibility of this transfer is demonstrated in a number of existing SRs on environmental topics. We propose that environmental practitioners should revise, test and adopt the best practice quality assessment tools used in healthcare as a recommended approach for application to environmental science. We provide pilot versions of quality assessment tools, modified from the best practice tools used in healthcare, for application on studies from environmental science.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305
GRADE guidelines: 3. Rating the quality of evidence
Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., Kunz, R., Brozek, J., Vist, G. E., Falck-Ytter, Y., Meerpohl, J., & Norris, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.015 · Cited by 8,285
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 2×
“(1) randomization (minimises the risk of selection bias) [29-31], (2) allocation concealment and (3) blinding (minimises the risk of performance bias and detection bias due to participants’ or investigators’ expectations) [15,22,32-37], (4) follow-up (high follow-up of participants from enrolment to study completion m…”
BackgroundPublication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 1×
“(1) randomization (minimises the risk of selection bias) [29-31], (2) allocation concealment and (3) blinding (minimises the risk of performance bias and detection bias due to participants’ or investigators’ expectations) [15,22,32-37], (4) follow-up (high follow-up of participants from enrolment to study completion m…”
BackgroundRaise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“Background There are significant concerns about the low reproducibility of scientific studies [1,2], and the limited effectiveness of peer-review as a quality-control mechanism [3,4].”
BackgroundEvaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907 · cited 21×
“It is therefore very important to consider carefully, the choice of quality assessment tool to be used in SRs [19].”
BackgroundMethodInfluentialWood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981 · cited 2×
“For example, in the healthcare field, a meta-analysis of 17 trials comparing the effectiveness of low-molecular-weight heparin (LMWH) with standard heparin for prevention of post-operative thrombosis, trials that were identified as ‘high quality’ by some of the 25 quality scales interrogated, indicated that LWMH was n…”
BackgroundMethod
Cited by
Making literature reviews more reliable through application of lessons from systematic reviews
Haddaway, N. R., Woodcock, P., Macura, B., & Collins, A. (2015) · Conservation Biology · doi:10.1111/cobi.12541 · Cited by 416
“Critical appraisal is the process by which individual studies are assessed for external (generalizability) and internal (quality) validity (Bilotta et al. 2014).”
BackgroundSimple study designs in ecology produce inaccurate estimates of biodiversity responses
Christie, A. P., Amano, T., Martin, P. A., Shackelford, G. E., Simmons, B. I., & Sutherland, W. J. (2019) · Journal of Applied Ecology · doi:10.1111/1365-2664.13499 · Cited by 375
Microplastic contamination of drinking water: A systematic review
Danopoulos, E., Twiddy, M., & Rotchell, J. M. (2020) · PLOS ONE · doi:10.1371/journal.pone.0236838 · Cited by 341
Danopoulos, E., Jenner, L. C., Twiddy, M., & Rotchell, J. M. (2020) · Environmental Health Perspectives · doi:10.1289/ehp7171 · Cited by 302
“Overall assessment of the certainty of the evidence was based on the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) framework (Higgins et al. 2019) and the Environmental-GRADE (Bilotta et al. 2014) across fi ve domains, categorized into four certainty ratings: high, moderate, low, and very…”
GRADE: Assessing the quality of evidence in environmental and occupational health
Morgan, R. L., Thayer, K. A., Bero, L., Bruce, N., Falck-Ytter, Y., Ghersi, D., Guyatt, G., Hooijmans, C., Langendam, M., Mandrioli, D., et al. (2016) · Environment International · doi:10.1016/j.envint.2016.01.004 · Cited by 277
Metadata
- Authors
- Bilotta, G. S., Milner, A. M., & Boyd, I. L.
- Venue
- Environmental Evidence
- Published
- 2014
- Volume
- 3
- Issue
- 1
- Pages
- 14
- ISSN
- 2047-2382 (ISSN-L)
- OpenAlex
- W2161725535
- Cited by
- 107
- References
- 29
- FWCI
- 9.29
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Quality assessment tools for evidence from environmental science. https://app.cochat.ai/papers/10.1186/2047-2382-3-14
- The paper
- Bilotta, G. S., Milner, A. M., & Boyd, I. L. (2014). Quality assessment tools for evidence from environmental science. Environmental Evidence. https://doi.org/10.1186/2047-2382-3-14
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.