Software support for environmental evidence synthesis
Open accessThe DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index; 5 of the 32 sources its claims cite were checked for support.
Overview
CoChat Summary
TL;DR
This commentary argues that environmental evidence synthesis faces a growing “synthesis gap” as scientific literature expands, and that better software support—especially machine learning and open science practices—could help close it. It proposes five actions for validating, communicating, adopting, and improving synthesis tools and metadata.
Key findings
- The paper introduces the “synthesis gap”: policy-relevant information is lost among websites, reports, and peer-reviewed articles, risking duplicated research and wasted global investment in environmental science.
- In medicine, the average systematic review takes five people 67 weeks, extrapolating to nearly 12,000 person hours; text-mining approaches can reduce literature-sorting effort by 30% to more than 90% relative to manual sorting, but risk missing up to 5% of relevant studies (95% recall).
- The authors propose five actions: better validation of software tools; rapid communication of novel methods; broader adoption of open science principles; investigating “completeness” in evidence synthesis; and improved article-level metadata.
- Table 1 maps emerging methods to synthesis stages, including web scraping, duplicate detection algorithms, topic modelling, machine learning classification, automated image and natural language processing, and meta-analysis tools.
Methods
- Not stated in the paper; this is a commentary/perspective, not an empirical study.
- It synthesizes prior literature and presents a conceptual argument with a figure and table; no sample, data collection, or formal analysis is described.
Limitations
- Stated: there are no widely agreed-upon standards for testing synthesis tools and no organisation capable of routinely providing independent validation; central organisations such as the Collaboration for Environmental Evidence and the Campbell and Cochrane Collaborations are not equipped for independent validation and are composed largely of volunteer researchers.
- Stated: text-mining approaches risk missing up to 5% of relevant studies (95% recall); most new papers introduce new approaches rather than evaluate existing methods, and users rarely report enough information to replicate or evaluate software performance.
- Stated: there is no standard format for storing or sharing evidence-synthesis decision data, and it is unknown whether cases where a single new study materially affects review outcomes are common or rare.
- Evident: as a commentary, the proposed actions are not empirically tested here, and no quantitative evaluation of their effectiveness is provided.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 2 Not valid · 19 Not checked · 13 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Locating scientific articles by keyword-based searching is particularly inefficient because it returns a large amount of irrelevant information 27”
What the sources say
- Could not verifycites [27] Haddaway, N. R., Collins, A. M., Coughlin, D. & Kirk, S. Environ. Evid. 6, 1 (2017). sourceExperimental — accuracy not yet measured
Our copy of Haddaway et al. (2017) frames inefficiency in terms of extracting citations individually from Google Scholar and organisation websites ('often prohibitively time consuming', Abstract p. 1; IP-blocking after manual extraction of several hundred citations, p. 5) and in terms of transparency/reproducibility, not the relevance of keyword search results. The advantage/disadvantage comparison (Table 1, p. 2) lists duplicates, incomplete citations, subscription costs, slow inclusion and low reproducibility of search-engine searches, but no statement that keyword searching returns a large amount of irrelevant information. Searches for 'irrelevant records', 'precision of search strings', 'keyword search terms ... not relevant' and 'screening large number of hits' returned nothing on that point; the closest is the tillage example (p. 9) where results were 'screened at title, abstract and full text stages', which implies irrelevant hits but does not assert the claim. Hence the claim is not addressed by our copy rather than contradicted. (LLM, conf=0.65)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This gap manifests as policyrelevant information being lost amongst a sea of websites, reports and peer-reviewed articles 9,10 .”
What the sources say
- Not checked yetcites [9] Olorisade, B. K., de Quincey, E., Brereton, P. & Andras, P. A critical analysis of studies that address the use of text mining for citation screening in systematic reviews. In Proc. 20th International Conference on Evaluation and Assessment in Software Engineering 14 (ACM, New York, 2016). source
- Partly supportedcites [10] O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M. & Ananiadou, S. Syst. Rev. 4, 5 (2015). sourceExperimental — accuracy not yet measured
Our copy of O'Mara-Eves et al. (2015) supports only the surrounding idea: the Background section 'The problem: lack of precision in systematic searches' (p. 1-2) states that 'the large and growing number of published studies, and their increasing rate of publication, makes the task of identifying relevant studies in an unbiased way both complex and time consuming', that 'relevant research is probably missed' when searches are limited, and that reviews thereby 'risk being unable to inform policy and practice'. The same section notes the growth in 'the number of databases to search and the number of papers and journals being published' and the need to search 'specialised registers and websites'. However, the paper nowhere describes a 'synthesis gap', nor does it say policy-relevant information is 'lost amongst a sea of websites, reports and peer-reviewed articles' — grey literature/'reports' as a locus of lost information is not addressed, and its concern is study identification/screening for reviews rather than information going unreached by policy. That framing (the synthesis gap and the 'sea of websites, reports and peer-reviewed articles') is the part this source does not support. (LLM, conf=0.65)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“and this leads to enormous increases in the cost of bridging the synthesis gap 28 .”
What the sources say
- Partly supportedcites [28] Rathbone, J., Carter, M., Hoffmann, T. & Glasziou, P. System. Rev. 4, 6 (2015). sourceExperimental — accuracy not yet measured
The Background describes duplicate screening as an onerous task, and the Discussion identifies inefficient labour as a form of research waste. The paper supports the idea that this work consumes reviewers’ time and resources, but it does not establish an enormous increase in cost or discuss the cost of bridging a synthesis gap. Page numbers are unavailable in our copy. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“there is a risk that untested approaches may introduce new forms of bias 18 .”
What the sources say
- Partly supportedcites [18] Turner, R. M., Bird, S. M. & Higgins, J. P. PLoS ONE 8, e59202 (2013). sourceExperimental — accuracy not yet measured
Our copy of Turner et al. (2013) contains only a narrower, related caution: the Introduction states that 'Cut-offs based on study size, although scientifically cost-efficient, introduce an extra element of subjectivity and might not ameliorate bias if the remaining large studies are insufficiently critiqued' — i.e. one specific analytic approach can add a new problem and may fail to remove bias, and even that argument is credited to reference [11] (Turner et al. 2009, Bias modelling in evidence synthesis). Nothing in our copy speaks of 'untested approaches' generally, nor of methods introducing 'new forms of bias'; the Discussion instead notes that small-study effects can be addressed by statistical adjustment methods [14;15] and refers to Sterne et al. [24] guidelines. So the cited sentence's generalisation extends beyond what this paper states. (LLM, conf=0.65)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“suggesting that completeness of the evidence base can be important in some instances.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
15 more claims
Create a free account to see all 37 claims and to verify the 27 references still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 11
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Without access to robust time-saving software tools, evidence synthesis researchers risk facing a marked increase in the so-called 'synthesis gap'; an incapacity of the scientific establishment to synthesize its' own research for application to societal problems 7 .”
From revtools: An R package to support article screening for evidence synthesis (2019) · cites this paper as [7]
Citations
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Biodiversity loss and its impact on humanity
Cardinale, B. J., Duffy, J. E., Gonzalez, A., Hooper, D. U., Perrings, C., Venail, P., Narwani, A., Mace, G. M., Tilman, D., Wardle, D. A., et al. (2012) · Nature · doi:10.1038/nature11148 · Cited by 7,457
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
The need for evidence-based conservation
Sutherland, W. J., Pullin, A. S., Dolman, P. M., & Knight, T. M. (2004) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2004.03.018 · Cited by 1,822
How Many Studies Do You Need?: A Primer on Statistical Power for Meta-Analysis
Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998609346961 · Cited by 1,294
An Introduction to Systematic Reviews
Gough, D., Oliver, S., & Thomas, J. (2017) · doi:10.4135/9781036234942 · Cited by 1,239
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews
Turner, R. M., Bird, S. M., & Higgins, J. P. T. (2013) · PLoS ONE · doi:10.1371/journal.pone.0059202 · Cited by 672
Why Most Clinical Research Is Not Useful
Ioannidis, J. P. A. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1002049 · Cited by 655
Research works printed but not linked: 2
- Blei, D. M., Ng, A. Y. & Jordan, M. I. J. Mach. Learn. Res. 3, 993–1022 (2003).
- Chang, J., Boyd-Graber, J. L., Gerrish, S., Wang, C. & Blei, D. M. in Advances in Neural Information Processing Systems Vol. 22. (eds Bengio, Y., Schuurmans, D., Lafferty, J. D., Williams, C. K. I. & Culotta, A.) 288–296 (Curran Associates, Red Hook, 2009).
Cited by
revtools: An R package to support article screening for evidence synthesis
Westgate, M. J. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1374 · Cited by 261
“7 While revtools is already a well-developed software package, there are a number of ways in which its functionality could be expanded to improve efficiencies in evidence synthesis.”
BackgroundSelf-citationResearch Weaving: Visualizing the Future of Research Synthesis
Nakagawa, S., Samarasinghe, G., Haddaway, N. R., Westgate, M. J., O’Dea, R. E., Noble, D. W. A., & Lagisz, M. (2019) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2018.11.007 · Cited by 261
Self-citationMachine-learning-based evidence and attribution mapping of 100,000 climate impact studies
Callaghan, M., Schleussner, C.-F., Nath, S., Lejeune, Q., Knutson, T. R., Reichstein, M., Hansen, G., Theokritoff, E., Andrijevic, M., Brecha, R. J., et al. (2021) · Nature Climate Change · doi:10.1038/s41558-021-01168-6 · Cited by 251
Sutherland, W. J., Taylor, N. G., MacFarlane, D., Amano, T., Christie, A. P., Dicks, L. V., Lemasson, A. J., Littlewood, N. A., Martin, P. A., Ockendon, N., et al. (2019) · Biological Conservation · doi:10.1016/j.biocon.2019.108199 · Cited by 232
Predicting the time needed for environmental systematic reviews and systematic maps
Haddaway, N. R., & Westgate, M. J. (2019) · Conservation Biology · doi:10.1111/cobi.13231 · Cited by 120
“Furthermore, the procedures we identified as the most time-consuming should be seen as important areas for methodological and technological development to increase efficiency (e.g., using machine learning [Westgate et al. 2018]).”
MethodSelf-citation
Metadata
- Authors
- Westgate, M. J., Haddaway, N. R., Cheng, S. H., McIntosh, E. J., Marshall, C., & Lindenmayer, D. B.
- Affiliations
- Australian National University (AU)
- Stockholm Environment Institute (SE)
- University of California, Santa Barbara (US)
- University of Oxford (GB)
- University of York (GB)
- Corresponding author
- Westgate, Martin J.
- Venue
- Nature Ecology & Evolution
- Published
- 2018-03-23
- Volume
- 2
- Issue
- 4
- Pages
- 588-590
- ISSN
- 2397-334X (ISSN-L)
- OpenAlex
- W2793459609
- Semantic Scholar
- db3d3da44bd9ec5b0547251bf77775a257cbc819
- Cited by
- 51
- References
- 40
- Citations by year
- 2026: 3 · 2025: 1 · 2024: 3 · 2023: 1 · 2022: 8 · 2021: 6 · 2020: 12 · 2019: 8 · 2018: 9
- FWCI
- 6.67
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature Ecology & Evolution (publishedVersion)
- PubMed (publishedVersion)
- ANU Open Research (Australian National University) (submittedVersion)
- Newcastle University ePrints (Newcastle Univesity)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Physical Sciences › Environmental Science › Management, Monitoring, Policy and Law › Environmental and Social Impact Assessments
- Keywords
- software tools, data collation, data sharing, information synthesis, environmental data
- MeSH terms
- Ecology, Software, Information Dissemination
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Software support for environmental evidence synthesis. https://app.cochat.ai/papers/10.1038/s41559-018-0502-x
- The paper
- Westgate, M. J., Haddaway, N. R., Cheng, S. H., McIntosh, E. J., Marshall, C., & Lindenmayer, D. B. (2018). Software support for environmental evidence synthesis. Nature Ecology & Evolution. https://doi.org/10.1038/s41559-018-0502-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.