The difficulties of systematic reviews
Open accessAbstract
The need for robust evidence to support conservation actions has driven the adoption of systematic approaches to research synthesis in ecology. However, applying systematic review to complex or open questions remains challenging, and this task is becoming more difficult as the quantity of scientific literature increases. We drew on the science of linguistics for guidance as to why the process of identifying and sorting information during systematic review remains so labor intensive, and to provide potential solutions. Several linguistic properties of peer‐reviewed corpora—including nonrandom selection of review topics, small‐world properties of semantic networks, and spatiotemporal variation in word meaning—greatly increase the effort needed to complete the systematic review process. Conversely, the resolution of these semantic complexities is a common motivation for narrative reviews, but this process is rarely enacted with the rigor applied during linguistic analysis. Therefore, linguistics provides a unifying framework for understanding some key challenges of systematic review and highlights 2 useful directions for future research. First, in cases where semantic complexity generates barriers to synthesis, ecologists should consider drawing on existing methods—such as natural language processing or the construction of research thesauri and ontologies—that provide tools for mapping and resolving that complexity. These tools could help individual researchers classify research material in a more robust manner and provide valuable guidance for future researchers on that topic. Second, a linguistic perspective highlights that scientific writing is a rich resource worthy of detailed study, an observation that can sometimes be lost during the search for data during systematic review or meta‐analysis. For example, mapping semantic networks can reveal redundancy and complementarity among scientific concepts, leading to new insights and research questions. Consequently, wider adoption of linguistic approaches may facilitate improved rigor and richness in research synthesis.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This condition (all relevant studies must be included) could hinder wider adoption of new software tools.
“a condition that could hinder wider adoption of new software tools 23 .”
From Software support for environmental evidence synthesis (2018) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Collective dynamics of ‘small-world’ networks
Watts, D. J., & Strogatz, S. H. (1998) · Nature · doi:10.1038/30918 · Cited by 43,060 · see citing works
Human behavior and the principle of least effort
Zipf, G. K. (1949) · Max Planck Digital Library · Cited by 6,598 · see citing works
(1998) · The MIT Press eBooks · doi:10.7551/mitpress/7287.001.0001 · Cited by 4,849 · see citing works
The role of google scholar in evidence reviews and its applicability to grey literature searching
Haddaway, N. R., Collins, A. M., Coughlin, D., & Kirk, S. (2015) · PLOS ONE · doi:10.1371/journal.pone.0138237 · Cited by 2,387 · see citing works
PLANT FUNCTIONAL MARKERS CAPTURE ECOSYSTEM PROPERTIES DURING SECONDARY SUCCESSION
Garnier, E., Cortez, J., Billès, G., Navas, M.-L., Roumet, C., Debussche, M., Laurent, G., Blanchard, A., Aubry, D., Bellmann, A., et al. (2004) · Ecology · doi:10.1890/03-0799 · Cited by 2,313 · see citing works
The need for evidence-based conservation
Sutherland, W. J., Pullin, A. S., Dolman, P. M., & Knight, T. M. (2004) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2004.03.018 · Cited by 1,822 · see citing works
Ecological and evolutionary traps.
Schlaepfer, M., Runge, M. C., & Sherman, P. (2002) · Trends in Ecology & Evolution · doi:10.1016/s0169-5347(02)02580-6 · Cited by 1,449 · see citing works
Extraneous factors in judicial decisions
Danziger, S., Levav, J., & Avnaim-Pesso, L. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1018033108 · Cited by 1,431 · see citing works
The Large‐Scale Structure of Semantic Networks: Statistical Analyses and a Model of Semantic Growth
Steyvers, M., & Tenenbaum, J. B. (2005) · Cognitive Science · doi:10.1207/s15516709cog2901_3 · Cited by 1,396 · see citing works
Guidelines for systematic review in Conservation and environmental management
PULLIN, A. S., & STEWART, G. B. (2006) · Conservation Biology · doi:10.1111/j.1523-1739.2006.00485.x · Cited by 1,137 · see citing works
Adaptive monitoring: a new paradigm for long-term research and monitoring.
Lindenmayer, D., & Likens, G. (2009) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2009.03.005 · Cited by 802 · see citing works
Self-citationArtificial reefs: a review of their design, application, management and performance
Baine, M. S. P. (2001) · Ocean & Coastal Management · doi:10.1016/s0964-5691(01)00048-5 · Cited by 585 · see citing works
The ecology, distribution, conservation and management of large old trees
Lindenmayer, D., & Laurance, W. (2017) · Biological Reviews of The Cambridge Philosophical Society · doi:10.1111/brv.12290 · Cited by 482 · see citing works
Self-citationBeyond species: why ecological interaction networks vary through space and time
Poisot, T., Stouffer, D. B., & Gravel, D. (2014) · Oikos · doi:10.1111/oik.01719 · Cited by 475 · see citing works
Making literature reviews more reliable through application of lessons from systematic reviews
Haddaway, N. R., Woodcock, P., Macura, B., & Collins, A. (2015) · Conservation Biology · doi:10.1111/cobi.12541 · Cited by 416 · see citing works
Adaptive management of biological systems: A review
Westgate, M. J., Likens, G. E., & Lindenmayer, D. B. (2012) · Biological Conservation · doi:10.1016/j.biocon.2012.08.016 · Cited by 396 · see citing works
Self-citationSpecialization and Rarity Predict Nonrandom Loss of Interactions from Mutualist Networks
Aizen, M. A., Sabatino, M. C., & Tylianakis, J. M. (2012) · Science · doi:10.1126/science.1215320 · Cited by 330 · see citing works
Doing more good than harm – Building an evidence-base for conservation and environmental management
Pullin, A. S., & Knight, T. (2009) · Biological Conservation · doi:10.1016/j.biocon.2009.01.010 · Cited by 295 · see citing works
Vive la résistance: reviving resistance for 21st century conservation
Nimmo, D. G., Nally, R. M., Cunningham, S. C., Haslem, A., & Bennett, A. F. (2015) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2015.07.008 · Cited by 267 · see citing works
Organising evidence for environmental management decisions: a ‘4S’ hierarchy
Dicks, L. V., Walsh, J. C., & Sutherland, W. J. (2014) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2014.09.004 · Cited by 249 · see citing works
A succession of theories: purging redundancy from disturbance theory
Pulsford, S. A., Lindenmayer, D., & Driscoll, D. (2016) · Biological Reviews of The Cambridge Philosophical Society · doi:10.1111/brv.12163 · Cited by 229 · see citing works
Self-citationLanguages cool as they expand: Allometric scaling and the decreasing need for new words
Petersen, A. M., Tenenbaum, J. N., Havlin, S., Stanley, H. E., & Perc, M. (2012) · Scientific Reports · doi:10.1038/srep00943 · Cited by 192 · see citing works
The benefits of systematic mapping to evidence-based environmental management
Haddaway, N., Bernes, C., Jonsson, B. G., & Hedlund, K. (2016) · AMBIO · doi:10.1007/s13280-016-0773-x · Cited by 192 · see citing works
Automated content analysis: addressing the big literature challenge in ecology and evolution
Nunez‐Mir, G. C., Iannone, B. V., Pijanowski, B. C., Kong, N., & Fei, S. (2016) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12602 · Cited by 185 · see citing works
Languages cool as they expand: Allometric scaling and the decreasing need for new words
Petersen, A. M., Tenenbaum, J., Havlin, S., Stanley, E., & Perc, M. (2012) · University of Maribor digital library (University of Maribor) · Cited by 172 · see citing works
Integrating plant- and animal-based perspectives for more effective restoration of biodiversity
McAlpine, C., Catterall, C. P., Nally, R. M., Lindenmayer, D., Reid, J. L., Holl, K. D., Bennett, A. F., Runting, R. K., Wilson, K., Hobbs, R. J., et al. (2016) · Frontiers in Ecology and the Environment · doi:10.1002/16-0108.1 · Cited by 170 · see citing works
Self-citationInternational Code of Zoological Nomenclature
Faust, E. C. (1962) · American Journal of Tropical Medicine and Hygiene · doi:10.4269/ajtmh.1962.11.3.tm0110030425a · Cited by 166 · see citing works
Density dependence: an ecological Tower of Babel
Herrando‐Pérez, S., Delean, S., Brook, B. W., et al. (2012) · Oecologia · doi:10.1007/s00442-012-2347-3 · Cited by 140 · see citing works
Evaluating the efficacy of adaptive management approaches: Is there a formula for success?
McFadden, J. E., Hiller, T. L., & Tyre, A. J. (2010) · Journal of Environmental Management · doi:10.1016/j.jenvman.2010.10.038 · Cited by 134 · see citing works
Text analysis tools for identification of emerging topics and research gaps in conservation science
Westgate, M. J., Barton, P. S., Pierson, J. C., & Lindenmayer, D. B. (2015) · Conservation Biology · doi:10.1111/cobi.12605 · Cited by 123 · see citing works
Self-citationGentile, J. H., Harwell, M. A., Cropper Jr, W., Harwell, C. C., DeAngelis, D., Davis, S., Ogden, J. C., & Lirman, D. (2001) · Science of The Total Environment · doi:10.1016/s0048-9697(01)00746-x · Cited by 99 · cited 1× · see citing works
“More broadly, ecologists often use conceptual models to define the key components of a given system and to determine how those components relate to one another (Gentile et al. 2001).”
BackgroundFormalized synthesis opportunities for ecology: systematic reviews and meta‐analyses
Lortie, C. J. (2014) · Oikos · doi:10.1111/j.1600-0706.2013.00970.x · Cited by 74 · see citing works
Framework to improve the application of theory in ecology and conservation
Driscoll, D. A., & Lindenmayer, D. B. (2011) · Ecological Monographs · doi:10.1890/11-0916.1 · Cited by 72 · see citing works
Self-citationOptimal taxonomic groups for biodiversity assessment: a meta‐analytic approach
Westgate, M. J., TULLOCH, A. I. T., Barton, P. S., Pierson, J. C., & Lindenmayer, D. B. (2016) · Ecography · doi:10.1111/ecog.02318 · Cited by 47 · see citing works
Self-citationFerreira, C., Bastille‐Rousseau, G., Bennett, A. M., Ellington, E. H., Terwissen, C. V., Austin, C., Borlestean, A., Boudreau, M. R., Chan, K., Forsythe, A., et al. (2015) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.12185 · Cited by 34 · see citing works
Maximizing the value of systematic reviews in ecology when data or resources are limited
Doerr, E. D., Dorrough, J. W., Davies, M., Doerr, V. A. J., & McIntyre, S. (2014) · Austral Ecology · doi:10.1111/aec.12179 · Cited by 27 · see citing works
Controlled vocabularies for scientific data: Users and desired functionalities
Zhang, Y., Ogletree, A., Greenberg, J., et al. (2015) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.2015.145052010054 · Cited by 15 · see citing works
Automated tagging of environmental data using a novel SKOS formatted environmental thesaurus
Wright, D. G., Harrison, K. A., & Watkins, J. W. (2014) · Earth Science Informatics · doi:10.1007/s12145-014-0183-1 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- Collaboration for Environmental Evidence. 2013. Guidelines for systematic review and evidence synthesis in environmental management. Version 4.2. Environmental Evidence, Bangor, United Kingdom. Available from http://www.environmentalevidence.org/wp-content/uploads/2014/06/Review- guidelines-version-4.2-final.pdf (accessed November 2016).
- Fellbaum, C., editor. 1998. Wordnet: an electronic lexical database. MIT Press, Cambridge, Massachusetts.
- Garnier, E., et al. 2016. Towards a thesaurus of plant characteristics: an ecological contribution. Journal of Ecology:in press.
- National Information Standards Organization 2005. Guidelines for the construction, format, and management of monolingual controlled vocabularies. NISO Press, Baltimore, Maryland.
Cited by
Sorted by citations
revtools: An R package to support article screening for evidence synthesis
Westgate, M. J. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1374 · Cited by 261 · see citing works
Self-citationPredicting the time needed for environmental systematic reviews and systematic maps
Haddaway, N. R., & Westgate, M. J. (2019) · Conservation Biology · doi:10.1111/cobi.13231 · Cited by 120 · see citing works
Self-citationSoftware support for environmental evidence synthesis
Westgate, M. J., Haddaway, N. R., Cheng, S. H., McIntosh, E. J., Marshall, C., & Lindenmayer, D. B. (2018) · Nature Ecology & Evolution · doi:10.1038/s41559-018-0502-x · Cited by 51 · see citing works
Self-citationUsing machine learning to disentangle homonyms in large text corpora
Roll, U., Correia, R. A., & Berger‐Tal, O. (2018) · Conservation Biology · doi:10.1111/cobi.13044 · Cited by 46 · see citing works
Lagisz, M., Yang, Y., Young, S., & Nakagawa, S. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2024.6 · Cited by 24 · see citing works
- Authors
- Westgate, M. J., & Lindenmayer, D. B.
- Affiliations
- Australian National University (AU)
- Corresponding author
- Westgate, Martin J.
- Venue
- Conservation Biology
- Publisher
- Wiley
- Published
- 2017-06-20
- Volume
- 31
- Issue
- 5
- Pages
- 1002-1007
- ISSN
- 0888-8892 (ISSN-L)
- 1523-1739
- OpenAlex
- W2567560848
- Cited by
- 62
- References
- 38
- Citations by year
- 2026: 6 · 2025: 3 · 2024: 5 · 2023: 1 · 2022: 20 · 2021: 3 · 2020: 7 · 2019: 11 · 2018: 4 · 2017: 2
- FWCI
- 1.61
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- Locations
- Conservation Biology (publishedVersion)
- PubMed (publishedVersion)
- ANU Open Research (Australian National University) (submittedVersion)
- Funders
- Australian Research Council (fl120100108)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- systematic reviews, evidence synthesis, semantic networks, small-world networks, natural language processing, ontology (information science), scientific writing
- MeSH terms
- Conservation of Natural Resources, Forecasting, Humans, Linguistics, Research Personnel, Review Literature as Topic, Meta-Analysis as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
