The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Simple study designs in ecology produce inaccurate estimates of biodiversity responses
Open accessAbstract
Monitoring the impacts of anthropogenic threats and interventions to mitigate these threats is key to understanding how to best conserve biodiversity. Ecologists use many different study designs to monitor such impacts. Simpler designs lacking controls (e.g. Before–After (BA) and After) or pre‐impact data (e.g. Control–Impact (CI)) are considered to be less robust than more complex designs (e.g. Before–After Control‐Impact (BACI) or Randomized Controlled Trials (RCTs)). However, we lack quantitative estimates of how much less accurate simpler study designs are in ecology. Understanding this could help prioritize research and weight studies by their design's accuracy in meta‐analysis and evidence assessment. We compared how accurately five study designs estimated the true effect of a simulated environmental impact that caused a step‐change response in a population's density. We derived empirical estimates of several simulation parameters from 47 ecological datasets to ensure our simulations were realistic. We measured design performance by determining the percentage of simulations where: (a) the true effect fell within the 95% Confidence Intervals of effect size estimates, and (b) each design correctly estimated the true effect's direction and magnitude. We also considered how sample size affected their performance. We demonstrated that BACI designs performed: 1.3–1.8 times better than RCTs; 2.9–4.2 times versus BA; 3.2–4.6 times versus CI; and 7.1–10.1 times versus After designs (depending on sample size), when correctly estimating true effect's direction and magnitude to within ±30%. Although BACI designs suffered from low power at small sample sizes, they outperformed other designs for almost all performance measures. Increasing sample size improved BACI design accuracy, but only increased the precision of simpler designs around biased estimates. Synthesis and applications . We suggest that more investment in more robust designs is needed in ecology since inferences from simpler designs, even with large sample sizes may be misleading. Facilitating this requires longer‐term funding and stronger research–practice partnerships. We also propose ‘accuracy weights’ and demonstrate how they can weight studies in three recent meta‐analyses by accounting for study design and sample size. We hope these help decision‐makers and meta‐analysts better account for study design when assessing evidence.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
Conditional logit analysis of qualitative choice behavior
McFadden, D. L. (1972) · eScholarship (California Digital Library) · Cited by 14,203
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Intervention Analysis with Applications to Economic and Environmental Problems
Box, G. E. P., & Tiao, G. C. (1975) · Journal of the American Statistical Association · doi:10.1080/01621459.1975.10480264 · Cited by 2,383
Environmental Impact Assessment: "Pseudoreplication" in Time?
Stewart-Oaten, A., Murdoch, W. W., & Parker, K. R. (1986) · Ecology · doi:10.2307/1939815 · Cited by 1,331
STATISTICAL ISSUES IN ECOLOGICAL META-ANALYSES
Gurevitch, J., & Hedges, L. V. (1999) · Ecology · doi:10.1890/0012-9658(1999)080[1142:siiema]2.0.co;2 · Cited by 1,081
Comparative evaluation of experimental approaches to the study of habitat fragmentation effects
McGarigal, K., & Cushman, S. A. (2002) · Ecological Applications · doi:10.1890/1051-0761(2002)012[0335:ceoeat]2.0.co;2 · Cited by 731
RESAMPLING TESTS FOR META-ANALYSIS OF ECOLOGICAL DATA
Adams, D. C., Gurevitch, J., & Rosenberg, M. S. (1997) · Ecology · doi:10.1890/0012-9658(1997)078[1277:rtfmao]2.0.co;2 · Cited by 688
Underreporting Research Is Scientific Misconduct
Chalmers, L. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100121018 · Cited by 551
Foundations of Restoration Ecology
Reid, J. L., Acha, S., Becknell, R. E., Domene, F., Figueiredo, A., Haake, D., Huang, W., Loza, I., Mashburn, B., Moreira, J. I., et al. (2017) · Restoration Ecology · doi:10.1111/rec.12592 · Cited by 493
Uses and misuses of meta‐analysis in plant ecology
Koricheva, J., & Gurevitch, J. (2014) · Journal of Ecology · doi:10.1111/1365-2745.12224 · Cited by 460
Clinical Epidemiology: How to Do Clinical Practice Research
Heneghan, C. (2006) · ACP Journal Club · doi:10.7326/acpjc-2006-145-1-a11 · Cited by 403
TEMPORAL AND SPATIAL VARIATION IN ENVIRONMENTAL IMPACT ASSESSMENT
Stewart-Oaten, A., & Bence, J. R. (2001) · Ecological Monographs · doi:10.1890/0012-9615(2001)071[0305:tasvie]2.0.co;2 · Cited by 355
Block, W. M., Franklin, A. B., Ward, J. P., Ganey, J. L., & White, G. C. (2001) · Restoration Ecology · doi:10.1046/j.1526-100x.2001.009003293.x · Cited by 336
A Critique of the Space-for-Time Substitution Practice in Community Ecology
Damgaard, C. (2019) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2019.01.013 · Cited by 302
The effects of fungal root endophytes on plant growth: a meta-analysis
Mayerhofer, M. S., Kernaghan, G., & Harper, K. A. (2013) · Mycorrhiza · doi:10.1007/s00572-012-0456-9 · Cited by 269
The Importance of Replication in Wildlife Research
Johnson, D. H. (2002) · The Journal of Wildlife Management · doi:10.2307/3802926 · Cited by 256
Sutherland, W. J., Taylor, N. G., MacFarlane, D., Amano, T., Christie, A. P., Dicks, L. V., Lemasson, A. J., Littlewood, N. A., Martin, P. A., Ockendon, N., et al. (2019) · Biological Conservation · doi:10.1016/j.biocon.2019.108199 · Cited by 232
Self-citationMeta-analysis in applied ecology
Stewart, G. (2010) · Biology Letters · doi:10.1098/rsbl.2009.0546 · Cited by 214
Weighting by Inverse Variance or by Sample Size in Random-Effects Meta-Analysis
Marín-Martínez, F., & Sánchez-Meca, J. (2010) · Educational and Psychological Measurement · doi:10.1177/0013164409344534 · Cited by 213
The Sutton Trust-Education Endowment Foundation Teaching and Learning Toolkit.
Higgins, S., Katsipataki, M., Aguilera, A. B. V., Coleman, R. A., Henderson, P., Major, L. E., Coe, R., & Mason, D. S. (2013) · Durham Research Online (Durham University) · Cited by 208
Bernes, C., Macura, B., Jonsson, B. G., Junninen, K., Müller, J. C., Sandström, J., Lõhmus, A., & Macdonald, S. E. (2018) · Environmental Evidence · doi:10.1186/s13750-018-0125-3 · Cited by 186
Smokorowski, K. E., & Randall, R. G. (2017) · FACETS · doi:10.1139/facets-2016-0058 · Cited by 183
Evaluating impacts using a BACI design, ratios, and a Bayesian approach with a focus on restoration
Conner, M. M., Saunders, W. C., Bouwes, N., & Jordan, C. E. (2016) · Environmental Monitoring and Assessment · doi:10.1007/s10661-016-5526-6 · Cited by 178
Sandström, J., Bernes, C., Junninen, K., Lõhmus, A., Macdonald, E., Müller, J., & Jonsson, B. G. (2019) · Journal of Applied Ecology · doi:10.1111/1365-2664.13395 · Cited by 170
Moland, E., Olsen, E. M., Knutsen, H., Garrigou, P., Espeland, S. H., Kleiven, A. R., André, C., & Knutsen, J. A. (2013) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2012.2679 · Cited by 155
Causal analysis in control–impact ecological studies with observational data
Larsen, A., Meng, K. C., & Kendall, B. E. (2019) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13190 · Cited by 149
França, F. M., Louzada, J., Korasaki, V., Griffiths, H. M., Silveira, J. M., & Barlow, J. (2016) · Journal of Applied Ecology · doi:10.1111/1365-2664.12657 · Cited by 147
Power, precaution, Type II error and sampling design in assessment of environmental impacts
Underwood, A. J. (2003) · Journal of Experimental Marine Biology and Ecology · doi:10.1016/s0022-0981(03)00304-6 · Cited by 138
Palma, A. D., Sánchez-Ortiz, K., Martin, P. A., Chadwick, A. J., Gilbert, G., Bates, A. E., Börger, L., Contu, S., Hill, S. L. L., & Purvis, A. (2018) · Advances in ecological research/Advances in Ecological Research · doi:10.1016/bs.aecr.2017.12.004 · Cited by 138
Self-citationASSESSING ENVIRONMENTAL IMPACTS: EFFECTS OF SPATIAL AND TEMPORAL VARIABILITY AT LIKELY IMPACT SCALES
Hewitt, J. E., Thrush, S., & Cummings, V. J. (2001) · Ecological Applications · doi:10.1890/1051-0761(2001)011[1502:aeieos]2.0.co;2 · Cited by 136
Wolfe, D. A., Champ, M. A., Flemer, D. A., & Mearns, A. J. (1987) · Estuaries · doi:10.2307/1351847 · Cited by 125
Webb, J. A., Miller, K. A., King, E. L., de Little, S. C., Stewardson, M. J., Zimmerman, J. K. H., & Poff, N. L. (2013) · Freshwater Biology · doi:10.1111/fwb.12234 · Cited by 123
Analyzing cause and effect in environmental assessments: using weighted evidence from the literature
Norris, R. H., Webb, J. A., Nichols, S. J., Stewardson, M. J., & Harrison, E. T. (2012) · Freshwater Science · doi:10.1899/11-027.1 · Cited by 113
Quality assessment tools for evidence from environmental science
Bilotta, G. S., Milner, A. M., & Boyd, I. L. (2014) · Environmental Evidence · doi:10.1186/2047-2382-3-14 · Cited by 113
Detecting Ecological Impacts Caused by Human Activities
Osenberg, C. W., & Schmitt, R. J. (1996) · Elsevier eBooks · doi:10.1016/b978-012627255-0/50003-3 · Cited by 105
Eales, J. F., Haddaway, N., Bernes, C., Cooke, S. J., Jonsson, B. G., Kouki, J., Petrokofsky, G., & Taylor, J. J. (2018) · Environmental Evidence · doi:10.1186/s13750-018-0131-5 · Cited by 100
Wauchope, H. S., Amano, T., Sutherland, W. J., & Johnston, A. (2019) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13302 · Cited by 89
Self-citationBernes, C., Jonsson, B. G., Junninen, K., Lõhmus, A., Macdonald, S. E., Müller, J. C., & Sandström, J. (2015) · Environmental Evidence · doi:10.1186/s13750-015-0050-7 · Cited by 79
Chevalier, M., Russell, J. C., & Knape, J. (2018) · Ecological Applications · doi:10.1002/eap.1838 · Cited by 72
An evidence assessment tool for ecosystem services and conservation studies
Mupepele, A., Walsh, J. C., Sutherland, W. J., & Dormann, C. F. (2015) · Ecological Applications · doi:10.1890/15-0595 · Cited by 68
Self-citationResampling Tests for Meta-Analysis of Ecological Data
Adams, D. C., Gurevitch, J., & Rosenberg, M. S. (1997) · Ecology · doi:10.2307/2265879 · Cited by 64
Progressive‐Change BACIPS: a flexible approach for environmental impact assessment
Thiault, L., Kernaléguen, L., Osenberg, C. W., & Claudet, J. (2016) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12655 · Cited by 63
Use of meta-analysis in forest biodiversity research: key challenges and considerations
Spake, R., & Doncaster, C. P. (2017) · Forest Ecology and Management · doi:10.1016/j.foreco.2017.05.059 · Cited by 60
Problems in the Analysis of Environmental Monitoring Data
Stewart‐Oaten, A. (1996) · Elsevier eBooks · doi:10.1016/b978-012627255-0/50009-4 · Cited by 57
Bernes, C., Bullock, J. M., Jakobsson, S., Rundlöf, M., Verheyen, K., & Lindborg, R. (2017) · Environmental Evidence · doi:10.1186/s13750-017-0103-1 · Cited by 51
A new method for identifying rapid decline dynamics in wild vertebrate populations
Fonzo, M. M. I. D., Collen, B., & Mace, G. M. (2013) · Ecology and Evolution · doi:10.1002/ece3.596 · Cited by 48
Monitoring Ecological Impacts: Concepts and Practice in Flowing Waters
Downes, B. J., Barmuta, L. A., Fairweather, P. G., Faith, D. P., Keough, M. J., Lake, P. S., Mapstone, B. D., & Quinn, G. P. (2002) · doi:10.1017/cbo9780511542015 · Cited by 42
ECOLOGY – Assessing effects of marine protected areas: confounding in space and possible solutions
Osenberg, C. W., Shima, J., Miller, S. L., & Stier, A. C. (2011) · Cambridge University Press eBooks · doi:10.1017/cbo9781139049382.010 · Cited by 42
Ecological Responses to Flow Alteration: Assessing Causal Relationships with Eco Evidence
Webb, J. A., Nichols, S. J., Norris, R. H., Stewardson, M. J., Wealands, S. R., & Lea, P. (2011) · Wetlands · doi:10.1007/s13157-011-0249-5 · Cited by 41
Removal of corallivorous snails as a proactive tool for the conservation of acroporid corals
Williams, D. E., Miller, M. W., Bright, A., & Cameron, C. (2014) · PeerJ · doi:10.7717/peerj.680 · Cited by 27
Influence of Forest Harvest on Nitrate Concentration in Temperate Streams—A Meta-Analysis
Mupepele, A., & Dormann, C. F. (2016) · Forests · doi:10.3390/f8010005 · Cited by 27
Sepúlveda, R. D., & Valdivia, N. (2016) · PLoS ONE · doi:10.1371/journal.pone.0157910 · Cited by 12
When can we trust population trends? Quantifying the effects of sampling interval and durationPreprint
Wauchope, H. S., Johnston, A., Amano, T., & Sutherland, W. J. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/498170 · Cited by 3
Self-citation
Cited by
COVID-19 lockdown allows researchers to quantify the effects of human activity on wildlife
Rutz, C., Loretto, M.-C., Bates, A. E., Davidson, S. C., Duarte, C. M., Jetz, W., Johnson, M., Kato, A., Kays, R., Mueller, T., et al. (2020) · Nature Ecology & Evolution · doi:10.1038/s41559-020-1237-z · Cited by 658
The challenge of biased evidence in conservation
Christie, A. P., Amano, T., Martin, P. A., Petrovan, S. O., Shackelford, G. E., Simmons, B. I., Smith, R. K., Williams, D. R., Wordley, C. F. R., & Sutherland, W. J. (2021) · Conservation Biology · doi:10.1111/cobi.13577 · Cited by 202
Self-citationTeixeira, D., Roe, P., van Rensburg, B. J., Linke, S., McDonald, P. G., Tucker, D., & Fuller, S. (2024) · Conservation Science and Practice · doi:10.1111/csp2.13132 · Cited by 38
Metadata
- Authors
- Christie, A. P., Amano, T., Martin, P. A., Shackelford, G. E., Simmons, B. I., & Sutherland, W. J.
- Venue
- Journal of Applied Ecology
- Publisher
- Wiley
- Published
- 2019-09-19
- Volume
- 56
- Issue
- 12
- Pages
- 2742-2754
- ISSN
- 0021-8901 (ISSN-L)
- 1365-2664
- OpenAlex
- W2971333488
- Cited by
- 375
- References
- 56
- FWCI
- 26.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.