An assessment of statistical methods for nonindependent data in ecological meta‐analyses
Abstract
In ecological meta-analyses, nonindependence among observed effect sizes from the same source paper is common. If not accounted for, nonindependence can seriously undermine inferences. We compared the performance of four meta-analysis methods that attempt to address such nonindependence and the standard random-effect model that ignores nonindependence. We simulated data with various types of within-paper nonindependence, and assessed the standard deviation of the estimated mean effect size and Type I error rate of each method. Although all four methods performed substantially better than the standard random-effects model that assumes independence, there were differences in performance among the methods. A two-step method that first summarizes the multiple observed effect sizes per paper using a weighted mean and then analyzes the reduced data in a standard random-effects model, and a robust variance estimation method performed consistently well. A hierarchical model with both random paper and study effects gave precise estimates but had a higher Type I error rates, possibly reflecting limitations of currently available meta-analysis software. Overall, we advocate the use of the two-step method with a weighted paper mean and the robust variance estimation method as reliable ways to handle within-paper nonindependence in ecological meta-analyses.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The multilevel structure effectively partitions the correlation structures within studies and allows the inclusion of multiple effect sizes per study.
“The multilevel structure effectively partitions the correlation structures within studies and allows the inclusion of multiple effect sizes per study (Nakagawa & Santos, 2012; Song et al., 2020).”
From Acoustic indices as proxies for biodiversity: a meta‐analysis (2022) · cites this paper as Song et al., 2020
Cites
Small sample inference for fixed effects from restricted maximum likelihood
Kenward, M. G., & Roger, J. H. (1997) · Biometrics · doi:10.2307/2533558 · Cited by 4,552 · see citing works
Meta-analysis and the science of research synthesis
Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018) · Nature · doi:10.1038/nature25753 · Cited by 1,709 · cited 1× · see citing works
“…(Jarvinen 1991, Arnqvist and Wooster 1995), meta-analysis has played an increasingly influential role in the field, such as testing ecological theories, identifying research directions, and informing conservation and management strategies (Stewart 2009, Cadotte et al. 2012, Gurevitch et al. 2018).”
BackgroundMethods to estimate the between‐study variance and its uncertainty in meta‐analysis
Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P., Langan, D., & Salanti, G. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1164 · Cited by 1,685 · see citing works
Methodological issues and advances in biological meta-analysis
Nakagawa, S., & Santos, E. S. A. (2012) · Evolutionary Ecology · doi:10.1007/s10682-012-9555-5 · Cited by 1,032 · see citing works
Meta-evaluation of meta-analysis: ten appraisal questions for biologists
Nakagawa, S., Noble, D. W. A., Senior, A. M., & Lagisz, M. (2017) · BMC Biology · doi:10.1186/s12915-017-0357-7 · Cited by 579 · see citing works
Meta-analysis: synthesizing research findings in ecology and evolution
Arnqvist, G., & Wooster, D. (1995) · Trends in Ecology & Evolution · doi:10.1016/s0169-5347(00)89073-4 · Cited by 570 · see citing works
A simple confidence interval for meta-analysis
Sidik, K., & Jonkman, J. N. (2002) · Statistics in Medicine · doi:10.1002/sim.1262 · Cited by 431 · cited 1× · see citing works
“We constructed confidence intervals based on the adjustment proposed by Hartung and Knapp (2001) and Sidik and Jonkman (2002).”
MethodHeterogeneity in ecological and evolutionary meta‐analyses: its magnitude and implications
Senior, A. M., Grueber, C. E., Kamiya, T., Lagisz, M., O'Dwyer, K., Santos, E. S. A., & Nakagawa, S. (2016) · Ecology · doi:10.1002/ecy.1591 · Cited by 299 · see citing works
How to critically read ecological meta-analyses
Lortie, C. J., Stewart, G., Rothstein, H., & Lau, J. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1109 · Cited by 39 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Methods for testing publication bias in ecological and evolutionary meta‐analyses
Nakagawa, S., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Sánchez‐Tójar, A., Yang, Y., & O’Dea, R. E. (2021) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13724 · Cited by 455 · see citing works
“Overall means will generally not be biased using aggregated or single sample/study effect sizes (Song et al. , 2020).”
Acoustic indices as proxies for biodiversity: a meta‐analysis
Alcocer, I., Lima, H., Sugai, L. S. M., & Llusia, D. (2022) · Biological Reviews · doi:10.1111/brv.12890 · Cited by 273 · see citing works
“The multilevel structure effectively partitions the correlation structures within studies and allows the inclusion of multiple effect sizes per study (Nakagawa & Santos, 2012; Song et al., 2020).”
Method<scp>EEG</scp> power spectral measures of cognitive workload: A meta‐analysis
Chikhi, S., Matton, N., & Blanchet, S. (2022) · Psychophysiology · doi:10.1111/psyp.14009 · Cited by 258 · see citing works
Lu, X., Hou, E., Guo, J., Gilliam, F. S., Li, J., Tang, S., & Kuang, Y. (2021) · Global Change Biology · doi:10.1111/gcb.15604 · Cited by 228 · see citing works
“A two- step method described by Song et al. (2020) was used to handle within- paper non- independence in observations, that is, a F I G U R E 1 Distribution of N manipulation experiments included in the meta- analysis.”
MethodShi, T.-S., Collins, S. L., Yu, K., Peñuelas, J., Sardans, J., Li, H., & Ye, J. (2024) · Nature Communications · doi:10.1038/s41467-024-47829-w · Cited by 179 · see citing works
“Because multiple treatments may share a single control, we added “ site ” as a random factor in the meta-analysis model to account for non-independence of observations collected from the same site 68,69 .”
Lehmann, A., Leifheit, E. F., Gerdawischke, M., & Rillig, M. C. (2021) · Microplastics and Nanoplastics · doi:10.1186/s43591-021-00007-x · Cited by 213 · see citing works
“To overcome this bias of nonindependent data, we used a two-step approach [33].”
MethodCinar, O., Nakagawa, S., & Viechtbauer, W. (2021) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13760 · Cited by 102 · see citing works
Bekchanova, M., Campion, L., Bruns, S. B., Kuppens, T. E., Lehmann, J. C., Jozefczak, M., Cuypers, A., & Malina, R. M. (2024) · Environmental Evidence · doi:10.1186/s13750-024-00326-5 · Cited by 93 · see citing works
“Each study may report multiple effect sizes for the same SPP [61], violating the traditional independency assumption [62].”
BackgroundResponse of soil microbial community parameters to plastic film mulch: A meta-analysis
Li, Y., Xie, H., Ren, Z., Ding, Y., Long, M., Zhang, G., Qin, X., Siddique, K. H. M., & Liao, Y. (2022) · Geoderma · doi:10.1016/j.geoderma.2022.115851 · Cited by 79 · see citing works
Comparing Productivity of Organic and Conventional Farming Systems: A Quantitative Review
Álvarez, R. (2021) · Archives of Agronomy and Soil Science · doi:10.1080/03650340.2021.1946040 · Cited by 77 · see citing works
“…mainly because the use of the same control that was compared with several organic treatments (multiple comparisons), was corrected by the classical method of averaging the data from the organic treatments to generate only a couple of comparisons per study (Noble et al. 2017; Song et al. 2020).”
MethodNessel, M. P., Konnovitch, T., Romero, G. Q., & González, A. L. (2021) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.12771 · Cited by 88 · see citing works
“We also generated models with a hierarchical random effect structure using study as a random effect to address nonindependence without decreasing sample size (Song et al., 2020).”
MethodMeta‐analysis of ecosystem services associated with oyster restoration
Smith, R. S., Cheng, S. L., & Castorani, M. C. N. (2022) · Conservation Biology · doi:10.1111/cobi.13966 · Cited by 75 · see citing works
Wu, S., Shi, Z., Chen, X., Gao, J., & Wang, X. (2022) · PeerJ · doi:10.7717/peerj.12861 · Cited by 63 · see citing works
Petsch, D. K., Bertoncin, A. P. D. S., Ortega, J. C. G., & Thomaz, S. M. (2022) · Oikos · doi:10.1111/oik.08768 · Cited by 69 · see citing works
Yang, Y., Macleod, M. R., Pan, J., Lagisz, M., & Nakagawa, S. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.105016 · Cited by 49 · see citing works
Francksen, R. M., Turnbull, S., Rhymer, C. M., Hiron, M., Bufe, C., Klaus, V. H., Newell‐Price, P., Stewart, G. B., & Whittingham, M. J. (2022) · Agronomy · doi:10.3390/agronomy12122928 · Cited by 30 · see citing works
Decline of insects and arachnids driven by nutrient enrichment: A meta‐analysis
Nessel, M. P., Konnovitch, T., Romero, G. Q., & González, A. L. (2022) · Ecology · doi:10.1002/ecy.3897 · Cited by 31 · see citing works
“The benefit of this approach compared with others is it allows each study to contribute one effect size to the overall mean effect size estimations, which is ideal when you have a wide variability in the number of case studies per paper (Nakagawa et al., 2017; Song et al., 2020).”
BackgroundCrop diversification and parasitic weed abundance: a global meta-analysis
Scott, D., & Freckleton, R. P. (2022) · Scientific Reports · doi:10.1038/s41598-022-24047-2 · Cited by 27 · see citing works
An assessment of statistical methods for nonindependent data in ecological meta‐analyses: Comment
Nakagawa, S., Senior, A. M., Viechtbauer, W., & Noble, D. W. A. (2021) · Ecology · doi:10.1002/ecy.3490 · Cited by 24 · see citing works
“OPEN RESEARCH Code (Nakagawa et al. 2021) is available on Zenodo: https://doi.org/10.5281/zenodo.4774381.”
BackgroundMethodInfluentialA quantitative review of the effects of residue removing on soil organic carbon in croplands
Álvarez, R. (2024) · Soil and Tillage Research · doi:10.1016/j.still.2024.106098 · Cited by 29 · see citing works
de Vries, N., Meeter, M., & Huizinga, M. (2024) · Educational Research and Reviews · doi:10.1016/j.edurev.2024.100619 · Cited by 22 · see citing works
Noble, D. W. A., Kar, F., Bush, A., Seebacher, F., & Nakagawa, S. (2025) · Functional Ecology · doi:10.1111/1365-2435.70031 · Cited by 22 · see citing works
“In addition, Bayesian methods better protect against type I errors in the presence of complex sources of non-independence (Noble et al. 2017; Nakagawa et al. 2021b; Song et al. 2021).”
Fire decreases soil respiration and its components in terrestrial ecosystems
Zhou, L., Liu, S.-S., Gu, Y., Wu, L., Hu, H., & He, J.-Z. (2023) · Functional Ecology · doi:10.1111/1365-2435.14443 · Cited by 17
Zhang, X., Jia, S., You, C., Xu, H., Yuan, Y., Li, J., Liu, S., Tan, B., Xu, Z., Sardans, J., et al. (2025) · Soil and Tillage Research · doi:10.1016/j.still.2025.106752 · Cited by 12 · see citing works
Quality of meta‐analyses in freshwater ecology: A systematic review
Lodi, S., Godoy, B. S., Ortega, J. C. G., & Bini, L. M. (2021) · Freshwater Biology · doi:10.1111/fwb.13695 · Cited by 12 · see citing works
“Accordingly, a metaanalysis ignoring the fact that multiple effect sizes from the same primary studies are not independent would produce results with inflated type I error rates (see also Song et al., 2020; Mengersen et al., 2013; Van den Noortgate et al., 2014).”
BackgroundOduro, C., Geng, Y., Dalu, T., Okrah, A., Kyei–Nuamah, D., Khan, S., Chen, J., Mu, H., & Wu, N. (2025) · Environmental Sciences Europe · doi:10.1186/s12302-025-01227-1 · Cited by 7 · see citing works
“These categorization was chosen based on their ecological relevance and the frequency with which they were reported in the studies, allowing for a comprehensive assessment of the combined effects of warming and microplastic pollution on freshwater macroinvertebrates [75].”
Courtice, G., Bauer, B., Cahill, C. L., Naser, G., & Paul, A. (2021) · Science of the Total Environment · doi:10.1016/j.scitotenv.2021.150844 · Cited by 6
Biodiversity survey and estimation for line-transect sampling
Chen, Y. F., Wang, R.-H., & Shen, T. (2023) · Frontiers in Plant Science · doi:10.3389/fpls.2023.1159090 · Cited by 5 · see citing works
“To this end, it is necessary to develop statistical methods that can consider non-independent sampling of individuals (Chen et al., 2019; Song et al., 2020).”
BackgroundQiao, B.-Y., He, Y.-C., Wang, S.-Y., Li, M.-L., Han, B., Wang, Q., Zhang, R.-Z., & Shao, X.-Q. (2025) · CATENA · doi:10.1016/j.catena.2025.109599 · Cited by 3
Álvarez, R. (2025) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2025.179933 · Cited by 2 · see citing works
Negative effects of elevated ozone levels on soil microbial characteristics: a meta-analysis
Lu, X., Li, J., Zhou, X., & Yue, X. (2025) · Plant and Soil · doi:10.1007/s11104-025-07309-6 · Cited by 3 · see citing works
“Here, we used a “two-step” method described by Song et al. (2020) to handle the nonindependence of within-paper data if one study included more than one observation.”
A meta-evaluation of the quality of reporting and execution in ecological meta-analyses
Pappalardo, P., Song, C., Hungate, B. A., & Osenberg, C. W. (2023) · PLoS ONE · doi:10.1371/journal.pone.0292606 · Cited by 4 · see citing works
Self-citationWilkinson, E. M., Jones, H. G., Kohler, M. J., Khalsa, S. S., Moseley, G. L., & Berryman, C. F. (2026) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2026.04.012 · Cited by 2 · see citing works
Gong, W., Wen, Z.-Y., Liang, G., Bu, Q., Liu, X., Lu, A., & Geng, D. (2026) · Agriculture Communications · doi:10.1016/j.agrcom.2026.100124 · Cited by 2 · see citing works
Pesta, B. J., Nijenhuis, J. T., Lasker, J., Kirkegaard, E. O. W., & Fuerst, J. G. R. (2023) · Intelligence · doi:10.1016/j.intell.2023.101737 · Cited by 3 · see citing works
“7 However, our focus was on variance component means, for which the averaging method produces unbiased estimates (Moeyaert et al., 2017; Song et al., 2020).”
Qamar, A., Shad, N., Ali, A., Nawaz, N., & Guo, C. (2026) · Plants · doi:10.3390/plants15213301
Li, D., Gao, M., Yu, D., Mao, J., & Bai, Y. (2026) · Journal of Hazardous Materials · doi:10.1016/j.jhazmat.2026.143763
Ecological impacts of woody plant invasions on soil biochemical properties: A meta-analysis
Pu, J., Cai, L., & Mak‐Mensah, E. (2026) · Biological Invasions · doi:10.1007/s10530-026-03875-7
“[ ( ) ] This structure accounts for non-independence of data arising from multiple species, fields, or variables within studies, following best practices as outlined in Song et al. (2020).”
Implications of Maternal Toxicant Effects on Size‐Dependent Fisheries' Management: A Meta‐Analysis
Sosa, B. M., Malinowski, C. R., & Höök, T. O. (2026) · Fish and Fisheries · doi:10.1111/faf.70089
Du, C.-J., Wu, X., Shi, H., Fu, B.-J., Li, T., Lu, Y.-H., Wan, L., Zhang, S., Huang, B., & Shi, Y. (2025) · Agriculture, Ecosystems & Environment · doi:10.1016/j.agee.2025.109880
- Authors
- Song, C., Peacor, S. D., Osenberg, C. W., & Bence, J. R.
- Affiliations
- Michigan State University (US)
- University of Georgia (US)
- Corresponding author
- Song, Chao
- Venue
- Ecology
- Publisher
- Wiley
- Published
- 2020-10-07
- Volume
- 101
- Issue
- 12
- Pages
- e03184-e03184
- ISSN
- 0012-9658 (ISSN-L)
- 1939-9170
- OpenAlex
- W4206730696
- Cited by
- 71
- References
- 0
- Citations by year
- 2026: 7 · 2025: 7 · 2024: 20 · 2023: 4 · 2022: 21 · 2021: 12
- FWCI
- 1.91
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Ecology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Division of Environmental Biology (DEB–1655394, DEB–1655426)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- robust variance estimation, two-stage algorithm, hierarchical models, random effects models, type I error rate, meta-analysis software
- MeSH terms
- Research Design, Software, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
