Meta-analysis and the science of research synthesis
Open accessThe DOI matches its Crossref record; 102 of the 102 references it cites were found in a registry or an index; 9 of the 69 sources its claims cite were checked for support.
Abstract
Meta-analysis is the quantitative, scientific synthesis of research results. Since the term and modern approaches to research synthesis were first introduced in the 1970s, meta-analysis has had a revolutionary effect in many scientific fields, helping to establish evidence-based practice and to resolve seemingly contradictory research outcomes. At the same time, its implementation has engendered criticism and controversy, in some cases general and others specific to particular disciplines. Here we take the opportunity provided by the recent fortieth anniversary of meta-analysis to reflect on the accomplishments, limitations, recent advances and directions for future developments in the field of research synthesis.
Overview
CoChat Summary
TL;DR
This review reflects on meta-analysis as quantitative research synthesis since the 1970s, its contributions to evidence-based practice, and its methodological development across medicine, social sciences, and ecology/evolution/conservation. It argues that meta-analysis has resolved contradictory evidence and enabled broad generalizations, but faces a “midlife crisis” from poor-quality applications, publication/research bias, and misuse of the term.
Key findings
- Meta-analysis was first introduced in the 1970s; the term was coined by Gene Glass in 1976, and the paper marks its 40th anniversary. It has helped establish evidence-based practice, resolve seemingly contradictory results, and spread across many scientific fields.
- In medicine, a meta-analysis of 20 clinical studies found maternal obesity associated with increased neural tube defect risk: a 1.7-fold rise for moderately obese women and over a 3-fold risk for severely obese women. In social science, a meta-analysis of 35 studies found no significant differences between multisystemic therapy and conventional social services for serious juvenile offenders.
- In ecology, evolution and conservation, meta-analysis was adopted in the early 1990s. Parmesan and Yohe used data on over 1,700 species to show significant range shifts and advancement of spring events; Benayas and colleagues found ecological restoration increased biodiversity and ecosystem services, though values remained lower in restored versus intact reference ecosystems.
- Methodological advances include fixed- and random-effects models, meta-regression, heterogeneity quantification (τ and I²), publication-bias tools, network meta-analysis, phylogenetic meta-analysis, and software such as MetaWin and OpenMEE. PRISMA includes a 27-item checklist for systematic review reporting.
Methods
- Design: narrative review/commentary on meta-analysis and research synthesis, not a new meta-analysis. The authors state they highlight principles, limitations, utility, and future developments.
- Data or sample: draws on published literature and illustrative examples across medicine, social sciences, and ecology/evolution/conservation; includes Box 1 on meta-analytic models and Table 1 on EEC milestones.
- Main analysis: conceptual and methodological synthesis; no primary data collection, systematic search protocol, or quantitative pooling is reported in the paper.
Limitations
- The paper states that meta-analysis and systematic reviews can highlight evidence deficiencies but cannot overcome them; publication bias and research bias may be suspected and sometimes estimated but cannot truly be corrected; selective or incomplete reporting can constrain synthesis.
- The paper states that too many published meta-analyses are of low quality, the term “meta-analysis” is frequently misapplied, and vote-counting persists despite being discredited; it also notes concerns about academic inequality and the “research parasites” controversy.
- The text makes evident that, as a narrative review, it does not provide a systematic search strategy, formal inclusion criteria, or a quantitative synthesis; its examples are illustrative rather than exhaustive.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 4 Not valid · 37 Not checked · 33 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Similarly, a synthesis may be constrained by either selective or incomplete reporting 66,67 .”
What the sources say
- Not checked yetcites [66] Stewart, L. A . et al. Preferred reporting items for a systematic review and meta-analysis of individual participant data: the PRISMA-IPD statement. J. Am. Med. Assoc. 313, 1657–1665 (2015) source
- Partly supportedcites [67] Saldanha, I. J . et al. Evaluating Data Abstraction Assistant, a novel software application for data abstraction during systematic reviews: protocol for a randomized controlled trial. Syst. Rev. 5, 196 (2016) sourceExperimental — accuracy not yet measured
The Background (page unavailable) reports that omissions and errors during data abstraction changed summary meta-analytic results, supporting the general point that a synthesis can be affected by missing information. It describes information present in articles but omitted by abstractors, not selective or incomplete reporting by the studies themselves; that specific mechanism is unsupported. (LLM, conf=0.78)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“When the number of studies is an order of magnitude greater than that, the inability of narrative processes to adequately summarize the outcomes increases exponentially 2,3 .”
What the sources say
- Not checked yetcites [2] Roberts, P. D ., Stewart, G. B. & Pullin, A. S. Are review articles a reliable source of evidence to support conservation and environmental management? A comparison with medicine. Biol. Conserv. 132, 409–423 (2006) source
- Partly supportedcites [3] Bastian, H ., Glasziou, P . & Chalmers, I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 7, e1000326 (2010) sourceExperimental — accuracy not yet measured
Our copy supports the qualitative half of the claim: the 'How Close Are We to Archie Cochrane's Goal?' section cites Goldschmidt and Mulrow for 'how great the potential is for error in reviews of health literature that were not conducted systematically' and states that 'the staple of medical literature synthesis remains the non-systematic narrative review', while the same section (Figure 4 discussion) and 'Where to Now?' document the growth from ~14 trials a day in 1979 to 75 trials and 11 systematic reviews a day, with 'the task keeps increasing in size and complexity' ('Where Are We Now?'). What the source does NOT support is the quantitative assertion attributed to it: nowhere does our copy say the inability of narrative processes to summarise outcomes 'increases exponentially', nor tie it to study counts an order of magnitude beyond a dozen — the only dozen-scale figure in the paper is the median number of trials per systematic review (six to 16, Cochrane reviews averaging over 12), which is offered as a measure of reviewing workload and coverage, not of narrative-review failure. The exponential scaling is therefore the citing authors' characterisation, not this paper's. (LLM, conf=0.72)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“However, as the number of publications and research results has mushroomed across scientific fields, the difficulties encountered in carrying out narrative reviews to identify and summarize evidence in a transparent manner has become increasingly apparent 1 .”
What the sources say
- Not checked yetcites [1] Jennions, M. D ., Lortie, C. J. & Koricheva, J. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 23, 364–380 (Princeton Univ. Press, 2013) source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, in many cases of the resolution of scientific problems, when the results of an individual study are combined with those of other studies, the resulting synthesis provides a more general and complete picture of the questions being addressed.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A statement the paper makes itself — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Sometimes, the striking results of initial studies are not confirmed by those of subsequent studies or by syntheses of a body of research.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
15 more claims
Create a free account to see all 78 claims and to verify the 60 references still open.
- It may state two separate claims as one. 32
- Probably not a substantial claim. 48
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The trade-off with our method is that it does not allow for determining the magnitude of the effect of the practices33–36 but only the proportion of impacts33,37 as positive, negative or neutral.”
From Farming practices to enhance biodiversity across biomes: a systematic review (2024) · cites this paper as [36]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Besides, publication biases that have been documented in several scientific fields90,91 can also affect ML-based science.92,93”
From Leakage and the reproducibility crisis in machine-learning-based science (2023) · cites this paper as [91]
Experimental — accuracy not yet measuredIn the citing paper
“Moreover, since a meta-analysis can be conducted only when sufficient and appropriate quantitative data from original studies are available, the number of published meta-analyses about a certain topic is an indicator of the overall understanding of the subject, allowing us to reveal research gaps (25, 26).”
From Agricultural diversification promotes multiple ecosystem services without compromising yield (2020) · cites this paper as [25]
In the citing paper
“Meta-analysis optimises statistical power for detecting treatment effects by combining data from multiple similar studies [22, 23].”
From Does the growing of Bt maize change abundance or ecological function of non-target animals compared to the growing of non-GM maize? A systematic review (2022) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Box 1: Meta-analytic models and heterogeneity
“𝜀 is the within-study sampling error (solid black lines), the deviation from the study-specific mean.”
§ Box 1: Meta-analytic models and heterogeneity
“the solid black lines are deviations from the overall mean (panels a,b) or slope (panels c,d)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Box 1: Meta-analytic models and heterogeneity
“The assumption of the fixed-effect meta-regression is that a moderator(s) can explain all between-study heterogeneity”
§ Box 1: Meta-analytic models and heterogeneity
“the blue distributions the between-study variance (panels a-d).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Box 1: Meta-analytic models and heterogeneity
“Two basic models of meta-analysis are the fixed-effect model (Fig 1a) and the random-effects model (Fig 1b) 6 .”
§ Box 1: Meta-analytic models and heterogeneity
“Figure 1. A variety of charts and plots common in meta-analysis. a. PRISMA diagram 20 , b. a forest plot showing overall means based on a fixed-effect model and random-effects model”
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042 · cited 1×
“Procedural protocols for systematic reviews are well established especially in the medical and social sciences (Preferred Reporting Items for Systematic Reviews, or PRISMA; www.prisma-statement.org) 20 .”
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464 · cited 1×
“…the use of machine learning to screen studies 82 for inclusion in systematic reviews and meta-analyses for large reviews, increasingly sophisticated software and models for complex meta-regression 53,83 , and integration of meta-analysis with decision support in medicine and other domains 84 .”
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
A globally coherent fingerprint of climate change impacts across natural systems
Parmesan, C., & Yohe, G. (2003) · Nature · doi:10.1038/nature01286 · Cited by 11,265 · cited 2×
“…of meta-analysis and systematic reviews in EEC have allowed for assessment of impacts of major environmental drivers (e.g., climate change 38 ), evaluation of the evidence for ecological and evolutionary theories 39 , assessment of effectiveness of conservation and management strategies…”
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
The combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
The meta-analysis of response ratios in experimental ecology
Hedges, L. V., Gurevitch, J., & Curtis, P. S. (1999) · Ecology · doi:10.1890/0012-9658(1999)080[1150:tmaorr]2.0.co;2 · Cited by 4,704
Self-citationMeta-Analytic Procedures for Social Research
Rosenthal, R. (1991) · doi:10.4135/9781412984997 · Cited by 4,635
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“…of flawed methodology and applications in other fields typically prompts calls for improvement rather than demolition of the field (e.g., papers noting serious problems with pre-clinical cancer studies have not called for an end to pre-cancer clinical research but rather for its improvement 18 ).”
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · cited 1×
“Open science practices have been strongly advocated particularly in the past few years to facilitate full access to scientific data, particularly in an unbiased manner 79 .”
Quantifying the evidence for biodiversity effects on ecosystem functioning and services
Balvanera, P., Pfisterer, A. B., Buchmann, N., He, J., Nakashizuka, T., Raffaelli, D., & Schmid, B. (2006) · Ecology Letters · doi:10.1111/j.1461-0248.2006.00963.x · Cited by 2,650 · cited 1×
“Examples of influential ecological meta-analyses include, for instance, assessment of effects of biodiversity on ecosystem functioning and services 41,42 , which have demonstrated that losses of species richness have negative impact on functioning of different ecosystems.”
GCTE-NEWS, Rustad, L., Campbell, J., Marion, G., Norby, R., Mitchell, M., Hartley, A., Cornelissen, J., & Gurevitch, J. (2001) · Oecologia · doi:10.1007/s004420000544 · Cited by 2,185 · cited 1×
“Meta-analysis has also been a valuable tool for practitioners in EEC involved in collaborative research who needed to combine results from experiments carried out across multiple study sites 45,46 .”
Self-citationMeta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154 · cited 1×
“However, it was not until the insight of psychologists Gene Glass and Mary Smith, that outcome measures from different experiments could be standardized and put on the same scale 34 that meta-analysis began to really impact scientific research.”
Stewart, L. A., Clarke, M., Rovers, M., Riley, R. D., Simmonds, M., Stewart, G., & Tierney, J. F. (2015) · JAMA · doi:10.1001/jama.2015.3656 · Cited by 2,134
Self-citationOn the Generality of the Latitudinal Diversity Gradient
Hillebrand, H. (2004) · The American Naturalist · doi:10.1086/381004 · Cited by 2,085
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033
Effects of biodiversity on the functioning of trophic groups and ecosystems
Cardinale, B. J., Srivastava, D. S., Emmett Duffy, J., Wright, J. P., Downing, A. L., Sankaran, M., & Jouseau, C. (2006) · Nature · doi:10.1038/nature05202 · Cited by 1,871 · cited 1×
“Examples of influential ecological meta-analyses include, for instance, assessment of effects of biodiversity on ecosystem functioning and services 41,42 , which have demonstrated that losses of species richness have negative impact on functioning of different ecosystems.”
The need for evidence-based conservation
Sutherland, W. J., Pullin, A. S., Dolman, P. M., & Knight, T. M. (2004) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2004.03.018 · Cited by 1,822
Enhancement of Biodiversity and Ecosystem Services by Ecological Restoration: A Meta-Analysis
Benayas, J. M. R., Newton, A. C., Diaz, A., & Bullock, J. M. (2009) · Science · doi:10.1126/science.1172460 · Cited by 1,796 · cited 1×
“From an applied angle, Benayas and colleagues 43 considered the degree to which ecological restoration reverses environmental degradation caused by human activities.”
The case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483 · cited 2×
“While the number of published meta-analyses has continued to increase rapidly within and among disciplines, too many meta-analyses are of low quality 13-15 .”
Network meta‐analysis for indirect treatment comparisons
Lumley, T. (2002) · Statistics in Medicine · doi:10.1002/sim.1201 · Cited by 1,206 · cited 1×
“Network meta-analyses seek to provide comparisons of multiple interventions, including indirect comparisons 87 .”
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188
Handbook of Meta-analysis in Ecology and Evolution
(2013) · doi:10.1515/9781400846184 · Cited by 1,183 · cited 1×
“…meta-regression and model selection to evaluate the influence of more complex data structures and multiple covariates, and increasingly generalized network meta-analysis and hierarchical modelling of multi-level data, including that from individual participant data in medicine 21,85,86 and in EEC.”
STATISTICAL ISSUES IN ECOLOGICAL META-ANALYSES
Gurevitch, J., & Hedges, L. V. (1999) · Ecology · doi:10.1890/0012-9658(1999)080[1142:siiema]2.0.co;2 · Cited by 1,081 · cited 2×
“…meta-analysis and systematic reviews include publication bias, i.e. the underreporting of non-significant results or disconfirming evidence 61,62 , and research bias 56 , where populations, species, or systems are over-or under-represented in the literature, giving a biased view of the totality.”
Self-citationMethodological issues and advances in biological meta-analysis
Nakagawa, S., & Santos, E. S. A. (2012) · Evolutionary Ecology · doi:10.1007/s10682-012-9555-5 · Cited by 1,032 · cited 2×
“…meta-regression and model selection to evaluate the influence of more complex data structures and multiple covariates, and increasingly generalized network meta-analysis and hierarchical modelling of multi-level data, including that from individual participant data in medicine 21,85,86 and in EEC.”
Self-citationHow general are positive relationships between plant population size, fitness and genetic variation?
LEIMU, R., MUTIKAINEN, P., KORICHEVA, J., & FISCHER, M. (2006) · Journal of Ecology · doi:10.1111/j.1365-2745.2006.01150.x · Cited by 904 · cited 1×
“…correlations between measures of genetic diversity, fitness and population size, that have demonstrated that reduction in population size due to habitat fragmentation reduces genetic variation, and that these losses of genetic diversity have a negative impact on fitness in affected populations 44 .”
Self-citationThe analysis of groups of experiments
Yates, F., & Cochran, W. G. (1938) · The Journal of Agricultural Science · doi:10.1017/s0021859600050978 · Cited by 900
A Meta-Analysis of Competition in Field Experiments
Gurevitch, J., Morrow, L. L., Wallace, A., & Walsh, J. S. (1992) · The American Naturalist · doi:10.1086/285428 · Cited by 865
Self-citationHadfield, J. D., & Nakagawa, S. (2010) · Journal of Evolutionary Biology · doi:10.1111/j.1420-9101.2009.01915.x · Cited by 822
Self-citationMcAuley, L., Pham, B., Tugwell, P., & Moher, D. (2000) · The Lancet · doi:10.1016/s0140-6736(00)02786-0 · Cited by 812
RESAMPLING TESTS FOR META-ANALYSIS OF ECOLOGICAL DATA
Adams, D. C., Gurevitch, J., & Rosenberg, M. S. (1997) · Ecology · doi:10.1890/0012-9658(1997)078[1277:rtfmao]2.0.co;2 · Cited by 688 · cited 1×
“…in meta-analysis in EEC include the meta-analysis of factorial experiments 54 , introduction of randomization (permutation) tests in meta-analysis 55 , early embrace of random-effects and mixed models when these were still highly controversial in other disciplines 56 , and methods for inclusion…”
Self-citationProductivity is a poor predictor of plant species richness
Adler, P. B., Seabloom, E. W., Borer, E. T., Hillebrand, H., Hautier, Y., Hector, A., Harpole, W. S., O’Halloran, L. R., Grace, J. B., Anderson, T. M., et al. (2011) · Science · doi:10.1126/science.1204498 · Cited by 629
Living Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap
Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins, J. P. T., Mavergames, C., & Gruen, R. L. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001603 · Cited by 596
Meta-evaluation of meta-analysis: ten appraisal questions for biologists
Nakagawa, S., Noble, D. W. A., Senior, A. M., & Lagisz, M. (2017) · BMC Biology · doi:10.1186/s12915-017-0357-7 · Cited by 579
Self-citationMeta-analysis: synthesizing research findings in ecology and evolution
Arnqvist, G., & Wooster, D. (1995) · Trends in Ecology & Evolution · doi:10.1016/s0169-5347(00)89073-4 · Cited by 570
What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology.
Schmidt, F. L. (1992) · American Psychologist · doi:10.1037/0003-066x.47.10.1173 · Cited by 564 · cited 1×
“Instead of looking at ground breaking individual studies as the basis for revealing the conclusive understanding of a scientific problem, individual studies are viewed by meta-analysts as contributions toward the accumulation of evidence 24,27,28 .”
Relative Citation Impact of Various Study Designs in the Health Sciences
Patsopoulos, N. A. (2005) · JAMA · doi:10.1001/jama.293.19.2362 · Cited by 498
Arnqvist, G. (1998) · The Quarterly Review of Biology · doi:10.1086/420179 · Cited by 475
An exercise in mega-silliness.
Eysenck, H. J. (1978) · American Psychologist · doi:10.1037//0003-066x.33.5.517.a · Cited by 466
The Interaction between Competition and Predation: A Meta‐analysis of Field Experiments
Gurevitch, J., Morrison, J. A., & Hedges, L. V. (2000) · The American Naturalist · doi:10.1086/303337 · Cited by 465 · cited 1×
“Methodological innovations incorporated or developed in meta-analysis in EEC include the meta-analysis of factorial experiments 54 , introduction of randomization (permutation) tests in meta-analysis 55 , early embrace of random-effects and mixed models when these were still highly controversial in…”
Self-citationUses and misuses of meta‐analysis in plant ecology
Koricheva, J., & Gurevitch, J. (2014) · Journal of Ecology · doi:10.1111/1365-2745.12224 · Cited by 460 · cited 1×
“Statistically flawed procedures such as vote-counting 14 , a biased statistical technique that provides only limited information about study outcomes, have been long discredited but are still being employed in published papers.”
Self-citationGurevitch, J., Curtis, P. S., & Jones, M. H. (2001) · Advances in Ecological Research · doi:10.1016/s0065-2504(01)32013-5 · Cited by 421
Self-citationOpen MEE : Intuitive, open‐source software for meta‐analysis in ecology and evolutionary biology
Wallace, B. C., Lajeunesse, M. J., Dietz, G., Dahabreh, I. J., Trikalinos, T. A., Schmid, C. H., & Gurevitch, J. (2017) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12708 · Cited by 421
Self-citationLongo, D. L., & Drazen, J. M. (2016) · New England Journal of Medicine · doi:10.1056/nejme1516564 · Cited by 419
Maternal obesity and risk of neural tube defects: a metaanalysis
Rasmussen, S. A., Chu, S. Y., Kim, S. Y., Schmid, C. H., & Lau, J. (2008) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2008.04.021 · Cited by 390 · cited 2×
“For example, a number of individual studies had indicated that maternal obesity was associated with neural tube defects (NTDs) in newborns, but studies ranged from showing no effect to a 3-fold increase in the risk of NTDs with obesity 25 .”
InfluentialGauch Jr, H. G. (2002) · doi:10.1017/cbo9780511815034 · Cited by 385 · cited 1×
“For hundreds of years, scientists collected data in individual studies, based on observations and experimentation 77 .”
Tipton, E., & Pustejovsky, J. E. (2015) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998615606099 · Cited by 362
Biological invasions: a field synopsis, systematic review, and database of the literature
Lowry, E., Rollinson, E. J., Laybourn, A. J., Scott, T. E., Aiello‐Lammens, M. E., Gray, S. M., Mickley, J., & Gurevitch, J. (2013) · Ecology and Evolution · doi:10.1002/ece3.431 · Cited by 332
Self-citationHeterogeneity in ecological and evolutionary meta‐analyses: its magnitude and implications
Senior, A. M., Grueber, C. E., Kamiya, T., Lagisz, M., O'Dwyer, K., Santos, E. S. A., & Nakagawa, S. (2016) · Ecology · doi:10.1002/ecy.1591 · Cited by 299
Self-citationReport on Certain Enteric Fever Inoculation Statistics
(1904) · BMJ · doi:10.1136/bmj.2.2288.1243 · Cited by 298
Bayesian statistics in medicine: a 25 year review
Ashby, D. (2006) · Statistics in Medicine · doi:10.1002/sim.2672 · Cited by 238
Graphical displays for meta-analysis: An overview with suggestions for practice
Anzures‐Cabrera, J., & Higgins, J. P. T. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.6 · Cited by 233
Whittaker, R. J. (2010) · Ecology · doi:10.1890/08-0968.1 · Cited by 223
Meta-analysis in applied ecology
Stewart, G. (2010) · Biology Letters · doi:10.1098/rsbl.2009.0546 · Cited by 214 · cited 2×
“…and systematic reviews in conservation and applied ecology has been facilitated by the promotion of evidence-based approaches in this field 36,37 , especially through organizations such as the Centre for Evidence Based Conservation (www.cebc.bangor.ac.uk) and the Collaboration for…”
Self-citationTransparency in Ecology and Evolution: Real Problems, Real Solutions
Parker, T. H., Forstmeier, W., Koricheva, J., Fidler, F., Hadfield, J. D., Chee, Y. E., Kelly, C. D., Gurevitch, J., & Nakagawa, S. (2016) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2016.07.002 · Cited by 201 · cited 2×
“These align with the more generalized open data movement which has considerable scope for remedying publication biases with pre-registration, minimising selective reporting with open data and facilitating direct generation of effects or multi-level analysis based on raw data 68,80,81 .”
Self-citationAn historical perspective on meta-analysis: Dealing quantitatively with varying study results
O'rourke, K. (2007) · Journal of the Royal Society of Medicine · doi:10.1177/0141076807100012020 · Cited by 200
Multisystemic Therapy for social, emotional, and behavioral problems in youth aged 10-17
Littell, J., Popa, M., & Forsythe, B. (2005) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004797.pub3 · Cited by 186
Meta‐Analysis and the Comparative Phylogenetic Method
Lajeunesse, M. J. (2009) · The American Naturalist · doi:10.1086/603628 · Cited by 160 · cited 1×
“…is on a single species, the multi-species nature of much of EEC research and therefore of meta-analyses has led biologists to combine phylogenetic comparative methods with meta-analytic models to take into account potential non-independence among lineages due to shared evolutionary history 49-51 .”
Leimu, R., & Koricheva, J. (2004) · Proceedings of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rspb.2004.2828 · Cited by 156 · cited 1×
“…for meta-analysis and systematic reviews include publication bias, i.e. the underreporting of non-significant results or disconfirming evidence 61,62 , and research bias 56 , where populations, species, or systems are over-or under-represented in the literature, giving a biased view of the…”
Self-citationMeta‐analysis of magnitudes, differences and variation in evolutionary parameters
Morrissey, M. B. (2016) · Journal of Evolutionary Biology · doi:10.1111/jeb.12950 · Cited by 149 · cited 2×
“Concerns have been raised that “the surge in popularity of meta-analysis may be occurring at the cost of qualitative synthesis” 16 .”
Roberts, P. D., Stewart, G. B., & Pullin, A. S. (2006) · Biological Conservation · doi:10.1016/j.biocon.2006.04.034 · Cited by 137
Self-citationMeta‐analysis: A need for well‐defined usage in ecology and conservation biology
Vetter, D., Rücker, G., & Storch, I. (2013) · Ecosphere · doi:10.1890/es13-00062.1 · Cited by 126 · cited 1×
“…using data from a number of primary publications, regardless of whether appropriate statistical procedures such as effect size calculation and heterogeneity analysis have been employed 69 or appropriate statistical models used that take into account the distinct structure of meta-analytic data.”
Adams, D. C. (2008) · Evolution · doi:10.1111/j.1558-5646.2007.00314.x · Cited by 126 · cited 1×
“…is on a single species, the multi-species nature of much of EEC research and therefore of meta-analyses has led biologists to combine phylogenetic comparative methods with meta-analytic models to take into account potential non-independence among lineages due to shared evolutionary history 49-51 .”
Staff, S. (2011) · Science · doi:10.1126/science.331.6018.692 · Cited by 123
Synthesizing Evidence: Shifting the Focus From Individual Studies to the Body of Evidence
Murad, M. H., & Montori, V. M. (2013) · JAMA · doi:10.1001/jama.2013.5616 · Cited by 104 · cited 2×
“…two basic goals, meta-analysis has been a more powerful and less biased means for clarifying, quantifying and disproving (or confirming) assumed wisdom than have previous conventional approaches 24 , such as narrative reviews and flawed quantitative methods such as ‘vote counts’ (discussed below).”
Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review
Zarin, W., Veroniki, A. A., Nincic, V., Vafaei, A., Reynen, E., Motiwala, S. S., Antony, J., Sullivan, S. M., Rios, P., Daly, C., et al. (2017) · BMC Medicine · doi:10.1186/s12916-016-0764-6 · Cited by 97 · cited 1×
“Developments in and applications of this powerful approach has increased dramatically in clinical medicine over the last 10 years 88 .”
Meta‐analysis at middle age: a personal history
Glass, G. V. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1133 · Cited by 88 · cited 2×
“It may be that the time was right, because modern meta-analysis was initiated almost simultaneously in the medical and social sciences in the late 1970s 11 .”
Multisystemic Therapy for social, emotional, and behavioral problems in youth aged 10-17
Littell, J. H., Campbell, M., Green, S., & Toews, B. (2005) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004797.pub4 · Cited by 84 · cited 1×
“…significant differences between MST and conventional social services in the success of outcomes (e.g. subsequent arrests or convictions) when the results of the different studies were synthesized, with no support for either clinically significant advantages or harmful outcomes of this approach 26 .”
Gauging the impact of meta-analysis on ecology
Cadotte, M. W., Mehrkens, L. R., & Menge, D. N. L. (2012) · Evolutionary Ecology · doi:10.1007/s10682-012-9585-z · Cited by 77 · cited 1×
“Overall, however, introduction of meta-analysis in EEC has been enthusiastically embraced by majority of scientists in these disciplines with the number of meta-analyses published increasing exponentially 59,60 .”
Vandvik, P. O., Brignardello-Petersen, R., & Guyatt, G. H. (2016) · BMC Medicine · doi:10.1186/s12916-016-0596-4 · Cited by 69
Shadish, W. R., & Lecy, J. D. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1132 · Cited by 63
A critique for meta‐analyses and the productivity– diversity relationship
Hillebrand, H., & Cardinale, B. J. (2010) · Ecology · doi:10.1890/09-0070.1 · Cited by 53
Fame, glory and neglect in meta-analyses
Kueffer, C., Niinemets, Ü., Drenovsky, R. E., Kattge, J., Milberg, P., Poorter, H., Reich, P. B., Werner, C., Westoby, M., & Wright, I. J. (2011) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2011.07.007 · Cited by 41 · cited 1×
“…outcome of the high impact and growing prestige of meta-analyses 71 , coupled with use of metrics such as citation numbers and h -indices in evaluations of research accomplishments, resulting in unease among some primary researchers about the fairness and rewards of the scientific process 72 .”
How to critically read ecological meta-analyses
Lortie, C. J., Stewart, G., Rothstein, H., & Lau, J. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1109 · Cited by 39 · cited 1×
“This approach is common in the fields of EEC, and in some social sciences, where meta-analysis has been used to address fundamental problems, weigh the evidence for prominent theories or hypotheses, and document the generality of important phenomena 12,23 .”
Self-citation16. Statistical Models for the Meta-analysis of Nonindependent Data
Mengersen, K. L., Jennions, M. D., & Schmid, C. H. (2013) · Princeton University Press eBooks · doi:10.1515/9781400846184-018 · Cited by 38
15. Temporal Trends in Effect Sizes: Causes, Detection, and Implications
(2013) · Handbook of Meta-analysis in Ecology and Evolution · doi:10.1515/9781400846184-017 · Cited by 36
Self-citationphyloMeta: a program for phylogenetic comparative analyses with meta-analysis
Lajeunesse, M. J. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btr438 · Cited by 32
INFANTILE HYPERTROPHIC STENOSIS OF THE PYLORUS
Paterson, H. J. (1906) · BMJ · doi:10.1136/bmj.2.2392.1243-b · Cited by 29
25. History and Progress of Meta-analysis
(2013) · Handbook of Meta-analysis in Ecology and Evolution · doi:10.1515/9781400846184-027 · Cited by 27
Self-citationSaldanha, I. J., Schmid, C. H., Lau, J., Dickersin, K., Berlin, J. A., Jap, J., Smith, B. T., Carini, S., Chan, W., De Bruijn, B., et al. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0373-7 · Cited by 15 · cited 1×
“…information, advances in meta-analysis are being propelled by methodological developments, and include the use of machine learning to screen studies 82 for inclusion in systematic reviews and meta-analyses for large reviews, increasingly sophisticated software and models for complex…”
Rejoinder to Simberloff (2006): Don't calculate effect sizes; study ecological effects
Simberloff, D. (2006) · Ecology Letters · doi:10.1111/j.1461-0248.2006.00952.x · Cited by 14
Littell, J. H., & Shlonsky, A. (2011) · Clinical Social Work Journal · doi:10.1007/s10615-010-0308-z · Cited by 13 · cited 1×
“While the number of published meta-analyses has continued to increase rapidly within and among disciplines, too many meta-analyses are of low quality 13-15 .”
Schmid, C. H., & Mengersen, K. L. (2013) · Princeton University Press eBooks · doi:10.1515/9781400846184-013 · Cited by 11
Mengersen, K., MacNeil, M. A., & Caley, M. J. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1105 · Cited by 10 · cited 1×
“…the use of machine learning to screen studies 82 for inclusion in systematic reviews and meta-analyses for large reviews, increasingly sophisticated software and models for complex meta-regression 53,83 , and integration of meta-analysis with decision support in medicine and other domains 84 .”
19. Meta-analysis of Results from Multisite Studies
Gurevitch, J. (2013) · Princeton University Press eBooks · doi:10.1515/9781400846184-021 · Cited by 10
Self-citation24. Using Meta-analysis to Test Ecological and Evolutionary Theory
(2013) · Handbook of Meta-analysis in Ecology and Evolution · doi:10.1515/9781400846184-026 · Cited by 9
Self-citation23. Role of Meta-analysis in Interpreting the Scientific Literature
(2013) · Handbook of Meta-analysis in Ecology and Evolution · doi:10.1515/9781400846184-025 · Cited by 7
Self-citationReview Articles: The Black-Market of Scientific Currency
Cohnstaedt, L. W., & Poland, J. (2017) · Annals of the Entomological Society of America · doi:10.1093/aesa/saw061 · Cited by 5 · cited 1×
“For instance, review articles have been recently decried “the black-market of scientific currency” with calls to replace citations to reviews and meta-analyses by citations of primary studies 73 .”
Pearson, K. (1904) · BMJ · doi:10.1136/bmj.2.2294.1667 · Cited by 5
Research works printed but not linked: 3
- Hedges, L. & Olkin, I. Statistical Methods for Meta-analysis (Academic Press, 1985)
- Higgins, J. P. T . & Green, S. (eds) Cochrane Handbook for Systematic Reviews of Interventions: Version 5.1.0 (Wiley, 2011).This large collaborative work provides definitive guidance for the production of systematic reviews in medicine and is of broad interest for methods development outside the medical field.
- Cooper, H. M ., Hedges, L. V . & Valentine, J. C. (eds) The Handbook of Research Synthesis and Meta-analysis 2nd edn (Russell Sage Foundation, 2009).This book is an important compilation that builds on the ground-breaking first edition to set the standard for best practice in meta-analysis, primarily in the social sciences but with applications to medicine and other fields.
Cited by
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01626-4 · Cited by 13,874
The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · International Journal of Surgery · doi:10.1016/j.ijsu.2021.105906 · Cited by 11,748
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · PLoS Medicine · doi:10.1371/journal.pmed.1003583 · Cited by 5,148
“Systematic reviews therefore generate various types of knowledge for different users of reviews (such as patients, healthcare providers, researchers, and policy makers).[1,2]To ensure a systematic review is valuable to users, authors should prepare a transparent, complete, and accurate account of why the review was do…”
BackgroundThe PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/v7gm2 · Cited by 5,177
Metadata
- Authors
- Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G.
- Affiliations
- Stony Brook University (US)
- Royal Holloway University of London (GB)
- Garvan Institute of Medical Research (AU)
- Newcastle University (GB)
- Corresponding author
- Gurevitch, Jessica
- Koricheva, Julia
- Nakagawa, Shinichi
- Stewart, Gavin
- Venue
- Nature
- Published
- 2018-03
- Volume
- 555
- Issue
- 7695
- Pages
- 175-182
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2791681873
- Semantic Scholar
- ac5ad1932e31c3ea141366e8e073d6d95869c4ef
- Cited by
- 1,709
- References
- 101
- Citations by year
- 2026: 177 · 2025: 230 · 2024: 231 · 2023: 199 · 2022: 234 · 2021: 235 · 2020: 204 · 2019: 130 · 2018: 64
- FWCI
- 31.20
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- UNSWorks (UNSW Sydney) (submittedVersion, cc-by-nc-nd)
- RePEc: Research Papers in Economics (submittedVersion)
- Newcastle University ePrints (Newcastle Univesity)
- Funders
- National Science Foundation (1262402)
- Laura and John Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- meta-analysis, evidence synthesis, history of meta-analysis, evolutionary biology, conservation biology, artificial intelligence, evidence-based practice
- MeSH terms
- Animals, Biological Science Disciplines, Ecology, Biological Evolution, Research, Meta-Analysis as Topic, History, 20th Century, History, 21st Century
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.