A comparison of statistical methods for meta‐analysis
Abstract
Meta‐analysis may be used to estimate an overall effect across a number of similar studies. A number of statistical techniques are currently used to combine individual study results. The simplest of these is based on a fixed effects model, which assumes the true effect is the same for all studies. A random effects model, however, allows the true effect to vary across studies, with the mean true effect the parameter of interest. We consider three methods currently used for estimation within the framework of a random effects model, and illustrate them by applying each method to a collection of six studies on the effect of aspirin after myocardial infarction. These methods are compared using estimated coverage probabilities of confidence intervals for the overall effect. The techniques considered all generally have coverages below the nominal level, and in particular it is shown that the commonly used DerSimonian and Laird method does not adequately reflect the error associated with parameter estimation, especially when the number of studies is small. Copyright © 2001 John Wiley & Sons, Ltd.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This test-based approach was often criticised because of low power to detect heterogeneity in case of few (< 10) included studies.
“This approach was, however, often criticised because of low power to detect heterogeneity in case of few (< 10) included studies [52, 77, 101, 104, 116, 118].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [116]
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · see citing works
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · Cited by 15,411 · see citing works
(1992) · doi:10.1142/1490 · Cited by 1,551 · see citing works
Meta-Analysis: Cumulating Research Findings across Studies
Feldman, J. M., Hunter, J. E., Schmidt, F. L., & Jackson, G. B. (1984) · Academy of Management Review · doi:10.2307/258247 · Cited by 1,199 · see citing works
Arrowsmith, D. K. (1987) · Bulletin of the London Mathematical Society · doi:10.1112/blms/19.4.404 · Cited by 1,180 · see citing works
Meta-Analysis: Cumulating Research Findings across Studies
Hunter, J. E., Schmidt, F. L., & Jackson, G. B. (1986) · Educational Researcher · doi:10.2307/1175263 · Cited by 792 · see citing works
A LIKELIHOOD APPROACH TO META-ANALYSIS WITH RANDOM EFFECTS
Hardy, R., & Thompson, S. G. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19960330)15:6 · Cited by 517 · see citing works
Problems Arising in the Analysis of a Series of Similar Experiments
Cochran, W. G. (1937) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.2307/2984123 · Cited by 461 · see citing works
Combining Information: Statistical Issues and Opportunities for Research
Draper, D. L., Gaver, D. P., Goel, P., Greenhouse, J. B., Hedges, L. V., Morris, C., Tucker, J., & Waternaux, C. (1993) · Cited by 279 · see citing works
Combining Information: Statistical Issues and Opportunities for Research
Srinivasan, C. (1993) · Technometrics · doi:10.2307/1270296 · Cited by 213 · see citing works
Lee, W. (1997) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19970415)16:7 · Cited by 74 · see citing works
Aspirin after myocardial infarction
Glasziou, P. (1989) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.1989.tb136344.x · Cited by 74 · see citing works
An overview of statistical issues and methods of meta-analysis
Schmid, E. J., Koch, G. G., & LaVange, L. M. (1991) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543409108835008 · Cited by 55 · see citing works
Aspirin after Myocardial Infarction
(1980) · The Lancet · doi:10.1016/s0140-6736(80)91626-8 · Cited by 53 · see citing works
The bias of the commonly-used estimate of variance in meta-analysis
Li, Y., Shi, L., & Roth, H. D. (1994) · Communication in Statistics- Theory and Methods · doi:10.1080/03610929408831305 · Cited by 45 · see citing works
Olkin, I. (1992) · Statistical Science · doi:10.1214/ss/1177011361 · Cited by 19 · see citing works
Elementary Mathematical Statistics
M., K. (1939) · Nature · doi:10.1038/143960a0 · Cited by 15 · see citing works
Elementary Mathematical Statistics
Robinson, H. A., & Baten, W. D. (1939) · National Mathematics Magazine · doi:10.2307/3028862 · Cited by 13 · see citing works
Detecting and describing heterogeneity in meta-analysis
Hardy, R. J., & Thompson, S. G. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980430)17:8
Flagged first, then by citations
Cited by
Sorted by citations
Assessing heterogeneity in meta-analysis: Q statistic or I² index?
Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.2.193 · Cited by 4,256 · see citing works
Methods to estimate the between‐study variance and its uncertainty in meta‐analysis
Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P., Langan, D., & Salanti, G. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1164 · Cited by 1,685 · see citing works
Controlled Interventions to Reduce Burnout in Physicians: A Systematic Review and Meta-analysis
Panagioti, M., Panagopoulou, E., Bower, P., Lewith, G., Kontopantelis, E., Chew-Graham, C., Dawson, S., van Marwijk, H., Geraghty, K., & Esmail, A. (2017) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.7674 · Cited by 1,395 · see citing works
Sex Differences in Insomnia: A Meta-Analysis
Zhang, B., & Wing, Y.-K. (2006) · Sleep · doi:10.1093/sleep/29.1.85 · Cited by 1,286 · see citing works
Destination image and tourist loyalty: a meta-analysis
Zhang, H., Fu, X., Cai, L. A., & Lu, L. (2014) · Tourism Management · doi:10.1016/j.tourman.2013.06.006 · Cited by 1,098 · see citing works
Dignath, C., & Büttner, G. (2008) · Metacognition and Learning · doi:10.1007/s11409-008-9029-x · Cited by 1,085 · see citing works
Tufanaru, C., Munn, Z., Stephenson, M., & Aromataris, E. (2015) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000065 · Cited by 1,057 · see citing works
Association between body-mass index and risk of death in more than 1 million Asians
Zheng, W., McLerran, D. F., Rolland, B., Zhang, X., Inoue, M., Matsuo, K., He, J., Gupta, P. C., Ramadas, K., Tsugane, S., et al. (2011) · New England Journal of Medicine · doi:10.1056/nejmoa1010679 · Cited by 885 · see citing works
Panagioti, M., Khan, K., Keers, R. N., Abuzour, A., Phipps, D., Kontopantelis, E., Bower, P., Campbell, S., Haneef, R., Avery, A. J., et al. (2019) · BMJ · doi:10.1136/bmj.l4185 · Cited by 874 · see citing works
Fu, R., Gartlehner, G., Grant, M., Shamliyan, T., Sedrakyan, A., Wilt, T. J., Griffith, L., Oremus, M., Raina, P., Ismaila, A., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.08.010 · Cited by 730 · see citing works
Stevens, G. A., Bennett, J. E., Hennocq, Q., Lu, Y., De-Regil, L. M., Rogers, L., Danaei, G., Li, G., White, R. A., Flaxman, S. R., et al. (2015) · The Lancet Global Health · doi:10.1016/s2214-109x(15)00039-x · Cited by 572 · see citing works
Random-Effects Meta-analysis of Inconsistent Effects: A Time for Change
Cornell, J. E., Mulrow, C. D., Localio, R., Stack, C. B., Meibohm, A. R., Guallar, E., & Goodman, S. N. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2886 · Cited by 453 · see citing works
A simple confidence interval for meta-analysis
Sidik, K., & Jonkman, J. N. (2002) · Statistics in Medicine · doi:10.1002/sim.1262 · Cited by 431 · see citing works
Mueller, M., D’Addario, M., Egger, M., Cevallos, M., Dekkers, O., Mugglin, C., & Scott, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0495-9 · Cited by 400 · see citing works
Mannan, M., Mamun, A., Doi, S., & Clavarino, A. (2016) · PLoS ONE · doi:10.1371/journal.pone.0157240 · Cited by 421 · see citing works
A comparison of heterogeneity variance estimators in combining results of studies
Sidik, K., & Jonkman, J. N. (2007) · Statistics in Medicine · doi:10.1002/sim.2688 · Cited by 380 · see citing works
A comprehensive meta-analysis of interpretation biases in depression
Everaert, J., Podina, I. R., & Koster, E. H. W. (2017) · Clinical Psychology Review · doi:10.1016/j.cpr.2017.09.005 · Cited by 378 · see citing works
Mannan, M., Mamun, A., Doi, S., & Clavarino, A. (2016) · Asian Journal of Psychiatry · doi:10.1016/j.ajp.2015.12.008 · Cited by 280 · see citing works
Masked hypertension: A systematic review
Bobrie, G., Clerson, P., Ménard, J., Postel-Vinay, N., Chatellier, G., & Plouin, P.-F. (2008) · Journal of Hypertension · doi:10.1097/hjh.0b013e3282fbcedf · Cited by 279 · see citing works
Veroniki, A. A., Jackson, D., Bender, R., Kuss, O., Langan, D., Higgins, J. P. T., Knapp, G., & Salanti, G. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1319 · Cited by 236 · see citing works
Advances in the meta-analysis of heterogeneous clinical trials II: The quality effects model
Doi, S. A. R., Barendregt, J. J., Khan, S., Thalib, L., & Williams, G. M. (2015) · Contemporary Clinical Trials · doi:10.1016/j.cct.2015.05.010 · Cited by 227 · see citing works
Jackson, D., Bowden, J., & Baker, R. (2010) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2009.09.017 · Cited by 226 · see citing works
Coventry, P. A., Hudson, J. L., Kontopantelis, E., Archer, J., Richards, D. A., Gilbody, S., Lovell, K., Dickens, C., Gask, L., Waheed, W., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0108114 · Cited by 221 · see citing works
Karkou, V., Aithal, S., Zubala, A., & Meekums, B. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00936 · Cited by 186 · see citing works
Antihistamines for the common cold
De Sutter, A. I., Saraswat, A., & van Driel, M. L. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009345.pub2 · Cited by 177 · see citing works
Publication Bias in Meta-Analysis: Confidence Intervals for Rosenthal’s Fail-Safe Number
Fragkos, K. C., Tsagris, M., & Frangos, C. C. (2014) · International Scholarly Research Notices · doi:10.1155/2014/825383 · Cited by 159 · see citing works
Nam, I., Mengersen, K., & Garthwaite, P. (2003) · Statistics in Medicine · doi:10.1002/sim.1410 · Cited by 153 · see citing works
Prevalence of epilepsy in autism spectrum disorders: A systematic review and meta-analysis
Liu, X., Sun, X., Sun, C., Zou, M., Chen, Y., Huang, J., Wu, L., & Chen, W.-X. (2022) · Autism · doi:10.1177/13623613211045029 · Cited by 145 · see citing works
Bayesian Meta-Analysis with Weakly Informative Prior Distributions
Williams, D. R., Rast, P., & Bürkner, P. C. (2018) · doi:10.31234/osf.io/7tbrm · Cited by 124 · see citing works
Shuster, J. J., Jones, L. S., & Salmon, D. A. (2007) · Statistics in Medicine · doi:10.1002/sim.3060 · Cited by 105 · see citing works
Genome-wide association analysis of age-at-onset in Alzheimer's disease
Kamboh, M. I., Barmada, M. M., Demirci, F. Y., Minster, R. L., Carrasquillo, M. M., Pankratz, V. S., Younkin, S. G., Saykin, A. J., Initiative, T. A. D. N., Sweet, R. A., et al. (2012) · Molecular Psychiatry · doi:10.1038/mp.2011.135 · Cited by 100 · see citing works
Current methods for meta-analysis
Schulze, R. (2007) · Zeitschrift für Psychologie / Journal of Psychology · doi:10.1027/0044-3409.215.2.90 · Cited by 79 · see citing works
Bayesian estimation in random effects meta‐analysis using a non‐informative prior
Bodnar, O., Link, A., Arendacká, B., Possolo, A., & Elster, C. (2017) · Statistics in Medicine · doi:10.1002/sim.7156 · Cited by 73 · see citing works
Pöchmüller, M., Schwingshackl, L., Colombani, P. C., & Hoffmann, G. (2016) · Journal of the International Society of Sports Nutrition · doi:10.1186/s12970-016-0139-6 · Cited by 56 · see citing works
The meaning of distal sensory loss and absent ankle reflexes in relation to age: A meta–analysis
Vrancken, A. F. J. E., Kalmijn, S., Brugman, F., Rinkel, G. J. E., & Notermans, N. C. (2006) · Journal of Neurology · doi:10.1007/s00415-005-0064-0 · Cited by 50 · see citing works
Placebo Responses in Genetically Determined Intellectual Disability: A Meta-Analysis
Curie, A., Yang, K., Kirsch, I., Gollub, R. L., des Portes, V., Kaptchuk, T. J., & Jensen, K. B. (2015) · PLOS ONE · doi:10.1371/journal.pone.0133316 · Cited by 43 · see citing works
Pfefferbaum, B., Nitiéma, P., & Newman, E. (2019) · Journal of Traumatic Stress · doi:10.1002/jts.22391 · Cited by 41 · see citing works
Xiong, Q., Tang, F., Li, Y., Xie, F., Yuan, L., Yao, C., Wu, R., Wang, J., Wang, Q., & Feng, P. (2022) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2022.110983 · Cited by 33 · see citing works
HSV-1 cellular model reveals links between aggresome formation and early step of Alzheimer’s disease
Albaret, M. A., Textoris, J., Dalzon, B., Lambert, J., Linard, M., Helmer, C., Hacot, S., Ghayad, S. E., Ferréol, M., Mertani, H. C., et al. (2023) · Translational Psychiatry · doi:10.1038/s41398-023-02376-8 · Cited by 25 · see citing works
Lee, J. Y., & Woo, S. H. (2019) · The Asian Journal of Shipping and Logistics · doi:10.1016/j.ajsl.2019.12.006 · Cited by 11 · see citing works
- Authors
- Brockwell, S. E., & Gordon, I. R.
- Affiliations
- The University of Melbourne (AU)
- Corresponding author
- Gordon, Ian R.
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2001-03-05
- Volume
- 20
- Issue
- 6
- Pages
- 825-840
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- OpenAlex
- W2112885721
- Cited by
- 573
- References
- 19
- Citations by year
- 2026: 18 · 2025: 29 · 2024: 18 · 2023: 35 · 2022: 21 · 2021: 39 · 2020: 49 · 2019: 49 · 2018: 46 · 2017: 32 · 2016: 33 · 2015: 37 · 2014: 13 · 2013: 26 · 2012: 19
- FWCI
- 8.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Statistics in Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- meta-analysis, random effects models, fixed effects model, confidence interval coverage, method comparison
- MeSH terms
- Anti-Inflammatory Agents, Non-Steroidal, Aspirin, Clinical Trials as Topic, Computer Simulation, Humans, Models, Biological, Myocardial Infarction, Meta-Analysis as Topic, Models, Statistical, Likelihood Functions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
