Empirical vs natural weighting in random effects meta‐analysis
Open accessThe DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
This article brings into serious question the validity of empirically based weighting in random effects meta‐analysis. These methods treat sample sizes as non‐random, whereas they need to be part of the random effects analysis. It will be demonstrated that empirical weighting risks substantial bias. Two alternate methods are proposed. The first estimates the arithmetic mean of the population of study effect sizes per the classical model for random effects meta‐analysis. We show that anything other than an unweighted mean of study effect sizes will risk serious bias for this targeted parameter. The second method estimates a patient level effect size, something quite different from the first. To prevent inconsistent estimation for this population parameter, the study effect sizes must be weighted in proportion to their total sample sizes for the trial. The two approaches will be presented for a meta‐analysis of a nasal decongestant, while at the same time will produce counter‐intuitive results for the DerSimonian–Laird approach, the most popular empirically based weighted method. It is concluded that all past publications based on empirically weighted random effects meta‐analysis should be revisited to see if the qualitative conclusions hold up under the methods proposed herein. It is also recommended that empirically based weighted random effects meta‐analysis not be used in the future, unless strong cautions about the assumptions underlying these analyses are stated, and at a minimum, some form of secondary analysis based on the principles set forth in this article be provided to supplement the primary analysis. Copyright © 2009 John Wiley & Sons, Ltd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Effect of Rosiglitazone on the Risk of Myocardial Infarction and Death from Cardiovascular Causes
Nissen, S. E., & Wolski, K. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa072761 · Cited by 4,675 · cited 2×
“[2] and Nissen–Wolski [6], where a very few studies play a dominant role.”
Shuster, J. J., Jones, L. S., & Salmon, D. A. (2007) · Statistics in Medicine · doi:10.1002/sim.3060 · Cited by 105 · cited 1×
“See Shuster and colleagues [7], who provide a parallel development for an unweighted random effects meta-analysis to cover that situation.”
Self-citationKollar, C., Schneider, H., Waksman, J., & Krusinska, E. (2007) · Clinical Therapeutics · doi:10.1016/j.clinthera.2007.05.021 · Cited by 39
Cited by
A basic introduction to fixed-effect and random-effects models for meta-analysis
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.12 · Cited by 6,793
IntHout, J., Ioannidis, J. P., & Borm, G. F. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-25 · Cited by 2,419
“Several other methods have been developed, like the Quantile Approximation (QA) method [28], the Profile Likelihood approach [29], natural weighting instead of empirically based weighting of studies [30], use of fixed effects estimates with a random effects approach to heterogeneity [31] and more recently, higher-orde…”
MethodHoaglin, D. C., Hawkins, N., Jansen, J. P., Scott, D. A., Itzler, R., Cappelleri, J. C., Boersma, C., Thompson, D., Larholt, K. M., Diaz, M., et al. (2011) · Value in Health · doi:10.1016/j.jval.2011.01.011 · Cited by 770
Meta-analysis methods for genome-wide association studies and beyond
Evangelou, E., & Ioannidis, J. P. A. (2013) · Nature Reviews Genetics · doi:10.1038/nrg3472 · Cited by 596
Doi, S. A. R., Barendregt, J. J., Khan, S., Thalib, L., & Williams, G. M. (2015) · Contemporary Clinical Trials · doi:10.1016/j.cct.2015.05.009 · Cited by 581
Evidence Base Update: 50 Years of Research on Treatment for Child and Adolescent Anxiety
Higa-McMillan, C. K., Francis, S. E., Rith-Najarian, L., & Chorpita, B. F. (2016) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2015.1046177 · Cited by 557
PRISMA harms checklist: improving harms reporting in systematic reviews
Zorzela, L., Loke, Y. K., Ioannidis, J. P., Golder, S., Santaguida, P., Altman, D. G., Moher, D., Vohra, S., & PRISMA harms group (2016) · BMJ · doi:10.1136/bmj.i157 · Cited by 557
Bowden, J., Tierney, J. F., Copas, A. J., & Burdett, S. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-41 · Cited by 553
Evaluating gender similarities and differences using metasynthesis.
Zell, E., Krizan, Z., & Teeter, S. R. (2015) · American Psychologist · doi:10.1037/a0038208 · Cited by 434
Do People Have Insight Into Their Abilities? A Metasynthesis
Zell, E., & Krizan, Z. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613518075 · Cited by 431
Zell, E., Strickhouser, J. E., Sedikides, C., & Alicke, M. D. (2020) · Psychological Bulletin · doi:10.1037/bul0000218 · Cited by 400
Bias caused by sampling error in meta-analysis with small sample sizes
Lin, L. (2018) · PLOS ONE · doi:10.1371/journal.pone.0204056 · Cited by 335
Misunderstandings about Q and ‘Cochran's Q test' in meta‐analysis
Hoaglin, D. C. (2016) · Statistics in Medicine · doi:10.1002/sim.6632 · Cited by 310
Shuster, J. J. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.38 · Cited by 297
Self-citationA re‐evaluation of fixed effect(s) meta‐analysis
Rice, K., Higgins, J. P. T., & Lumley, T. (2018) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12275 · Cited by 287
Sequential methods for random‐effects meta‐analysis
Higgins, J. P. T., Whitehead, A., & Simmonds, M. (2011) · Statistics in Medicine · doi:10.1002/sim.4088 · Cited by 258
Jackson, D., Bowden, J., & Baker, R. (2010) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2009.09.017 · Cited by 226
Kowalewski, M., Pawliszak, W., Malvindi, P. G., Bokszanski, M. P., Perlinski, D., Raffa, G. M., Kowalkowska, M. E., Zaborowska, K., Navarese, E. P., Kolodziejczak, M., et al. (2016) · The Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/j.jtcvs.2015.08.042 · Cited by 225
Fixed- and random-effects meta-analytic structural equation modeling: Examples and analyses in R
Cheung, M. W.-L. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0361-y · Cited by 208
Conducting a meta-analysis: Basics and good practices
Cheung, M. W. ‐., Ho, R. C. M., Lim, Y., & Mak, A. (2012) · International Journal of Rheumatic Diseases · doi:10.1111/j.1756-185x.2012.01712.x · Cited by 172
Obtaining Unbiased Results in Meta-Analysis: The Importance of Correcting for Statistical Artifacts
Wiernik, B. M., & Dahlke, J. A. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919885611 · Cited by 135
“As a result, weighting effect sizes by inverse sampling error variance introduces dependency between study weights and study effect sizes, which can bias meta-analysis results (Hedges & Olkin, 1985, p. 110; Kulinskaya & Bakbergenuly, 2018; Schmidt & Hunter, 2015; Shuster, 2010).”
BackgroundFry, D., Krzeczkowska, A., Ren, J., Lu, M., Fang, X., Fry, D., Krzeczkowska, A., Anderson, N., Ren, J., Lu, M., et al. (2025) · The Lancet Child & Adolescent Health · doi:10.1016/s2352-4642(24)00329-8 · Cited by 53
“32 As the field grows with more studies and a shift towards country-level estimates, alternative analytical approaches might become more suitable.”
Metadata
- Authors
- Shuster, J. J.
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2010-05-19
- Volume
- 29
- Issue
- 12
- Pages
- 1259-1265
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- OpenAlex
- W2144426506
- Cited by
- 122
- References
- 3
- FWCI
- 11.94
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.