The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
Bias caused by sampling error in meta-analysis with small sample sizes
Open accessAbstract
BACKGROUND: Meta-analyses frequently include studies with small sample sizes. Researchers usually fail to account for sampling error in the reported within-study variances; they model the observed study-specific effect sizes with the within-study variances and treat these sample variances as if they were the true variances. However, this sampling error may be influential when sample sizes are small. This article illustrates that the sampling error may lead to substantial bias in meta-analysis results. METHODS: We conducted extensive simulation studies to assess the bias caused by sampling error. Meta-analyses with continuous and binary outcomes were simulated with various ranges of sample size and extents of heterogeneity. We evaluated the bias and the confidence interval coverage for five commonly-used effect sizes (i.e., the mean difference, standardized mean difference, odds ratio, risk ratio, and risk difference). RESULTS: Sampling error did not cause noticeable bias when the effect size was the mean difference, but the standardized mean difference, odds ratio, risk ratio, and risk difference suffered from this bias to different extents. The bias in the estimated overall odds ratio and risk ratio was noticeable even when each individual study had more than 50 samples under some settings. Also, Hedges' g, which is a bias-corrected estimate of the standardized mean difference within studies, might lead to larger bias than Cohen's d in meta-analysis results. CONCLUSIONS: Cautions are needed to perform meta-analyses with small sample sizes. The reported within-study variances may not be simply treated as the true variances, and their sampling error should be fully considered in such meta-analyses.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
A basic introduction to fixed-effect and random-effects models for meta-analysis
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.12 · Cited by 6,793
Casella, G., & Berger, R. (2024) · Chapman & Hall/CRC eBooks · doi:10.1201/9781003456285 · Cited by 5,711
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Funnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Comparison of two methods to detect publication bias in meta-analysis.
Peters, J. L. (2006) · JAMA · doi:10.1001/jama.295.6.676 · Cited by 2,278
What to add to nothing? Use and avoidance of continuity corrections in meta‐analysis of sparse data
J. Sweeting, M., J. Sutton, A., & C. Lambert, P. (2004) · Statistics in Medicine · doi:10.1002/sim.1761 · Cited by 1,734
Meta-analysis and the science of research synthesis
Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018) · Nature · doi:10.1038/nature25753 · Cited by 1,709
Methods to estimate the between‐study variance and its uncertainty in meta‐analysis
Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P., Langan, D., & Salanti, G. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1164 · Cited by 1,685
Quantifying publication bias in meta-analysis
Lin, L., & Chu, H. (2018) · Biometrics · doi:10.1111/biom.12817 · Cited by 1,674
Self-citationStatistics Notes: The odds ratio
Bland, J. M. (2000) · BMJ · doi:10.1136/bmj.320.7247.1468 · Cited by 1,294
A comparison of methods to detect publication bias in meta‐analysis
Macaskill, P., Walter, S. D., & Irwig, L. (2001) · Statistics in Medicine · doi:10.1002/sim.698 · Cited by 1,286
Kirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133
THE ESTIMATION AND SIGNIFICANCE OF THE LOGARITHM OF A RATIO OF FREQUENCIES
HALDANE, B. J. B. S. (1956) · Annals of Human Genetics · doi:10.1111/j.1469-1809.1955.tb01285.x · Cited by 1,076
Language bias in randomised controlled trials published in English and German
Egger, M., Zellweger-Zähner, T., Schneider, M., Junker, C., Lengeler, C., & Antes, G. (1997) · The Lancet · doi:10.1016/s0140-6736(97)02419-7 · Cited by 1,028
Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events
Bradburn, M. J., Deeks, J. J., Berlin, J. A., & Russell Localio, A. (2007) · Statistics in Medicine · doi:10.1002/sim.2528 · Cited by 938
Stijnen, T., Hamza, T. H., & Özdemir, P. (2010) · Statistics in Medicine · doi:10.1002/sim.4040 · Cited by 757
Effect sizes for research: A broad practical approach.
Grissom, R. J., & Kim, J. J. (2005) · Cited by 732
Meta-analysis: Potentials and promise
Egger, M., & Smith, G. D. (1997) · BMJ · doi:10.1136/bmj.315.7119.1371 · Cited by 646
Recent developments in meta‐analysis
Sutton, A. J., & Higgins, J. P. T. (2008) · Statistics in Medicine · doi:10.1002/sim.2934 · Cited by 590
Consensus Values and Weighting Factors
Paule, R. C., & Mandel, J. (1982) · Journal of Research of the National Bureau of Standards · doi:10.6028/jres.087.022 · Cited by 572
Bayesian approaches to random-effects meta-analysis: a comparative study
Smith, T. C., Spiegelhalter, D. J., & Thomas, A. (1995) · Statistics in Medicine · doi:10.1002/sim.4780142408 · Cited by 568
Empirical Comparison of Publication Bias Tests in Meta-Analysis
Lin, L., Chu, H., Murad, M. H., Hong, C., Qu, Z., Cole, S. R., & Chen, Y. (2018) · Journal of General Internal Medicine · doi:10.1007/s11606-018-4425-7 · Cited by 464
Self-citationReasons or excuses for avoiding meta-analysis in forest plots
Ioannidis, J. P. A., Patsopoulos, N. A., & Rothstein, H. R. (2008) · BMJ · doi:10.1136/bmj.a117 · Cited by 457
Davey, J., Turner, R. M., Clarke, M. J., & Higgins, J. P. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-160 · Cited by 404
A comparison of heterogeneity variance estimators in combining results of studies
Sidik, K., & Jonkman, J. N. (2007) · Statistics in Medicine · doi:10.1002/sim.2688 · Cited by 380
Misunderstandings about Q and ‘Cochran's Q test' in meta‐analysis
Hoaglin, D. C. (2016) · Statistics in Medicine · doi:10.1002/sim.6632 · Cited by 310
Modelling publication bias in meta-analysis: a review
Sutton, A. J., Song, F., Gilbody, S. M., & Abrams, K. R. (2000) · Statistical Methods in Medical Research · doi:10.1177/096228020000900503 · Cited by 288
Warn, D. E., Thompson, S. G., & Spiegelhalter, D. J. (2002) · Statistics in Medicine · doi:10.1002/sim.1189 · Cited by 277
Rücker, G., Schwarzer, G., Carpenter, J., & Olkin, I. (2009) · Statistics in Medicine · doi:10.1002/sim.3511 · Cited by 252
Jackson, D., Bowden, J., & Baker, R. (2010) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2009.09.017 · Cited by 226
Reference bias in reports of drug trials.
Gotzsche, P. C. (1987) · BMJ · doi:10.1136/bmj.295.6599.654 · Cited by 222
A comparison of seven random‐effects models for meta‐analyses that estimate the summary odds ratio
Jackson, D., Law, M., Stijnen, T., Viechtbauer, W., & White, I. R. (2018) · Statistics in Medicine · doi:10.1002/sim.7588 · Cited by 203
The effect of publication bias magnitude and direction on the certainty in evidence
Murad, M. H., Chu, H., Lin, L., & Wang, Z. (2018) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2018-110891 · Cited by 203
Self-citation(2002) · Understanding Social Science Research · doi:10.4135/9780857020208.n7 · Cited by 195
Is There Gender Bias in JAMA's Peer Review Process?
Gilbert, J. R., Williams, E. S., & Lundberg, G. D. (1994) · JAMA · doi:10.1001/jama.1994.03520020065018 · Cited by 181
Meta-analysis as Evidence: Building a Better Pyramid
Berlin, J. A., & Golub, R. M. (2014) · JAMA · doi:10.1001/jama.2014.8167 · Cited by 160
Lin, L., Chu, H., & Hodges, J. S. (2016) · Biometrics · doi:10.1111/biom.12543 · Cited by 143
Self-citationIs there gender bias in JAMA's peer review process?
Gilbert, J. R. (1994) · JAMA · doi:10.1001/jama.272.2.139 · Cited by 135
Empirical vs natural weighting in random effects meta‐analysis
Shuster, J. J. (2009) · Statistics in Medicine · doi:10.1002/sim.3607 · Cited by 125
Some general points in estimating heterogeneity variance with the DerSimonian-Laird estimator
Böhning, D. (2002) · Biostatistics · doi:10.1093/biostatistics/3.4.445 · Cited by 122
Cai, T., Parast, L. M., & RYAN, L. M. (2010) · Statistics in Medicine · doi:10.1002/sim.3964 · Cited by 116
A Sampling Investigation of the Efficiency of Weighting Inversely as the Estimated Variance
Cochran, W. G., & Carroll, S. P. (1953) · Biometrics · doi:10.2307/3001436 · Cited by 81
Chu, H., Nie, L., Chen, Y., Huang, Y., & Sun, W. (2010) · Statistical Methods in Medical Research · doi:10.1177/0962280210393712 · Cited by 80
Malzahn, U. (2000) · Biometrika · doi:10.1093/biomet/87.3.619 · Cited by 75
The implications of publication bias for meta-analysis' other parameter
Jackson, D. (2005) · Statistics in Medicine · doi:10.1002/sim.2293 · Cited by 59
An unbiased correction for sampling error in validity generalization studies.
Hedges, L. V. (1989) · Journal of Applied Psychology · doi:10.1037/0021-9010.74.3.469 · Cited by 53
Multistep estimators of the between‐study variance: The relationship with the Paule‐Mandel estimator
van Aert, R. C. M., & Jackson, D. (2018) · Statistics in Medicine · doi:10.1002/sim.7665 · Cited by 48
The Bias and Higher Cumulants of the Logarithm of a Binomial Variate
Pettigrew, H. M., Gart, J. J., & Thomas, D. G. (1986) · Biometrika · doi:10.1093/biomet/73.2.425 · Cited by 44
Weighting bias in meta-analysis of binary outcomes
Tang, J.-L. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00237-7 · Cited by 31
Practical challenges of I2 as a measure of heterogeneity
Hoaglin, D. C. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1251 · Cited by 16
Meta-analysis: formulating, evaluating, combining, and reporting
Normand, S.-L. T. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19990215)18:3<321::aid-sim28>3.0.co;2-p
(1985)
Cited by
Shi, L., & Lin, L. (2019) · Medicine · doi:10.1097/md.0000000000015987 · Cited by 818
Self-citationArcsine‐based transformations for meta‐analysis of proportions: Pros, cons, and alternatives
Lin, L., & Xu, C. (2020) · Health Science Reports · doi:10.1002/hsr2.178 · Cited by 339
Self-citationEvaluation of various estimators for standardized mean difference in meta‐analysis
Lin, L., & Aloe, A. M. (2021) · Statistics in Medicine · doi:10.1002/sim.8781 · Cited by 297
Self-citationSun, Y., Ji, M., Leng, M., Li, X., Zhang, X., & Wang, Z. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2022.104204 · Cited by 176
Serrano-Ripoll, M. J., Zamanillo-Campos, R., Fiol-DeRoque, M. A., Castro, A., & Ricci-Cabello, I. (2022) · JMIR mHealth and uHealth · doi:10.2196/29621 · Cited by 96
Metadata
- Authors
- Lin, L.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2018-09-13
- Volume
- 13
- Issue
- 9
- Pages
- e0204056
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2892325866
- Cited by
- 335
- References
- 65
- FWCI
- 14.06
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.