P value–driven methods were underpowered to detect publication bias: analysis of Cochrane review meta-analyses
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
OBJECTIVE To investigate the effect of number of studies in a meta-analysis on the detection of publication bias using p-value driven methods. METHODS The proportion of meta-analyses detected by Egger's, Harbord's, Peters', and Begg's tests to have asymmetry suggestive of publication bias were examined in 5014 meta-analyses from Cochrane reviews. P-values were also assessed in meta-analyses with varying number of studies, while symmetry was held constant. A simulation study was conducted to investigate if the above tests underestimate or overestimate the presence of publication bias. RESULTS The proportion of meta-analyses detected as asymmetrical via Egger's, Harbord's, Peters', and Begg's tests decreased by 42.6%, 41.1%, 29.3%, and 28.3%, when the median number of studies in the meta-analysis decreased from 87 to 14. P-values decreased as the number of studies increased in the meta-analysis, despite the level of symmetry remaining constant. The simulation study confirmed that when publication bias is present, p-value tests underestimate the presence of publication bias, particularly when study numbers are small. CONCLUSIONS P-value based tests used for detection of publication bias related asymmetry in meta-analysis require careful examination as they underestimate asymmetry. Alternative methods not dependent on the number of studies are preferable.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Sterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709
Scientists rise up against statistical significance
Amrhein, V., Greenland, S., & McShane, B. (2019) · Nature · doi:10.1038/d41586-019-00857-9 · Cited by 2,983
Moving to a World Beyond “ p < 0.05”
Wasserstein, R. L., Schirm, A. L., & Lazar, N. A. (2019) · The American Statistician · doi:10.1080/00031305.2019.1583913 · Cited by 2,976
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282
A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints
Harbord, R. M., Egger, M., & Sterne, J. A. C. (2006) · Statistics in Medicine · doi:10.1002/sim.2380 · Cited by 2,267
What to add to nothing? Use and avoidance of continuity corrections in meta‐analysis of sparse data
J. Sweeting, M., J. Sutton, A., & C. Lambert, P. (2004) · Statistics in Medicine · doi:10.1002/sim.1761 · Cited by 1,734
Empirical assessment of effect of publication bias on meta-analyses
Sutton, A. J. (2000) · BMJ · doi:10.1136/bmj.320.7249.1574 · Cited by 1,398
A new improved graphical and quantitative method for detecting bias in meta-analysis
Furuya-Kanamori, L., Barendregt, J. J., & Doi, S. A. R. (2018) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000141 · Cited by 1,160
Self-citationMuch ado about nothing: a comparison of the performance of meta-analytical methods with rare events
Bradburn, M. J., Deeks, J. J., Berlin, J. A., & Russell Localio, A. (2007) · Statistics in Medicine · doi:10.1002/sim.2528 · Cited by 938
Arbyn, M., Xu, L., Simoens, C., & Martin-Hirsch, P. P. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009069.pub3 · Cited by 621
Cited by
A historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
Doi, S. A., Furuya-Kanamori, L., Xu, C., Lin, L., Chivese, T., & Thalib, L. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.08.019 · Cited by 150
Self-citationA proposed framework to guide evidence synthesis practice for meta-analysis with zero-events studies
Xu, C., Furuya-Kanamori, L., Zorzela, L., Lin, L., & Vohra, S. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.012 · Cited by 117
Self-citationThe cognitive effects of computational thinking: a systematic review and meta-analytic study
Montuori, C., Gambarota, F., Altoé, G., & Arfé, B. (2024) · Computers & Education · doi:10.1016/j.compedu.2023.104961 · Cited by 76
Physical activity in pregnancy prevents gestational diabetes: A meta-analysis
Doi, S. A. R., Furuya-Kanamori, L., Toft, E., Musa, O. A. H., Mohamed, A. M., Clark, J., & Thalib, L. (2020) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2020.108371 · Cited by 33
Self-citation
Metadata
- Authors
- Furuya-Kanamori, L., Xu, C., Lin, L., Doan, T., Chu, H., Thalib, L., & Doi, S. A. R.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2020-02
- Volume
- 118
- Pages
- 86-92
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2988302981
- Cited by
- 123
- References
- 12
- FWCI
- 9.85
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: P value–driven methods were underpowered to detect publication bias: analysis of Cochrane review meta-analyses. https://app.cochat.ai/papers/10.1016/j.jclinepi.2019.11.011
- The paper
- Furuya-Kanamori, L., Xu, C., Lin, L., Doan, T., Chu, H., Thalib, L., & Doi, S. A. R. (2020). P value–driven methods were underpowered to detect publication bias: analysis of Cochrane review meta-analyses. Journal of Clinical Epidemiology. https://doi.org/10.1016/j.jclinepi.2019.11.011
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.