The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey
Open accessAbstract
BACKGROUND: Statistical tests for funnel-plot asymmetry are common in meta-analyses. Inappropriate application can generate misleading inferences about publication bias. We aimed to measure, in a survey of meta-analyses, how frequently the application of these tests would be not meaningful or inappropriate. METHODS: We evaluated all meta-analyses of binary outcomes with é 3 studies in the Cochrane Database of Systematic Reviews (2003, issue 2). A separate, restricted analysis was confined to the largest meta-analysis in each of the review articles. In each meta-analysis, we assessed whether criteria to apply asymmetry tests were met: no significant heterogeneity, I2 < 50%, é 10 studies (with statistically significant results in at least 1) and ratio of the maximal to minimal variance across studies > 4. We performed a correlation and 2 regression asymmetry tests and evaluated their concordance. Finally, we sampled 60 meta-analyses from print journals in 2005 that cited use of the standard regression test. RESULTS: A total of 366 of 6873 (5%) and 98 of 846 meta-analyses (12%) in the wider and restricted Cochrane data set, respectively, would have qualified for use of asymmetry tests. Asymmetry test results were significant in 7%-18% of the meta-analyses. Concordance between the 3 tests was modest (estimated k 0.33-0.66). Of the 60 journal meta-analyses, 7 (12%) would qualify for asymmetry tests; all 11 claims for identification of publication bias were made in the face of large and significant heterogeneity. INTERPRETATION: Statistical conditions for employing asymmetry tests for publication bias are absent from most meta-analyses; yet, in medical journals these tests are performed often and interpreted erroneously.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, publication bias tests might be problematic for IN-OT studies due to small sample sizes resulting in insufficient statistical power and sample size variability for the tests to yield reliable and significant results (26).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredIn the citing paper
“The small number of studies and presence of significant heterogeneity precluded formal testing for publication bias[35].”
From Daily Sitting Time and All-Cause Mortality: A Meta-Analysis (2013) · cites this paper as [35]
In the citing paper
“Of 60 meta-analyses in 24 articles published in 2005 in which formal assessments were reported, most were based on fewer than ten studies; most displayed statistically significant heterogeneity; and many reviewers misinterpreted the results of the tests employed [123].”
From The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration (2009) · cites this paper as [123]
In the citing paper
“Although evidence that smaller studies had larger estimated effects than large ones may suggest the possibility that the available evidence is biased, misinterpretation of such data is common [123].”
From The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration (2009) · cites this paper as [123]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationCochrane Handbook for Systematic Reviews of Interventions
Tarsilla, M. (2010) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v6i14.284 · Cited by 5,232
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Self-citationFunnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Quantitative Synthesis in Systematic Reviews
Lau, J., Ioannidis, J. P. A., & Schmid, C. H. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-9-199711010-00008 · Cited by 2,517
Self-citationPublication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282
A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints
Harbord, R. M., Egger, M., & Sterne, J. A. C. (2006) · Statistics in Medicine · doi:10.1002/sim.2380 · Cited by 2,267 · cited 2×
“, the natural logarithm of the odds ratio divided by its standard error] against its precision [the inverse of the standard error](8)) and a new variant, a modified version of the regression test, which has a lower false-positive rate.(10) For all tests, statistical significance was claimed for p < 0.”
BackgroundMethodCompeting interests: none declared.
Hirsch, L. (2006) · Cited by 1,396
Publication bias in meta-analysis its causes and consequences
Thornton, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00161-4 · Cited by 1,291
A comparison of methods to detect publication bias in meta‐analysis
Macaskill, P., Walter, S. D., & Irwig, L. (2001) · Statistics in Medicine · doi:10.1002/sim.698 · Cited by 1,286
Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors
De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., et al. (2004) · New England Journal of Medicine · doi:10.1056/nejme048225 · Cited by 1,241
Publication bias: evidence of delayed publication in a cohort study of clinical research projects
Stern, J. M., & Simes, R. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.640 · Cited by 818
Adjusting for publication bias in the presence of heterogeneity
Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003) · Statistics in Medicine · doi:10.1002/sim.1461 · Cited by 809
Ioannidis, J. P. A. (1998) · JAMA · doi:10.1001/jama.279.4.281 · Cited by 632
Self-citationChan, A.-W., & Altman, D. G. (2005) · BMJ · doi:10.1136/bmj.38356.424606.8f · Cited by 613
Publication Bias in Meta‐Analysis
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch1 · Cited by 603
DeAngelis, C. D., Drazen, J. M., Frizelle, F., Haug, C. J., Hoey, J., Horton, R. C., Kotzin, S., Laine, C. M., Marušić, A., Overbeke, A. J. P. M., et al. (2004) · JAMA · doi:10.1001/jama.292.11.1363 · Cited by 579
Jadad, A. R., Cook, D. J., Jones, A., Klassen, T. P., Tugwell, P., Moher, M., & Moher, D. (1998) · JAMA · doi:10.1001/jama.280.3.278 · Cited by 534
Terrin, N. C., Schmid, C. H., & Lau, J. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.01.006 · Cited by 502
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2019) · CHANCE · doi:10.1080/09332480.2019.1579573 · Cited by 474
Self-citationOutcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research
Chan, A.-W. (2004) · Canadian Medical Association Journal · doi:10.1503/cmaj.1041086 · Cited by 444
Publication Bias: The Problem That Won't Go Away
DICKERSIN, K., & MIN, Y. (1993) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1993.tb26343.x · Cited by 427
Misleading funnel plot for detection of bias in meta-analysis
Tang, J.-L., & Liu, J. L. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00204-8 · Cited by 420
Clinical trial registration: a statement from the International Committee of Medical Journal Editors
De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P., et al. (2004) · The Lancet · doi:10.1016/s0140-6736(04)17034-7 · Cited by 408
Ioannidis, J., & Trikalinos, T. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.019 · Cited by 352
Self-citationSelective reporting biases in cancer prognostic factor studies.
Kyzas, P., Loizou, K. T., & Ioannidis, J. (2005) · Journal of the National Cancer Institute · doi:10.1093/jnci/dji184 · Cited by 226
Self-citationClinical trial registration: a statement from the International Committee of Medical Journal Editors
DeAngelis, C. (2004) · Canadian Medical Association Journal · doi:10.1503/cmaj.1041281 · Cited by 118
Shea, B., Moher, D., Graham, I. D., Pham, B., & Tugwell, P. (2002) · Evaluation & the Health Professions · doi:10.1177/0163278702025001008 · Cited by 109
Mallett, S. V. (2003) · BMJ Evidence-Based Medicine · doi:10.1136/ebm.8.4.100 · Cited by 71
Extreme between-study homogeneity in meta-analyses could offer useful insights
Ioannidis, J. P. A., Trikalinos, T. A., & Zintzaras, E. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.02.013 · Cited by 58
Self-citationBeyond Trial Registration: A Global Trial Bank for Clinical Trial Reporting
Sim, I., & Detmer, D. E. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020365 · Cited by 46
Assessment of publication bias in meta-analyses of cardiovascular diseases
Palma, S., & Delgado‐Rodríguez, M. (2005) · Journal of Epidemiology & Community Health · doi:10.1136/jech.2005.033027 · Cited by 45
Ioannidis, J. P. A. (2005) · doi:10.1002/0470870168.ch15 · Cited by 44
Self-citationSusan Mallett, D., & Mike Clarke, D. (2003) · ACP Journal Club · doi:10.7326/acpjc-2003-139-1-a11 · Cited by 37
Flagged first, then by citations
Cited by
Sorted by citations
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349
Self-citationSterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709
“After reviewing the results of these studies, and based on theoretical considerations, we formulated recommendations on testing for funnel plot asymmetry (box 2). The appendix describes the proposed tests, explains the reasons that some were not recommended, and discusses funnel plots for intervention effects measured…”
MethodSelf-citation
Metadata
- Authors
- Ioannidis, J. P. A., & Trikalinos, T. A.
- Affiliations
- Tufts University (US)
- Venue
- Canadian Medical Association Journal
- Publisher
- CMA Joule Inc.
- Published
- 2007-04-10
- Volume
- 176
- Issue
- 8
- Pages
- 1091-1096
- ISSN
- 0820-3946 (ISSN-L)
- 1488-2329
- PMC
- PMC1839799
- OpenAlex
- W2164093802
- Cited by
- 1,052
- References
- 40
- Citations by year
- 2026: 49 · 2025: 48 · 2024: 64 · 2023: 49 · 2022: 81 · 2021: 73 · 2020: 83 · 2019: 81 · 2018: 73 · 2017: 55 · 2016: 62 · 2015: 64 · 2014: 68 · 2013: 38 · 2012: 39
- FWCI
- 31.66
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Canadian Medical Association Journal (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, funnel plot asymmetry, meta-analysis, heterogeneity, Cochrane Database of Systematic Reviews, binary outcomes, test concordance
- MeSH terms
- Data Interpretation, Statistical, Humans, Periodicals as Topic, Meta-Analysis as Topic, Databases, Bibliographic, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
