A new improved graphical and quantitative method for detecting bias in meta-analysis
The DOI matches its Crossref record; 38 of the 40 references it cites were found in a registry or an index.
Abstract
Detection of publication and related biases remains suboptimal and threatens the validity and interpretation of meta-analytical findings. When bias is present, it usually differentially affects small and large studies manifesting as an association between precision and effect size and therefore visual asymmetry of conventional funnel plots. This asymmetry can be quantified and Egger's regression is, by far, the most widely used statistical measure for quantifying funnel plot asymmetry. However, concerns have been raised about both the visual appearance of funnel plots and the sensitivity of Egger's regression to detect such asymmetry, particularly when the number of studies is small. In this article, we propose a new graphical method, the Doi plot, to visualize asymmetry and also a new measure, the LFK index, to detect and quantify asymmetry of study effects in Doi plots. We demonstrate that the visual representation of asymmetry was better for the Doi plot when compared with the funnel plot. We also show that the diagnostic accuracy of the LFK index in discriminating between asymmetry due to simulated publication bias versus chance or no asymmetry was also better with the LFK index which had areas under the receiver operating characteristic curve of 0.74-0.88 with simulations of meta-analyses with five, 10, 15, and 20 studies. The Egger's regression result had lower areas under the receiver operating characteristic curve values of 0.58-0.75 across the same simulations. The LFK index also had a higher sensitivity (71.3-72.1%) than the Egger's regression result (18.5-43.0%). We conclude that the methods proposed in this article can markedly improve the ability of researchers to detect bias in meta-analysis.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Based on the evidence, a hierarchy of selection was determined to prefer PCR data, followed by microscopy, and then rapid diagnostic tests [13, 14].”
From Malaria in pregnancy: Meta-analyses of prevalence and associated complications (2024) · cites this paper as [14]
In the citing paper
“Publication biases were assessed using DOI plots and LFK index [14].”
From Malaria in pregnancy: Meta-analyses of prevalence and associated complications (2024) · cites this paper as [14]
In the citing paper
“However, this natural localization of malaria is compounded by a lack of robust and resilient health systems in many of the affected countries, where poverty, conflict, and natural disasters often further limit the impact of concerted public health efforts to tackle the disease [14, 15].”
From Malaria in pregnancy: Meta-analyses of prevalence and associated complications (2024) · cites this paper as [14]
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Sterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709 · cited 2×
“Although the focus of asymmetry plots and tests has been on biases that impact on the association between precision and effect in meta-analysis, it must be kept in mind that asymmetry can also occur due to heterogeneity,(40) and this of course is due to factors that lead to heterogeneity including study quality (weake…”
BackgroundFunnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 1×
“If indeed SSE is present, it then calls into question the validity of the results.(7,34,35) Although evidence for SSE has traditionally been graphically assessed by inspection of the funnel plot, research has shown that the latter can be misleading.”
BackgroundEFFECTIVENESS OF INTRAVENOUS THROMBOLYTIC TREATMENT IN ACUTE MYOCARDIAL INFARCTION
GRUPPOITALIANOPERLOSTUDIODELL (1986) · The Lancet · doi:10.1016/s0140-6736(86)92368-8 · Cited by 2,440
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282 · cited 6×
“The funnel plot and Egger’s regression are also known to have limited sensitivity for moderate amounts of asymmetry especially for meta-analyses based on a few studies.(3) It has been recommended that the classic funnel plot should not be used when the number of studies included in a meta-analysis is less than 10(37)…”
BackgroundMethodInfluentialBarendregt, J. J., Doi, S. A., Lee, Y. Y., Norman, R. E., & Vos, T. (2013) · Journal of Epidemiology and Community Health · doi:10.1136/jech-2013-203104 · Cited by 2,169
Self-citationThe case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766
Empirical assessment of effect of publication bias on meta-analyses
Sutton, A. J. (2000) · BMJ · doi:10.1136/bmj.320.7249.1574 · Cited by 1,398
Hunter, J. P., Saratzis, A., Sutton, A. J., Boucher, R. H., Sayers, R. D., & Bown, M. J. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.03.003 · Cited by 786
Research works printed but not linked: 3
- Cates, 2000, Lung cancer and passive smoking. Scales for visual test of publication bias are unfair, BMJ, 321, 1222
- Seagroatt, 1998, Bias in meta-analysis detected by a simple, graphical test. Test had 10% false positive rate, BMJ, 316, 470
- Balint, 2016, Which endovenous ablation method does offer a better long-term technical success in the treatment of the incompetent great saphenous vein?, Rev Vasc, 24, 649
Cited by
Rogers, J. P., Chesney, E., Oliver, D., Pollak, T. A., McGuire, P., Fusar-Poli, P., Zandi, M. S., Lewis, G., & David, A. S. (2020) · The Lancet Psychiatry · doi:10.1016/s2215-0366(20)30203-0 · Cited by 2,407
Vijenthira, A., Gong, I. Y., Fox, T. A., Booth, S., Cook, G., Fattizzo, B., Martín-Moro, F., Razanamahery, J., Riches, J. C., Zwicker, J., et al. (2020) · Blood · doi:10.1182/blood.2020008824 · Cited by 625
Cheng, C., Lau, Y.-C., Chan, L., & Luk, J. W. (2021) · Addictive Behaviors · doi:10.1016/j.addbeh.2021.106845 · Cited by 571
Tapia-Serrano, M. A., Sevil-Serrano, J., Sánchez-Miguel, P. A., López-Gil, J. F., Tremblay, M. S., & García-Hermoso, A. (2022) · Journal of Sport and Health Science · doi:10.1016/j.jshs.2022.01.005 · Cited by 333
López-Gil, J. F., García-Hermoso, A., Smith, L., Firth, J., Trott, M., Mesas, A. E., Jiménez-López, E., Gutiérrez-Espinoza, H., Tárraga-López, P. J., & Victoria-Montesinos, D. (2023) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2022.5848 · Cited by 276
Metadata
- Authors
- Furuya-Kanamori, L., Barendregt, J. J., & Doi, S. A. R.
- Affiliations
- EpiGear (Australia) (AU)
- Australian National University (AU)
- Corresponding author
- Barendregt, Jan J.
- ORCID
- Doi, Suhail A.R.
- Venue
- International Journal of Evidence-Based Healthcare
- Published
- 2018-12
- Volume
- 16
- Issue
- 4
- Pages
- 195-203
- ISSN
- 1744-1595 (ISSN-L)
- 1744-1609
- OpenAlex
- W2799972028
- Cited by
- 1,160
- References
- 40
- Citations by year
- 2026: 206 · 2025: 286 · 2024: 218 · 2023: 150 · 2022: 116 · 2021: 97 · 2020: 49 · 2019: 31 · 2018: 7
- FWCI
- 35.03
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Evidence-Based Healthcare (publishedVersion)
- PubMed (publishedVersion)
- ANU Open Research (Australian National University) (submittedVersion)
- Qatar University QSpace (Qatar University) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- publication bias, funnel plot asymmetry, small-study effects, diagnostic accuracy
- MeSH terms
- Computer Simulation, ROC Curve, Meta-Analysis as Topic, Models, Statistical, Bias, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A new improved graphical and quantitative method for detecting bias in meta-analysis. https://app.cochat.ai/papers/10.1097/xeb.0000000000000141
- The paper
- Furuya-Kanamori, L., Barendregt, J. J., & Doi, S. A. R. (2018). A new improved graphical and quantitative method for detecting bias in meta-analysis. International Journal of Evidence-Based Healthcare. https://doi.org/10.1097/xeb.0000000000000141
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.