A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints
Abstract
Publication bias and related bias in meta-analysis is often examined by visually checking for asymmetry in funnel plots of treatment effect against its standard error. Formal statistical tests of funnel plot asymmetry have been proposed, but when applied to binary outcome data these can give false-positive rates that are higher than the nominal level in some situations (large treatment effects, or few events per trial, or all trials of similar sizes). We develop a modified linear regression test for funnel plot asymmetry based on the efficient score and its variance, Fisher's information. The performance of this test is compared to the other proposed tests in simulation analyses based on the characteristics of published controlled trials. When there is little or no between-trial heterogeneity, this modified test has a false-positive rate close to the nominal level while maintaining similar power to the original linear regression test ('Egger' test). When the degree of between-trial heterogeneity is large, none of the tests that have been proposed has uniformly good properties.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Small-study effects were assessed using comparison-adjusted funnel plots and Egger's and Begg's tests.
“Small-study effects were assessed using comparison-adjusted funnel plots in the network, along with Egger's and Begg's tests for statistical significance.”
For odds ratios, methods by Harbord et al. and Peters et al. address this artefactual correlation.
“For odds ratios, methods proposed by Harbord et al. 64 and Peters et al. 65 overcome this problem”
From Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: a review (2021) · cites this paper as [65]
Graphical methods (funnel plot) and analytic methods (Egger's test) are often used to investigate small study bias, though interpretation can be problematic.
“Graphical methods, especially the funnel plot [212], and analytic methods (e.g., Egger's test) are often used [213],[214],[215], although their interpretation can be problematic [216],[217].”
From The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration (2009) · cites this paper as [213]
Cites
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139 · see citing works
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · see citing works
Self-citationMeta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · see citing works
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763 · see citing works
(2005) · Springer eBooks · doi:10.1007/0-387-27255-0_9 · Cited by 20,333 · see citing works
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955 · see citing works
McCullagh, P., & Nelder, J. A. (1989) · doi:10.1007/978-1-4899-3242-6 · Cited by 16,836 · see citing works
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099 · see citing works
Funnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779 · see citing works
Self-citationBeta blockade during and after myocardial infarction: An overview of the randomized trials
Yusuf, S., Peto, R., Lewis, J., Collins, R., & Sleight, P. (1985) · Progress in Cardiovascular Diseases · doi:10.1016/s0033-0620(85)80003-7 · Cited by 3,205 · see citing works
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · see citing works
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955 · see citing works
ON ESTIMATING THE RELATION BETWEEN BLOOD GROUP AND DISEASE
Woolf, B. (1955) · Annals of Human Genetics · doi:10.1111/j.1469-1809.1955.tb01348.x · Cited by 2,739 · see citing works
Moher, D. (2001) · JAMA · doi:10.1001/jama.285.15.1987 · Cited by 2,545 · see citing works
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282 · see citing works
Self-citationInvestigating and dealing with publication and other biases in meta-analysis
Sterne, J. A. C., Egger, M., & Smith, G. D. (2001) · BMJ · doi:10.1136/bmj.323.7304.101 · Cited by 2,225 · see citing works
Self-citationWhat to add to nothing? Use and avoidance of continuity corrections in meta‐analysis of sparse data
J. Sweeting, M., J. Sutton, A., & C. Lambert, P. (2004) · Statistics in Medicine · doi:10.1002/sim.1761 · Cited by 1,734 · see citing works
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,422 · see citing works
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · see citing works
A comparison of methods to detect publication bias in meta‐analysis
Macaskill, P., Walter, S. D., & Irwig, L. (2001) · Statistics in Medicine · doi:10.1002/sim.698 · Cited by 1,286 · see citing works
Egger, M., Jüni, P., Bartlett, C., Holenstein, F., & Sterne, J. (2003) · Health Technology Assessment · doi:10.3310/hta7010 · Cited by 1,181 · see citing works
Self-citationNicotine replacement therapy for smoking cessation
Silagy, C., Lancaster, T., Stead, L., Mant, D., & Fowler, G. (2004) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000146.pub2 · Cited by 972 · see citing works
Publication bias: evidence of delayed publication in a cohort study of clinical research projects
Stern, J. M., & Simes, R. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.640 · Cited by 818 · see citing works
Publication bias—a problem in interpreting medical data
Begg, C. B., & Berlin, J. A. (1988) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982993 · Cited by 814 · see citing works
Adjusting for publication bias in the presence of heterogeneity
Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003) · Statistics in Medicine · doi:10.1002/sim.1461 · Cited by 809 · see citing works
A general parametric approach to the meta‐analysis of randomized clinical trials
Whitehead, A., & Whitehead, J. (1991) · Statistics in Medicine · doi:10.1002/sim.4780101105 · Cited by 750 · see citing works
Publication bias: the case for an international registry of clinical trials.
Simes, R. J. (1986) · Journal of Clinical Oncology · doi:10.1200/jco.1986.4.10.1529 · Cited by 747 · see citing works
A note on graphical presentation of estimated odds ratios from several clinical trials
Galbraith, R. F. (1988) · Statistics in Medicine · doi:10.1002/sim.4780070807 · Cited by 663 · see citing works
The Design and Analysis of Sequential Clinical Trials.
Harrington, D. T., & Whitehead, J. (1985) · Journal of the American Statistical Association · doi:10.2307/2288093 · Cited by 637 · see citing works
A meta-analysis of 50 years of ruptured abdominal aortic aneurysm repair
Bown, M. J., Sutton, A. J., BELL, P. R. F., & Sayers, R. D. (2002) · British journal of surgery · doi:10.1046/j.1365-2168.2002.02122.x · Cited by 561 · see citing works
The Design and Analysis of Sequential Clinical Trials.
Whitehead, J. (1997) · Biometrics · doi:10.2307/2533535 · Cited by 524 · see citing works
Misleading funnel plot for detection of bias in meta-analysis
Tang, J.-L., & Liu, J. L. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00204-8 · Cited by 420 · see citing works
Selection Models and the File Drawer Problem
Iyengar, S., & Greenhouse, J. B. (1988) · Statistical Science · doi:10.1214/ss/1177013012 · Cited by 399 · see citing works
Graphical Display of Estimates Having Differing Standard Errors
Galbraith, R. F. (1988) · Technometrics · doi:10.1080/00401706.1988.10488400 · Cited by 300 · see citing works
Graphical Display of Estimates Having Differing Standard Errors
Galbraith, R. F. (1988) · Technometrics · doi:10.2307/1270081 · Cited by 268 · see citing works
A sensitivity analysis for publication bias in systematic reviews
Copas, J. B., & Shi, J. Q. (2001) · Statistical Methods in Medical Research · doi:10.1177/096228020101000402 · Cited by 260 · see citing works
Bias in meta-analysis detected by a simple, graphical test. Graphical test is itself biased.
Irwig, L., Macaskill, P., Berry, G., & Glasziou, P. (1998) · PubMed Central · Cited by 240 · see citing works
An Approach for Assessing Publication Bias Prior to Performing a Meta-Analysis
Dear, K., & Begg, C. (1992) · Statistical Science · doi:10.1214/ss/1177011363 · Cited by 202 · see citing works
Sackett, D. L., Deeks, J. J., & Altman, D. G. (1996) · BMJ Evidence-Based Medicine · doi:10.1136/ebm.1996.1.164 · Cited by 178 · see citing works
Meta-analysis. Beyond the grand mean?
Smith, G. D., Egger, M., & Phillips, A. N. (1997) · BMJ · doi:10.1136/bmj.315.7122.1610 · Cited by 174 · see citing works
Self-citationBias in the one‐step method for pooling study results
Greenland, S., & Salván, A. (1990) · Statistics in Medicine · doi:10.1002/sim.4780090307 · Cited by 165 · see citing works
Egger, M., Ebrahim, S., & Smith, G. D. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.1.1 · Cited by 160 · see citing works
Self-citationSchwarzer, G., Antes, G., & Schumacher, M. (2002) · Statistics in Medicine · doi:10.1002/sim.1224 · Cited by 91 · see citing works
Registering clinical trials: an essential role for WHO
Evans, T. W., Gülmezoglu, M., & Pang, T. (2004) · The Lancet · doi:10.1016/s0140-6736(04)16136-9 · Cited by 48 · see citing works
The importance of critically interpreting simulation studies.
Maldonado, G., & Greenland, S. (1997) · PubMed · Cited by 31 · see citing works
Explaining heterogeneity in meta-analysis: a comparison of methods
Thompson, S. G., & Sharp, S. J. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19991030)18:20
Flagged first, then by citations
Cited by
Sorted by citations
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164 · see citing works
Conducting Meta-Analyses in <i>R</i> with the <b>metafor</b> Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464 · see citing works
“Extensions, modifications, and further developments of the regression test are described (among others) by Macaskill et al. (2001), Sterne and Egger (2005), Harbord et al. (2006), Peters et al. (2006), R¨ucker et al. (2008), and Moreno et al. (2009).”
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648 · see citing works
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578 · see citing works
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349 · see citing works
Sterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709 · see citing works
Self-citationHow to perform a meta-analysis with R: a practical tutorial
Balduzzi, S., Rücker, G., & Schwarzer, G. (2019) · Evidence Based Mental Health · doi:10.1136/ebmental-2019-300117 · Cited by 5,827 · see citing works
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Italian Journal of Public Health · doi:10.2427/5768 · Cited by 3,374 · see citing works
Systematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,237 · see citing works
What Is the Male-to-Female Ratio in Autism Spectrum Disorder? A Systematic Review and Meta-Analysis
Loomes, R., Hull, L., & Mandy, W. P. L. (2017) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2017.03.013 · Cited by 2,804 · see citing works
Comparison of two methods to detect publication bias in meta-analysis.
Peters, J. L. (2006) · JAMA · doi:10.1001/jama.295.6.676 · Cited by 2,278 · see citing works
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053 · see citing works
Early skin-to-skin contact for mothers and their healthy newborn infants
Moore, E. R., Bergman, N., Anderson, G. C., & Medley, N. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003519.pub4 · Cited by 1,797 · see citing works
Quantifying publication bias in meta-analysis
Lin, L., & Chu, H. (2018) · Biometrics · doi:10.1111/biom.12817 · Cited by 1,674 · see citing works
“…statistical tests have been proposed for publication bias in the funnel plot, such as Begg’s rank test (Begg and Mazumdar, 1994) and Egger’s regression test (Egger et al., 1997) and its extensions (e.g., Macaskill et al., 2001; Rothstein et al., 2005; Harbord et al., 2006; Peters et al., 2006).”
MethodResultMitchell, A. J., Beaumont, H., Ferguson, D. Y., Yadegarfar, M., & Stubbs, B. (2014) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.12336 · Cited by 1,171 · see citing works
“To assess the risk of bias, we undertook a visual inspection of funnel plots and calculated the Harbord bias test (32).”
MethodAllen, N. C., Bagade, S., McQueen, M. B., Ioannidis, J. P. A., Kavvoura, F. K., Khoury, M. J., Tanzi, R. E., & Bertram, L. (2008) · Nature Genetics · doi:10.1038/ng.171 · Cited by 1,054 · see citing works
“For the analyses in this manuscript, we also carried out a modified regression procedure recently propose”
The appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey
Ioannidis, J. P. A., & Trikalinos, T. A. (2007) · Canadian Medical Association Journal · doi:10.1503/cmaj.060410 · Cited by 1,052 · see citing works
“, the natural logarithm of the odds ratio divided by its standard error] against its precision [the inverse of the standard error](8)) and a new variant, a modified version of the regression test, which has a lower false-positive rate.(10) For all tests, statistical significance was claimed for p < 0.”
BackgroundMethodOutcome of delirium in critically ill patients: systematic review and meta-analysis
Salluh, J. I., Wang, H., Schneider, E. B., Nagaraja, N., Yenokyan, G., Damluji, A. A., Serafim, R. B., & Stevens, R. D. (2015) · BMJ · doi:10.1136/bmj.h2538 · Cited by 1,057 · see citing works
Fluoride varnishes for preventing dental caries in children and adolescents
Marinho, V. C., Worthington, H. V., Walsh, T., & Clarkson, J. E. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002279.pub2 · Cited by 1,022 · see citing works
Chapter 10: Addressing reporting biases
Egger, M., & Moher, D. (2011) · Cited by 977 · see citing works
Self-citationLim, S. S., Davies, M. J., Norman, R. J., & Moran, L. J. (2012) · Human Reproduction Update · doi:10.1093/humupd/dms030 · Cited by 939 · see citing works
“The arcsine test proposed by Rucker et al. (2008) was conducted since neither the Egger’s (Egger et al., 1997) nor Harbord’s (Harbord et al., 2006) test was appropriate.”
MethodBowden, J., Spiller, W., Del Greco M, F., Sheehan, N., Thompson, J., Minelli, C., & Davey Smith, G. (2018) · International Journal of Epidemiology · doi:10.1093/ije/dyy101 · Cited by 880 · see citing works
Haider, B. A., Olofin, I., Wang, M., Spiegelman, D. L., Ezzati, M., Fawzi, W. W., & on behalf of Nutrition Impact Model Study Group (anaemia) (2013) · BMJ · doi:10.1136/bmj.f3443 · Cited by 895 · see citing works
Type 2 diabetes and cancer: umbrella review of meta-analyses of observational studies
Tsilidis, K. K., Kasimis, J. C., López, D. S., Ntzani, E., & Ioannidis, J. P. A. (2015) · BMJ · doi:10.1136/bmj.g7607 · Cited by 861 · see citing works
Luppa, M., Sikorski, C., Luck, T., Ehreke, L., Konnopka, A., Wiese, B., Weyerer, S., König, H.-H., & Riedel-Heller, S. G. (2012) · Journal of Affective Disorders · doi:10.1016/j.jad.2010.11.033 · Cited by 855 · see citing works
Strunz, E. C., Addiss, D. G., Stocks, M. E., Ogden, S., Utzinger, J., & Freeman, M. C. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001620 · Cited by 807 · see citing works
“Heterogeneity creates great difficulty in assessing publication bias accurately with statistical tests, so it is impossible to know how pronounced publication bias may be throughout our metaanalyses [100].”
BackgroundVitamin D supplementation for women during pregnancy
De‐Regil, L. M., Palacios, C. S., Lombardo, L. K., & Peña‐Rosas, J. P. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008873.pub3 · Cited by 768 · see citing works
Probiotics for the prevention of Clostridium difficile-associated diarrhea in adults and children
Goldenberg, J. Z., Ma, S. S., Saxton, J. D., Martzen, M. R., Vandvik, P. O., Thorlund, K., Guyatt, G. H., & Johnston, B. C. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006095.pub3 · Cited by 741 · see citing works
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2007) · Statistics in Medicine · doi:10.1002/sim.2889 · Cited by 716 · see citing works
Interpretation of tests of heterogeneity and bias in meta-analysis
Ioannidis, J. P. A. (2008) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2008.00986.x · Cited by 696 · see citing works
Support for healthy breastfeeding mothers with healthy term babies
Renfrew, M. J., McCormick, F. M., Wade, A. M., Quinn, B., & Dowswell, T. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001141.pub4 · Cited by 700 · see citing works
Rabe, H., Gyte, G. M., Díaz-Rossello, J. L., & Duley, L. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003248.pub4 · Cited by 674 · see citing works
Muka, T., Glišić, M., Milić, J., Verhoog, S., Bohlius, J., Bramer, W. M., Chowdhury, R., & Franco, O. H. (2019) · European Journal of Epidemiology · doi:10.1007/s10654-019-00576-5 · Cited by 682 · see citing works
Moja, L., Piatti, A., Pecoraro, V. L., Ricci, C., Virgili, G., Salanti, G., Germagnoli, L., Liberati, A., & Banfi, G. (2012) · PLoS ONE · doi:10.1371/journal.pone.0046175 · Cited by 651 · see citing works
“Nevertheless Harbord’s test was not statistically significant (P = 0.173).”
MethodInfluentialPopulations at risk for severe or complicated influenza illness: systematic review and meta-analysis
Mertz, D., Kim, T. H., Johnstone, J., Lam, P.-P., Science, M. E., Kuster, S. P., Fadel, S. A., Tran, D. T., Fernández, E., Bhatnagar, N., et al. (2013) · BMJ · doi:10.1136/bmj.f5061 · Cited by 632 · see citing works
van Kempen, E., Casas, M., Pershagen, G., & Foraster, M. (2018) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph15020379 · Cited by 620 · see citing works
“Also we applied Egger’s test of publication bias using the STATA-commands METAFUNNEL and METABIAS [144,145].”
MethodArbyn, M., Verdoodt, F., Snijders, P. J. F., Verhoef, V. M. J., Suonio, E., Dillner, L., Minozzi, S., Bellisario, C., Banzi, R., Zhao, F.-H., et al. (2014) · The Lancet Oncology · doi:10.1016/s1470-2045(13)70570-9 · Cited by 674 · see citing works
Rapid response systems: a systematic review and meta-analysis
Maharaj, R., Raffaele, I., & Wendon, J. A. (2015) · Critical Care · doi:10.1186/s13054-015-0973-y · Cited by 606 · see citing works
Recent developments in meta‐analysis
Sutton, A. J., & Higgins, J. P. T. (2007) · Statistics in Medicine · doi:10.1002/sim.2934 · Cited by 583 · see citing works
“This has led to a number of modified tests being developed for binary outcomes [122, 123, 125, 126], and more generally [127].”
BackgroundRoy, A., Eisenhut, M., Harris, R., Rodrigues, L. C., Sridhar, S., Habermann, S., Snell, L. B., Mangtani, P., Adetifa, I., Lalvani, A., et al. (2014) · BMJ · doi:10.1136/bmj.g4643 · Cited by 586 · see citing works
Pijls, B. G. C. W., Jolani, S., Atherley, A., Derckx, R. T., Dijkstra, J. I. R., Franssen, G. H. L. M., Hendriks, S., Richters, A., Jellema, A., Zalpuri, S., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-044640 · Cited by 580 · see citing works
Paravertebral block versus thoracic epidural for patients undergoing thoracotomy
Yeung, J., Gates, S., Naidu, B. V., Wilson, M., & Smith, F. G. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009121.pub2 · Cited by 577 · see citing works
Lauersen, J. B., Bertelsen, D. M., & Andersen, L. B. (2013) · British Journal of Sports Medicine · doi:10.1136/bjsports-2013-092538 · Cited by 580 · see citing works
Antibiotic prophylaxis versus no prophylaxis for preventing infection after cesarean section
Smaill, F. M., & Grivell, R. M. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007482.pub3 · Cited by 575 · see citing works
Navarese, E. P., Robinson, J. G., Kowalewski, M., Kolodziejczak, M., Andreotti, F., Bliden, K., Tantry, U., Kubica, J., Raggi, P., & Gurbel, P. A. (2018) · JAMA · doi:10.1001/jama.2018.2525 · Cited by 549 · see citing works
Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study
Nuesch, E., Trelle, S., Reichenbach, S., Rutjes, A. W. S., Tschannen, B., Altman, D. G., Egger, M., & Juni, P. (2010) · BMJ · doi:10.1136/bmj.c3515 · Cited by 559 · see citing works
“and might be affected by misclassification, whereas sample size or standard error might be extracted more easily. Sample size or statistical precision might therefore be the best single proxy for the cumulative effect of the different sources of bias in randomised trials in osteoarthritis and probably also in other fi…”
BackgroundSelf-citationMechanical methods for induction of labour
Jozwiak, M., Bloemenkamp, K. W. M., Kelly, A. J., Mol, B. W., Irion, O. P., & Boulvain, M. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001233.pub2 · Cited by 555 · see citing works
Probiotics for the prevention of Clostridium difficile-associated diarrhea in adults and children
Goldenberg, J. Z., Yap, C., Lytvyn, L., Lo, C. K.-F., Beardsley, J., Mertz, D., & Johnston, B. C. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006095.pub4 · Cited by 550 · see citing works
Bachmann, S., Finger, C., Huss, A., Egger, M., Stuck, A. E., & Clough‐Gorr, K. M. (2010) · BMJ · doi:10.1136/bmj.c1718 · Cited by 542 · see citing works
Self-citationKirkbride, J. B., Errázuriz, A., Croudace, T. J., Morgan, C., Jackson, D., Boydell, J. E., Murray, R. M., & Jones, P. B. (2012) · PLoS ONE · doi:10.1371/journal.pone.0031660 · Cited by 535 · see citing works
“Given such tests are sensitive to the number of data points and between-study heterogeneity [39], we restricted such tests to meta-analyses with a minimum of ten citations.”
MethodNavaneethan, U., Njei, B. M., Lourdusamy, V., Konjeti, V. R., Vargo, J. J., & Parsi, M. A. (2014) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2014.09.017 · Cited by 535 · see citing works
“The robustness of the meta-analysis to publication bias was assessed by funnel plots and bias indicators, including the Begg-Mazumdar test and the Harbord–Egger test.(14,15)”
Lill, C., Roehr, J. T., McQueen, M., Kavvoura, F., Bagade, S., Schjeide, B., Schjeide, L. M., Meissner, E., Zauft, U., Allen, N. C., et al. (2012) · PLoS Genetics · doi:10.1371/journal.pgen.1002548 · Cited by 575 · see citing works
“Thirdly, we directly assessed the evidence for small study bias by applying a recently proposed regression test [32] on all nominally significant (P,0.”
MethodLai, C. C. H., Ardern, C. L., Feller, J. A., & Webster, K. E. (2017) · British Journal of Sports Medicine · doi:10.1136/bjsports-2016-096836 · Cited by 527 · see citing works
Rabe, H., Díaz-Rossello, J. L., Duley, L., & Dowswell, T. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003248.pub3 · Cited by 527 · see citing works
Ferrante, G., Rao, S. V., Jüni, P., da Costa, B. R., Reimers, B., Condorelli, G., Anzuini, A., Jolly, S. S., Bertrand, O. F., Krucoff, M. W., et al. (2016) · JACC: Cardiovascular Interventions · doi:10.1016/j.jcin.2016.04.014 · Cited by 524 · see citing works
Chaimani, A., & Salanti, G. (2012) · Research Synthesis Methods · doi:10.1002/jrsm.57 · Cited by 521 · see citing works
Clarke, H., Bonin, R. P., Orser, B. A., Englesakis, M., Wijeysundera, D. N., & Katz, J. (2012) · Anesthesia & Analgesia · doi:10.1213/ane.0b013e318249d36e · Cited by 481 · see citing works
Prevalence of statin intolerance: a meta-analysis
Bytyçi, I., Penson, P. E., Mikhailidis, D. P., Wong, N. D., Hernandez, A. V., Sahebkar, A., Thompson, P. D., Mazidi, M., Rysz, J., Pella, D., et al. (2022) · European Heart Journal · doi:10.1093/eurheartj/ehac015 · Cited by 481 · see citing works
“05 (two-tailed) was considered statistically significant.(18)”
Poggio, F. B., Bruzzone, M., Ceppi, M., Ponde, N. F., Valle, G. L., Mastro, L. D., de Azambuja, E., & Lambertini, M. (2018) · Annals of Oncology · doi:10.1093/annonc/mdy127 · Cited by 475 · see citing works
“The likelihood of publication bias was assessed by visual inspection of funnel plot for study size against treatment effect [18] and with Harbords’ asymmetry test [19].”
MethodEarly skin-to-skin contact for mothers and their healthy newborn infants
Moore, E. R., Anderson, G. C., Bergman, N., & Dowswell, T. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003519.pub3 · Cited by 474 · see citing works
Malin, G. L., Morris, R. K., & Khan, K. S. (2010) · BMJ · doi:10.1136/bmj.c1471 · Cited by 461 · see citing works
Brunström, M., & Carlberg, B. (2017) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2017.6015 · Cited by 468 · see citing works
The effect of smoking on COVID‐19 severity: A systematic review and meta‐analysis
Reddy, R. K., Charles, W. N., Sklavounos, A., Dutt, A., Seed, P. T., & Khajuria, A. (2020) · Journal of Medical Virology · doi:10.1002/jmv.26389 · Cited by 466 · see citing works
Chu, D. K., Wood, R. A., French, S., Fiocchi, A. G., Jordana, M., Waserman, S., Brożek, J., & Schünemann, H. J. (2019) · The Lancet · doi:10.1016/s0140-6736(19)30420-9 · Cited by 448 · see citing works
Active versus expectant management for women in the third stage of labour
Begley, C., Gyte, G. M., Devane, D., McGuire, W. L., & Weeks, A. D. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007412.pub4 · Cited by 448 · see citing works
Kenet, G., Lütkhoff, L. K., Albisetti, M., Bernard, T. J., Bonduel, M. M., Brandão, L. R., Chabrier, S., Chan, A. K. C., deVeber, G., Fiedler, B. J., et al. (2010) · Circulation · doi:10.1161/circulationaha.109.913673 · Cited by 446 · see citing works
Arcsine test for publication bias in meta‐analyses with binary outcomes
Rücker, G., Schwarzer, G., & Carpenter, J. R. (2007) · Statistics in Medicine · doi:10.1002/sim.2971 · Cited by 444 · see citing works
Goldenberg, J. Z., Day, A., Brinkworth, G. D., Sato, J., Yamada, S., Jönsson, T., Beardsley, J., Johnson, J. A., Thabane, L., & Johnston, B. C. (2021) · BMJ · doi:10.1136/bmj.m4743 · Cited by 428 · see citing works
Perineal techniques during the second stage of labour for reducing perineal trauma
Aasheim, V., Nilsen, A. B. V., Reinar, L. M., & Lukasse, M. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006672.pub3 · Cited by 439 · see citing works
Fetal and umbilical Doppler ultrasound in high-risk pregnancies
Alfirević, Ž., Stampalija, T., & Gyte, G. M. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007529.pub3 · Cited by 440 · see citing works
van Enst, W. A., Ochodo, E., Scholten, R. J. P. M., Hooft, L., & Leeflang, M. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-70 · Cited by 432 · see citing works
“The applied tests were Egger’s test (n = 18), Deeks’ test (n = 12), Begg’s test (n = 5), both the Egger and Begg test (n = 4), and both the Begg-Mazumdar and Harbord’s test [70].”
MethodFrequency of Progression From Acute to Chronic Pancreatitis and Risk Factors: A Meta-analysis
Sankaran, S. J., Xiao, A., Wu, L. M., Windsor, J. A., Forsmark, C. E., & Petrov, M. S. (2015) · Gastroenterology · doi:10.1053/j.gastro.2015.07.066 · Cited by 423 · see citing works
Empirical Comparison of Publication Bias Tests in Meta-Analysis
Lin, L., Chu, H., Murad, M. H., Hong, C., Qu, Z., Cole, S. R., & Chen, Y. (2018) · Journal of General Internal Medicine · doi:10.1007/s11606-018-4425-7 · Cited by 464 · see citing works
“, design flaws such as inadequate allocation concealment) and they commonly enroll high-risk individuals, the funnel plot can be viewed as an approach to evaluating small study effects in general, rather than publication bias in particular.(14,31,32) In addition, the P value plots in Figures 2 and 3 indicate that some…”
BackgroundMitchell, A. J., Sheth, B., Gill, J., Yadegarfar, M., Stubbs, B., Yadegarfar, M., & Meader, N. (2017) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2017.04.001 · Cited by 414 · see citing works
Arias, Á., Pérez‐Martínez, I., Tenías, J. M., & Lucendo, A. J. (2015) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.13441 · Cited by 412 · see citing works
“Publication bias was evaluated with the aid of a funnel plot, the asymmetry of which was assessed with both the Begg-Mazumda’s rank test(16) and the Harbord test.(17) For the primary outcome, planned subgroup analyses were performed based on the geographic origin of patients (comparisons of figures were made by contin…”
MethodE-learning for health professionals
Vaona, A., Banzi, R., Kwag, K. H., Rigon, G., Cereda, D., Pecoraro, V. L., Tramacere, I., & Moja, L. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011736.pub2 · Cited by 406 · see citing works
Newly diagnosed diabetes mellitus after acute pancreatitis: a systematic review and meta-analysis
Das, S. L. M., Singh, P. P., Phillips, A. R. J., Murphy, R., Windsor, J. A., & Petrov, M. S. (2013) · Gut · doi:10.1136/gutjnl-2013-305062 · Cited by 401 · see citing works
Publication Bias in the Organizational Sciences
Kepes, S., Banks, G. C., McDaniel, M. A., & Whetzel, D. L. (2012) · Organizational Research Methods · doi:10.1177/1094428112452760 · Cited by 400 · see citing works
“Overall, the modified tests may not necessarily outperform the original ones, but they can under certain conditions (e.g., Harbord et al., 2006; Macaskill et al., 2001), especially when the outcome measure is the natural log of the odds ratio (lnOR) (Moreno, Sutton, Ades, et al., 2009).”
BackgroundMethodKeum, N., Greenwood, D. C., Lee, D. H., Kim, R., Aune, D., Ju, W., Hu, F. B., & Giovannucci, E. L. (2015) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djv088 · Cited by 398 · see citing works
“Potential for small study effects (37), such as publication bias, was tested using Egger’s test (38).”
MethodPublication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471 · see citing works
Puli, S. R., Bechtold, M. L., Buxbaum, J. L., & Eloubeidi, M. A. (2012) · Pancreas · doi:10.1097/mpa.0b013e3182546e79 · Cited by 389 · see citing works
“The effect of publication and selection bias on the summary estimates was tested by Egger bias indicator.(11) Also, funnel plots were constructed to evaluate potential publication bias using the SE and diagnostic odds ratio.”
MethodEffects of vitamin E on stroke subtypes: meta-analysis of randomised controlled trials
Schürks, M., Glynn, R. J., Rist, P. M., Tzourio, C., & Kurth, T. (2010) · BMJ · doi:10.1136/bmj.c5702 · Cited by 383 · see citing works
“We formally tested for small study effects (such as publication bias) by using Harbord’s test.”
MethodCervical stitch (cerclage) for preventing preterm birth in singleton pregnancy
Alfirević, Ž., Stampalija, T., & Medley, N. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008991.pub3 · Cited by 380 · see citing works
Puli, S. R., Reddy, J. B., Bechtold, M. L., Antillon, D., Ibdah, J. A., & Antillon, M. R. (2008) · World Journal of Gastroenterology · doi:10.3748/wjg.14.1479 · Cited by 381 · see citing works
Sarcopenia and Postoperative Complication Risk in Gastrointestinal Surgical Oncology
Simonsen, C., de Heer, P., Bjerre, E. D., Suetta, C., Højman, P., Pedersen, B. K., Svendsen, L. B., & Christensen, J. F. (2018) · Annals of Surgery · doi:10.1097/sla.0000000000002679 · Cited by 378 · see citing works
Schwingshackl, L., Knüppel, S., Schwedhelm, C., Hoffmann, G., Missbach, B., Stelmach-Mardas, M., Dietrich, S., Eichelmann, F., Kontopanteils, E., Iqbal, K., et al. (2016) · Advances in Nutrition · doi:10.3945/an.116.013052 · Cited by 360 · see citing works
Moreno, S. G., Sutton, A. J., Ades, A., Stanley, T. D., Abrams, K. R., Peters, J. L., & Cooper, N. J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-2 · Cited by 373 · see citing works
Planned caesarean section for term breech delivery
Hofmeyr, G. J., Hannah, M., & Lawrie, T. A. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000166.pub2 · Cited by 370 · see citing works
Kramer, M. S., & Kakuma, R. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000133.pub3 · Cited by 372 · see citing works
Palmer, S. C., Craig, J. C., Navaneethan, S. D., Tonelli, M., Pellegrini, F., & Strippoli, G. F. M. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-4-201208210-00007 · Cited by 359 · see citing works
“To assess potential bias from small study effects, we constructed funnel plots for the log risk ratio in individual studies against the SE of the risk ratio and formally assessed for plot asymmetry by using the Egger regression test (106).”
MethodThe evidence for improving housing to reduce malaria: a systematic review and meta-analysis
Tusting, L. S., Ippolito, M. M., Willey, B. A., Kleinschmidt, I., Dorsey, G., Gosling, R. D., & Lindsay, S. W. (2015) · Malaria Journal · doi:10.1186/s12936-015-0724-1 · Cited by 355 · see citing works
“Risk of bias across studies (publication bias) was assessed using funnel plots and Egger’s test for funnel plot asymmetry [24].”
BackgroundMethodPuli, S. R., Reddy, J. B., Bechtold, M. L., Antillon, M. R., & Brugge, W. R. (2009) · Digestive Diseases and Sciences · doi:10.1007/s10620-008-0651-x · Cited by 355 · see citing works
“In this meta-analysis and systematic review, bias calculations using the Harbord-Egger bias indicator [37] and Begg-Mazumdar bias indicator [38] showed no statistically significant bias.”
MethodLucendo, A. J., Arias, Á., & Molina‐Infante, J. (2015) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2015.07.041 · Cited by 350 · see citing works
Assessing risk of bias due to missing results in a synthesis
Page, M. J., Higgins, J. P., & Sterne, J. A. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch13 · Cited by 402 · see citing works
Self-citationImmersion in water during labour and birth
Cluett, E. R., Burns, E. E., & Cuthbert, A. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000111.pub4 · Cited by 344 · see citing works
Tranexamic acid for preventing postpartum haemorrhage
Novikova, N., Hofmeyr, G. J., & Cluver, C. A. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007872.pub3 · Cited by 339 · see citing works
Crowther, C. A., McKinlay, C. J. D., Middleton, P., & Harding, J. E. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003935.pub3 · Cited by 337 · see citing works
Mitchell, A., Yadegarfar, M., Gill, J., & Stubbs, B. (2016) · BJPsych Open · doi:10.1192/bjpo.bp.115.001685 · Cited by 322 · see citing works
Navarese, E. P., Andreotti, F., Schulze, V. T., odziejczak, M. K., Buffon, A., Brouwer, M. A., Costa, F., Kowalewski, M., Parati, G., Lip, G. Y. H., et al. (2015) · BMJ · doi:10.1136/bmj.h1618 · Cited by 335 · see citing works
- Authors
- Harbord, R. M., Egger, M., & Sterne, J. A. C.
- Affiliations
- University of Bristol (GB)
- University of Bern (CH)
- Corresponding author
- Harbord, Roger M.
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2005-12-12
- Volume
- 25
- Issue
- 20
- Pages
- 3443-3457
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- OpenAlex
- W2039418122
- Cited by
- 2,267
- References
- 46
- Citations by year
- 2026: 83 · 2025: 96 · 2024: 136 · 2023: 140 · 2022: 142 · 2021: 159 · 2020: 109 · 2019: 132 · 2018: 111 · 2017: 144 · 2016: 135 · 2015: 137 · 2014: 130 · 2013: 127 · 2012: 190
- FWCI
- 15.70
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Statistics in Medicine (publishedVersion)
- PubMed (publishedVersion)
- Bern Open Repository and Information System (University of Bern) (submittedVersion)
- Bristol Research (University of Bristol) (submittedVersion)
- Explore Bristol Research (submittedVersion)
- Funders
- University of Bristol
- Medical Research Council
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- publication bias, small-study effects, funnel plot asymmetry, binary outcomes, meta-analysis, efficient score, Fisher information, false positive rate, between-trial heterogeneity
- MeSH terms
- Humans, Meta-Analysis as Topic, Randomized Controlled Trials as Topic, Publication Bias, Endpoint Determination
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
