The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Interpretation of tests of heterogeneity and bias in meta-analysis
Abstract
Statistical tests of heterogeneity and bias, in particular publication bias, are very popular in meta-analyses. These tests use statistical approaches whose limitations are often not recognized. Moreover, it is often implied with inappropriate confidence that these tests can provide reliable answers to questions that in essence are not of statistical nature. Statistical heterogeneity is only a correlate of clinical and pragmatic heterogeneity and the correlation may sometimes be weak. Similarly, statistical signals may hint to bias, but seen in isolation they cannot fully prove or disprove bias in general, let alone specific causes of bias, such as publication bias in particular. Both false-positive and false-negative signals of heterogeneity and bias can be common and their prevalence may be anticipated based on some rational considerations. Here I discuss the major common challenges and flaws that emerge in using and interpreting statistical tests of heterogeneity and bias in meta-analyses. I discuss misinterpretations that can occur at the level of statistical inference, clinical/pragmatic inference and specific cause attribution. Suggestions are made on how to avoid these flaws, use these tests properly and learn from them.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These limitations are common in meta-analyses of cross-sectional studies (Ioannidis, 2008).”
From Decision making capacity for treatment in psychiatric inpatients: a systematic review and meta-analysis (2024) · cites this paper as Ioannidis, 2008
Citations
Cites
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationThe combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Self-citationAssessing heterogeneity in meta-analysis: Q statistic or I² index?
Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.2.193 · Cited by 4,256
Funnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779
Cited by
Sorted by citations
Sterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709
Self-citationGloy, V. L., Briel, M., Bhatt, D. L., Kashyap, S. R., Schauer, P. R., Mingrone, G., Bucher, H. C., & Nordmann, A. J. (2013) · BMJ · doi:10.1136/bmj.f5934 · Cited by 1,351
Sonuga-Barke, E. J. S., Brandeis, D., Cortese, S., Daley, D., Ferrin, M., Holtmann, M., Stevenson, J., Danckaerts, M., van der Oord, S., Döpfner, M., et al. (2013) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2012.12070991 · Cited by 1,167
“Given the relatively small number of methodologically sound studies, the field is not yet mature enough for the investigation of publication bias using funnel plots—the interpretation of which, moreover, is equivocal when based on a small number of studies (29).”
BackgroundA systematic review of personality trait change through intervention.
Roberts, B. W., Luo, J., Briley, D. A., Chow, P. I., Su, R., & Hill, P. L. (2017) · Psychological Bulletin · doi:10.1037/bul0000088 · Cited by 1,047
“Most critically, Ioannidis (2008) argued that when effect sizes are even moderately heterogeneous, regression-based tests are “inappropriate, meaningless, or both” (p. 954).”
BackgroundZachariae, R., Lyby, M. S., Ritterband, L. M., & O'Toole, M. S. (2016) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2015.10.004 · Cited by 704
Probiotics for the prevention of pediatric antibiotic-associated diarrhea
Goldenberg, J. Z., Lytvyn, L., Steurich, J., Parkin, P. C., Mahant, S., & Johnston, B. C. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004827.pub4 · Cited by 635
Taylor, B., Irving, H. M., Kanteres, F., Room, R., Borges, G., Cherpitel, C., Greenfield, T., & Rehm, J. (2010) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2010.02.011 · Cited by 509
Global epidemiology of Duchenne muscular dystrophy: an updated systematic review and meta-analysis
Crisafulli, S., Sultana, J., Fontana, A., Salvo, F., Messina, S., & Trifirò, G. (2020) · Orphanet Journal of Rare Diseases · doi:10.1186/s13023-020-01430-8 · Cited by 505
“In response to this shortcoming, I(2) estimates were also accompanied by their associated 95% CI [70, 71] because imprecise or biased estimates of heterogeneity can have serious consequences: for instance its overestimation may trigger inappropriate exploration of the cause(s) of heterogeneity.”
BackgroundBedaso, A., Adams, J., Peng, W., & Sibbritt, D. W. (2021) · Reproductive Health · doi:10.1186/s12978-021-01209-5 · Cited by 628
“Publication bias, heterogeneity, and subgroup analysis Possible publication bias was assessed through inspection of the funnel plot and Egger’s regression tests [53, 54].”
MethodMorris, M. C., Compas, B. E., & Garber, J. (2012) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.02.002 · Cited by 466
Patra, J., Bakker, R., Irving, H., Jaddoe, V., Malini, S., & Rehm, J. (2011) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.2011.03050.x · Cited by 555
Evolution of Heterogeneity (I2) Estimates and Their 95% Confidence Intervals in Large Meta-Analyses
Thorlund, K., Imberger, G. L., Johnston, B. C., Walsh, M. W., Awad, T., Thabane, L., Gluud, C. N., Devereaux, P. J., & Wetterslev, J. (2012) · PLoS ONE · doi:10.1371/journal.pone.0039471 · Cited by 448
“In response to the above identified shortcomings, it has been proposed that reported I(2) estimates should be accompanied by their associated 95% confidence interval (CI) [6,12].”
BackgroundLopez-Duran, N. L., Kovacs, M., & George, C. J. (2009) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2009.03.016 · Cited by 425
“…factors that explain differences between studies even if heterogeneity was not observed, as estimates of heterogeneity have some limitations (Ioannidis, 2008), and significant moderating factors may be present even when the variability between studies is not large (see Hall and Rosenthal,…”
BackgroundZhang, L., Rana, I., Shaffer, R. M., Taioli, E., & Sheppard, L. A. (2019) · Mutation Research/Reviews in Mutation Research · doi:10.1016/j.mrrev.2019.02.001 · Cited by 381
Meta-assessment of bias in science
Fanelli, D., Costas, R., & Ioannidis, J. P. A. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1618569114 · Cited by 364
“Moreover, small-study effects may result not necessarily from QRP but also from legitimate choices made in designing small studies (17, 36).”
BackgroundSelf-citationZymperdikas, V. F., Koretsi, V., Papageorgiou, S. N., & Papadopoulos, M. A. A. P. (2015) · European Journal of Orthodontics · doi:10.1093/ejo/cjv034 · Cited by 306
“All P values were 2-sided with α = 5%, except for the test of between-studies or betweensubgroups heterogeneity (α = 10%) (55).”
MethodTreatment-effect estimates adjusted for small-study effects via a limit meta-analysis
Rücker, G., Schwarzer, G., Carpenter, J. R., Binder, H., & Schumacher, M. C. (2010) · Biostatistics · doi:10.1093/biostatistics/kxq046 · Cited by 295
“We acknowledge that some authors take a censorious attitude to such tests, believing them misleading (Lau and others, 2006; Terrin and others, 2003; Tang and Liu, 2000), stigmatizing them as “pseudo tests” (Ioannidis, 2008), and questioning whether funnel plots are a suited means at all for judging small-study effects…”
BackgroundEffect of Hospital Volume on Surgical Outcomes After Pancreaticoduodenectomy
Hata, T., Motoi, F., Ishida, M., Naitoh, T., Katayose, Y., Egawa, S., & Unno, M. (2015) · Annals of Surgery · doi:10.1097/sla.0000000000001437 · Cited by 267
“10 in the Q statistic (x(2)test).(20) I(2) values ranged from 0% to 100% and were interpreted as follows: 0 to 24%, no heterogeneity; 25% to 49%, moderate heterogeneity; 50% to 74% high level of heterogeneity; 75% to 100%, extreme heterogeneity.”
BackgroundNigg, J. T., Lewis, K., Edinger, T., & Falk, M. (2012) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2011.10.015 · Cited by 260
Probiotics for the prevention of pediatric antibiotic-associated diarrhea
Johnston, B. C., Goldenberg, J. Z., Vandvik, P. O., Sun, X., & Guyatt, G. H. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004827.pub3 · Cited by 264
Wang, Q., Zhu, H., Liu, H., & Zhang, B. (2012) · Hepatology · doi:10.1002/hep.25610 · Cited by 246
Carter, E. C., & McCullough, M. E. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00823 · Cited by 381
“Data were obtained from Martin Hagger, and further uses of these data should be acknowledged as such (Hagger et al., 2010).”
MethodInfluentialDadi, A. F., Miller, E. R., & Mwanri, L. (2020) · PLoS ONE · doi:10.1371/journal.pone.0227323 · Cited by 260
Vassos, E., Collier, D. A., & Fazel, S. (2014) · Molecular Psychiatry · doi:10.1038/mp.2013.31 · Cited by 234
Knops, A. M., Legemate, D. A., Goossens, A., Bossuyt, P. M. M., & Ubbink, D. T. (2013) · Annals of Surgery · doi:10.1097/sla.0b013e3182864fd6 · Cited by 231
Coles, N. A., Larsen, J. T., & Lench, H. C. (2019) · Psychological Bulletin · doi:10.1037/bul0000194 · Cited by 255
Meta‐research: The art of getting it wrong
Ioannidis, J. P. A. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.19 · Cited by 237
“2010, 1 169--184 1 7 5 A number of tests have been developed by meta-analysts to detect and exclude publication bias, but I am afraid that they are used beyond their real capacity (Ioannidis, 2008a; Ioannidis and Trikalinos, 2007b; Lau et al., 2006).”
BackgroundInfluentialSelf-citationGagnier, J. J., Moher, D., Boon, H., Beyene, J., & Bombardier, C. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-111 · Cited by 222
Papageorgiou, S. N., Koletsi, D., Iliadi, A., Peltomäki, T., & Eliades, T. (2019) · European Journal of Orthodontics · doi:10.1093/ejo/cjz094 · Cited by 243
Chen, D. T., Jiang, X., Akula, N., Shugart, Y. Y., Wendland, J. R., Steele, C. J. M., Kassem, L., Park, J.-H., Chatterjee, N., Jamain, S., et al. (2011) · Molecular Psychiatry · doi:10.1038/mp.2011.157 · Cited by 220
Wu, B., Gu, D.-Z., Yu, J., Yang, J., & Shen, W. (2020) · Infection Genetics and Evolution · doi:10.1016/j.meegid.2020.104485 · Cited by 217
Koretsi, V., Zymperdikas, V. F., Papageorgiou, S. N., & Papadopoulos, M. A. (2015) · The European Journal of Orthodontics · doi:10.1093/ejo/cju071 · Cited by 191
“Additional analyses Possible sources of heterogeneity were sought through pre-specified mixed-effects subgroup analyses and random-effects meta-regression with the Knapp–Hartung adjustment (60).”
MethodBerlim, M. T., Neufeld, N. H., & Van den Eynde, F. (2013) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2013.03.022 · Cited by 219
“The lack of significant overall heterogeneity among the included RCTs and our ability to discern the likely source of heterogeneity for pre-post Y-BOCS results shows that our findings are reliable (Ioannidis, 2008).”
ResultMaxwell, L., Devries, K. M., Zionts, D. L., Alhusen, J. L., & Campbell, J. C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0118234 · Cited by 188
“The heterogeneity of our exposure and outcome and the small number of included studies necessarily complicates the assessment of publication bias [60].”
BackgroundMethodHusebø, A. M. L., Dyrstad, S. M., Søreide, J. A., & Bru, E. (2013) · Journal of Clinical Nursing · doi:10.1111/j.1365-2702.2012.04322.x · Cited by 177
“patients decrease their physical activity levels after diagnosis, the importance of a positive attitude towards exercise including an established exercise behaviour before the cancer diagnosis has been made, increase the adherence to exercise intervention programmes (Midtgaard et al. 2009). In a review of 38 studies i…”
BackgroundMethodInfluentialComparison of four heterogeneity measures for meta‐analysis
Lin, L. (2019) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.13159 · Cited by 186
Moreno, S. G., Sutton, A. J., Turner, E. H., Abrams, K. R., Cooper, N. J., Palmer, T. M., & Ades, A. E. (2009) · BMJ · doi:10.1136/bmj.b2981 · Cited by 184
Revisiones sistemáticas de la literatura. Qué se debe saber acerca de ellas
Manterola, C., Astudillo, P., Arias, E., & Claros, N. (2011) · Cirugía Española · doi:10.1016/j.ciresp.2011.07.009 · Cited by 184
Interpreting Meta-Analyses of Genome-Wide Association Studies
Han, B., & Eskin, E. (2012) · PLoS Genetics · doi:10.1371/journal.pgen.1002555 · Cited by 177
“Another standard approach is to plot observed effect sizes and their confidence intervals of all studies [17, 25, 26].”
BackgroundAntidepressants for the treatment of depression in people with cancer
Ostuzzi, G., Matcham, F., Dauchy, S., Barbui, C., & Hotopf, M. H. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011006.pub2 · Cited by 184
“Ioannidis 2008 Ioannidis JP. Interpretation of tests of heterogeneity and bias in meta-analysis.”
BackgroundMethodFisher, D. J., Copas, A. J., Tierney, J. F., & Parmar, M. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.11.016 · Cited by 180
McMahon, F. J., Akula, N., Schulze, T. G., Detera‐Wadleigh, S. D., Steele, C. J. M., Wendland, J. R., Breuer, R., Strohmaier, J., Rietschel, M., Muglia, P., et al. (2010) · Nature Genetics · doi:10.1038/ng.523 · Cited by 180
Autotransplantation of teeth in humans: a systematic review and meta-analysis
Almpani, K., Papageorgiou, S. N., & Papadopoulos, M. A. A. P. (2015) · Clinical Oral Investigations · doi:10.1007/s00784-015-1473-9 · Cited by 180
Pabalan, N., Singian, E., Tabangay, L., Jarjanazi, H., Boivin, M. J., & Ezeamama, A. E. (2018) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0005523 · Cited by 176
Kuramatsu, J. B., Biffi, A., Gerner, S. T., Sembill, J. A., Sprügel, M. I., Leasure, A. C., Sansing, L. H., Matouk, C. C., Falcone, G. J., Endres, M., et al. (2019) · JAMA · doi:10.1001/jama.2019.13014 · Cited by 167
Moore, K. A., Simpson, J. A., Scoullar, M. J. L., McGready, R., & Fowkes, F. J. I. (2017) · The Lancet Global Health · doi:10.1016/s2214-109x(17)30340-6 · Cited by 165
The validity of circulating microRNAs in oncology: Five years of challenges and contradictions
Jarry, J., Schadendorf, D., Greenwood, C. M. T., Spatz, A., & van Kempen, L. C. L. T. (2014) · Molecular Oncology · doi:10.1016/j.molonc.2014.02.009 · Cited by 163
“the results identified as “most significant” in the study are subject to winner’s curse (Ioannidis, 2008; Faye et al., 2011).”
Pereira, T., Patsopoulos, N. A., Salanti, G., & Ioannidis, J. P. A. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.13 · Cited by 161
Self-citationLow birth weight and its associated factors in Ethiopia: a systematic review and meta-analysis
Endalamaw, A., Engeda, E. H., Ekubagewargies, D. T., Belay, G. M., & Tefera, M. A. (2018) · The Italian Journal of Pediatrics/Italian journal of pediatrics · doi:10.1186/s13052-018-0586-6 · Cited by 167
“Heterogeneity of studies was quantified using the I-squared statistic, in which 25, 50, and 75% represented low, moderate and high heterogeneity respectively [35, 36].”
MethodHow effective are expressive writing interventions for adolescents? A meta-analytic review
Travagin, G., Margola, D., & Revenson, T. A. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.01.003 · Cited by 186
Cystatin C and Cardiovascular Disease
van der Laan, S. W., Fall, T., Soumaré, A., Teumer, A., Sedaghat, S., Baumert, J., Zabaneh, D., van Setten, J., Išgum, I., Galesloot, T. E., et al. (2016) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2016.05.092 · Cited by 173
“Adding more studies to the observational analysis (46) or stratifying on the basis of these subgroups (29) might reduce heterogeneity and/or identify potential characteristics that account for heterogeneity.”
BackgroundAntidepressants for the treatment of depression in people with cancer
Ostuzzi, G., Matcham, F., Dauchy, S., Barbui, C., & Hotopf, M. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011006.pub3 · Cited by 158
Oral Anticoagulation and Functional Outcome after Intracerebral Hemorrhage
Biffi, A., Kuramatsu, J. B., Leasure, A. C., Kamel, H., Kourkoulis, C. E., Schwab, K. M., Ayres, A. M., Elm, J., Gurol, M. E., Greenberg, S. M., et al. (2017) · Annals of Neurology · doi:10.1002/ana.25079 · Cited by 151
“Mortality 105 (31) 60 (30) 55 (34) 38 (37) 30 (22) 25 (32)”
Dadi, A. F., Miller, E. R., & Mwanri, L. (2020) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-020-03092-7 · Cited by 175
“Visual inspection of a funnel plot asymmetry and Egger’s regression test were used to check for potential publication bias [32, 33].”
MethodLinden, A. H., & Hönekopp, J. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620964193 · Cited by 159
“(2014) reported a statistically significant moderator, but given its atheoretical nature and the number of moderators tested, it remains debatable whether this reflects progress in understanding or successful capitalization on chance (Ioannidis, 2008).”
BackgroundLeichsenring, F., Abbass, A. A., Heim, N., Keefe, J. R., Kisely, S. R., Luyten, P., Rabung, S., & Steinert, C. (2023) · World Psychiatry · doi:10.1002/wps.21104 · Cited by 157
Berlim, M. T., McGirr, A., den Eynde, F. V., Fleck, M. P., & Giacobbe, P. (2014) · Journal of Affective Disorders · doi:10.1016/j.jad.2014.02.016 · Cited by 140
“Fifth, the reliability of the analyses of heterogeneity and publication bias might have been relatively low owing to the small number of included studies (Ioannidis, 2008).”
ResultThe link between dissociation and both suicide attempts and non-suicidal self-injury: Meta-analyses
Calati, R., Bensassi, I., & Courtet, P. (2017) · Psychiatry Research · doi:10.1016/j.psychres.2017.01.035 · Cited by 138
Detecting and adjusting for small‐study effects in meta‐analysis
Rücker, G., Carpenter, J. R., & Schwarzer, G. (2011) · Biometrical Journal · doi:10.1002/bimj.201000151 · Cited by 132
“It is questioned whether funnel plots are a suited means for judging publication bias (Tang and Liu, 2000; Terrin et al., 2003, 2005; Lau et al., 2006; Ioannidis and Trikalinos, 2007; Ioannidis, 2008).”
BackgroundInterventions for orthodontically induced white spot lesions: a systematic review and meta-analysis
Höchli, D., Hersberger-Zurfluh, M. A., Papageorgiou, S. N., & Eliades, T. (2016) · European Journal of Orthodontics · doi:10.1093/ejo/cjw065 · Cited by 141
“10 used for the test of heterogeneity (28).”
MethodGarcía‐Robles, P., Cortés‐Pérez, I., Nieto-Escámez, F. A., García‐López, H., Obrero‐Gaitán, E., & Osuna‐Pérez, M. C. (2024) · Anatomical Sciences Education · doi:10.1002/ase.2397 · Cited by 165
“49,50 The risk of publication bias was assessed through the visualiza - tion of the funnel plots, with asymmetric funnel plots indicating a risk of publication bias, 51 and the p - value of the Egger test (present if p 0.1).”
MethodInfluentialUS studies may overestimate effect sizes in softer research
Fanelli, D., & Ioannidis, J. P. A. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1302997110 · Cited by 133
“Thepatterns observed in this study seem tobedifferent from, and independent of, the classic file-drawer problem, which is often incorrectly equated with small-study effects (inverse correlation between study size and magnitude of reporting) (44).”
BackgroundSelf-citationZumsteg, J. M., Cooper, J. S., & Noon, M. S. (2012) · Journal of Industrial Ecology · doi:10.1111/j.1530-9290.2012.00476.x · Cited by 137
“Bias Across Studies Publication bias is often the main type of bias across studies; it can be a concern when published studies do not accurately reflect the actual range of studies that have been performed (Ioannidis 2008).”
Light therapy in non-seasonal depression: An update meta-analysis
Tao, L., Jiang, R., Zhang, K., Qian, Z., Chen, P., Lv, Y., & Yao, Y. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2020.113247 · Cited by 129
Scientists Admitting to Plagiarism: A Meta-analysis of Surveys
Pupovac, V., & Fanelli, D. (2014) · Science and Engineering Ethics · doi:10.1007/s11948-014-9600-6 · Cited by 129
“…be emphasized, however, that the funnel plot test is of limited use when the number of primary studies is small and between-study heterogeneity is large, because these conditions make the interpretation of funnel plots very ambiguous (Terrin et al. 2003; Macaskill et al. 2001; Ioannidis 2008).”
A meta-analysis on verbal working memory in children and adolescents with ADHD
Ramos, A. A., Hamdan, A. C., & Machado, L. (2020) · The Clinical Neuropsychologist · doi:10.1080/13854046.2019.1604998 · Cited by 125
“In consideration of the fact that a bias analysis-based exclusively on funnel plots is primarily visual (Egger, Smith, et al., 1997), precarious (Ioannidis, 2008) and subjective (Duval, 2005; Sterne, Becker, & Egger, 2005), its use was complemented with Duval and Tweedie’s trim-and-fill method…”
MethodGedefaw, G. A., Demis, A., Alemnew, B. T. T., Wondmieneh, A. B., Getie, ፟., & Waltengus, F. (2020) · Patient Safety in Surgery · doi:10.1186/s13037-020-00236-8 · Cited by 127
“Cochrane Q-test and Isquared statistics were computed to check the heterogeneity of studies [15, 16].”
MethodNaenni, N., Lim, H., Papageorgiou, S. N., & Hämmerle, C. H. F. (2019) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.13052 · Cited by 128
“10 used for tests of between-studies or between-subgroups heterogeneity (Ioannidis, 2008).”
MethodInfluentialKitsiou, S., Paré, G., & Jaana, M. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2770 · Cited by 118
“However, the limitations of these metrics [64-67,70,71] and the uncertainty around the I point estimates, which can be expressed with 95% confidence intervals, were not considered in any of the included MAs.”
BackgroundMethodInfluentialThe role of the funnel plot in detecting publication and related biases in meta-analysis
Liu, J. L. (2011) · Evidence-Based Dentistry · doi:10.1038/sj.ebd.6400831 · Cited by 118
Bedaso, A., Mekonnen, N., & Duko, B. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-058187 · Cited by 160
The pulpal anesthetic efficacy of articaine versus lidocaine in dentistry
Brandt, R. G., Anderson, P. F., McDonald, N. J., Sohn, W., & Peters, M. C. (2011) · The Journal of the American Dental Association · doi:10.14219/jada.archive.2011.0219 · Cited by 119
Albaladejo-Saura, M., Vaquero‐Cristóbal, R., González-Gálvez, N., & Esparza-Ros, F. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18010328 · Cited by 120
“The DerSimonianLaird (Cohen) pooling method was used, and heterogeneity was assessed using the Cochrane Q test (Chi2), Higgins I2, and significance (p) to determine the appropriateness of the application of a fixed or random-effect model for the pooled analysis [50].”
MethodCoronado-Montoya, S., Levis, A. W., Kwakkenbos, L., Steele, R. J., Turner, E. H., & Thombs, B. D. (2016) · PLOS ONE · doi:10.1371/journal.pone.0153220 · Cited by 107
“Generally, risk of reporting bias will be higher in fields with small, underpowered trials [68]; when there is a strong incentive for reporting positive results, which is often the case when professionals who practice a given psychological treatment also test it [69, 70]; and when there is prior documentation of bias…”
BackgroundApplication of MALDI-TOF MS to rapid identification of anaerobic bacteria
Li, Y., Shan, M., Zhu, Z., Mao, X., Yan, M., Chen, Y., Zhu, Q., Li, H., & Gu, B. (2019) · BMC Infectious Diseases · doi:10.1186/s12879-019-4584-0 · Cited by 111
“This result is called “publication bias” in statistics [17].”
BackgroundBasic study design influences the results of orthodontic clinical investigations
Papageorgiou, S. N., Xavier, G. M., & Cobourne, M. T. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.03.008 · Cited by 108
“10 used for the test of heterogeneity and reporting biases [30].”
MethodMartín-Saavedra, J. S., Vergara-Méndez, L. D., Pradilla, I., Vélez-van-Meerbeke, A., & Talero-Gutiérrez, C. (2018) · Complementary Therapies in Medicine · doi:10.1016/j.ctim.2018.07.008 · Cited by 82
(ICCBD), M. O. T. I. C. C. F. B. D., Song, J., Bergen, S. E., Florio, A. D., Karlsson, R., Charney, A. W., Ruderfer, D. M., Stahl, E. A., Chambert, K. D., Moran, J. L., et al. (2015) · Molecular Psychiatry · doi:10.1038/mp.2015.165 · Cited by 105
Ardeshirlarijani, E., Tabatabaei‐Malazy, O., Mohseni, S., Qorbani, M., Larijani, B. A., & Jalili, R. B. (2019) · DARU Journal of Pharmaceutical Sciences · doi:10.1007/s40199-019-00302-2 · Cited by 108
Empirically Supported Treatment: Recommendations for a New Model
Tolin, D. F., McKay, D., Forman, E. M., Klonsky, E. D., & Thombs, B. D. (2015) · Clinical Psychology: Science and Practice · doi:10.1111/cpsp.12122 · Cited by 219
Rieker, J. A., Avilés, J. M. R., Muiños, M., & Ballesteros, S. (2022) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2022.838968 · Cited by 117
“Nonetheless, it has been suggested that large estimates of between-study heterogeneity can cause regression asymmetry (Ioannidis and Trikalinos, 2007; Ioannidis, 2008).”
BackgroundEvidence‐based healthcare, clinical knowledge and the rise of personalised medicine
Miles, A., Loughlin, M., & Polychronis, A. (2008) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2008.01094.x · Cited by 99
“The ultimate paper which closes the 2008 Thematic Edition on EBM has been contributed by John Ioannidis and is concerned with the interpretation of tests of heterogeneity and bias in metaanalysis [273].”
BackgroundKonstantonis, D., Vasileiou, D., Papageorgiou, S. N., & Eliades, T. (2018) · European Journal Of Oral Sciences · doi:10.1111/eos.12409 · Cited by 100
“10 that was used for the test of heterogeneity and reporting biases because of low power (33).”
MethodEffects of exercise programs on kyphosis and lordosis angle: A systematic review and meta-analysis
González-Gálvez, N., Gea-García, G. M., & Marcos-Pardo, P. J. (2019) · PLoS ONE · doi:10.1371/journal.pone.0216180 · Cited by 98
Mitsikostas, D., Chalarakis, N., Mantonakis, L., Delicha, E. M., & Sfikakis, P. P. (2011) · European Journal of Neurology · doi:10.1111/j.1468-1331.2011.03528.x · Cited by 97
Wang, S., Zhang, L., Pan, W., & Hua, H. (2017) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-017-1543-z · Cited by 94
“25, the effect size heterogeneity was considered statistically significant [45].”
BackgroundTsilidis, K. K., Papatheodorou, S., Εvangelou, Ε., & Ioannidis, J. P. A. (2012) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djs437 · Cited by 92
“Heterogeneity may often be a manifestation of bias in some studies of a metaanalysis, and this may be difficult to dissect and differentiate from genuine differences across studies (65).”
BackgroundSelf-citationDepression among caregivers of cancer patients: Updated systematic review and meta‐analysis
Bedaso, A., Kibret, G. D., & Duko, B. (2022) · Psycho-Oncology · doi:10.1002/pon.6045 · Cited by 107
“We have examined publication bias by visual inspection of a funnel plot and Egger's regression test.(29,30) A p‐value <0.”
MethodWang, C., Yu, J., Wang, H.-F., Tan, C., Meng, X., & Tan, L. (2014) · Journal of Alzheimer s Disease · doi:10.3233/jad-140660 · Cited by 94
“Considering the relatively small number of methodologically sound studies, it is not yet mature enough for the investigation of publication bias using funnel plots [13].”
BackgroundCutts, J. C., Agius, P. A., Lin, Z., Powell, R., Moore, K. A., Draper, B. L., Simpson, J. A., & Fowkes, F. J. I. (2020) · BMC Medicine · doi:10.1186/s12916-019-1467-6 · Cited by 90
“If the I(2) statistic was ≤ 75%, a metaanalysis based on a random effects model was conducted; when the I(2) statistic was > 75% and/or the lower 95% confidence limit was between 50 and 100%, the studies were not combined [49, 50].”
MethodPeripheral vascular endothelial growth factor as a novel depression biomarker: A meta-analysis
Carvalho, A. F., Köhler, C. A., McIntyre, R. S., Knöchel, C., Brunoni, A. R., Thase, M. E., Quevedo, J., Fernandes, B. S., & Berk, M. (2015) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2015.07.002 · Cited by 88
“The I2 measure was used to quantify between-study heterogeneity, and considered to be high when I2 > 50% (Ioannidis, 2008).”
MethodMcFarland, L. V., & Goh, S. (2019) · Travel Medicine and Infectious Disease · doi:10.1016/j.tmaid.2018.09.007 · Cited by 86
Azevedo, L. B., Stephenson, J. R., Ells, L. J., Adu‐Ntiamoah, S., DeSmet, A., Giles, E. L., Haste, A., O’Malley, C. L., Jones, D., Chai, L. K., et al. (2021) · Obesity Reviews · doi:10.1111/obr.13373 · Cited by 87
Nocebo in clinical trials for depression: A meta-analysis
Mitsikostas, D. D., Mantonakis, L., & Chalarakis, N. (2013) · Psychiatry Research · doi:10.1016/j.psychres.2013.10.019 · Cited by 86
“Data was analyzed using a random effects model, due to heterogeneity between studies, which was assessed by I2 and Cochran Q tests (Cochran, 1954; Higgins and Thompson, 2002; Ioannidis, 2008).”
Papageorgiou, S. N., Gölz, L., Jäger, A., Eliades, T., & Bourauel, C. P. (2016) · European Journal Of Oral Sciences · doi:10.1111/eos.12250 · Cited by 86
Berlim, M. T., & Van den Eynde, F. (2014) · The Canadian Journal of Psychiatry · doi:10.1177/070674371405900905 · Cited by 96
Exposure to pesticides and childhood leukemia risk: A systematic review and meta-analysis
Karalexi, M. A., Tagkas, C. F., Markozannes, G., Tseretopoulou, X., Hernández, A. F., Schüz, J., Halldorsson, T. I., Ψαλτοπούλου, Θ., Petridou, E. T., Tzoulaki, I., et al. (2021) · Environmental Pollution · doi:10.1016/j.envpol.2021.117376 · Cited by 93
Who benefits from individual placement and support? A meta-analysis
de Winter, L., Couwenbergh, C., van Weeghel, J., Sanches, S. A., Michon, H. W. C., & Bond, G. R. (2022) · Epidemiology and Psychiatric Sciences · doi:10.1017/s2045796022000300 · Cited by 88
Buck, L. M., Dalci, O., Darendeliler, M. A., Papageorgiou, S. N., & Papadopoulou, A. K. (2016) · European Journal of Orthodontics · doi:10.1093/ejo/cjw048 · Cited by 90
“Statistical significance was set at a two-sided α of 5 per cent, except for test of heterogeneity, where α was set at 10 per cent, due to low power (48).”
BackgroundEfficacy and safety of miconazole for oral candidiasis: a systematic review and meta‐analysis
Zhang, L., Fu, J., Hua, H., & Yan, Z. (2015) · Oral Diseases · doi:10.1111/odi.12380 · Cited by 84
Metadata
- Authors
- Ioannidis, J. P. A.
- Affiliations
- University of Ioannina (GR)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- Journal of Evaluation in Clinical Practice
- Publisher
- Wiley
- Published
- 2008-10
- Volume
- 14
- Issue
- 5
- Pages
- 951-957
- ISSN
- 1356-1294 (ISSN-L)
- 1365-2753
- OpenAlex
- W2144112749
- Cited by
- 696
- References
- 57
- Citations by year
- 2026: 52 · 2025: 69 · 2024: 98 · 2023: 72 · 2022: 50 · 2021: 64 · 2020: 49 · 2019: 34 · 2018: 30 · 2017: 23 · 2016: 23 · 2015: 26 · 2014: 27 · 2013: 22 · 2012: 15
- FWCI
- 4.52
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Evaluation in Clinical Practice (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- heterogeneity testing, publication bias, meta-analysis, statistical heterogeneity, clinical heterogeneity, false positive signals, bias detection
- MeSH terms
- Data Interpretation, Statistical, Humans, Regression Analysis, Research Design, Risk, Sensitivity and Specificity, Meta-Analysis as Topic, Reproducibility of Results, Bias, Confidence Intervals, Publication Bias, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
