The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Interpretation of tests of heterogeneity and bias in meta-analysis
Abstract
Statistical tests of heterogeneity and bias, in particular publication bias, are very popular in meta-analyses. These tests use statistical approaches whose limitations are often not recognized. Moreover, it is often implied with inappropriate confidence that these tests can provide reliable answers to questions that in essence are not of statistical nature. Statistical heterogeneity is only a correlate of clinical and pragmatic heterogeneity and the correlation may sometimes be weak. Similarly, statistical signals may hint to bias, but seen in isolation they cannot fully prove or disprove bias in general, let alone specific causes of bias, such as publication bias in particular. Both false-positive and false-negative signals of heterogeneity and bias can be common and their prevalence may be anticipated based on some rational considerations. Here I discuss the major common challenges and flaws that emerge in using and interpreting statistical tests of heterogeneity and bias in meta-analyses. I discuss misinterpretations that can occur at the level of statistical inference, clinical/pragmatic inference and specific cause attribution. Suggestions are made on how to avoid these flaws, use these tests properly and learn from them.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These limitations are common in meta-analyses of cross-sectional studies (Ioannidis, 2008).”
From Decision making capacity for treatment in psychiatric inpatients: a systematic review and meta-analysis (2024) · cites this paper as Ioannidis, 2008
Citations
Cites
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationThe combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Self-citationAssessing heterogeneity in meta-analysis: Q statistic or I² index?
Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.2.193 · Cited by 4,256
Funnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779
Cited by
Sorted by citations
Items 201–300 of 695
Rush, C. J., Campbell, R. T., Jhund, P. S., Connolly, E. C., Preiss, D., Gardner, R. S., Petrie, M. C., & McMurray, J. J. V. (2015) · JACC Heart Failure · doi:10.1016/j.jchf.2015.03.013 · Cited by 37
Elkordy, S. A., Aboalnaga, A. A., Fayed, M. M. S., Aboulfotouh, M. H., & Abouelezz, A. M. (2015) · European Journal of Orthodontics · doi:10.1093/ejo/cjv081 · Cited by 37
Evaluation of root resorption following orthodontic intrusion: a systematic review and meta-analysis
Bellini‐Pereira, S. A., de Almeida, J. C., Castillo, A. A.-D., dos Santos, C. C. O., Henriques, J. F. C., & Janson, G. (2020) · European Journal of Orthodontics · doi:10.1093/ejo/cjaa054 · Cited by 37
Biresaw, H., Mulugeta, H., Endalamaw, A., Yesuf, N. N., & Alemu, Y. (2021) · Health Services Insights · doi:10.1177/11786329211040689 · Cited by 37
Agegnehu, C. D., Techane, M. A., Mersha, A. T., & Atalell, K. A. (2022) · AIDS and Behavior · doi:10.1007/s10461-022-03610-y · Cited by 36
Jiang, W., Liang, Y., Long, Z., Hu, M., Han, Y., & Qin, X. (2024) · Journal of Vascular Surgery Venous and Lymphatic Disorders · doi:10.1016/j.jvsv.2024.101842 · Cited by 36
Kowash, M., Alayyan, W., Halabi, M. A., Hussein, I., & Khamis, A. H. (2017) · Journal of International Society of Preventive and Community Dentistry · doi:10.4103/jispcd.jispcd_237_17 · Cited by 36
“10 used for the heterogeneity (Ioannidis, 2008).[14] Visual inspection”
MethodInfluentialKindu, M., Gezie, L. D., Gelaw, B., & Moges, F. (2020) · International Journal of Microbiology · doi:10.1155/2020/9461901 · Cited by 36
Naaktgeboren, C. A., van Enst, W. A., Ochodo, E., de Groot, J. A. H., Hooft, L., Leeflang, M., Bossuyt, P. M., Moons, K. G. M., & Reitsma, J. B. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.05.018 · Cited by 37
Zhou, Z., Liu, H., Ju, H., Chen, H., Jin, H., & Sun, M. (2023) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2023.01.026 · Cited by 35
Magnitude of urban household food insecurity in East Africa: a systematic review and meta-analysis
Gebremichael, B., Abate, B. B., Bimerew, M., Haile, D., Biadgilign, S., & Baye, K. (2021) · Public Health Nutrition · doi:10.1017/s1368980021003529 · Cited by 34
“The I2 statistical value of 0 in this analysis shows homogeneity, while the values of 25, 50 and 75 percent showed medium, moderate and strong heterogeneity, respectively(32,33).”
BackgroundSubgroup effects despite homogeneous heterogeneity test results
Groenwold, R. H. H., Rovers, M. M., Lubsen, J. C., & van der Heijden, G. J. M. G. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-43 · Cited by 34
Papageorgiou, S. N., Xavier, G. M., Cobourne, M. T., & Eliades, T. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.04.017 · Cited by 33
“10 used for the test of heterogeneity and reporting biases, due to low power [28].”
MethodAlayyan, W., Halabi, M. A., Hussein, I., Khamis, A., & Kowash, M. (2018) · The Saudi Dental Journal · doi:10.1016/j.sdentj.2018.05.002 · Cited by 34
2D:4D Suggests a Role of Prenatal Testosterone in Gender Dysphoria
Sadr-Bazzaz, M., Khorashad, B. S., Talaeı, A., Fazeli, N., & Hönekopp, J. (2020) · Archives of Sexual Behavior · doi:10.1007/s10508-020-01630-0 · Cited by 35
“Tests for this kind of publication bias have low statistical power when the number of primary studies is small (Ioannidis, 2008), as was the case here.”
Bidjan, D., Sallmann, R., Eliades, T., & Papageorgiou, S. N. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9123806 · Cited by 37
“between-subgroups heterogeneity where α-value was set as 10% [28].”
MethodJin, H., Yoo, H. J., Kim, Y. A., Lee, J., Lee, Y., Kwon, S., Seo, Y. J., Lee, S. H., Koh, J., Ji, Y., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-07567-9 · Cited by 35
Global COVID-19 vaccine acceptance level and its determinants: an umbrella review
Abate, B. B., Tilahun, B. D., & Yayeh, B. M. (2024) · BMC Public Health · doi:10.1186/s12889-023-17497-4 · Cited by 39
“In this study, the I 2 statistic value of zero indicates true homogeneity, whereas the value 25, 50, and 75% represented low, moderate and high heterogeneity respectively [27, 37].”
BackgroundAccuracy of electronic apex locators in primary teeth: a meta‐analysis
Ahmad, I. A., & Pani, S. C. (2014) · International Endodontic Journal · doi:10.1111/iej.12315 · Cited by 35
Tamir, T. T., Tekeba, B., Mekonen, E. G., Gebrehana, D. A., & Zegeye, A. F. (2025) · Child and Adolescent Psychiatry and Mental Health · doi:10.1186/s13034-025-00879-4 · Cited by 40
“Heterogeneity across the included studies was classified as low, moderate, or high when the values of I-square were 25%, 50-75%, and 75%, respectively [19].”
Ai, R., Wei, J., Ma, D., Jiang, L., Dan, H., Zhou, Y. J., Ji, N., Zeng, X. Y., & Chen, Q. (2017) · Archives of Oral Biology · doi:10.1016/j.archoralbio.2017.04.030 · Cited by 33
“According to Cochran, I2 ranges from 0% to 100%, and cut-offs are used to assess whether heterogeneity is present; 50% is taken as the cut-off for marked heterogeneity, 25% indicates little heterogeneity (Higgins et al., 2003; Ioannidis, 2008), and 0% indicates that there is no heterogeneity.”
Treatment of signs and symptoms of the common cold using EPs 7630 - results of a meta-analysis
Schapowal, A., Dobos, G., Cramer, H., Ong, K. C., Adler, M., Zimmermann, A., Brandes-Schramm, J., & Lehmacher, W. (2019) · Heliyon · doi:10.1016/j.heliyon.2019.e02904 · Cited by 34
“5 supported by the fact that benefits of the herbal preparation were consistently observed in four of the five clinical trials included in the review although the treatment effect sizes in two of these trials were admittedly small to moderate (Ioannidis, 2008).”
BackgroundResultMeta-analysis of small for gestational age births and disinfection byproduct exposures
Summerhayes, R. J., Rahman, B., Morgan, G. G., Beresin, G., Moreno, C., & Wright, J. M. (2020) · Environmental Research · doi:10.1016/j.envres.2020.110280 · Cited by 34
Implications of the open science era for educational psychology research syntheses
Patall, E. A. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1897009 · Cited by 37
“Second, meta-analyses that rely on a small number of studies may be particularly likely to detect a false positive effect, as indicated by research suggesting an inverse relationship between the number of studies in a research synthesis and the pooled effect size (Ioannidis, 2008c).”
BackgroundZhang, Q., Wu, R., Zhu, S., Le, J., Chen, Y., Lan, C., Yao, S., Zhao, W., & Kendrick, K. M. (2021) · Autism Research · doi:10.1002/aur.2565 · Cited by 40
Survival outcomes for radiotherapy treatment of epidermoid tumors with malignant transformation
Nagasawa, D. T., Yew, A. Y., Spasić, M., Choy, W., Gopen, Q. S., & Yang, I. (2011) · Journal of Clinical Neuroscience · doi:10.1016/j.jocn.2011.06.002 · Cited by 33
Nocebo in Alzheimer's disease; meta-analysis of placebo-controlled clinical trials
Zis, P., & Mitsikostas, D. D. (2015) · Journal of the Neurological Sciences · doi:10.1016/j.jns.2015.05.029 · Cited by 32
“Data were analyzed using a random effects model, because of heterogeneity between studies, which was assessed by I and Cochran Q tests [20–22].”
MethodKaklamanos, E. G., Lazaridou, D. C., Tsiantou, D., Kotsanos, N., & Athanasiou, A. E. (2016) · Odontology · doi:10.1007/s10266-016-0281-2 · Cited by 32
“10 used for the heterogeneity tests [29].”
Cheng, Y.-X., Tao, W. Y., Zhang, H., Peng, D., & Wei, Z. (2021) · World Journal of Surgical Oncology · doi:10.1186/s12957-021-02267-6 · Cited by 32
Sitole, M., Silva, M. J., Spooner, L. M., Comee, M., & Malloy, M. (2013) · Clinical Therapeutics · doi:10.1016/j.clinthera.2012.12.017 · Cited by 31
Pabalan, N., Kunjantarachot, A., Ruangpratheep, C., Jarjanazi, H., Christofolini, D. M., Barbosa, C. P., & Bianco, B. A. V. (2017) · Gynecologic Oncology · doi:10.1016/j.ygyno.2017.06.017 · Cited by 31
Bennett, G., King, N., Redfern, K. M., & Breese, B. C. (2022) · The Journal of Maternal-Fetal & Neonatal Medicine · doi:10.1080/14767058.2022.2155043 · Cited by 33
“It scores heterogeneity between 0% and 100%, with 25%, 50%, and 75% representing low, moderate and high heterogeneity, respectively [17].”
Tusa, B. S., Weldesenbet, A. B., Tola, A., Merga, B. T., & Regassa, L. D. (2021) · Health and Quality of Life Outcomes · doi:10.1186/s12955-020-01655-y · Cited by 30
“Heterogeneity of studies was observed using forest plot and quantified using the I-squared statistic, in which 25, 50, and 75% represented low, moderate and high heterogeneity respectively [27, 28].”
MethodWilliams, A. O., Makinde, O., & Ojo, M. (2016) · Global Health Research and Policy · doi:10.1186/s41256-016-0010-y · Cited by 23
“The number of covariates was limited to avoid the problem of multiplicity and false-positive results [26].”
BackgroundMethodTorre, G. L. (2016) · Mediterranean Journal of Hematology and Infectious Diseases · doi:10.4084/mjhid.2016.044 · Cited by 29
Dawood, Z. S., Alaimo, L., Lima, H. A., Moazzam, Z., Shaikh, C. F., Ahmed, A. S., Munir, M. M., Endo, Y., & Pawlik, T. M. (2022) · Annals of Surgical Oncology · doi:10.1245/s10434-022-12641-7 · Cited by 29
“Heterogeneity may introduce bias; however, its major role is to enable meta-analysts to identify which results may be combined for analytic purposes.(62) Owing to the high heterogeneity in the current study, the HSROC model was used to obtain the pooled sensitivity, specificity, and DOR, which uses random effects to p…”
MethodBrown, S. D., Furrow, D. R., Hill, D. F., Gable, J. C., Porter, L. P., & Jacobs, W. J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614551749 · Cited by 31
“…claimed that journal editors selectively publish positive findings and discriminate against study replications, permitting errors in data and theory to enjoy a long half-life (see also Ferguson & Brannick, 2012; Ioannidis, 2008, 2012; Shadish, Doherty, & Montgomery, 1989; Stroebe & Strack, 2014).”
BackgroundRezaei, Y., Fesharaki, M. G., Dehghani, M. R., Arya, A., Haghjoo, M., & Arjmand, N. (2015) · Journal of Cardiovascular Pharmacology and Therapeutics · doi:10.1177/1074248415602557 · Cited by 29
“Furthermore, statistical heterogeneity is a correlate of clinical heterogeneity, and this relationship may sometimes be weak.(37) Third, we did not make an analysis regarding the duration of statin therapy.”
BackgroundChelkeba, L., Melaku, T., & Mega, T. A. (2021) · Infection and Drug Resistance · doi:10.2147/idr.s289687 · Cited by 30
Paredes, A. C., Teixeira, P., Almeida, A., & Pinto, P. R. (2021) · Journal of Pain · doi:10.1016/j.jpain.2021.03.157 · Cited by 28
Walji, L. T., Murchie, P., Lip, G., Speirs, V., & Iversen, L. (2021) · Cancer Epidemiology · doi:10.1016/j.canep.2021.101995 · Cited by 28
Musleh, R., Schlattmann, P., Caldonazo, T. M. M., Kirov, H., Witte, O. W., Doenst, T., Günther, A., & Diab, M. (2022) · Journal of the American Heart Association · doi:10.1161/jaha.121.024401 · Cited by 30
Getawa, S., Bayleyegn, B., Aynalem, M., Worku, Y. B., & Adane, T. (2022) · Journal of International Medical Research · doi:10.1177/03000605221129547 · Cited by 28
“05).(18,19) Sub-group and sensitivity analyses were not performed because of the absence of significant heterogeneity.”
BackgroundHussain, U., Alam, S., Rehman, K., Antonoglou, G. N., & Papageorgiou, S. N. (2023) · European Journal of Orthodontics · doi:10.1093/ejo/cjac044 · Cited by 30
Havakeshian, G., Koretsi, V., Eliades, T., & Papageorgiou, S. N. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9093015 · Cited by 29
“All p values were two-sided with α = 5%, except for the test of between-studies or between-subgroups heterogeneity, where α-value was set at 10% [27].”
Degarege, A., Gebrezgi, M. T., Beck‐Sagué, C. M., Wahlgren, M., de Mattos, L. C., & Madhivanan, P. (2019) · BMC Infectious Diseases · doi:10.1186/s12879-019-3730-z · Cited by 27
Ren, S., Li, Y., Pu, L., & Feng, Y. (2023) · Worldviews on Evidence-Based Nursing · doi:10.1111/wvn.12663 · Cited by 26
Peng, J., Wang, J., Shu, Q., Luo, Y., Wang, S., & Liu, Z. (2024) · Scientific Reports · doi:10.1038/s41598-024-69754-0 · Cited by 26
“Interstudy heterogeneity was assessed using the Cochran Q statistic ( χ 2 test) 12 , along with the I2 statistic as a measure describing degrees of heterogeneity 13 .”
MethodWang, J., Jin, T., Chen, B., Pan, Y., & Shao, C. (2022) · Journal of Vascular Surgery · doi:10.1016/j.jvs.2022.03.876 · Cited by 26
Epidemiology of child psychopathology: major milestones
Verhulst, F. C., & Tiemeier, H. W. (2015) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-015-0681-9 · Cited by 27
Li, L., Zhai, M., Wang, M., Cui, S., Chen, C., Wang, J., & Wei, F. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12051790 · Cited by 28
“10 was used for heterogeneity testing and publication bias [28].”
MethodBae, J. (2014) · Epidemiology and Health · doi:10.4178/epih/e2014019 · Cited by 25
Cao, D.-H., Gu, W.-B., Zhao, H.-Y., Hu, J.-L., & Yuan, H. (2024) · Archives of Osteoporosis · doi:10.1007/s11657-024-01400-8 · Cited by 25
Sobre las revisiones sistemáticas y narrativas de la literatura en Medicina
F, M. S. (2020) · Revista chilena de enfermedades respiratorias · doi:10.4067/s0717-73482020000100026 · Cited by 24
The safety of remimazolam versus propofol in gastroscopic sedation: a meta-analysis
An, X., Shen, T., Yin, X., Jin, X., Zhang, Y., & Wang, T. (2024) · BMC Anesthesiology · doi:10.1186/s12871-024-02422-y · Cited by 24
Publication Bias & Small-study Effects in Pediatric Dentistry Meta-analyses
Papageorgiou, S. N., Dimitraki, D., Coolidge, T., & Kotsanos, N. (2014) · Journal of Evidence Based Dental Practice · doi:10.1016/j.jebdp.2014.09.001 · Cited by 25
Li, M.-L., Chen, Z., Liu, Z.-Z., Zhang, N., Liu, J., Wang, H.-R., Wang, W., Liang, Y., Chen, J.-W., Liu, Z., et al. (2019) · BMC Complementary and Alternative Medicine · doi:10.1186/s12906-019-2441-8 · Cited by 22
“We investigated heterogeneity between trials using the I(2) statistic [40, 41] and by visual inspection of the forest plots.”
MethodWang, Y., Jiang, T., Jiang, M., & Shuijing, G. (2019) · BMC Cancer · doi:10.1186/s12885-019-5385-y · Cited by 23
Sathornviriyapong, S., Matsuda, A., Miyashita, M., Matsumoto, S., Sakurazawa, N., Kawano, Y., Yamada, M., & Uchida, E. (2016) · Annals of Surgical Oncology · doi:10.1245/s10434-016-5298-9 · Cited by 23
Abate, B. B., Sendekie, A. K., Alamaw, A. W., Tegegne, K. M., Kitaw, T. A., Azmeraw, M., Kassaw, A., Yilak, G., Zemariam, A. B., & Tilahun, B. D. (2025) · AJOG Global Reports · doi:10.1016/j.xagr.2025.100441 · Cited by 22
Percutaneous Vascular Interventions Versus Bypass Surgeries in Patients With Critical Limb Ischemia
Wang, J., Shu, C., Wu, Z., Zhao, J., Ma, Y., Huang, B., Yuan, D., Yang, Y., Bian, H., He, Y., et al. (2017) · Annals of Surgery · doi:10.1097/sla.0000000000002344 · Cited by 22
The impact of music training on executive functions in childhood—a systematic review
Degé, F., & Frischen, U. (2022) · Zeitschrift für Erziehungswissenschaft · doi:10.1007/s11618-022-01102-2 · Cited by 22
“However, unfortunately there is no single statistical test that can guarantee a bias free systematic review or document an existing bias with certainty (Ioannidis 2008; Peters et al. 2010).”
BackgroundWang, S., Wang, M., Leung, I. S. H., Song, G., Xu, X., Chen, S., & Yin, Y. (2023) · International Journal of Nursing Practice · doi:10.1111/ijn.13181 · Cited by 22
Phase 4 Studies in Heart Failure - What is Done and What is Needed?
Iyngkaran, P., Liew, D., McDonald, P., Thomas, M. C., Reid, C. M., Chew, D. P., & Hare, D. L. (2016) · Current Cardiology Reviews · doi:10.2174/1573403x12666160606121458 · Cited by 21
“systematic reviews and meta-analyses in clinical practice has however varied [149-162].”
BackgroundAlyammahi, A. S., Kaklamanos, E. G., & Athanasiou, A. E. (2017) · European Journal of Orthodontics · doi:10.1093/ejo/cjx042 · Cited by 22
Fact or fiction: reducing the proportion and impact of false positives
Stahl, D. R., & Pickles, A. R. (2017) · Psychological Medicine · doi:10.1017/s003329171700294x · Cited by 23
Techane, M. A., Alemu, T. G., Wubneh, C. A., Belay, G. M., Tamir, T. T., Muhye, A. B., Kassie, D. G., Wondim, A., Terefe, B., Tarekegn, B. T., et al. (2022) · The Italian Journal of Pediatrics/Italian journal of pediatrics · doi:10.1186/s13052-022-01307-5 · Cited by 21
Cleach, L. L., Doney, L., Katz, K. A., Williams, H. C., & Trinquart, L. (2016) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2016.09.019 · Cited by 21
“But, their use is inappropriate in most meta-analyses because of too few trials or excessive heterogeneity (Ioannidis, 2008).”
BackgroundZhu, X., Dai, W.-Z., & Ma, T. (2018) · Annals of Translational Medicine · doi:10.21037/atm.2018.06.43 · Cited by 21
Bedaso, A., Mekonnen, N., & Duko, B. (2021) · Aging & Mental Health · doi:10.1080/13607863.2021.1932740 · Cited by 26
“Possible publication bias was examined by inspection of the visual inspection of a funnel and conducting egger’s regression tests (egger et al., 1997; ioannidis, 2008).”
MethodDawood, Z. S., Khalil, M., Waqar, U., Banani, I., Alidina, Z., & Pawlik, T. M. (2025) · Journal of Gastrointestinal Surgery · doi:10.1016/j.gassur.2025.102005 · Cited by 20
Meta-analysis of Diffusion-Weighted Magnetic Resonance Imaging in Detecting Prostate Cancer
Jin, G., Su, D. K., Luo, N., Liu, L. D., Zhu, X., & Huang, X. Y. (2013) · Journal of Computer Assisted Tomography · doi:10.1097/rct.0b013e3182801ae1 · Cited by 20
“The overall performance of diagnostic studies can be visualized and reflected by an sROC curve without being affected by a change in threshold values.(37,38) The best diagnostic modality would yield a point in the upper left corner or coordinate (0,1) of the sROC space, representing 100% sensitivity (no falsenegative…”
BackgroundIs red blood cell distribution width a prognostic factor for colorectal cancer? A meta-analysis
Wen, Z., Zhou, X., & Xiao, D. (2022) · Frontiers in Surgery · doi:10.3389/fsurg.2022.945126 · Cited by 20
“The I(2) value and the results of the chi-squared test were used to assess the statistical heterogeneity (29, 30).”
MethodLi, H., Wang, Y., Xi, H., Zhang, J., Zhao, M., & Jia, X. (2023) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2023.07.036 · Cited by 22
Ortiz, H. A. D. L. B., Arias, M., & Liebano, R. E. (2024) · Lasers in Medical Science · doi:10.1007/s10103-024-04069-0 · Cited by 20
“[26].”
Pasha, S., & Baldwin, K. D. (2018) · Spine · doi:10.1097/brs.0000000000002748 · Cited by 20
Performance of Hawley-type retainers: a systematic review of randomized clinical trials
Rahma, W. J. A., Kaklamanos, E. G., & Athanasiou, A. E. (2017) · European Journal of Orthodontics · doi:10.1093/ejo/cjx036 · Cited by 19
Sladekova, M., Webb, L. E. A., & Field, A. P. (2022) · Psychological Methods · doi:10.1037/met0000470 · Cited by 18
“…addressing publication bias, as funnel plot asymmetry can also be the result of missing moderators and heterogeneity among the effects, especially when the primary effect sizes are correlated with their standard errors (Ioannidis, 2008; Lau et al., 2006; Peters et al., 2010; Sterne et al., 2011).”
BackgroundMarcó-García, S., Ariyo, K., Owen, G. S., & David, A. S. (2024) · Psychological Medicine · doi:10.1017/s0033291724000242 · Cited by 18
“These limitations are common in meta-analyses of cross-sectional studies (Ioannidis, 2008).”
Effects of Temperature Fluctuations on the Growth Cycle of Rice
Liu, Z., Xu, D., Wang, R., Guo, X., Song, Y., Wang, M., & Cai, Y. (2025) · Agriculture · doi:10.3390/agriculture15010099 · Cited by 18
“The initial step in the analysis involved assessing whether temperature changes significantly affected rice growth and development [17].”
de Sena, I. G., da Costa, A. V., dos Santos, Í. K., de Araújo, D. P., Gomes, F. T. D. S., Cavalcanti, J. R. L. D. P., Knackfuss, M. I., de Andrade, M. F., Melo, P. K. M., & Fonseca, I. A. T. (2023) · PLoS ONE · doi:10.1371/journal.pone.0293357 · Cited by 18
“The heterogeneity magnitude was assessed by calculating the I 2 , which ranges from 0% to 100%, indicating substantial heterogeneity when greater than 50% and considerable heterogeneity when greater than 75% [33].”
Sheikh, M., & Ammar, M. (2024) · Frontiers in Neuroscience · doi:10.3389/fnins.2024.1398952 · Cited by 23
“This allowed SPSS to identify heterogeneity between studies using the Q test, p -value, and I 2 , which shows the proportion of overall variance caused by heterogeneity across trials (Higgins et al., 2003; Ioannidis, 2008; DerSimonian and Laird, 2015).”
MethodWeigang, T., Xu, W., Lifeng, G., Lu, J., Li, Y., Huaqin, J., & Li, H. (2021) · International Urology and Nephrology · doi:10.1007/s11255-020-02691-9 · Cited by 18
Is red cell distribution width a prognostic factor in patients with breast cancer? A meta-analysis
Yin, J.-M., Zhu, K.-P., Guo, Z., Wen, Y., He, Y., & Du, G.-C. (2023) · Frontiers in Surgery · doi:10.3389/fsurg.2023.1000522 · Cited by 17
“The I(2) value and the chi-squared test were used to assess the statistical heterogeneity (28, 29).”
MethodLai, S., Hwang, B., Kuo, Y., Liu, C.-S., & Liao, K. (2023) · Frontiers in Endocrinology · doi:10.3389/fendo.2023.1158153 · Cited by 17
Cardioembolic Stroke Risk and Recovery After Anticoagulation-Related Intracerebral Hemorrhage
Murphy, M. P., Kuramatsu, J. B., Leasure, A. C., Falcone, G. J., Kamel, H., Sansing, L. H., Kourkoulis, C. E., Schwab, K. M., Elm, J., Gurol, M. E., et al. (2018) · Stroke · doi:10.1161/strokeaha.118.021799 · Cited by 17
“Results from multivariable analyses from each individual study were then metaanalyzed using a conservative inverse variance-based random effects method (DerSimonian-Laird).21 Cochran Q test was used to estimate heterogeneity, followed by calculation of I2.”
InfluentialEdessa, D., Sisay, M., & Dessie, Y. (2020) · PLoS ONE · doi:10.1371/journal.pone.0237534 · Cited by 17
“01) even though the studies included were> 10 in number and had more than one studies with significant results [42, 43].”
ResultAssociations between XPD polymorphisms and risk of breast cancer: a meta-analysis
Jiang, Z., Li, C., Xu, Y., Cai, S., & Wang, X. (2010) · Breast Cancer Research and Treatment · doi:10.1007/s10549-010-0751-0 · Cited by 17
“Because publication bias may cause heterogeneity in the results [49], both Begg’s funnel”
ResultSalmen, M. R., Ewy, G. A., & Sasson, C. (2012) · BMJ Open · doi:10.1136/bmjopen-2012-001273 · Cited by 16
“0004), so a pooled OR for survival was not calculated (figure 5).(30) Subgroup analysis of the patients with a presenting rhythm of witnessed VF/VT showed a trend towards improved survival for the AHA/ERC 2005 Guidelines as compared to the AHA/ERC 2000 Guidelines.”
BackgroundResultChinzowu, T., Roy, S., & Nishtala, P. (2021) · BMC Geriatrics · doi:10.1186/s12877-021-02512-3 · Cited by 12
“Several authors, for example, Ioannidis [53] and Huedo-Medina et al.”
BackgroundPabalan, N., Singian, E., Jarjanazi, H., & Steiner, T. S. (2012) · European Journal of Clinical Microbiology & Infectious Diseases · doi:10.1007/s10096-012-1779-3 · Cited by 16
“Heterogeneity refers to diversity which may influence the manner in which the data are treated [32].”
BackgroundLin, X., Tao, H., Cai, M., Liao, A., Cheng, Z., & Lin, H. (2016) · Medicine · doi:10.1097/md.0000000000002687 · Cited by 16
“Heterogeneity was quantified using the Q-statistic and I(2) test.(15,16) We conducted a sensitivity analysis to explore possible explanations for heterogeneity and to assess the impact of various subgroups.”
MethodGuo, X., Jiao, D., Zhu, J., Xiao, H. F., Zhao, X., Yang, Y., Zhao, Y., & Liu, Z. (2021) · Gland Surgery · doi:10.21037/gs-21-186 · Cited by 16
Observed and unobserved heterogeneity in failure data analysis
Zaki, R., Barabadi, A., Barabady, J., & Qarahasanlou, A. N. (2021) · Proceedings of the Institution of Mechanical Engineers Part O Journal of Risk and Reliability · doi:10.1177/1748006x211022538 · Cited by 16
“Each test is the optimum one for detecting the heterogeneity of a specific form.(54,55) For example, a shortcoming of the Q statistic is that it has poor power to detect true heterogeneity among studies when the meta-analysis includes a small number of studies and excessive power to detect negligible variability with…”
BackgroundKaklamanos, E. G., & Kolokitha, O.-E. (2016) · Journal of Cranio-Maxillofacial Surgery · doi:10.1016/j.jcms.2016.01.005 · Cited by 16
“10 used for the heterogeneity tests (Ioannidis, 2008).”
Workineh, Y., Kasa, A. S., Ayalew, E., & Temesgen, W. A. (2019) · Advances in Preventive Medicine · doi:10.1155/2019/3602585 · Cited by 18
Assessing the incidence of gastroenteritis among elderly people living in long term care facilities
Kirk, M., HALL, G., Veitch, M., & Becker, N. G. (2010) · Journal of Hospital Infection · doi:10.1016/j.jhin.2010.04.009 · Cited by 16
Mirzania, M., Shakibazadeh, E., Maleki, A., Noorafrooz, M., Karimi, N., Mayra, K., Dashtkoohi, M., & Hantoushzadeh, S. (2025) · Reproductive Health · doi:10.1186/s12978-025-02119-6 · Cited by 16
“The I 2 values of 25%, 50%, and 75% were considered as low, moderate, and high heterogeneity, respectively [28, 29].”
Wang, X., Ou, H., Wu, Y., & Xing, Z. (2023) · Translational Pediatrics · doi:10.21037/tp-23-134 · Cited by 14
“The I 2 value and the chi-squared test were used to evaluate the statistical heterogeneity (41,42); the I²<30% was considered non-important, 30–60% was considered moderate, and >60% was considered substantial.”
Li, Z., Shu, X., Wen, Z., Liu, F., Liu, X., Lv, Q., Liu, X., Zhang, W., & Peng, D. (2023) · Biomedical Reports · doi:10.3892/br.2023.1710 · Cited by 15
“The fixed‑effects model was used in this study if I 2 ≤50% and P 0.05 was considered statistically significant (16).”
MethodInfluential
Metadata
- Authors
- Ioannidis, J. P. A.
- Affiliations
- University of Ioannina (GR)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- Journal of Evaluation in Clinical Practice
- Publisher
- Wiley
- Published
- 2008-10
- Volume
- 14
- Issue
- 5
- Pages
- 951-957
- ISSN
- 1356-1294 (ISSN-L)
- 1365-2753
- OpenAlex
- W2144112749
- Cited by
- 696
- References
- 57
- Citations by year
- 2026: 52 · 2025: 69 · 2024: 98 · 2023: 72 · 2022: 50 · 2021: 64 · 2020: 49 · 2019: 34 · 2018: 30 · 2017: 23 · 2016: 23 · 2015: 26 · 2014: 27 · 2013: 22 · 2012: 15
- FWCI
- 4.52
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Evaluation in Clinical Practice (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- heterogeneity testing, publication bias, meta-analysis, statistical heterogeneity, clinical heterogeneity, false positive signals, bias detection
- MeSH terms
- Data Interpretation, Statistical, Humans, Regression Analysis, Research Design, Risk, Sensitivity and Specificity, Meta-Analysis as Topic, Reproducibility of Results, Bias, Confidence Intervals, Publication Bias, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
