The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
A comparison of heterogeneity variance estimators in simulated random‐effects meta‐analyses
Open accessAbstract
Studies combined in a meta‐analysis often have differences in their design and conduct that can lead to heterogeneous results. A random‐effects model accounts for these differences in the underlying study effects, which includes a heterogeneity variance parameter. The DerSimonian‐Laird method is often used to estimate the heterogeneity variance, but simulation studies have found the method can be biased and other methods are available. This paper compares the properties of nine different heterogeneity variance estimators using simulated meta‐analysis data. Simulated scenarios include studies of equal size and of moderate and large differences in size. Results confirm that the DerSimonian‐Laird estimator is negatively biased in scenarios with small studies and in scenarios with a rare binary outcome. Results also show the Paule‐Mandel method has considerable positive bias in meta‐analyses with large differences in study size. We recommend the method of restricted maximum likelihood (REML) to estimate the heterogeneity variance over other methods. However, considering that meta‐analyses of health studies typically contain few studies, the heterogeneity variance estimate should not be used as a reliable gauge for the extent of heterogeneity in a meta‐analysis. The estimated summary effect of the meta‐analysis and its confidence interval derived from the Hartung‐Knapp‐Sidik‐Jonkman method are more robust to changes in the heterogeneity variance estimate and show minimal deviation from the nominal coverage of 95% under most of our simulated scenarios.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Meta-analysis was conducted using a random-effect model with restricted maximum likelihood, as the estimator of true heterogeneity, which seems to provide us with less biased estimates than other methods.33”
From Psychological interventions for depression and anxiety: a systematic review and meta-analysis of Iranian chronic pain trials. (2020) · cites this paper as [33]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For random-effects models, many methods are available, and their performance has been shown to differ depending on the characteristics of the meta-analysis (such as the number and size of the included studies113 114).”
From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [114]
The cited source says what the claim attributes to it.
In the citing paper
“a random-effects model typically cannot adequately estimate heterogeneity with only two effects (Langan et al., 2019)”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Langan et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“This method assumes a t-distribution of the pooled effect rather than a normal distribution, which reduces the chance of obtaining false-positive results (Langan et al., 2019).”
From The state of hiring discrimination: A meta-analysis of (almost) all recent correspondence experiments (2023) · cites this paper as Langan et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Self-citationConducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
Self-citationBorenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Self-citationBorenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
Self-citationHunter, J., & Schmidt, F. (2004) · doi:10.4135/9781412985031 · Cited by 3,023
Maximum Likelihood Approaches to Variance Component Estimation and to Related Problems
Harville, D. A. (1977) · Journal of the American Statistical Association · doi:10.1080/01621459.1977.10480998 · Cited by 2,667
IntHout, J., Ioannidis, J. P., & Borm, G. F. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-25 · Cited by 2,419
Random-effects model for meta-analysis of clinical trials: An update
DerSimonian, R., & Kacker, R. (2007) · Contemporary Clinical Trials · doi:10.1016/j.cct.2006.04.004 · Cited by 2,374
Improved tests for a random effects meta‐regression with a single covariate
Knapp, G., & Hartung, J. (2003) · Statistics in Medicine · doi:10.1002/sim.1482 · Cited by 2,075
Flagged first, then by citations
Cited by
Sorted by citations
Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262
Analysing data and undertaking meta‐analyses
Deeks, J. J., Higgins, J. P., Altman, D. G., & on behalf of the Cochrane Statistical Methods Group (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch10 · Cited by 2,672
Self-citationAnalysing Data and Undertaking Meta‐Analyses
Deeks, J. J., Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch9 · Cited by 1,800
Self-citationPage, M. J., Moher, D., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/gwdhk · Cited by 916
Estimating the sample mean and standard deviation from commonly reported quantiles in meta-analysis
McGrath, S., Zhao, X., Steele, R. J., Thombs, B. D., Benedetti, A., Levis, B., Riehm, K. E., Saadat, N., Levis, A. W., Azar, M., et al. (2020) · Statistical Methods in Medical Research · doi:10.1177/0962280219889080 · Cited by 869
“The restricted maximum likelihood method was used to estimate heterogeneity in all meta-analyses.(30) The QE and BC methods were less biased for estimating the pooled mean compared to the 2528 Statistical Methods in Medical Research 29(9)”
MethodThe WHO Rapid Evidence Appraisal for COVID-19 Therapies (REACT) Working Group, Domingo, P., Mur, I., Mateo, G. M., Gutierrez, M. D. M., Pomar, V., de Benito, N., Corbacho, N., Herrera, S., Millan, L., et al. (2021) · JAMA · doi:10.1001/jama.2021.11330 · Cited by 678
Self-citationLiu, W., Zhang, J., Liu, H., Guo, X., Zhang, X., Yao, X., Cao, Z., & Zhang, T. (2021) · Environment International · doi:10.1016/j.envint.2020.106277 · Cited by 688
Unger, J. M., Vaidya, R., Hershman, D. L., Minasian, L. M., & Fleury, M. E. (2018) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djy221 · Cited by 604
“Given its advantageous properties (both efficient and approximately unbiased), we used the restricted maximum-likelihood estimator (35,36).”
MethodYao, H., Zhang, A., Li, D., Wu, Y., Wang, C.-Z., Wan, J.-Y., & Yuan, C.-S. (2024) · BMJ · doi:10.1136/bmj-2023-076410 · Cited by 558
Neuenschwander, M., Ballon, A., Weber, K. S., Norat, T., Aune, D., Schwingshackl, L., & Schlesinger, S. (2019) · BMJ · doi:10.1136/bmj.l2368 · Cited by 531
Su, W., Han, X., Yu, H., Wu, Y., & Potenza, M. N. (2020) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106480 · Cited by 476
Ludyga, S., Gerber, M., Pühse, U., Looser, V. N., & Kamijo, K. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0851-8 · Cited by 522
Prevention of Type 2 Diabetes by Lifestyle Changes: A Systematic Review and Meta-Analysis
Uusitupa, M. I. J., Khan, T. A., Viguiliouk, E., Kahleová, H., Rivellese, A. A., Hermansen, K., Pfeiffer, A. F. H., Thanopoulou, A. C., Salas‐Salvadó, J., Schwab, U., et al. (2019) · Nutrients · doi:10.3390/nu11112611 · Cited by 531
“Data were expressed as risk ratios (RRs) with 95% confidence intervals (CIs) and pooled using the restricted maximum likelihood (REML) random-effects models [29].”
MethodMunshi, N. C., Avet-Loiseau, H., Anderson, K. C., Neri, P., Paiva, B., Samur, M., Dimopoulos, M., Kulakova, M., Lam, A., Hashim, M., et al. (2020) · Blood Advances · doi:10.1182/bloodadvances.2020002827 · Cited by 431
“Heterogeneity in design and population among the studies eligible for meta-analysis was assessed with multiple estimators.(16) Differences within subgroups were tested for statistical significance with the (2-sided) log-rank test, and HRs with 2-sided 95% CIs were estimated with a Cox regression model.”
MethodInvestigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
“Also, a random- effects model typically cannot adequately estimate heterogeneity with only two effects (Langan et al., 2019).”
BackgroundMeta-analysis of proportions using generalized linear mixed models.
Lin, L., & Chu, H. (2020) · Epidemiology · doi:10.1097/ede.0000000000001232 · Cited by 345
Morze, J., Wittenbecher, C., Schwingshackl, L., Danielewicz, A., Rynkiewicz, A., Hu, F. B., & Guasch‐Ferré, M. (2022) · Diabetes Care · doi:10.2337/dc21-1705 · Cited by 316
Evaluation of various estimators for standardized mean difference in meta‐analysis
Lin, L., & Aloe, A. M. (2021) · Statistics in Medicine · doi:10.1002/sim.8781 · Cited by 297
Tsang, N. N. Y., So, H. C., Ng, K. Y., Cowling, B. J., Leung, G. M., & Ip, D. (2021) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(21)00146-8 · Cited by 289
Hickin, N., Käll, A., Shafran, R., Sutcliffe, S., Manzotti, G., & Langan, D. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2021.102066 · Cited by 295
Self-citationUnger, J. M., Hershman, D. L., Till, C., Minasian, L. M., Osarogiagbon, R. U., Fleury, M. E., & Vaidya, R. (2020) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djaa155 · Cited by 270
“A restricted maximumlikelihood estimator of the between-study variance was used (23,24).”
MethodVeroniki, A. A., Jackson, D., Bender, R., Kuss, O., Langan, D., Higgins, J. P. T., Knapp, G., & Salanti, G. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1319 · Cited by 236
“It performs well even in meta‐analyses with fewer than 10 studies(28) and is robust to the use of different estimators for the between‐study variance and to changes in the magnitude of the between‐study variance.(45,113) However, it should be considered that HKSJ is not always conservative compared with a fixed‐effect…”
BackgroundMethodInfluentialSelf-citationPapageorgiou, S. N., Koletsi, D., Iliadi, A., Peltomäki, T., & Eliades, T. (2019) · European Journal of Orthodontics · doi:10.1093/ejo/cjz094 · Cited by 243
Tan, B. K. J., Han, R., Zhao, J. J., Tan, N. K. W., Quah, E. S. H., Tan, C. J., Chan, Y. H., Teo, N. W. Y., Charn, T. C., See, A., et al. (2022) · BMJ · doi:10.1136/bmj-2021-069503 · Cited by 241
“29 The pooled proportion, and corresponding 95% confidence intervals and 95% prediction intervals, of patients with persistent smell or taste dysfunction were then retrieved after back transformation.”
Lehrer, E. J., Singh, R. B., Wang, M., Chinchilli, V. M., Trifiletti, D. M., Ost, P., Siva, S., Meng, M.-B., Tchelebi, L. T., & Zaorsky, N. G. (2020) · JAMA Oncology · doi:10.1001/jamaoncol.2020.6146 · Cited by 198
Baccolini, V., Rosso, A., Paolo, C. D., Isonne, C., Salerno, C., Migliara, G., Prencipe, G. P., Massimi, A., Marzuillo, C., Vito, C. D., et al. (2021) · Journal of General Internal Medicine · doi:10.1007/s11606-020-06407-8 · Cited by 219
Comparison of four heterogeneity measures for meta‐analysis
Lin, L. (2019) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.13159 · Cited by 186
“Alternative estimates, such as those based on the restricted maximum likelihood or Bayesian analysis, may be better options to be used in the heterogeneity measures.(43,44) Third, as in conventional meta‐analysis methods, the within‐study variances were treated as true values in our analysis and the four heterogeneity…”
MethodKoutoukidis, D. A., Koshiaris, C., Henry, J. A., Noreik, M., Morris, E., Manoharan, I., Tudor, K., Bodenham, E., Dunnigan, A., Jebb, S. A., et al. (2021) · Metabolism · doi:10.1016/j.metabol.2020.154455 · Cited by 186
How work engagement relates to performance and absenteeism: a meta-analysis
Neuber, L., Englitz, C., Schulte, N., Forthmann, B., & Holling, H. (2021) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2021.1953989 · Cited by 262
Ho, I., Azcoaga-Lorenzo, A., Akbari, A., Davies, J., Hodgins, P., Khunti, K., Kadam, U. T., Lyons, R. A., McCowan, C., Mercer, S. W., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-057017 · Cited by 193
Li, J., Luo, J. F., Pavlov, I., Perez, Y., Tan, W., Roca, O., Tavernier, E., Kharat, A., McNicholas, B., Ibarra-Estrada, M. Á., et al. (2022) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(22)00043-1 · Cited by 167
Baranyi, G., Fazel, S., Delhey, S., & Mundt, A. P. (2022) · The Lancet Public Health · doi:10.1016/s2468-2667(22)00093-7 · Cited by 174
Perspectives of Patients About Immediate Access to Test Results Through an Online Patient Portal
Steitz, B. D., Turer, R. W., Lin, C., MacDonald, S. T., Salmi, L., Wright, A. T., Lehmann, C. U., Langford, K., McDonald, S., Reese, T. J., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.3572 · Cited by 153
Internalization of body shape ideals and body dissatisfaction: A systematic review and meta‐analysis
Paterna, A., Alcaraz‐Ibáñez, M., Fuller‐Tyszkiewicz, M., & Sicilia, Á. (2021) · International Journal of Eating Disorders · doi:10.1002/eat.23568 · Cited by 178
“Random-effects models were estimated using the restricted maximum likelihood (REML) estimation method, which is robust even in the absence of normal data distributions (Langan et al., 2019).”
MethodInterventions for the management of long covid (post-covid condition): living systematic review
Zeraatkar, D., Ling, M., Kirsh, S., Jassal, T., Shahab, M., Movahed, H., Talukdar, J. R., Walch, A., Chakraborty, S. P., Turner, T., et al. (2024) · BMJ · doi:10.1136/bmj-2024-081318 · Cited by 149
“67 68 Overall, the diversity in interventions and outcome measures precluded meta-analyses.”
A meta-regression of the impact of policy on the efficacy of individual placement and support
Brinchmann, B., Widding‐Havneraas, T., Modini, M., Rinaldi, M., Moe, C. F., McDaid, D., Park, A., Killackey, E., Harvey, S. B., & Mykletun, A. (2020) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.13129 · Cited by 134
“the estimated variance of true effects) (39) Heterogeneity between studies in the meta-analysis was assessed with Cochran’s Q statistic and the I(2) statistic.”
MethodCurrent practices in meta‐regression in psychology, education, and medicine
Tipton, E., Pustejovsky, J. E., & Ahmadi, H. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1339 · Cited by 133
Leadership in the public sector: A meta‐analysis of styles, outcomes, contexts, and methods
Backhaus, L., & Vogel, R. (2022) · Public Administration Review · doi:10.1111/puar.13516 · Cited by 146
Tanriver-Ayder, E., Faes, C., de Casteele, T. V., McCann, S., & Macleod, M. R. (2021) · BMJ Open Science · doi:10.1136/bmjos-2020-100074 · Cited by 108
Hartman, C. A., Larsson, H., Vos, M., Bellato, A., Libutzki, B., Solberg, B. S., Chen, Q., Du Rietz, E., Mostert, J. C., Kittel-Schneider, S., et al. (2023) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105209 · Cited by 104
“The Restricted Maximum-Likelihood (REML) estimator is used with Knapp-Hartung confidence interval adjustment (Langan et al., 2019).”
de Jong, V. M. T., Moons, K. G. M., Riley, R. D., Smith, C. T., Marson, A. G., Eijkemans, M. J. C., & Debray, T. P. A. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1384 · Cited by 104
Zhang, L., Yao, H., Li, L., Rietz, E. D., Andell, P., Garcia‐Argibay, M., D'Onofrio, B. M., Cortese, S., Larsson, H., & Chang, Z. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.43597 · Cited by 112
Ferreira, G. E., Shaheed, C. A., Underwood, M. R., Finnerup, N. B., Day, R. O., McLachlan, A. J., Eldabe, S. S., Zadro, J. R., & Maher, C. G. (2023) · BMJ · doi:10.1136/bmj-2022-072415 · Cited by 116
McCrory, C., McLoughlin, S., Layte, R., Cheallaigh, C. N., O’Halloran, A. M., Barros, H., Berkman, L. F., Bochud, M., Crimmins, E. M., Farrell, M. T., et al. (2023) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2023.106117 · Cited by 108
Assessing Heterogeneity in Random-Effects Meta-analysis
Langan, D. (2021) · Methods in molecular biology · doi:10.1007/978-1-0716-1566-9_4 · Cited by 99
Self-citationPiechotta, V., Siemens, W., Thielemann, I., Toews, M., Koch, J., Vygen-Bonnet, S., Kothari, K. U., Grummich, K., Braun, C., Kapp, P., et al. (2023) · The Lancet Child & Adolescent Health · doi:10.1016/s2352-4642(23)00078-0 · Cited by 98
Shephard, E., Zuccolo, P. F., Idrees, I., Godoy, P. B. G., Salomone, E., Ferrante, C., Sorgato, P., Catão, L. F. C. C., Goodwin, A., Bolton, P., et al. (2021) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2021.03.016 · Cited by 96
Schweizer, C., Hanreich, C., Tscholl, P. M., Ristl, R., Apprich, S. R., Windhager, R. H., & Waldstein, W. (2021) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-021-06770-x · Cited by 96
Korang, S. K., von Rohden, E., Veroniki, A. A., Ong, G., Ngalamika, O., Siddiqui, F., Juul, S., Nielsen, E. E., Feinberg, J. B., Petersen, J. J., et al. (2022) · PLoS ONE · doi:10.1371/journal.pone.0260733 · Cited by 100
“We assessed heterogeneity through visual inspection of forest plots, using the I(2) statistic [11, 24, 25], and by estimating the between-study variance using the DerSimonian and Laird method [26, 27].”
MethodSelf-citationBrikell, I., Yao, H., Li, L., Astrup, A., Gao, L., Gillies, M. B., Xie, T., Zhang‐James, Y., Dalsgaard, S., Engeland, A., et al. (2023) · The Lancet Psychiatry · doi:10.1016/s2215-0366(23)00332-2 · Cited by 105
Kock, L. S., Brown, J., Hiscock, R., Tattan‐Birch, H., Smith, C. A., & Shahab, L. (2019) · The Lancet Public Health · doi:10.1016/s2468-2667(19)30220-8 · Cited by 94
Work Experiences and Self-Esteem Development: A Meta-Analysis of Longitudinal Studies
Krauss, S., & Orth, U. (2022) · European Journal of Personality · doi:10.1177/08902070211027142 · Cited by 92
Neo, W. S., Foti, D., Keehn, B., & Kelleher, B. (2023) · Translational Psychiatry · doi:10.1038/s41398-023-02681-2 · Cited by 114
“We used the restricted maximum likelihood estimator, which provides approximately unbiased estimates of between-study heterogeneity variance [55].”
MethodMoghaddam, L. F., Vettore, M. V., Bayani, A., Bayat, A.-H., Ahounbar, E., Hemmat, M., Armoon, B., & Fakhri, Y. (2020) · BMC Pediatrics · doi:10.1186/s12887-020-02371-8 · Cited by 98
Gaebe, K., Li, A. Y., Park, A., Parmar, A., Lok, B. H., Sahgal, A., Chan, K., Erickson, A. W., & Das, S. (2022) · The Lancet Oncology · doi:10.1016/s1470-2045(22)00271-6 · Cited by 97
Magnon, V., Vallet, G. T., Benson, A. C., Mermillod, M., Chaussé, P., Lacroix, A., Bouillon‐Minois, J., & Dutheil, F. (2022) · Cortex · doi:10.1016/j.cortex.2022.07.008 · Cited by 102
Heterogeneity in meta-analysis: a comprehensive overview
Stogiannis, D., Siannis, F., & Androulakis, E. (2023) · The International Journal of Biostatistics · doi:10.1515/ijb-2022-0070 · Cited by 96
Positive soil priming effects are the rule at a global scale
Xu, S., Delgado‐Baquerizo, M., Kuzyakov, Y. V., Wu, Y., Liu, L., Yang, Y., Li, Y., Yu, Y., Zhu, B., & Yao, H. (2024) · Global Change Biology · doi:10.1111/gcb.17502 · Cited by 90
Westwood, S. J., Parlatini, V., Rubia, K., Cortese, S., Sonuga-Barke, E. J. S., European ADHD Guidelines Group (EAGG), Banaschewski, T., Baeyens, D., Bölte, S., Brandeis, D., et al. (2023) · Molecular Psychiatry · doi:10.1038/s41380-023-02000-7 · Cited by 98
“Outcome domains were analysed only if fi ve or more relevant RCTs were available per outcome, in order to be consistent with previous EAGG meta-analyses [15, 33, 34] and to reduce between-SMD heterogeneity [35].”
MethodA synthesis of local adaptation to climate through reciprocal common gardens
Lortie, C. J., & Hierro, J. L. (2021) · Journal of Ecology · doi:10.1111/1365-2745.13664 · Cited by 84
Pangestu, S., & Rencz, F. (2022) · Value in Health · doi:10.1016/j.jval.2022.07.017 · Cited by 86
Seide, S., Jensen, K., & Kieser, M. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1397 · Cited by 86
Krüsi, M., Eliades, T., & Papageorgiou, S. N. (2019) · Progress in Orthodontics · doi:10.1186/s40510-019-0261-5 · Cited by 87
“The strengths of this systematic review consist of the registration of its a priori protocol in PROSPERO [38, 39], its exhaustive literature search, its improved analytical methods [22], the use of the GRADE approach [25] to assess the quality of the meta-evidence, and the transparent provision of the study’s data [24…”
MethodSensitivity analysis with iterative outlier detection for systematic reviews and meta‐analyses
Meng, Z., Wang, J., Lin, L., & Wu, C. (2024) · Statistics in Medicine · doi:10.1002/sim.10008 · Cited by 98
Problematic smartphone use and academic achievement: A systematic review and meta-analysis
Paterna, A., Alcaraz-Ibáñez, M., Aguilar-Parra, J. M., Salavera, C., Demetrovics, Z., & Griffiths, M. D. (2024) · Journal of Behavioral Addictions · doi:10.1556/2006.2024.00014 · Cited by 83
A contrast of meta and metafor packages for meta‐analyses in R
Lortie, C. J., & Filazzola, A. (2020) · Ecology and Evolution · doi:10.1002/ece3.6747 · Cited by 82
Effectiveness of Electronic Cigarettes in Smoking Cessation: A Systematic Review and Meta-analysis
Grabovac, I., Oberndorfer, M., Fischer, J., Wiesinger, W., Haider, S., & Dorner, T. E. (2020) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntaa181 · Cited by 83
Hardman, C. A., Jones, A., Burton, S. P., Duckworth, J. J., McGale, L. S., Mead, B. R., Roberts, C. A., Field, M., & Werthmann, J. (2020) · Appetite · doi:10.1016/j.appet.2020.104986 · Cited by 80
Meta‐analyzing partial correlation coefficients using Fisher's z transformation
van Aert, R. C. M. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1654 · Cited by 82
Sex, Racial, and Ethnic Representation in COVID-19 Clinical Trials
Xiao, H., Vaidya, R., Liu, F., Chang, X., Xia, X., & Unger, J. M. (2022) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2022.5600 · Cited by 81
Ludwig, D. S., Dickinson, S. L., Henschel, B., Ebbeling, C. B., & Allison, D. B. (2021) · The Journal of Nutrition · doi:10.1093/jn/nxaa350 · Cited by 71
Belloni, S., Arrigoni, C., & Caruso, R. (2021) · Acta Oncologica · doi:10.1080/0284186x.2021.1962543 · Cited by 74
“Given the high clinical heterogeneity between studies, a random-effects model of the inverse variance was used to calculate the pooled effect [32], applying the restricted maximum likelihood (REML) [33].”
MethodAssociation between ADHD and vision problems. A systematic review and meta-analysis
Bellato, A., Perna, J., Ganapathy, P. S., Solmi, M., Zampieri, A., Cortese, S., & Faraone, S. V. (2022) · Molecular Psychiatry · doi:10.1038/s41380-022-01699-0 · Cited by 82
“The Restricted Maximum-Likelihood (REML) estimator was used with Knapp-Hartung confidence interval adjustment [42].”
MethodPonsford, M., Ward, T., Stoneham, S. M., Dallimore, C. M., Sham, D., Osman, K., Barry, S. M., Jolles, S. R. A., Humphreys, I. R., & Farewell, D. M. (2021) · Frontiers in Immunology · doi:10.3389/fimmu.2021.744696 · Cited by 68
Prevention of orthodontic enamel demineralization: A systematic review with meta‐analyses
Tasios, T., Papageorgiou, S. N., Papadopoulos, M. A. A. P., Τσάπας, Α., & Haidich, A. (2019) · Orthodontics and Craniofacial Research · doi:10.1111/ocr.12322 · Cited by 74
Abbing, A., Koretsi, V., Eliades, T., & Papageorgiou, S. N. (2020) · Progress in Orthodontics · doi:10.1186/s40510-020-00334-4 · Cited by 79
“As duration of orthodontic treatment is bound to be affected by clinician-, appliance-, and patient-related characteristics, a random-effects model was deemed appropriate to calculate the average distribution of true effects, based on clinical and statistical reasoning [18], and a restricted maximum likelihood random-…”
MethodSantaniello, A., Sansone, M., Fioretti, A., & Menna, L. F. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17093278 · Cited by 76
“Moreover, heterogeneity of studies has been evaluated using I2 [57].”
MethodJuul, S., Nielsen, N., Bentzer, P., Veroniki, A. A., Thabane, L., Linder, A., Klingenberg, S. L., Gluud, C. N., & Jakobsen, J. C. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01371-0 · Cited by 71
Self-citationPabst, A., Proksch, S., Médé, B., Comstock, D. C., Ross, J. M., & Balasubramaniam, R. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104587 · Cited by 68
The Link Between Low Self-Esteem and Eating Disorders: A Meta-Analysis of Longitudinal Studies
Krauss, S., Dapp, L. C., & Orth, U. (2023) · Clinical Psychological Science · doi:10.1177/21677026221144255 · Cited by 71
“Between-studies heterogeneity (i.e., τ 2 ) was estimated with the method of restricted maximum likelihood (Harville, 1977), as recommended by Langan et al. (2019).”
MethodKindred, R., Bates, G. W., & McBride, N. L. (2022) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2022.102640 · Cited by 79
Eze, P., Lawani, L. O., & Acharya, Y. (2021) · BMJ Global Health · doi:10.1136/bmjgh-2021-005035 · Cited by 66
“44 Robustness of pooled estimate was also assessed for the influence of studies with less than 200 participants in either arm—as studies with small sample size are more likely to exaggerate the intervention’s effect size.”
Genetic heterogeneity: Challenges, impacts, and methods through an associative lens
Woodward, A., Urbanowicz, R. J., Naj, A. C., & Moore, J. H. (2022) · Genetic Epidemiology · doi:10.1002/gepi.22497 · Cited by 71
Selecting the best meta-analytic estimator for evidence-based practice: a simulation study
Doi, S. A. R., & Furuya‐Kanamori, L. (2019) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000207 · Cited by 66
“The only link between data generation and model input was that the quality rank input in themodel and systematic error were linked sowe have an issue here akin to simulations done for random effects models where data are generated assuming random effects and analysed in the same fashion.(20) However in this simulation…”
MethodEstimation in meta‐analyses of mean difference and standardized mean difference
Bakbergenuly, I., Hoaglin, D. C., & Kulinskaya, E. (2019) · Statistics in Medicine · doi:10.1002/sim.8422 · Cited by 65
Weatherald, J. C., Parhar, K. K. S., Duhailib, Z. A., Chu, D. K., Granholm, A., Solverson, K., Lewis, K. A., Møller, M. H., Alshahrani, M. S. S., Belley‐Côté, E. P., et al. (2022) · BMJ · doi:10.1136/bmj-2022-071966 · Cited by 65
“28 Because more randomized controlled trials were identified than anticipated, we modified the analysis plan post hoc to exclude any quasi-randomized trials from the primary and secondary outcome analyses and instead include such trials in a sensitivity analysis.”
MethodPapageorgiou, S. N., Antonoglou, G. N., Michelogiannakis, D., Kakali, L., Eliades, T., & Madianos, P. (2022) · Journal of Clinical Periodontology · doi:10.1111/jcpe.13529 · Cited by 75
Meta-analysis in vocational behavior: A systematic review and recommendations for best practices
Rudolph, C. W., Chang, C. K., Rauvola, R. S., & Zacher, H. (2020) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2020.103397 · Cited by 67
Karl, V., Thorn, S., Mathes, T., Heß, S., & Maegele, M. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.0625 · Cited by 64
Methods for estimating between‐study variance and overall effect in meta‐analysis of odds ratios
Bakbergenuly, I., Hoaglin, D. C., & Kulinskaya, E. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1404 · Cited by 63
Impellizzeri, F. M., McCall, A., & van Smeden, M. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.09.007 · Cited by 73
Tenets for the Proper Conduct and Use of Meta-Analyses: A Practical Guide for Neurosurgeons
Lee, K. S., Zhang, J. J. Y., Nga, V. D. W., Ng, C. H., Tai, B. C., Higgins, J. P. T., & Syn, N. L. X. (2022) · World Neurosurgery · doi:10.1016/j.wneu.2021.09.034 · Cited by 63
Self-citationScientific-Consensus Communication About Contested Science: A Preregistered Meta-Analysis
van Stekelenburg, A., Schaap, G., Veling, H., Riet, J. V. ‘., & Buijzen, M. (2022) · Psychological Science · doi:10.1177/09567976221083219 · Cited by 69
“estimators in meta-analysis can produce different results (Langan et al., 2019; e.g., van der Linden & Goldberg, 2020), we compared the results obtained using the REML estimator and HKSJ CIs with those obtained using a conventional random-effects analysis (using the DerSimonian-Laird estimator without the HKSJ method).”
MethodBerendsen, S., Berendse, S., van der Torren, J., Vermeulen, J. M., & de Haan, L. (2024) · EClinicalMedicine · doi:10.1016/j.eclinm.2023.102392 · Cited by 73
metamedian : An R package for meta‐analyzing studies reporting medians
McGrath, S., Zhao, X., Öztürk, Ö., Katzenschlager, S., Steele, R., & Benedetti, A. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1686 · Cited by 61
“See [20, 21, 22] for discussions of estimators of the between-study variance.”
BackgroundKabera, F., Roy, J., Afifi, M., Godden, S., Stryhn, H., Sánchez, J., & Dufour, S. (2021) · Frontiers in Veterinary Science · doi:10.3389/fvets.2021.688450 · Cited by 61
“A random effects approach was used, as it was described in the published protocol (14) and the between-study variability was estimated using the method of restricted maximum likelihood (REML) (17) and the Knapp–Hartung adjustment for random effects model (18).”
MethodCastrillon, J., Rele, S., Sumithran, P., Hii, M., Thuraisingam, S., Spelman, T., Phan, T., Choong, P., Dowsey, M., & Shadbolt, C. (2025) · Anaesthesia · doi:10.1111/anae.16601 · Cited by 65
“Pooled odds ratios and 95%CI were calculated via random effect meta-analysis using the restricted maximum likelihood heterogeneity variance estimator [41, 42].”
What aspects of online peer feedback robustly predict growth in students' task performance?
Zong, Z., Schunn, C. D., & Wang, Y. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106924 · Cited by 61
“Some of the heterogeneity of effect is explained by the context in which the peer review occurs (e.g., discipline, grade band, type of learning objects), but they account for relatively small amounts of variance (Huisman et al., 2019; Langan et al., 2019; Li et al., 2020).”
BackgroundRisk Stratification in Nonischemic Dilated Cardiomyopathy Using CMR Imaging
Eichhorn, C., Koeckerling, D., Reddy, R. K., Ardissino, M., Rogowski, M., Coles, B., Hunziker, L., Greulich, S., Shiri, I., Frey, N., et al. (2024) · JAMA · doi:10.1001/jama.2024.13946 · Cited by 65
Anomalous information reception by mediums: A meta-analysis of the scientific evidence
Sarraf, M. A., Menie, M. A. W. O., & Tressoldi, P. E. (2020) · EXPLORE · doi:10.1016/j.explore.2020.04.002 · Cited by 58
Metadata
- Authors
- Langan, D., Higgins, J. P. T., Jackson, D., Bowden, J., Veroniki, A. A., Kontopantelis, E., Viechtbauer, W., & Simmonds, M.
- Affiliations
- University of Manchester (GB)
- University of Bristol (GB)
- AstraZeneca (United Kingdom) (GB)
- St. Michael's Hospital (CA)
- Maastricht University (NL)
- University of York (GB)
- Corresponding author
- Langan, Dean
- ORCID
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2018-09-06
- Volume
- 10
- Issue
- 1
- Pages
- 83-98
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W2885675882
- Semantic Scholar
- 264ababb39dc4b8737425a30c9fb75c5d8eb2bf4
- Cited by
- 1,074
- References
- 51
- Citations by year
- 2026: 214 · 2025: 173 · 2024: 198 · 2023: 144 · 2022: 140 · 2021: 113 · 2020: 60 · 2019: 28 · 2018: 2
- FWCI
- 38.65
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Research Synthesis Methods (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Bristol Research (University of Bristol) (submittedVersion, other-oa)
- UCL Discovery (University College London) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Funders
- National Institute for Health and Care Research (NF-SI-0617-10145)
- Medical Research Council (MC_UU_00011/2, MR/K006665/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- random-effects meta-analysis, estimation bias, binary outcomes, confidence interval coverage
- MeSH terms
- Systematic Reviews as Topic, Algorithms, Analysis of Variance, Computer Simulation, Data Interpretation, Statistical, Humans, Probability, Research Design, Software, Meta-Analysis as Topic, Reproducibility of Results, Models, Statistical, Selection Bias, Likelihood Functions, Odds Ratio
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
