Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies
Open accessAbstract
BACKGROUND: Random-effects meta-analysis is commonly performed by first deriving an estimate of the between-study variation, the heterogeneity, and subsequently using this as the basis for combining results, i.e., for estimating the effect, the figure of primary interest. The heterogeneity variance estimate however is commonly associated with substantial uncertainty, especially in contexts where there are only few studies available, such as in small populations and rare diseases. METHODS: Confidence intervals and tests for the effect may be constructed via a simple normal approximation, or via a Student-t distribution, using the Hartung-Knapp-Sidik-Jonkman (HKSJ) approach, which additionally uses a refined estimator of variance of the effect estimator. The modified Knapp-Hartung method (mKH) applies an ad hoc correction and has been proposed to prevent counterintuitive effects and to yield more conservative inference. We performed a simulation study to investigate the behaviour of the standard HKSJ and modified mKH procedures in a range of circumstances, with a focus on the common case of meta-analysis based on only a few studies. RESULTS: The standard HKSJ procedure works well when the treatment effect estimates to be combined are of comparable precision, but nominal error levels are exceeded when standard errors vary considerably between studies (e.g. due to variations in study size). Application of the modification on the other hand yields more conservative results with error rates closer to the nominal level. Differences are most pronounced in the common case of few studies of varying size or precision. CONCLUSIONS: Use of the modified mKH procedure is recommended, especially when only a few studies contribute to the meta-analysis and the involved studies' precisions (standard errors) vary.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
We adopted HKSJ as it is recommended for meta-analysis with few studies.
“Since only 14 studies were included in the meta-analysis, we adopted HKSJ as it is recommended for meta-analysis with few studies [38].”
From Generative AI Mental Health Chatbots as Therapeutic Tools: Systematic Review and Meta-Analysis of Their Role in Reducing Mental Health Issues (2025) · cites this paper as [38]
The Abstract Conclusions recommend the modified mKH procedure—not standard HKSJ—especially when few studies have varying precision. The Conclusions say standard HKSJ achieves the intended coverage only when study standard errors are similar; its error probability can exceed the target in unbalanced settings. The paper therefore does not make the unqualified recommendation attributed to it. (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763 · see citing works
IntHout, J., Ioannidis, J. P., & Borm, G. F. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-25 · Cited by 2,419 · see citing works
Improved tests for a random effects meta‐regression with a single covariate
Knapp, G., & Hartung, J. (2003) · Statistics in Medicine · doi:10.1002/sim.1482 · Cited by 2,075 · see citing works
Self-citationBias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model
Viechtbauer, W. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030003261 · Cited by 1,699 · cited 1× · see citing works
“(DL) estimator, a moment estimator [25] with acknowledged shortcomings [26], the restricted maximum likelihood (REML) estimator [3, 27], and the Paule-Mandel (PM) estimator, an essentially heuristic approach [28, 29].”
MethodMethods to estimate the between‐study variance and its uncertainty in meta‐analysis
Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P., Langan, D., & Salanti, G. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1164 · Cited by 1,685 · see citing works
Self-citationControlling the risk of spurious findings from meta-regression
Higgins, J. P. T., & Thompson, S. G. (2004) · Statistics in Medicine · doi:10.1002/sim.1752 · Cited by 1,267 · cited 1× · see citing works
“While the normal model may bemotivated by asymptotic arguments, in actual applications the number of estimates to be combined is commonly small [13, 14] and hence the estimation uncertainty in the betweenstudy variance τ 2 is substantial, so that an adjustment is appropriate and in fact improves operating characterist…”
BackgroundMuch ado about nothing: a comparison of the performance of meta-analytical methods with rare events
Bradburn, M. J., Deeks, J. J., Berlin, J. A., & Russell Localio, A. (2007) · Statistics in Medicine · doi:10.1002/sim.2528 · Cited by 938 · see citing works
Turner, R. M., Davey, J., Clarke, M. J., Thompson, S. G., & Higgins, J. P. (2012) · International Journal of Epidemiology · doi:10.1093/ije/dys041 · Cited by 837 · see citing works
A refined method for the meta-analysis of controlled clinical trials with binary outcome
Hartung, J., & Knapp, G. (2001) · Statistics in Medicine · doi:10.1002/sim.1009 · Cited by 775 · see citing works
Self-citationDawid, A. P. (1982) · Journal of the American Statistical Association · doi:10.1080/01621459.1982.10477856 · Cited by 625 · cited 1× · see citing works
“Simulations Since a (1−α) confidence interval is supposed to cover the true parameter value with probability (1−α), the calibration of such intervals may be checked by repeatedly generating random data based on known parameter values and then determining the empirical frequency with which true values are actually cove…”
BackgroundA simple confidence interval for meta-analysis
Sidik, K., & Jonkman, J. N. (2002) · Statistics in Medicine · doi:10.1002/sim.1262 · Cited by 431 · cited 7× · see citing works
“Hartung and Knapp [8, 9] and Sidik and Jonkman [10] independently introduced an adjusted confidence interval.”
MethodInfluentialTurner, R. M., Jackson, D., Wei, Y., Thompson, S. G., & Higgins, J. P. T. (2015) · Statistics in Medicine · doi:10.1002/sim.6381 · Cited by 410 · see citing works
A comparison of heterogeneity variance estimators in combining results of studies
Sidik, K., & Jonkman, J. N. (2007) · Statistics in Medicine · doi:10.1002/sim.2688 · Cited by 380 · see citing works
Small studies are more heterogeneous than large ones: a meta-meta-analysis
IntHout, J., Ioannidis, J. P. A., Borm, G. F., & Goeman, J. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.03.017 · Cited by 314 · see citing works
Flagged first, then by citations
Research works printed but not linked: 23
- [1]Hedges LV Olkin I Statistical Methods for Meta-analysis 1985 San Diego, CA, USA Academic Press
- [2]Hartung J Knapp G Sinha BK Statistical Meta-analysis with Applications 2008 Hoboken, NJ, USA Wiley
- [7]Follmann DA Proschan MA Valid inference in random effects meta-analysis Biometrics 1999 55 3 732 7 10.1111/j.0006-341X.1999.00732.x 11315000 · DOI
- [8]Hartung J Knapp G On tests of the overall treatment effect in meta-analysis with normally distributed responses Stat Med 2001 20 12 1771 82 10.1002/sim.791 11406840 · DOI · Paper page
- [14]Kontopantelis E Springate DA Reeves D A re-analysis of the Cochrane Library data: The dangers of unobserved heterogeneity in meta-analyses PLoS ONE 2013 8 7 69930 10.1371/journal.pone.0069930 PMC3724681 23922860 · DOI
- [17]Gagne JJ Thompson L O’Keefe K Kesselheim AS Innovative research methods for studying treatments for rare diseases: methodological review BMJ 2014 349 6802 10.1136/bmj.g6802 PMC4242670 25422272 · DOI
- [19]Korn EL, McShane LM, Freidlin B. Statistical challenges in the evaluation of treatments for small patient populations. Sci Transl Med. 2013; 178. doi: 10.1126/scitranslmed.3004018. 23536014 · DOI
- [20]Kesselheim AS Myers JA Avorn J Characteristics of clinical trials to support approval of orphan vs nonorphan drugs for cancer J Am Med Assoc 2011 305 22 2320 6 10.1001/jama.2011.769 21642684 · DOI
- [22]Prakken B Albani S Martini A Juvenile idiopathic arthritis The Lancet 2011 377 9783 2138 149 10.1016/S0140-6736(11)60244-4 21684384 · DOI
- [23]Hinks A Martin P Flynn E Eyre S Packham J Barton A Association of the CCR5 gene with juvenile idiopathic arthritis Genes & Immunity 2010 11 7 584 9 10.1038/gene.2010.25 20463745 PMC2974598 · DOI
- [25]DerSimonian R Laird N Meta-analysis in clinical trials Control Clin Trials 1986 7 3 177 88 10.1016/0197-2456(86)90046-2 3802833 · DOI · Paper page
- [26]Böhning D Malzahn U Dietz E Schlattmann P Viwatwongkasem C Biggeri A Some general points in estimating heterogeneity variance with the DerSimonian-Laird estimator Biostat 2002 3 4 445 57 10.1093/biostatistics/3.4.445 12933591 · DOI
- [28]Paule RC Mandel J Consensus values and weighting factors J Res Natl Bur Stand 1982 87 5 377 85 10.6028/jres.087.022 PMC6768160 34566088 · DOI
- [29]Rukhin AL Biggerstaff BJ Vangel MG Restricted maximum-likelihood estimation of a common mean and the Mandel-Paule algorithm J Stat Plan Infer 2000 83 2 319 30 10.1016/S0378-3758(99)00098-1 · DOI
- [30]Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Soft.2010;36(3) doi: 10.18637/jss.v036.i03. · DOI · Paper page
- [32]Hartung J An alternative method for meta-analysis Biom J 1999 41 8 901 16 10.1002/(SICI)1521-4036(199912)41:8<901::AID-BIMJ901>3.0.CO;2-W · DOI
- [33]Jackson D Riley RD A refined method for multivariate meta-analysis and meta-regression Stat Med 2014 33 4 541 54 10.1002/sim.5957 23996351 PMC4285306 · DOI
- [36]Kuß O Statistical methods for meta-analyses including information from studies without any events—add nothing to nothing and succeed nevertheless Stat Med 2015 34 7 1097 116 10.1002/sim.6383 25446971 · DOI · Paper page
- [38]Sánchez-Meca J Marín-Martínez F Confidence intervals for the overall effect size in random-effects meta-analysis Psychol Methods 2008 13 1 31 48 10.1037/1082-989X.13.1.31 18331152 · DOI
- [39]Sidik K Jonkman JN Authors’ reply Stat Med 2004 23 1 159 62 10.1002/sim.1729 · DOI
- [40]Viechtbauer W Confidence intervals for the amount of heterogeneity in meta-analysis Stat Med 2007 26 1 37 52 10.1002/sim.2514 16463355 · DOI
- [41]Gonnermann A Framke T Großhennig A Koch A No solution yet for combining two independent studies in the presence of heterogeneity Stat Med 2015 34 16 2476 80 10.1002/sim.6473 26040434 PMC4471592 · DOI
- [42]Sutton AJ Abrams KR Bayesian methods in meta-analysis and evidence synthesis Stat Methods Med Res 2001 10 4 277 303 10.1191/096228001678227794 11491414 · DOI
Cited by
Sorted by citations
A comparison of heterogeneity variance estimators in simulated random‐effects meta‐analyses
Langan, D., Higgins, J. P. T., Jackson, D., Bowden, J., Veroniki, A. A., Kontopantelis, E., Viechtbauer, W., & Simmonds, M. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1316 · Cited by 1,074 · see citing works
Methods for evidence synthesis in the case of very few studies
Bender, R., Friede, T., Koch, A., Kuss, O., Schlattmann, P., Schwarzer, G., & Skipka, G. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1297 · Cited by 250 · see citing works
Self-citationAntipsychotics for Treating Delirium in Hospitalized Adults: A Systematic Review
Nikooie, R., Neufeld, K. J., Oh, E. S., Wilson, L. M., Zhang, A., Robinson, K. A., & Needham, D. M. (2019) · Annals of Internal Medicine · doi:10.7326/m19-1860 · Cited by 246 · see citing works
Veroniki, A. A., Jackson, D., Bender, R., Kuss, O., Langan, D., Higgins, J. P. T., Knapp, G., & Salanti, G. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1319 · Cited by 236 · see citing works
Self-citationPartlett, C., & Riley, R. D. (2017) · Statistics in Medicine · doi:10.1002/sim.7140 · Cited by 240 · see citing works
Peripheral blood cellular immunophenotype in depression: a systematic review and meta-analysis
Foley, É. M., Parkinson, J. T., Mitchell, R. E., Turner, L., & Khandaker, G. M. (2023) · Molecular Psychiatry · doi:10.1038/s41380-022-01919-7 · Cited by 138 · see citing works
Meta‐analysis of two studies in the presence of heterogeneity with applications in rare diseases
Friede, T., Röver, C., Wandel, S., & Neuenschwander, B. (2017) · Biometrical Journal · doi:10.1002/bimj.201500236 · Cited by 128 · see citing works
Self-citationAntipsychotics for Preventing Delirium in Hospitalized Adults: A Systematic Review
Oh, E. S., Needham, D. M., Nikooie, R., Wilson, L. M., Zhang, A., Robinson, K. A., & Neufeld, K. J. (2019) · Annals of Internal Medicine · doi:10.7326/m19-1859 · Cited by 108 · see citing works
Bayesian estimation in random effects meta‐analysis using a non‐informative prior
Bodnar, O., Link, A., Arendacká, B., Possolo, A., & Elster, C. (2017) · Statistics in Medicine · doi:10.1002/sim.7156 · Cited by 73 · see citing works
Zhang, Q., Zhang, R., Xiong, Y., Sui, Y., Tong, C., & Lin, F.-H. (2025) · Journal of Medical Internet Research · doi:10.2196/78238 · Cited by 56 · see citing works
“Since only 14 studies were included in the meta-analysis, we adopted HKSJ as it is recommended for meta-analysis with few studies [38].”
Georgescu, R., Fodor, L. A., Dobrean, A., & Cristea, I. A. (2020) · Psychological Medicine · doi:10.1017/s0033291719001855 · Cited by 53 · see citing works
Brini, S., Brudasca, N. I., Hodkinson, A., Kaluzinska, K., Wach, A., Storman, D., Prokop-Dorner, A., Jemioło, P., & Bala, M. M. (2023) · Clinical Psychology Review · doi:10.1016/j.cpr.2022.102236 · Cited by 44 · see citing works
Saueressig, T., Pedder, H., Bowe, S. J., Owen, P. J., & Belavy, D. L. (2021) · Journal of Orthopaedic & Sports Physical Therapy · doi:10.2519/jospt.2021.0107 · Cited by 41 · see citing works
Lorenzo-Luaces, L., Johns, E., & Keefe, J. R. (2018) · Journal of Medical Internet Research · doi:10.2196/10113 · Cited by 36 · see citing works
Ramezani-Jolfaie, N., Mohammadi, M., & Salehi-Abargouei, A. (2020) · Eating and Weight Disorders - Studies on Anorexia, Bulimia and Obesity · doi:10.1007/s40519-019-00773-x · Cited by 20 · see citing works
ChatGPT as a Tool for Biostatisticians: A Tutorial on Applications, Opportunities, and Limitations
Dobler, D., Binder, H., Boulesteix, A., Igelmann, J., Köhler, D., Mansmann, U., Pauly, M., Scherag, A., Schmid, M., Al Tawil, A., et al. (2025) · Statistics in Medicine · doi:10.1002/sim.70263 · Cited by 20 · see citing works
Kwan, Y. K., Lau, Y., Ang, W. W., & Lau, S. T. (2024) · Nicotine and Tobacco Research · doi:10.1093/ntr/ntad145 · Cited by 9 · see citing works
- Authors
- Röver, C., Knapp, G., & Friede, T.
- Affiliations
- Universitätsmedizin Göttingen (DE)
- TU Dortmund University (DE)
- Corresponding author
- Röver, Christian
- Venue
- BMC Medical Research Methodology
- Published
- 2015-11-14
- Volume
- 15
- Issue
- 1
- Pages
- 99-99
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC4647507
- OpenAlex
- W2199614199
- Cited by
- 527
- References
- 14
- Citations by year
- 2026: 212 · 2025: 72 · 2024: 42 · 2023: 64 · 2022: 35 · 2021: 32 · 2020: 24 · 2019: 17 · 2018: 14 · 2017: 8 · 2016: 6
- FWCI
- 8.82
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- GoeScholar The Publication Server of the Georg-August-Universität Göttingen (Georg-August-Universität Göttingen) (submittedVersion, cc-by)
- GRO (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Seventh Framework Programme (FP HEALTH 2013-602144)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- random-effects meta-analysis, study heterogeneity, confidence intervals, t-distribution, conservative inference, rare diseases, small populations
- MeSH terms
- Algorithms, Computer Simulation, Humans, Meta-Analysis as Topic, Models, Statistical, Confidence Intervals, Sample Size, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
