The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Small studies are more heterogeneous than large ones: a meta-meta-analysis
Open accessAbstract
Objectives Between-study heterogeneity plays an important role in random-effects models for meta-analysis. Most clinical trials are small, and small trials are often associated with larger effect sizes. We empirically evaluated whether there is also a relationship between trial size and heterogeneity (τ). Study design and setting We selected the first meta-analysis per intervention review of the Cochrane Database of Systematic Reviews Issues 2009-2013 with a dichotomous (n = 2,009) or continuous (n = 1,254) outcome. The association between estimated τ and trial size was evaluated across meta-analyses using regression and within meta-analyses using a Bayesian approach. Small trials were predefined as those having standard errors (SEs) over 0.2 standardized effects. Results Most meta-analyses were based on few (median 4) trials. Within the same meta-analysis, the small study τS(2) was larger than the large-study τL(2) [average ratio 2.11; 95% credible interval (1.05, 3.87) for dichotomous and 3.11 (2.00, 4.78) for continuous meta-analyses]. The imprecision of τS was larger than of τL: median SE 0.39 vs. 0.20 for dichotomous and 0.22 vs. 0.13 for continuous small-study and large-study meta-analyses. Conclusion Heterogeneity between small studies is larger than between larger studies. The large imprecision with which τ is estimated in a typical small-studies' meta-analysis is another reason for concern, and sensitivity analyses are recommended.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“but this is also influenced by unique patient characteristics, treatment techniques, and variability in measured outcomes.58”
From The Effect of Upper Cervical Mobilization/Manipulation on Temporomandibular Joint Pain, Maximal Mouth Opening, and Pressure Pain Thresholds: A Systematic Review and Meta-Analysis (2023) · cites this paper as [58]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This is due to the wide variety of study designs, the wide confidence intervals of individual studies, and the inclusion of low-weight studies, which can have a disproportionate effect on heterogeneity, especially in random-effects models [58].”
From Safety outcomes of statin vs non-statin lipid-lowering interventions in patients with prior statin-associated muscle symptoms: A systematic review and meta-analysis (2025) · cites this paper as [58]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,893
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“[11] argued that underpowered studies are prone to several analytical and reporting biases.”
BackgroundSelf-citationWinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility
Lunn, D. J., Thomas, A., Best, N., & Spiegelhalter, D. J. (2000) · Statistics and Computing · doi:10.1023/a:1008929526011 · Cited by 5,397
Assessing heterogeneity in meta-analysis: Q statistic or I² index?
Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.2.193 · Cited by 4,256
Wood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538
Random-effects model for meta-analysis of clinical trials: An update
DerSimonian, R., & Kacker, R. (2007) · Contemporary Clinical Trials · doi:10.1016/j.cct.2006.04.004 · Cited by 2,374
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282
R2WinBUGS : A Package for Running WinBUGS from R
Sturtz, S., Ligges, U., & Gelman, A. (2005) · Journal of Statistical Software · doi:10.18637/jss.v012.i03 · Cited by 1,654
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495
Self-citationUndue reliance on I 2 in assessing heterogeneity may mislead
Rücker, G., Schwarzer, G., Carpenter, J. R., & Schumacher, M. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-79 · Cited by 1,287
Uncertainty in heterogeneity estimates in meta-analyses
Ioannidis, J. P. A., Patsopoulos, N. A., & Evangelou, E. (2007) · BMJ · doi:10.1136/bmj.39343.408449.80 · Cited by 1,254
Self-citationTurner, R. M., Davey, J., Clarke, M. J., Thompson, S. G., & Higgins, J. P. (2012) · International Journal of Epidemiology · doi:10.1093/ije/dys041 · Cited by 837
The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews
Turner, R. M., Bird, S. M., & Higgins, J. P. T. (2013) · PLoS ONE · doi:10.1371/journal.pone.0059202 · Cited by 672
Consensus Values and Weighting Factors
Paule, R. C., & Mandel, J. (1982) · Journal of Research of the National Bureau of Standards · doi:10.6028/jres.087.022 · Cited by 580
Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study
Nuesch, E., Trelle, S., Reichenbach, S., Rutjes, A. W. S., Tschannen, B., Altman, D. G., Egger, M., & Juni, P. (2010) · BMJ · doi:10.1136/bmj.c3515 · Cited by 559
Characteristics of Clinical Trials Registered in ClinicalTrials.gov, 2007-2010
Califf, R. M. (2012) · JAMA · doi:10.1001/jama.2012.3424 · Cited by 516
Friedrich, J. O., Adhikari, N. K. J., & Beyene, J. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-5 · Cited by 460
Reasons or excuses for avoiding meta-analysis in forest plots
Ioannidis, J. P. A., Patsopoulos, N. A., & Rothstein, H. R. (2008) · BMJ · doi:10.1136/bmj.a117 · Cited by 457
Self-citationEvolution of Heterogeneity (I2) Estimates and Their 95% Confidence Intervals in Large Meta-Analyses
Thorlund, K., Imberger, G. L., Johnston, B. C., Walsh, M. W., Awad, T., Thabane, L., Gluud, C. N., Devereaux, P. J., & Wetterslev, J. (2012) · PLoS ONE · doi:10.1371/journal.pone.0039471 · Cited by 448
A comparison of heterogeneity variance estimators in combining results of studies
Sidik, K., & Jonkman, J. N. (2007) · Statistics in Medicine · doi:10.1002/sim.2688 · Cited by 380
Friedrich, J. O., Adhikari, N. K. J., & Beyene, J. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-32 · Cited by 333
Empirical Evaluation of Very Large Treatment Effects of Medical Interventions
Pereira, T. V., Horwitz, R. I., & Ioannidis, J. P. A. (2012) · JAMA · doi:10.1001/jama.2012.13444 · Cited by 257
Self-citationA comparison of procedures to test for moderators in mixed-effects meta-regression models
Viechtbauer, W., López-López, J. A., Sánchez-Meca, J., & Marín-Martínez, F. (2015) · Psychological Methods · doi:10.1037/met0000023 · Cited by 241
Predictive ability of meta-analyses of randomised controlled trials
Villar, J., Carroli, G., & Belizán, J. M. (1995) · The Lancet · doi:10.1016/s0140-6736(95)90646-0 · Cited by 223
Hennekens, C. H. (2009) · JAMA · doi:10.1001/jama.2009.1756 · Cited by 149
Obtaining evidence by a single well-powered trial or several modestly powered trials
IntHout, J., Ioannidis, J. P. A., & Borm, G. F. (2012) · Statistical Methods in Medical Research · doi:10.1177/0962280212461098 · Cited by 74
Self-citationEstimation of between-trial variance in sequential meta-analyses: A simulation study
Novianti, P. W., Roes, K. C. B., & van der Tweel, I. (2013) · Contemporary Clinical Trials · doi:10.1016/j.cct.2013.11.012 · Cited by 71
Planning future studies based on the conditional power of a meta‐analysis
Roloff, V., Higgins, J. P. T., & Sutton, A. J. (2012) · Statistics in Medicine · doi:10.1002/sim.5524 · Cited by 68
The evidence provided by a single trial is less reliable than its statistical analysis suggests
Borm, G. F., Lemmers, O., Fransen, J., & Donders, R. A. R. T. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.09.013 · Cited by 54
Self-citationKrum, H., & Tonkin, A. (2003) · Journal of Cardiac Failure · doi:10.1054/s1071-9164(03)00018-6 · Cited by 30
Publication bias was not a good reason to discourage trials with low power
Borm, G. F., den Heijer, M., & Zielhuis, G. A. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.02.017 · Cited by 18
Self-citationA treatment should be evaluated by small trials
Borm, G. F., & Donders, R. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.03.006 · Cited by 10
Self-citationEmergence of Large Treatment Effects From Small Trials—Reply
Ioannidis, J. P. A., Pereira, T., & Horwitz, R. I. (2013) · JAMA · doi:10.1001/jama.2012.208831 · Cited by 8
Self-citationEmergence of Large Treatment Effects From Small Trials
Batterham, A. M., & Hopkins, W. G. (2013) · JAMA · doi:10.1001/jama.2012.208828 · Cited by 8
A simple method for converting an odds ratio to effect size for use in meta-analysis
Chinn, S. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001130)19:22
Cited by
Sorted by citations
Plea for routinely presenting prediction intervals in meta-analysis.
IntHout, J., Ioannidis, J. P. A., Rovers, M. M., & Goeman, J. J. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010247 · Cited by 2,264
Self-citationPhysical activity and exercise for chronic pain in adults: an overview of Cochrane Reviews
Geneen, L. J., Moore, R. A., Clarke, C., Martin, D., Colvin, L. A., & Smith, B. H. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011279.pub3 · Cited by 1,222
Physical activity and exercise for chronic pain in adults: an overview of Cochrane Reviews
Geneen, L. J., Moore, R. A., Clarke, C., Martin, D., Colvin, L. A., & Smith, B. H. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011279.pub2 · Cited by 719
Röver, C., Knapp, G., & Friede, T. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0091-1 · Cited by 527
Music therapy for stress reduction: a systematic review and meta-analysis
de Witte, M., Pinho, A. D. S., Stams, G.-J., Moonen, X., Bos, A. E. R., & van Hooren, S. (2022) · Health Psychology Review · doi:10.1080/17437199.2020.1846580 · Cited by 444
Metadata
- Authors
- IntHout, J., Ioannidis, J. P. A., Borm, G. F., & Goeman, J. J.
- Affiliations
- Radboud University Medical Center (NL)
- Stanford Medicine (US)
- Corresponding author
- IntHout, Joanna
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2015-08
- Volume
- 68
- Issue
- 8
- Pages
- 860-869
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W1973158091
- Cited by
- 314
- References
- 42
- Citations by year
- 2026: 33 · 2025: 37 · 2024: 40 · 2023: 36 · 2022: 38 · 2021: 37 · 2020: 26 · 2019: 23 · 2018: 21 · 2017: 10 · 2016: 7 · 2015: 4
- FWCI
- 2.33
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion, other-oa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- study heterogeneity, small-study effects, trial size, random effects models, meta-analysis, sensitivity analysis, meta-meta-analysis
- MeSH terms
- Bayes Theorem, Clinical Trials as Topic, Epidemiologic Methods, Humans, Models, Theoretical, Research Design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
