CoChat
Sign Up
Cited by 314Meta-analysis and systematic reviewsStatistical Methods in Clinical TrialsDelphi Technique in Research

Small studies are more heterogeneous than large ones: a meta-meta-analysis

Open access
IntHout, Joanna, Ioannidis, John P.A., Borm, George F., Goeman, Jelle J.·2015·Journal of Clinical Epidemiology
DOI 10.1016/j.jclinepi.2015.03.017
Open this paper in CoChat — free accountOpen PDF

Abstract

Objectives Between-study heterogeneity plays an important role in random-effects models for meta-analysis. Most clinical trials are small, and small trials are often associated with larger effect sizes. We empirically evaluated whether there is also a relationship between trial size and heterogeneity (τ). Study design and setting We selected the first meta-analysis per intervention review of the Cochrane Database of Systematic Reviews Issues 2009-2013 with a dichotomous (n = 2,009) or continuous (n = 1,254) outcome. The association between estimated τ and trial size was evaluated across meta-analyses using regression and within meta-analyses using a Bayesian approach. Small trials were predefined as those having standard errors (SEs) over 0.2 standardized effects. Results Most meta-analyses were based on few (median 4) trials. Within the same meta-analysis, the small study τS(2) was larger than the large-study τL(2) [average ratio 2.11; 95% credible interval (1.05, 3.87) for dichotomous and 3.11 (2.00, 4.78) for continuous meta-analyses]. The imprecision of τS was larger than of τL: median SE 0.39 vs. 0.20 for dichotomous and 0.22 vs. 0.13 for continuous small-study and large-study meta-analyses. Conclusion Heterogeneity between small studies is larger than between larger studies. The large imprecision with which τ is estimated in a typical small-studies' meta-analysis is another reason for concern, and sensitivity analyses are recommended.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2015.03.017

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Heterogeneity is also influenced by unique patient characteristics, treatment techniques, and variability in measured outcomes.

    “but this is also influenced by unique patient characteristics, treatment techniques, and variability in measured outcomes.58”

    From The Effect of Upper Cervical Mobilization/Manipulation on Temporomandibular Joint Pain, Maximal Mouth Opening, and Pressure Pain Thresholds: A Systematic Review and Meta-Analysis (2023) · cites this paper as [58]

    The Discussion (page unavailable) suggests that differences in patient populations may contribute to heterogeneity. Another passage notes that heterogeneity may relate to intervention and outcome type, but the paper does not specifically establish effects of treatment techniques or variability in measured outcomes. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • The heterogeneity is due to the wide variety of study designs, the wide confidence intervals of individual studies, and the inclusion of low-weight studies, which can have a disproportionate effect on heterogeneity, especially in random-effects models.

    “This is due to the wide variety of study designs, the wide confidence intervals of individual studies, and the inclusion of low-weight studies, which can have a disproportionate effect on heterogeneity, especially in random-effects models [58].”

    From Safety outcomes of statin vs non-statin lipid-lowering interventions in patients with prior statin-associated muscle symptoms: A systematic review and meta-analysis (2025) · cites this paper as [58]

    The Discussion (page unavailable) says differences in patient populations and study quality may increase between-study heterogeneity. The implications section says small studies can have major influence in random-effects meta-analyses, but it describes influence on the pooled result, not low-weight studies causing heterogeneity. The paper does not identify wide confidence intervals of individual studies as a cause of heterogeneity. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Ioannidis JP, Pereira TV, Horwitz RI. Emergence of large treatment effects from small trialsdreply. JAMA 2013;309:768e9.

Cited by

Sorted by citations

Related

Authors
IntHout, J., Ioannidis, J. P. A., Borm, G. F., & Goeman, J. J.
Affiliations
  • Radboud University Medical Center (NL)
  • Stanford Medicine (US)
Corresponding author
IntHout, Joanna
ORCID
Venue
Journal of Clinical Epidemiology
Publisher
Elsevier BV
Published
2015-08
Volume
68
Issue
8
Pages
860-869
ISSN
0895-4356 (ISSN-L)
DOI
10.1016/j.jclinepi.2015.03.017
OpenAlex
W1973158091
Cited by
314
References
42
Citations by year
2026: 33 · 2025: 37 · 2024: 40 · 2023: 36 · 2022: 38 · 2021: 37 · 2020: 26 · 2019: 23 · 2018: 21 · 2017: 10 · 2016: 7 · 2015: 4 · 2014: 1
FWCI
2.33
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Clinical Epidemiology (publishedVersion)
  • PubMed (publishedVersion)
  • Radboud Repository (Radboud University) (submittedVersion)
  • Radboud Repository (Radboud University) (submittedVersion, other-oa)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Social Sciences › Sociology and Political Science › Delphi Technique in Research
Keywords
study heterogeneity, small-study effects, trial size, random effects models, meta-analysis, sensitivity analysis, meta-meta-analysis
MeSH terms
Bayes Theorem, Clinical Trials as Topic, Epidemiologic Methods, Humans, Models, Theoretical, Research Design

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review