CoChat
Sign Up
Cited by 195Meta-analysis and systematic reviewsStatistical Methods in Epidemiologyscientometrics and bibliometrics research

Most published meta-regression analyses based on aggregate data suffer from methodological pitfalls: a meta-epidemiological study

Open access
Geissbühler, Michael, Hincapié, Cesar A., Aghlmandi, Soheila, Zwahlen, Marcel, Jüni, Peter, da Costa, Bruno R.·2021·BMC Medical Research Methodology
DOI 10.1186/s12874-021-01310-0PMC PMC8207572
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Due to clinical and methodological diversity, clinical studies included in meta-analyses often differ in ways that lead to differences in treatment effects across studies. Meta-regression analysis is generally recommended to explore associations between study-level characteristics and treatment effect, however, three key pitfalls of meta-regression may lead to invalid conclusions. Our aims were to determine the frequency of these three pitfalls of meta-regression analyses, examine characteristics associated with the occurrence of these pitfalls, and explore changes between 2002 and 2012. Methods: A meta-epidemiological study of studies including aggregate data meta-regression analysis in the years 2002 and 2012. We assessed the prevalence of meta-regression analyses with at least 1 of 3 pitfalls: ecological fallacy, overfitting, and inappropriate methods to regress treatment effects against the risk of the analysed outcome. We used logistic regression to investigate study characteristics associated with pitfalls and examined differences between 2002 and 2012. Results: Our search yielded 580 studies with meta-analyses, of which 81 included meta-regression analyses with aggregated data. 57 meta-regression analyses were found to contain at least one pitfall (70%): 53 were susceptible to ecological fallacy (65%), 14 had a risk of overfitting (17%), and 5 inappropriately regressed treatment effects against the risk of the analysed outcome (6%). We found no difference in the prevalence of meta-regression analyses with methodological pitfalls between 2002 and 2012, nor any study-level characteristic that was clearly associated with the occurrence of any of the pitfalls. Conclusion: The majority of meta-regression analyses based on aggregate data contain methodological pitfalls that may result in misleading findings.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12874-021-01310-0

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Non-daily vs. daily statin dosing resulted in an odds ratio of 0.58 (95% CI 0.23-1.43) for incidence of muscle symptoms.

    “Non-daily vs. daily statin dosing resulted in an OR of 0.58 (95% CI 0.23–1.43). [34,51,52].”

    From Safety outcomes of statin vs non-statin lipid-lowering interventions in patients with prior statin-associated muscle symptoms: A systematic review and meta-analysis (2025) · cites this paper as [52]

    The Methods > Analysis passage describes odds ratios for associations between study characteristics and methodological pitfalls, not statin dosing or muscle symptoms. A search of our copy for non-daily versus daily statin dosing and for the reported OR and confidence interval found no passage addressing that comparison; page numbers were unavailable. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 25

  1. [2]da Costa BR Jüni P Systematic reviews and meta-analyses of randomized trials: principles and pitfalls Eur Heart J 2014 35 47 3336 3345 10.1093/eurheartj/ehu424 25416325 · DOI
  2. [3]Higgins JPT Thomas J Chandler J Cochrane handbook for systematic reviews of interventions 2019 2nd edn Chichester Wiley
  3. [4]Thompson SG Higgins JPT How should meta-regression analyses be undertaken and interpreted? Stat Med 2002 21 11 1559 1573 10.1002/sim.1187 12111920 · DOI · Paper page
  4. [5]Berlin JA Santanna J Schmid CH Szczech LA Feldman HI Anti-Lymphocyte antibody induction therapy study group. Individual patient- versus group-level data meta-regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head Stat Med 2002 21 3 371 87 10.1002/sim.1023 11813224 · DOI
  5. [6]Austin PC Steyerberg EW The number of subjects per variable required in linear regression analyses J Clin Epidemiol 2015 68 6 627 636 10.1016/j.jclinepi.2014.12.014 25704724 · DOI
  6. [8]Senn S Importance of trends in the interpretation of an overall odds ratio in the meta-analysis of clinical trials Stat Med 1994 13 3 293 296 10.1002/sim.4780130310 8202653 · DOI
  7. [9]Sharp SJ Thompson SG Altman DG The relation between treatment benefit and underlying risk in meta-analysis BMJ 1996 313 7059 735 738 10.1136/bmj.313.7059.735 8819447 PMC2352108 · DOI
  8. [10]McIntosh MW. The population risk as an explanatory variable in research synthesis of clinical trials. Stat Med. 1996;15(16):1713–28. 10.1002/(sici)1097-0258(19960830)15:16%3c1713::aid-sim331%3e3.0.co;2-d. 8870154 · DOI
  9. [11]Thompson SG, Smith TC, Sharp SJ. Investigating underlying risk as a source of heterogeneity in meta-analysis. Stat Med. 1997;16(23):2741–58. 10.1002/(sici)1097-0258(19971215)16:23%3c2741::aid-sim703%3e3.0.co;2-0. 9421873 · DOI
  10. [12]Sharp SJ, Thompson SG. Analysing the relationship between treatment effect and underlying risk in meta-analysis: comparison and development of approaches. Stat Med. 2000;19(23):3251–74. 10.1002/1097-0258(20001215)19:23%3c3251::aid-sim625%3e3.0.co;2-2. 11113958 · DOI
  11. [13]Arends LR, Hoes AW, Lubsen J, Grobbee DE, Stijnen T. Baseline risk as predictor of treatment benefit: three clinical meta-re-analyses. Stat Med. 2000;19(24):3497–518. 10.1002/1097-0258(20001230)19:24%3c3497::aid-sim830%3e3.0.co;2-h. 11122510 · DOI · Paper page
  12. [14]Song F Xiong T Parekh-Bhurke S Inconsistency between direct and indirect comparisons of competing interventions: meta-epidemiological study BMJ 2011 343 aug16 2 d4909 9 10.1136/bmj.d4909 21846695 PMC3156578 · DOI
  13. [15]Song F Loke YK Walsh T Glenny AM Eastwood AJ Altman DG Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews BMJ 2009 338 apr03 1 b1147 7 10.1136/bmj.b1147 19346285 PMC2665205 · DOI
  14. [16]Schmid CH, Lau J, McIntosh MW, Cappelleri JC. An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials. Stat Med. 1998;17(17):1923–42. 10.1002/(sici)1097-0258(19980915)17:17%3c1923::aid-sim874%3e3.0.co;2-6. 9777687 · DOI
  15. [17]Heinze G A comparative investigation of methods for logistic regression with separated or nearly separated data Stat Med 2006 25 24 4216 4226 10.1002/sim.2687 16955543 · DOI
  16. [19]Cammà C Schepis F Orlando A Transarterial chemoembolization for unresectable hepatocellular carcinoma: meta-analysis of randomized controlled trials Radiology 2002 224 1 47 54 10.1148/radiol.2241011262 12091661 · DOI
  17. [20]Najaka SS Gottfredson DC Wilson DB A meta-analytic inquiry into the relationship between selected risk factors and problem behavior Prev Sci 2001 2 4 257 271 10.1023/a:1013610115351 11833928 · DOI
  18. [21]Zapf A Rauch G Kieser M Why do you need a biostatistician? BMC Med Res Methodol 2020 20 1 23 26 10.1186/s12874-020-0916-4 32024478 PMC7003429 · DOI
  19. [22]Giraudeau B Higgins JPT Tavernier E Trinquart L Sample size calculation for meta-epidemiological studies Stat Med 2016 35 2 239 250 10.1002/sim.6627 26286683 · DOI
  20. [23]Lambert PC Sutton AJ Abrams KR Jones DR A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis J Clin Epidemiol 2002 55 1 86 94 10.1016/s0895-4356(01)00414-0 11781126 · DOI
  21. [24]Schmid CH Stark PC Berlin JA Landais P Lau J Meta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors J Clin Epidemiol 2004 57 7 683 697 10.1016/j.jclinepi.2003.12.001 15358396 · DOI
  22. [25]Fisher DJ Carpenter JR Morris TP Freeman SC Tierney JF Meta-analytical methods to identify who benefits most from treatments: daft, deluded, or deft approach? BMJ 2017 356 j573 10.1136/bmj.j573 28258124 PMC5421441 · DOI
  23. [26]Burke DL Ensor J Riley RD Meta-analysis using individual participant data: one-stage and two-stage approaches, and why they may differ Stat Med 2017 36 5 855 875 10.1002/sim.7141 27747915 PMC5297998 · DOI · Paper page
  24. [27]Sutton AJ Kendrick D Coupland CAC Meta-analysis of individual- and aggregate-level data Stat Med 2008 27 5 651 669 10.1002/sim.2916 17514698 · DOI
  25. [28]Rücker G Schwarzer G Carpenter JR Schumacher M Undue reliance on I 2 in assessing heterogeneity may mislead BMC Med Res Methodol 2008 8 1 841 849 10.1186/1471-2288-8-79 PMC2648991 19036172 · DOI · Paper page

Cited by

Sorted by citations

Authors
Geissbühler, M., Hincapié, C. A., Aghlmandi, S., Zwahlen, M., Jüni, P., & da Costa, B. R.
Affiliations
  • University of Bern (CH)
  • St. Michael's Hospital (CA)
Corresponding author
Jüni, Peter
ORCID
Venue
BMC Medical Research Methodology
Publisher
Springer Science and Business Media LLC
Published
2021-06-15
Volume
21
Issue
1
Pages
123-123
ISSN
1471-2288 (ISSN-L)
DOI
10.1186/s12874-021-01310-0
PMC
PMC8207572
OpenAlex
W3170269751
Cited by
195
References
3
Citations by year
2026: 53 · 2025: 39 · 2024: 51 · 2023: 35 · 2022: 16 · 2021: 1
FWCI
14.93
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Research Methodology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • TSpace (University of Toronto) (submittedVersion, other-oa)
  • BORIS (University Library Bern) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Canada Research Chairs (Tier1)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
Keywords
meta-regression, aggregated data, ecological fallacy, overfitting, methodological pitfalls, treatment effect heterogeneity, meta-epidemiology, publication trends
MeSH terms
Humans, Publications, Regression Analysis, Bias, Epidemiologic Studies

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review