CoChat
Sign Up
Cited by 108Meta-analysis and systematic reviewsStatistical Methods in EpidemiologyAdvanced Causal Inference Techniques

Synthesis of observational studies should consider credibility ceilings

Salanti, Georgia, Ioannidis, John P.A.·2009·Journal of Clinical Epidemiology
DOI 10.1016/j.jclinepi.2008.05.014
Open this paper in CoChat — free account

Abstract

Objective Meta-analyses of observational studies often get spuriously precise results. We aimed to factor this skepticism in meta-analysis calculations.Study design and setting We developed a simple sensitivity analysis starting from the assumption that any single observational study cannot give us more than a maximum certainty c% (called credibility ceiling) that an effect is in a particular direction and not in the other. Each study included in meta-analysis is adjusted for different credibility ceilings c and the consistency of the conclusion examined. We applied the method in three meta-analyses of observational studies with nominally statistically significant summary effects (mortality with teaching versus nonteaching health care; risk of non-Hodgkin's lymphoma with hair dyes; mortality with omega-3 fatty acids).Results Between-study heterogeneity I(2) estimates dropped from 36%-72% without a ceiling effect to 0% with ceilings of 9%, 4%, and 4% in the three meta-analyses, respectively. Nominal statistical significance was lost with ceilings of 10%, 8%, and 11%, respectively. The likelihood ratios suggested that even with minimal ceiling effects, there was no strong support for the credibility of each of these three associations.Conclusions Consideration of credibility ceilings allows conservative interpretation of observational evidence and can be applied routinely to meta-analyses of observational studies.

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2008.05.014

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Other topics included meta-regression, while few articles focused on more specific statistical issues, such as dose-response analysis, credibility ceilings, bias adjustment, hierarchical models, or regression bias.

    “Other topics included meta-regression, while few articles focused on more specific statistical issues, such as dose-response analysis [81, 109, 111, 113], credibility ceilings [110], bias adjustment [112], hierarchical models [117], or regression bias [108].”

    From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [110]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Salanti, G., & Ioannidis, J. P. A.
Affiliations
  • University of Ioannina (GR)
  • Tufts University (US)
Corresponding author
Ioannidis, John P.A.
ORCID
Venue
Journal of Clinical Epidemiology
Publisher
Elsevier BV
Published
2009-02
Volume
62
Issue
2
Pages
115-122
ISSN
0895-4356 (ISSN-L)
DOI
10.1016/j.jclinepi.2008.05.014
OpenAlex
W2010361292
Cited by
108
References
25
Citations by year
2026: 5 · 2025: 4 · 2024: 16 · 2023: 15 · 2022: 6 · 2021: 18 · 2020: 18 · 2019: 6 · 2018: 6 · 2017: 4 · 2016: 2 · 2014: 3 · 2012: 1
FWCI
1.93
Type
journal article
Language
en
Open access
closed
Licence
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
Locations
  • Journal of Clinical Epidemiology (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Keywords
observational studies, sensitivity analysis, study heterogeneity, direction of effect, likelihood ratios, non-Hodgkin lymphoma, hair dyes, omega-3 fatty acids, mortality
MeSH terms
Algorithms, Data Interpretation, Statistical, Hair Dyes, Hospitals, Teaching, Humans, Lymphoma, Non-Hodgkin, Research Design, Risk, Meta-Analysis as Topic, Fatty Acids, Omega-3, Bias, Likelihood Functions, Hospital Mortality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review