The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Synthesis of observational studies should consider credibility ceilings
Abstract
Objective Meta-analyses of observational studies often get spuriously precise results. We aimed to factor this skepticism in meta-analysis calculations.Study design and setting We developed a simple sensitivity analysis starting from the assumption that any single observational study cannot give us more than a maximum certainty c% (called credibility ceiling) that an effect is in a particular direction and not in the other. Each study included in meta-analysis is adjusted for different credibility ceilings c and the consistency of the conclusion examined. We applied the method in three meta-analyses of observational studies with nominally statistically significant summary effects (mortality with teaching versus nonteaching health care; risk of non-Hodgkin's lymphoma with hair dyes; mortality with omega-3 fatty acids).Results Between-study heterogeneity I(2) estimates dropped from 36%-72% without a ceiling effect to 0% with ceilings of 9%, 4%, and 4% in the three meta-analyses, respectively. Nominal statistical significance was lost with ceilings of 10%, 8%, and 11%, respectively. The likelihood ratios suggested that even with minimal ceiling effects, there was no strong support for the credibility of each of these three associations.Conclusions Consideration of credibility ceilings allows conservative interpretation of observational evidence and can be applied routinely to meta-analyses of observational studies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Other topics included meta-regression, while few articles focused on more specific statistical issues, such as dose-response analysis [81, 109, 111, 113], credibility ceilings [110], bias adjustment [112], hierarchical models [117], or regression bias [108].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [110]
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Self-citationEvaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Wacholder, S., Chanock, S. J., García‐Closas, M., ghormli, L. K. E., & Rothman, N. (2004) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djh075 · Cited by 1,753
Bayesian Approaches to Clinical Trials and Health‐Care Evaluation
Spiegelhalter, D. J., Abrams, K. R., & Myles, J. P. (2003) · Wiley eBooks · doi:10.1002/0470092602 · Cited by 1,687
Sterne, J. A. C. (2001) · BMJ · doi:10.1136/bmj.322.7280.226 · Cited by 1,466
Hooper, L., Thompson, R. L., Harrison, R. A., Summerbell, C. D., Ness, A. R., Moore, H. J., Worthington, H. V., Durrington, P. N., Higgins, J. P. T., Capps, N. E., et al. (2006) · BMJ · doi:10.1136/bmj.38755.366331.2f · Cited by 676
Greenland, S. (2001) · Biostatistics · doi:10.1093/biostatistics/2.4.463 · Cited by 359
Most published research findings are false-but a little replication goes a long way.
Moonesinghe, R., Khoury, M. J., & Janssens, A. C. J. W. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040028 · Cited by 341
Bayesian Approaches to Clinical Trials and Health-Care Evaluation
Best, M. A. (2004) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2004.00347_17.x · Cited by 314
Issues in the reporting of epidemiological studies: a survey of recent practice.
Pocock, S. J., Collier, T. J., Dandreo, K. J., de Stavola, B. L., Goldman, M. B., Kalish, L. A., Kasten, L. E., & McCormack, V. A. (2004) · BMJ · doi:10.1136/bmj.38250.571088.55 · Cited by 313
Kyzas, P., Cunha, I. W., & Ioannidis, J. P. A. (2005) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-04-1870 · Cited by 250
Self-citationAlmost all articles on cancer prognostic markers report statistically significant results
Kyzas, P., Denaxa-Kyza, D., & Ioannidis, J. P. A. (2007) · European Journal of Cancer · doi:10.1016/j.ejca.2007.08.030 · Cited by 233
Self-citationSynthesis of Environmental Evidence: Nitrogen Dioxide Epidemiology Studies
Hasselblad, V., Eddy, D. M., & Kotchmar, D. J. (1992) · Journal of the Air & Waste Management Association · doi:10.1080/10473289.1992.10467018 · Cited by 205
For Debate: The statistical basis of public policy: a paradigm shift is overdue
Lilford, R. J., & Braunholtz, D. (1996) · BMJ · doi:10.1136/bmj.313.7057.603 · Cited by 190
Personal Use of Hair Dyes and Risk of Cancer
Takkouche, B., Etminan, M., & Montes, A. (2005) · JAMA · doi:10.1001/jama.293.20.2516 · Cited by 151
Selection in Reported Epidemiological Risks: An Empirical Assessment
Kavvoura, F. K., Liberopoulos, G., & Ioannidis, J. P. A. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040079 · Cited by 115
Self-citationPatient Outcomes with Teaching Versus Nonteaching Healthcare: A Systematic Review
Papanikolaou, P. N., Christidi, G. D., & Ioannidis, J. P. A. (2006) · PLoS Medicine · doi:10.1371/journal.pmed.0030341 · Cited by 111
Self-citationA Bayesian Method for Synthesizing Evidence: The Confidence Profile Method
Eddy, D. M., Hasselblad, V., & Shachter, R. D. (1990) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300008928 · Cited by 101
Wolpert, R. L., & Mengersen, K. L. (2004) · Statistical Science · doi:10.1214/088342304000000350 · Cited by 60
(2003) · Bayesian Approaches to Clinical Trials and Health‐Care Evaluation · doi:10.1002/0470092602.ch8 · Cited by 10
Critchfield, G. C., & Eddy, D. M. (1987) · Medical Care · doi:10.1097/00005650-198712001-00004 · Cited by 6
Cited by
Sorted by citations
Adiposity and cancer at major anatomical sites: umbrella review of the literature
Kyrgiou, M., Kalliala, I., Markozannes, G., Gunter, M. J., Paraskevaidis, E., Gabra, H., Martin-Hirsch, P., & Tsilidis, K. K. (2017) · BMJ · doi:10.1136/bmj.j477 · Cited by 803
Cupp, M. A., Cariolou, M., Tzoulaki, I., Aune, D., Evangelou, E., & Berlanga-Taylor, A. J. (2020) · BMC Medicine · doi:10.1186/s12916-020-01817-1 · Cited by 697
Moja, L., Piatti, A., Pecoraro, V. L., Ricci, C., Virgili, G., Salanti, G., Germagnoli, L., Liberati, A., & Banfi, G. (2012) · PLoS ONE · doi:10.1371/journal.pone.0046175 · Cited by 677
Self-citationRisk factors for endometrial cancer: An umbrella review of the literature
Raglan, O., Kalliala, I. E. J., Markozannes, G., Cividini, S., Gunter, M. J., Nautiyal, J., Gabra, H., Paraskevaidis, E. A., Martin‐Hirsch, P. L., Tsilidis, K. K., et al. (2018) · International Journal of Cancer · doi:10.1002/ijc.31961 · Cited by 664
Mueller, M., D’Addario, M., Egger, M., Cevallos, M., Dekkers, O., Mugglin, C., & Scott, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0495-9 · Cited by 400
Metadata
- Authors
- Salanti, G., & Ioannidis, J. P. A.
- Affiliations
- University of Ioannina (GR)
- Tufts University (US)
- Corresponding author
- Ioannidis, John P.A.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2009-02
- Volume
- 62
- Issue
- 2
- Pages
- 115-122
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2010361292
- Cited by
- 108
- References
- 25
- Citations by year
- 2026: 5 · 2025: 4 · 2024: 16 · 2023: 15 · 2022: 6 · 2021: 18 · 2020: 18 · 2019: 6 · 2018: 6 · 2017: 4 · 2016: 2 · 2014: 3 · 2012: 1
- FWCI
- 1.93
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- observational studies, sensitivity analysis, study heterogeneity, direction of effect, likelihood ratios, non-Hodgkin lymphoma, hair dyes, omega-3 fatty acids, mortality
- MeSH terms
- Algorithms, Data Interpretation, Statistical, Hair Dyes, Hospitals, Teaching, Humans, Lymphoma, Non-Hodgkin, Research Design, Risk, Meta-Analysis as Topic, Fatty Acids, Omega-3, Bias, Likelihood Functions, Hospital Mortality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
