Selection in Reported Epidemiological Risks: An Empirical Assessment
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Epidemiological studies may be subject to selective reporting, but empirical evidence thereof is limited. We empirically evaluated the extent of selection of significant results and large effect sizes in a large sample of recent articles. METHODS AND FINDINGS: We evaluated 389 articles of epidemiological studies that reported, in their respective abstracts, at least one relative risk for a continuous risk factor in contrasts based on median, tertile, quartile, or quintile categorizations. We examined the proportion and correlates of reporting statistically significant and nonsignificant results in the abstract and whether the magnitude of the relative risks presented (coined to be consistently > or =1.00) differs depending on the type of contrast used for the risk factor. In 342 articles (87.9%), > or =1 statistically significant relative risk was reported in the abstract, while only 169 articles (43.4%) reported > or =1 statistically nonsignificant relative risk in the abstract. Reporting of statistically significant results was more common with structured abstracts, and was less common in US-based studies and in cancer outcomes. Among 50 randomly selected articles in which the full text was examined, a median of nine (interquartile range 5-16) statistically significant and six (interquartile range 3-16) statistically nonsignificant relative risks were presented (p = 0.25). Paradoxically, the smallest presented relative risks were based on the contrasts of extreme quintiles; on average, the relative risk magnitude was 1.41-, 1.42-, and 1.36-fold larger in contrasts of extreme quartiles, extreme tertiles, and above-versus-below median values, respectively (p < 0.001). CONCLUSIONS: Published epidemiological investigations almost universally highlight significant associations between risk factors and outcomes. For continuous risk factors, investigators selectively present contrasts between more extreme groups, when relative risks are inherently lower.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Self-citationThe Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration
Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., Gøtzsche, P. C., Lang, T., & for the CONSORT Group (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00012 · Cited by 3,835
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 1×
“While these biases have been studied predominantly in the randomized trials literature, selective reporting may be more prominent in epidemiological research [4].”
BackgroundChan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · cited 1×
“Bias may also affect the reporting of results within studies: ‘‘positive’’ outcomes may be reported preferentially over potentially less appealing ‘‘negative’’ analyses, even if this distorts the original analysis plan [7,8].”
ResultContradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465
Self-citationVisser, M., Goodpaster, B. H., Kritchevsky, S. B., Newman, A. B., Nevitt, M., Rubin, S. M., Simonsick, E. M., Harris, T. B., & Study, F. T. H. A. (2005) · The Journals of Gerontology Series A: Biological Sciences and Medical Sciences · doi:10.1093/gerona/60.3.324 · Cited by 1,403
Language bias in randomised controlled trials published in English and German
Egger, M., Zellweger-Zähner, T., Schneider, M., Junker, C., Lengeler, C., & Antes, G. (1997) · The Lancet · doi:10.1016/s0140-6736(97)02419-7 · Cited by 1,028
A consumer's guide to subgroup analyses
Oxman, A. D., & Guyatt, G. H. (1992) · Annals of Internal Medicine · doi:10.7326/0003-4819-116-1-78 · Cited by 914 · cited 1×
“Emphasis may be given to post-hoc subgroup analyses [9] or to dubious adjustments [10,11] that claim statistical significance [11].”
Background
Cited by
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495
“Empirical evidence has demonstrated across a large sample of 379 epidemiologic studies that investigators selected the contrasts for continuous variables so as to show effects as being larger: more extreme contrasts were presented, when effects were inherently smaller.(46) Post hoc demonstration of selective analysis…”
BackgroundSelf-citationDissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039
Type 2 diabetes and cancer: umbrella review of meta-analyses of observational studies
Tsilidis, K. K., Kasimis, J. C., Lopez, D. S., Ntzani, E. E., & Ioannidis, J. P. A. (2015) · BMJ · doi:10.1136/bmj.g7607 · Cited by 860
Self-citationInterpretation of tests of heterogeneity and bias in meta-analysis
Ioannidis, J. P. A. (2008) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2008.00986.x · Cited by 696
Self-citationDietary fiber and health outcomes: an umbrella review of systematic reviews and meta-analyses
Veronese, N., Solmi, M., Caruso, M. G., Giannelli, G., Osella, A. R., Evangelou, E., Maggi, S., Fontana, L., Stubbs, B., & Tzoulaki, I. (2018) · The American Journal of Clinical Nutrition · doi:10.1093/ajcn/nqx082 · Cited by 536
Why Current Publication Practices May Distort Science
Young, N. S., Ioannidis, J. P. A., & Al-Ubaydli, O. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050201 · Cited by 502
Self-citationReasons or excuses for avoiding meta-analysis in forest plots
Ioannidis, J. P. A., Patsopoulos, N. A., & Rothstein, H. R. (2008) · BMJ · doi:10.1136/bmj.a117 · Cited by 457
“confounding, and potentially stronger selective reporting biases in observational studies.”
BackgroundSelf-citationLi, X., Meng, X., Timofeeva, M., Tzoulaki, I., Tsilidis, K. K., Ioannidis, P. A., Campbell, H., & Theodoratou, E. (2017) · BMJ · doi:10.1136/bmj.j2376 · Cited by 385
Kyzas, P. A., Evangelou, E., Denaxa-Kyza, D., & Ioannidis, J. P. A. (2008) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djn125 · Cited by 364
Self-citationGuo, S.-W. (2009) · Molecular Human Reproduction · doi:10.1093/molehr/gap064 · Cited by 328
“Few, if any, positive findings from genetic association studies have been replicated (Altmuller et al., 2001; Hirschhorn et al., 2002; Chanock et al., 2007; Kavvoura et al., 2007), and those who tried to replicate previously reported positive findings often end up with negative results.”
BackgroundThe False-positive to False-negative Ratio in Epidemiologic Studies
Ioannidis, J. P. A., Tarone, R., & McLaughlin, J. K. (2011) · Epidemiology · doi:10.1097/ede.0b013e31821b506e · Cited by 321
“Empirical evaluations have shown that almost all published observational studies report some nominally statistically significant results.(28,59) An evaluation of 1915 publications on cancer prognostic markers shows that only 1.”
BackgroundSelf-citationWhen Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298
Self-citationGrading the strength of a body of evidence when assessing health care interventions: an EPC update
Berkman, N. D., Lohr, K. N., Ansari, M. T., Balk, E. M., Kane, R., McDonagh, M., Morton, S. C., Viswanathan, M., Bass, E. B., Butler, M., et al. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.023 · Cited by 279
False Alarms and Pseudo-Epidemics: The Limitations of Observational Epidemiology
Grimes, D. A., & Schulz, K. F. (2012) · Obstetrics & Gynecology · doi:10.1097/aog.0b013e31826af61a · Cited by 254
Ioannidis, J. P. A. (2011) · JAMA · doi:10.1001/jama.2011.713 · Cited by 252
Self-citationTelomere length behaves as biomarker of somatic redundancy rather than biological age
Boonekamp, J. J., Simons, M. J. P., Hemerik, L., & Verhulst, S. (2013) · Aging Cell · doi:10.1111/acel.12050 · Cited by 200
Use of benznidazole to treat chronic Chagas' disease: a systematic review with a meta-analysis
Perez-Molina, J. A., Perez-Ayala, A., Moreno, S., Fernandez-Gonzalez, M. C., Zamora, J., & Lopez-Velez, R. (2009) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkp357 · Cited by 199
Is everything we eat associated with cancer? A systematic cookbook review
Schoenfeld, J. D., & Ioannidis, J. P. (2013) · The American Journal of Clinical Nutrition · doi:10.3945/ajcn.112.047142 · Cited by 198
“It was previously shown in a variety of other fields that “negative” results are either less likely to be published (16–21) or misleadingly interpreted (19, 22).”
BackgroundResultInfluentialSelf-citationUnderstanding and Harnessing the Health Effects of Rapid Urbanization in China
Zhu, Y.-G., Ioannidis, J. P. A., Li, H., Jones, K. C., & Martin, F. L. (2011) · Environmental Science & Technology · doi:10.1021/es2004254 · Cited by 164
Self-citationMattes, R. D., Shikany, J. M., Kaiser, K. A., & Allison, D. B. (2011) · Obesity Reviews · doi:10.1111/j.1467-789x.2010.00755.x · Cited by 160
Synthesis of observational studies should consider credibility ceilings
Salanti, G., & Ioannidis, J. P. A. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.05.014 · Cited by 108
Self-citationMaking Prospective Registration of Observational Research a Reality
Dal-Ré, R., Ioannidis, J. P., Bracken, M. B., Buffler, P. A., Chan, A.-W., Franco, E. L., La Vecchia, C., & Weiderpass, E. (2014) · Science Translational Medicine · doi:10.1126/scitranslmed.3007513 · Cited by 123
“Indeed, nearly all published OS results report some significant findings, and many report implausibly large effects (11).”
BackgroundSelf-citationCope, M. B., & Allison, D. B. (2008) · Obesity Reviews · doi:10.1111/j.1467-789x.2008.00504.x · Cited by 110
Metadata
- Authors
- Kavvoura, F. K., Liberopoulos, G., & Ioannidis, J. P. A.
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2007-03-06
- Volume
- 4
- Issue
- 3
- Pages
- e79
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- OpenAlex
- W2044757014
- Cited by
- 115
- References
- 30
- FWCI
- 13.76
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.