Believability of relative risks and odds ratios in abstracts: Cross sectional study.
Open accessAbstract
OBJECTIVE: To compare the distribution of P values in abstracts of randomised controlled trials with that in observational studies, and to check P values between 0.04 and 0.06. DESIGN: Cross sectional study of all 260 abstracts in PubMed of articles published in 2003 that contained "relative risk" or "odds ratio" and reported results from a randomised trial, and random samples of 130 abstracts from cohort studies and 130 from case-control studies. P values were noted or calculated if unreported. MAIN OUTCOME MEASURES: Prevalence of significant P values in abstracts and distribution of P values between 0.04 and 0.06. RESULTS: The first result in the abstract was statistically significant in 70% of the trials, 84% of cohort studies, and 84% of case-control studies. Although many of these results were derived from subgroup or secondary analyses, or biased selection of results, they were presented without reservations in 98% of the trials. P values were more extreme in observational studies (P < 0.001) and in cohort studies than in case-control studies (P = 0.04). The distribution of P values around P = 0.05 was extremely skewed. Only five trials had 0.05 < or = P < 0.06, whereas 29 trials had 0.04 < or = P < 0.05. I could check the calculations for 27 of these trials. One of four non-significant results was significant. Four of the 23 significant results were wrong, five were doubtful, and four could be discussed. Nine cohort studies and eight case-control studies reported P values between 0.04 and 0.06, but in all 17 cases P < 0.05. Because the analyses had been adjusted for confounders, these results could not be checked. CONCLUSIONS: Significant results in abstracts are common but should generally be disbelieved.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Many debate the use and value of analyses restricted to subgroups of the study population.
“many debate the use and value of analyses restricted to subgroups of the study population [ 4 , 104 ].”
From Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration (2007) · cites this paper as [104]
Inclusion or exclusion of non-English language literature, unpublished data, or older data can influence effect estimates in meta-analyses.
“Inclusion or not of non-English language literature [51],[52],[53],[54],[55], unpublished data, or older data can influence the effect estimates in meta-analyses [56],[57],[58],[59].”
From The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration (2009) · cites this paper as [59]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Statistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · see citing works
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · see citing works
Self-citationSterne, J. A. C. (2001) · BMJ · doi:10.1136/bmj.322.7280.226 · Cited by 1,466 · see citing works
Subgroup analysis and other (mis)uses of baseline data in clinical trials
Assmann, S. F., Pocock, S. J., Enos, L. E., & Kasten, L. E. (2000) · The Lancet · doi:10.1016/s0140-6736(00)02039-0 · Cited by 1,167 · see citing works
Full publication of results initially presented in abstracts.
Scherer, R. W., Langenberg, P., & von Elm, E. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000005.pub3 · Cited by 997 · see citing works
Statistical Problems in the Reporting of Clinical Trials
Pocock, S., Hughes, M. D., & Lee, R. J. (1987) · New England Journal of Medicine · doi:10.1056/nejm198708133170706 · Cited by 660 · see citing works
Chan, A.-W., & Altman, D. G. (2005) · BMJ · doi:10.1136/bmj.38356.424606.8f · Cited by 613 · see citing works
Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research
Chan, A.-W. (2004) · Canadian Medical Association Journal · doi:10.1503/cmaj.1041086 · Cited by 444 · see citing works
Data dredging, bias, or confounding: They can all get you into the BMJ and the Friday papers
Smith, G. D., & Ebrahim, S. (2002) · BMJ · doi:10.1136/bmj.325.7378.1437 · Cited by 434 · see citing works
Randomization, Statistics, and Causal Inference
Greenland, S. (1990) · Epidemiology · doi:10.1097/00001648-199011000-00003 · Cited by 385 · see citing works
Gøtzsche, P. C. (1989) · Controlled Clinical Trials · doi:10.1016/0197-2456(89)90017-2 · Cited by 339 · see citing works
Self-citationIssues in the reporting of epidemiological studies: a survey of recent practice.
Pocock, S. J., Collier, T. J., Dandreo, K. J., de Stavola, B. L., Goldman, M. B., Kalish, L. A., Kasten, L. E., & McCormack, V. A. (2004) · BMJ · doi:10.1136/bmj.38250.571088.55 · Cited by 313 · see citing works
Sifting the evidence—what's wrong with significance tests?
Sterne, J. A. C., & Smith, G. D. (2001) · Physical Therapy · doi:10.1093/ptj/81.8.1464 · Cited by 301 · see citing works
Statistics Notes: The use of transformation when comparing two means
Bland, J. M., & Altman, D. G. (1996) · BMJ · doi:10.1136/bmj.312.7039.1153 · Cited by 270 · see citing works
Blinding during data analysis and writing of manuscripts
Gøtzsche, P. C. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00263-4 · Cited by 123 · see citing works
Self-citationCitation bias of hepato-biliary randomized clinical trials
Gluud, L. L., & Gluud, C. N. (2002) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00513-3 · Cited by 72 · see citing works
Full publication of results initially presented in abstracts
Scherer, R., Langenberg, P., & von Elm, E. (2005) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000005.pub2 · Cited by 41 · see citing works
Is the clinical trial evidence about new drugs statistically adequate?
Bland, J. M., Jones, D. D., Bennett, S., Cook, D. G., Haines, A., & MacFarlane, A. (1985) · British Journal of Clinical Pharmacology · doi:10.1111/j.1365-2125.1985.tb02626.x · Cited by 28 · see citing works
Sample size of randomized double-blind trials 1976-1991.
Mulward, S., & Gøtzsche, P. C. (1996) · PubMed · Cited by 17 · see citing works
Self-citation[Nasal eosinophilia after specific provocation in patients with perennial allergic rhinitis].
Vargas, Y. G. G., González, N., & Cairo-Cueto, S. M. (1997) · PubMed · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Taubes G. Epidemiology faces its limits. Science 1995;269:164-9.
- Smith GD, Ebrahim S. Data dredging, bias, or confounding. BMJ 2002;325:1437-8.
Cited by
Sorted by citations
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164 · see citing works
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648 · see citing works
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578 · see citing works
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349 · see citing works
Self-citationMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883 · see citing works
Self-citationVandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., Egger, M., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040297 · Cited by 8,613 · see citing works
“There is debate about the dangers associated with subgroup analyses, and multiplicity of analyses in general (4, 104).”
Self-citationVandenbroucke, J. P., Elm, E. V., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., Egger, M., & initiative, F. T. S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00010-w1 · Cited by 3,543 · see citing works
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Italian Journal of Public Health · doi:10.2427/5768 · Cited by 3,374 · see citing works
Self-citationVandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., & Egger, M. (2014) · International Journal of Surgery · doi:10.1016/j.ijsu.2014.07.014 · Cited by 2,969 · see citing works
Self-citationVandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., & Egger, M. (2007) · Epidemiology · doi:10.1097/ede.0b013e3181577511 · Cited by 2,779 · see citing works
Self-citationMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2012) · International Journal of Surgery · doi:10.1016/j.ijsu.2011.10.001 · Cited by 2,345 · see citing works
Self-citationHusereau, D., Drummond, M., Petrou, S., Carswell, C., Moher, D., Greenberg, D., Augustovski, F., Briggs, A. H., Mauskopf, J., & Loder, E. (2013) · Value in Health · doi:10.1016/j.jval.2013.02.002 · Cited by 1,998 · see citing works
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.004 · Cited by 1,858 · see citing works
Self-citationBoutron, I. (2010) · JAMA · doi:10.1001/jama.2010.651 · Cited by 842 · see citing works
Hopewell, S., Clarke, M., Moher, D., Wager, E., Middleton, P., Altman, D. G., Schulz, K. F., & and the CONSORT Group (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050020 · Cited by 674 · see citing works
Evolution of Reporting <i>P</i> Values in the Biomedical Literature, 1990-2015
Chavalarias, D., Wallach, J. D., Li, A. H. T., & Ioannidis, J. P. A. (2016) · JAMA · doi:10.1001/jama.2016.1952 · Cited by 370 · see citing works
Yavchitz, A., Boutron, I., Bafeta, A., Marroun, I., Charles, P., Mantz, J., & Ravaud, P. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001308 · Cited by 293 · see citing works
“Readers often base their initial assessment of a trial on the information reported in an abstract conclusion, and in some geographic areas, the abstract of an RCT report may be all that health professionals have easy access to [25,26].”
BackgroundData Extraction Errors in Meta-analyses That Use Standardized Mean Differences
Gøtzsche, P. C., Hróbjartsson, A., Marić, K., & Tendal, B. (2007) · JAMA · doi:10.1001/jama.298.4.430 · Cited by 238 · see citing works
Self-citationModelling the effects of subjective and objective decision making in scientific peer review
Park, I.-U., Peacey, M. W., & Munafò, M. R. (2014) · Nature · doi:10.1038/nature12786 · Cited by 103 · see citing works
Fernandes-Taylor, S., Hyun, J. K., Reeder, R. N., & Harris, A. H. (2011) · BMC Research Notes · doi:10.1186/1756-0500-4-304 · Cited by 75 · see citing works
de Winter, J. C., & Dodou, D. (2015) · PeerJ · doi:10.7717/peerj.733 · Cited by 78 · see citing works
Selective outcome reporting: telling and detecting true lies. The state of the science
Macura, A., Abraha, I., Kirkham, J., Gensini, G. F., Moja, L., & Iorio, A. (2010) · Internal and Emergency Medicine · doi:10.1007/s11739-010-0371-z · Cited by 19 · see citing works
- Authors
- Gøtzsche, P. C.
- Affiliations
- Rigshospitalet (DK)
- Corresponding author
- Gøtzsche, Peter C
- ORCID
- Gøtzsche, Peter C
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2006-07-19
- Volume
- 333
- Issue
- 7561
- Pages
- 231-234
- ISSN
- 0007-1447 (ISSN-L)
- 0959-8138
- 1468-5833
- PMC
- PMC1523498
- OpenAlex
- W2061717662
- Cited by
- 85
- References
- 21
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 1 · 2022: 2 · 2021: 5 · 2020: 1 · 2019: 4 · 2017: 4 · 2016: 2 · 2015: 5 · 2014: 3 · 2013: 3 · 2012: 3
- FWCI
- 12.15
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- BMJ (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- p-values, relative risk, odds ratio, abstracts, publication bias, randomized controlled trials, observational studies, cohort studies, case-control study, statistical significance, p-value distribution
- MeSH terms
- Abstracting and Indexing, Cross-Sectional Studies, Cohort Studies, Odds Ratio, Case-Control Studies, Randomized Controlled Trials as Topic, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
