CoChat
Sign Up
Cited by 1,466Statistical Methods in EpidemiologyMeta-analysis and systematic reviewsStatistical Methods in Clinical Trials

Sifting the evidence---what's wrong with significance tests? Another comment on the role of statistical methods

Open access
Sterne, J. A C·2001·BMJ
DOI 10.1136/bmj.322.7280.226PMC PMC1119478
Open this paper in CoChat — free account

Abstract

The findings of medical research are often met with considerable scepticism, even when they have apparently come from studies with sound methodologies that have been subjected to appropriate statistical analysis. This is perhaps particularly the case with respect to epidemiological findings that suggest that some aspect of everyday life is bad for people. Indeed, one recent popular history, the medical journalist James Le Fanu's The Rise and Fall of Modern Medicine , went so far as to suggest that the solution to medicine's ills would be the closure of all departments of epidemiology.1 One contributory factor is that the medical literature shows a strong tendency to accentuate the positive; positive outcomes are more likely to be reported than null results.2–4 By this means alone a host of purely chance findings will be published, as by conventional reasoning examining 20 associations will produce one result that is “significant at P=0.05” by chance alone. If only positive findings are published then they may be mistakenly considered to be of importance rather than being the necessary chance results produced by the application of criteria for meaningfulness based on statistical significance. As many studies contain long questionnaires collecting information on hundreds of variables, and measure a wide range of potential outcomes, several false positive findings are virtually guaranteed. The high volume and often contradictory nature5 of medical research findings, however, is not only because of publication bias. A more fundamental problem is the widespread misunderstanding of the nature of statistical significance. #### Summary points P values, or significance levels, measure the strength of the evidence against the null hypothesis; the smaller the P value, the stronger the evidence against the null hypothesis An arbitrary division of results, into “significant” or “non-significant” according to the P value, was not the intention of the …

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmj.322.7280.226

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • If studies in a field have 20% power, then out of 100 genuine non-null effects, they are expected to discover only 20.

    “When studies in a given field are designed with a power of 20%, it means that if there are 100 genuine non-null effects to be discovered in that field, these studies are expected to discover only 20 of them 11 .”

    From Power failure: why small sample size undermines the reliability of neuroscience (2013) · cites this paper as [11]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

2 of the 48 references couldn’t be matched to a known paper.

Research works printed but not linked: 5

  1. [38]Davey Smith G. Reflections on the limits to epidemiology. J Clin Epidemiol (in press).
  2. BMJ. 2001 Jan 27;322(7280):226–231.
  3. 1-1.Fisher RA. Statistical methods for research workers. 1st ed. Edinburgh: Oliver and Boyd; 1925. . Reprinted by Oxford University Press. [Google Scholar]
  4. 1-2.Fisher RA. Design of experiments. 1st ed. Edinburgh: Oliver and Boyd; 1935. . Reprinted by Oxford University Press. [Google Scholar]
  5. 1-4.Cox DR. Statistical significance tests. Br J Clin Pharmacol. 1982;14:325–331. doi: 10.1111/j.1365-2125.1982.tb01987.x. [DOI] [PMC free article] [PubMed] [Google Scholar] · DOI

Cited by

Sorted by citations

Related

Authors
Sterne, J. A. C.
Affiliations
University of Bristol (GB)
Corresponding author
Sterne, J. A C
ORCID
Sterne, J. A C
Venue
BMJ
Publisher
BMJ
Published
2001-01-27
Volume
322
Issue
7280
Pages
226-231
ISSN
0959-8138 (ISSN-L)
DOI
10.1136/bmj.322.7280.226
PMC
PMC1119478
OpenAlex
W2025651303
Cited by
1,466
References
48
Citations by year
2026: 48 · 2025: 64 · 2024: 75 · 2023: 63 · 2022: 63 · 2021: 82 · 2020: 72 · 2019: 82 · 2018: 75 · 2017: 79 · 2016: 74 · 2015: 48 · 2014: 56 · 2013: 50 · 2012: 53
FWCI
45.33
Type
journal article
Language
en
Open access
Locations
  • BMJ (publishedVersion)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • HighWire Press Open Archive (submittedVersion)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords
statistical significance, p-values, publication bias, false positives, null hypothesis, evidence evaluation
MeSH terms
Bayes Theorem, Data Interpretation, Statistical, Humans, Research Design, Statistics as Topic, History, 20th Century

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review