The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Sifting the evidence—what's wrong with significance tests?
Abstract
The findings of medical research are often met with considerable scepticism, even when they have apparently come from studies with sound methodologies that have been subjected to appropriate statistical analysis. This is perhaps particularly the case with respect to epidemiological findings that suggest that some aspect of everyday life is bad for people. Indeed, one recent popular history, the medical journalist James Le Fanu's The Rise and Fall of Modern Medicine, went so far as to suggest that the solution to medicine's ills would be the closure of all departments of epidemiology.1 One contributory factor is that the medical literature shows a strong tendency to accentuate the positive; positive outcomes are more likely to be reported than null results.2–4 By this means alone a host of purely chance findings will be published, as by conventional reasoning examining 20 associations will produce one result that is “significant at P = 0.05” by chance alone. If only positive findings are published then they may be mistakenly considered to be of importance rather than being the necessary chance results produced by the application of criteria for meaningfulness based on statistical significance. As many studies contain long questionnaires collecting information on hundreds of variables, and measure a wide range of potential outcomes, several false positive findings are virtually guaranteed. The high volume and often contradictory nature5 of medical research findings, however, is not only because of publication bias. A more fundamental problem is the widespread misunderstanding of the nature of statistical significance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
Casella, G., & Berger, R. (2024) · Chapman & Hall/CRC eBooks · doi:10.1201/9781003456285 · Cited by 5,711
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
IX. On the problem of the most efficient tests of statistical hypotheses
Neyman, J., & Pearson, E. S. (1933) · Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character · doi:10.1098/rsta.1933.0009 · Cited by 2,341
Confidence intervals rather than P values: estimation rather than hypothesis testing.
Gardner, M. J., & Altman, D. G. (1986) · BMJ · doi:10.1136/bmj.292.6522.746 · Cited by 2,010
Bayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734
Freiman, J. A., Chalmers, T. C., Smith, H., & Kuebler, R. R. (1978) · New England Journal of Medicine · doi:10.1056/nejm197809282991304 · Cited by 1,628
ON THE PROBLEM OF THE MOST EFFICIENT TESTS OF STATISTICAL HYPOTHESES
(1967) · Joint Statistical Papers · doi:10.1525/9780520339897-008 · Cited by 1,504
Statistics with Confidence: Confidence Intervals and Statistical Guidelines.
Jolliffe, F. R., Gardner, M. J., & Altman, D. G. (1990) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982808 · Cited by 1,319
Statistical Methods and Scientific Inference.
Bartlett, M. S., & Fisher, R. A. (1957) · Biometrika · doi:10.2307/2333276 · Cited by 1,278
Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy
Goodman, S. N. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-12-199906150-00008 · Cited by 1,185
Statistical Methods for Research Workers
Tukey, J., & Fisher, R. A. (1952) · Econometrica · doi:10.2307/1907425 · Cited by 949
Toward Evidence-Based Medical Statistics. 2: The Bayes Factor
Goodman, S. N. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-12-199906150-00019 · Cited by 912
Dickersin, K. (1992) · JAMA · doi:10.1001/jama.1992.03480030052036 · Cited by 854
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478397 · Cited by 775
Statistical Methods and Scientific Inference.
Bailey, N. T. J., & Fisher, R. (1957) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2342557 · Cited by 760
The fallacy of the null-hypothesis significance test.
Rozeboom, W. W. (1960) · Psychological Bulletin · doi:10.1037/h0042040 · Cited by 729
Why do we need some large, simple randomized trials?
Yusuf, S., Collins, R., & Peto, R. (1984) · Statistics in Medicine · doi:10.1002/sim.4780030421 · Cited by 699
Statistical guidelines for contributors to medical journals.
Altman, D. G., Gore, S. M., Gardner, M. J., & Pocock, S. J. (1983) · BMJ · doi:10.1136/bmj.286.6376.1489 · Cited by 661
Meta-analysis: Potentials and promise
Egger, M., & Smith, G. D. (1997) · BMJ · doi:10.1136/bmj.315.7119.1371 · Cited by 646
Self-citationStatistical Power, Sample Size, and Their Reporting in Randomized Controlled Trials
Moher, D. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03520020048013 · Cited by 608
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.2307/2289131 · Cited by 548
H., F. Z. (1932) · Agronomy Journal · doi:10.2134/agronj1932.00021962002400120011x · Cited by 515
The Fisher, Neyman-Pearson Theories of Testing Hypotheses: One Theory or Two?
Lehmann, E. L. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10476404 · Cited by 382
Statistical significance tests.
Cox, D. (1982) · British Journal of Clinical Pharmacology · doi:10.1111/j.1365-2125.1982.tb01987.x · Cited by 334
Goodman, S. N. (1993) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a116700 · Cited by 303
Rise and fall of modern medicine
Le Fanu, J. (1999) · The Lancet · doi:10.1016/s0140-6736(05)75559-8 · Cited by 296
Cornell, R. G., Wolf, F. M., Bailar, J. C., & Mosteller, F. (1989) · Journal of the American Statistical Association · doi:10.2307/2289981 · Cited by 295
The Fisher, Neyman-Peerson Theories of Testing Hypotheses: One Theory or Two?
Lehmann, E. L. (2012) · Selected Works of E. L. Lehmann · doi:10.1007/978-1-4614-1412-4_19 · Cited by 220
Browner, W. S. (1987) · JAMA · doi:10.1001/jama.257.18.2459 · Cited by 212
The Rise and Fall of Modern Medicine
(2000) · British Journal of Ophthalmology · doi:10.1136/bjo.84.5.554f · Cited by 189
For Debate: The statistical basis of public policy: a paradigm shift is overdue
Lilford, R. J., & Braunholtz, D. (1996) · BMJ · doi:10.1136/bmj.313.7057.603 · Cited by 188
Confounding in epidemiological studies: why "independent" effects may not be all they seem.
Smith, G. D., & Phillips, A. N. (1992) · BMJ · doi:10.1136/bmj.305.6856.757 · Cited by 186
Self-citationStatistics with Confidence. Confidence Intervals and Statistical Guidelines
Newcombe, R. (1990) · Journal of Medical Genetics · doi:10.1136/jmg.27.1.71 · Cited by 181
Tests of Significance Considered as Evidence
Berkson, J. (1942) · Journal of the American Statistical Association · doi:10.2307/2279000 · Cited by 174
Placing Trials in Context Using Bayesian Analysis
Brophy, J. M. (1995) · JAMA · doi:10.1001/jama.1995.03520350053028 · Cited by 154
Yates, F. (1951) · Journal of the American Statistical Association · doi:10.1080/01621459.1951.10500764 · Cited by 154
Rothman, K. J. (1986) · Annals of Internal Medicine · doi:10.7326/0003-4819-105-3-445 · Cited by 145
An Assessment of Publication Bias Using a Sample of Published Clinical Trials
Berlin, J. A., Begg, C. B., & Louis, T. A. (1989) · Journal of the American Statistical Association · doi:10.1080/01621459.1989.10478782 · Cited by 137
Clinical significance not statistical significance: a simple Bayesian alternative to p values.
Burton, P. R., Gurrin, L. C., & Campbell, M. J. (1998) · Journal of Epidemiology & Community Health · doi:10.1136/jech.52.5.318 · Cited by 112
P-Values and Confidence Intervals: Two Sides of the Same Unsatisfactory Coin
Feinstein, A. R. (1998) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(97)00295-3 · Cited by 98
A Collection of 56 Topics with Contradictory Results in Case-Control Research
Mayes, L. C., Horwitz, R. I., & FHNSTEIN, A. R. (1988) · International Journal of Epidemiology · doi:10.1093/ije/17.3.680 · Cited by 88
Identification and Relative Weight of Cardiovascular Risk Factors
Hopkins, P. N., & Williams, R. R. (1986) · Cardiology Clinics · doi:10.1016/s0733-8651(18)30632-5 · Cited by 83
The design of prospective epidemiological studies: More subjects or better measurements?
Phillips, A. N., & Smith, G. D. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90120-p · Cited by 68
Self-citationKiefer, J. (1985) · Springer eBooks · doi:10.1007/978-1-4613-8505-9_39 · Cited by 65
Freiman, J. A., Chalmers, T. C., Smith, H., & Kuebler, R. R. (2019) · CRC Press eBooks · doi:10.1201/9780429187445-19 · Cited by 45
The Hypothesis Generating Machine
Cole, P. (1993) · Epidemiology · doi:10.1097/00001648-199305000-00012 · Cited by 31
Detection and Surveillance of Colorectal Cancer-Reply
Goldberg, S. B. (1990) · JAMA · doi:10.1001/jama.1990.03440030057018 · Cited by 21
Hetzel, B. S. (1985) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.1985.tb133037.x · Cited by 20
Sample size of randomized double-blind trials 1976-1991.
Mulward, S., & Gøtzsche, P. C. (1996) · PubMed · Cited by 17
Cited by
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
The ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306
Sensitivity analysis in observational research: Introducing the E-value
VanderWeele, T. J., & Ding, P. (2017) · Annals of Internal Medicine · doi:10.7326/m16-2607 · Cited by 6,350
Making Meaningful Inferences About Magnitudes
Batterham, A. M., & Hopkins, W. G. (2006) · International Journal of Sports Physiology and Performance · doi:10.1123/ijspp.1.1.50 · Cited by 2,183
Theodoratou, E., Tzoulaki, I., Zgaga, L., & Ioannidis, J. P. A. (2014) · BMJ · doi:10.1136/bmj.g2035 · Cited by 956
Metadata
- Authors
- Sterne, J. A. C., & Smith, G. D.
- Venue
- Physical Therapy
- Publisher
- Oxford University Press (OUP)
- Published
- 2001-08-01
- Volume
- 81
- Issue
- 8
- Pages
- 1464-1469
- ISSN
- 0031-9023 (ISSN-L)
- 1538-6724
- OpenAlex
- W2176759697
- Cited by
- 301
- References
- 51
- FWCI
- 4.09
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.