The DOI matches its Crossref record; 130 of the 135 references it cites were found in a registry or an index, and 1 are not checked yet.
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations
Open accessAbstract
Misinterpretation and abuse of statistical tests, confidence intervals, and statistical power have been decried for decades, yet remain rampant. A key problem is that there are no interpretations of these concepts that are at once simple, intuitive, correct, and foolproof. Instead, correct use and interpretation of these statistics requires an attention to detail which seems to tax the patience of working scientists. This high cognitive demand has led to an epidemic of shortcut definitions and interpretations that are simply wrong, sometimes disastrously so-and yet these misinterpretations dominate much of the scientific literature. In light of this problem, we provide definitions and a discussion of basic statistics that are more general and critical than typically found in traditional introductory expositions. Our goal is to provide a resource for instructors, researchers, and consumers of statistics whose knowledge of statistical theory and technique may be limited but who wish to avoid and spot misinterpretations. We emphasize how violation of often unstated analysis protocols (such as selecting analyses for presentation based on the P values they produce) can lead to small P values even if the declared test hypothesis is correct, and can lead to large P values even if that hypothesis is incorrect. We then provide an explanatory list of 25 misinterpretations of P values, confidence intervals, and power. We conclude with guidelines for improving statistical interpretation and reporting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Major threats that have high prevalence include the misuse and misinterpretation of null-hypothesis testing methods19,20”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [20]
In the citing paper
“The use of statistical methods has been demonstrated to be suboptimal in some fields.”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
The environment and disease: association or causation?
Hill, A. B. (2015) · Journal of the Royal Society of Medicine · doi:10.1177/0141076814562718 · Cited by 7,804
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Sontrop, J. M., Dixon, S. N., Garg, A. X., Jiménez, I. B., Dohein, O., Huang, S. S., & Clark, W. F. (2013) · American Journal of Nephrology · doi:10.1159/000350377 · Cited by 5,903
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Cox, D. R., & Hinkley, D. V. (1974) · doi:10.1007/978-1-4899-2887-0 · Cited by 3,089
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490
Research works printed but not linked: 12
- Ashworth A. Veto on the use of null hypothesis testing and p intervals: right or wrong? Taylor & Francis Editor. 2015. Resources online, http://editorresources.taylorandfrancisgroup.com/veto-on-the-use-of-null-hypothesis-testing-and-p-intervals-right-or-wrong/ . Accessed 27 Feb 2016.
- Flanagan O. Journal’s ban on null hypothesis significance testing: reactions from the statistical arena. 2015. Stats Life online, https://www.statslife.org.uk/opinion/2114-journal-s-ban-on-null-hypothesis-significance-testing-reactions-from-the-statistical-arena . Accessed 27 Feb 2016.
- Cox DR. The role of significance tests (with discussion). Scand J Stat. 1977;4:49–70.
- Freedman DA, Pisani R, Purves R. Statistics. 4th ed. New York: Norton; 2007.
- Hogben L. Statistical theory. London: Allen and Unwin; 1957.
- Ware JH, Mosteller F, Ingelfinger JA. p-Values. In: Bailar JC, Hoaglin DC, editors. Ch. 8. Medical uses of statistics. 3rd ed. Hoboken, NJ: Wiley; 2009. p. 175–94.
- Gigerenzer G. Mindless statistics. J Socioecon. 2004;33:567–606.
- Greenland S, Poole C. Problems in common interpretations of statistics in scientific articles, expert reports, and testimony. Jurimetrics. 2011;51:113–29.
- Kaye DH. Is proof of statistical significance relevant? Wash Law Rev. 1986;61:1333–66.
- Royall R. Statistical evidence. New York: Chapman and Hall; 1997.
- Greenland S, O’Rourke K. Meta-analysis Ch. 33. In: Rothman KJ, Greenland S, Lash TL, editors. Modern epidemiology. 3rd ed. Philadelphia: Lippincott-Wolters-Kluwer; 2008. p. 682–5.
- Sterne JAC. Meta-analysis: an updated collection from the Stata journal. College Station, TX: Stata Press; 2009.
Cited by
Sorted by citations
Sensitivity analysis in observational research: Introducing the E-value
VanderWeele, T. J., & Ding, P. (2017) · Annals of Internal Medicine · doi:10.7326/m16-2607 · Cited by 6,350
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications
Wagenmakers, E., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., Gronau, Q. F., Šmíra, M., Epskamp, S., et al. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1343-3 · Cited by 1,767
Kruschke, J. K., & Liddell, T. M. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1221-4 · Cited by 1,288
Lederer, D. J., Bell, S. C., Branson, R. D., Chalmers, J. D., Marshall, R., Maslove, D. M., Ost, D. E., Punjabi, N. M., Schatz, M., Smyth, A. R., et al. (2019) · Annals of the American Thoracic Society · doi:10.1513/annalsats.201808-564ps · Cited by 715
Metadata
- Authors
- Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G.
- Affiliations
- University of California, Los Angeles (US)
- Luxembourg Institute of Health (LU)
- RTI International (US)
- The University of Melbourne (AU)
- University of North Carolina at Chapel Hill (US)
- Stanford Medicine (US)
- University of Oxford (GB)
- Corresponding author
- Greenland, Sander
- ORCID
- Venue
- European Journal of Epidemiology
- Published
- 2016-04
- Volume
- 31
- Issue
- 4
- Pages
- 337-350
- ISSN
- 0393-2990 (ISSN-L)
- 1573-7284
- PMC
- PMC4877414
- OpenAlex
- W2408017125
- Cited by
- 3,200
- References
- 135
- Citations by year
- 2026: 325 · 2025: 454 · 2024: 407 · 2023: 290 · 2022: 348 · 2021: 392 · 2020: 330 · 2019: 270 · 2018: 201 · 2017: 148 · 2016: 33
- FWCI
- 220.87
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- European Journal of Epidemiology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- Funders
- Seventh Framework Programme (602552)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- p-values, confidence intervals, statistical power, statistical misinterpretation, hypothesis testing, selective analysis, analysis protocol, statistical reporting guidelines, statistics education
- MeSH terms
- Data Interpretation, Statistical, Humans, Probability, Confidence Intervals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
