A reckless guide to P-values: local evidence, global errors
Open accessAbstract
This chapter demystifies P-values, hypothesis tests and significance tests, and introduces the concepts of local evidence and global error rates. The local evidence is embodied in \textit{this} data and concerns the hypotheses of interest for \textit{this} experiment, whereas the global error rate is a property of the statistical analysis and sampling procedure. It is shown using simple examples that local evidence and global error rates can be, and should be, considered together when making inferences. Power analysis for experimental design for hypothesis testing are explained, along with the more locally focussed expected P-values. Issues relating to multiple testing, HARKing, and P-hacking are explained, and it is shown that, in many situation, their effects on local evidence and global error rates are in conflict, a conflict that can always be overcome by a fresh dataset from replication of key experiments. Statistics is complicated, and so is science. There is no singular right way to do either, and universally acceptable compromises may not exist. Statistics offers a wide array of tools for assisting with scientific inference by calibrating uncertainty, but statistical inference is not a substitute for scientific inference. P-values are useful indices of evidence and deserve their place in the statistical toolbox of basic pharmacologists.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
If a researcher only makes claims about specific colors of jelly bean, they are not testing a joint hypothesis, and an unadjusted conventional alpha level of .050 is appropriate in each case.
“If the researcher only makes claims about specific colors of jelly bean (e.g., 'eating green jelly beans is linked to acne, p = . 023 '), then they are not testing a joint hypothesis, and an unadjusted conventional alpha level of .050 is appropriate in each case (Lew, 2019, pp. 21-22).”
From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Lew, 2019
In §4.2 (p. 21), Lew distinguishes a color-specific, local null hypothesis from the omnibus null addressed by a Bonferroni adjustment. But he does not endorse an unadjusted α = .05 cutoff as appropriate for each claim: §4.2 (p. 22) stresses that the elevated false-positive risk remains real, and §3.2 (p. 11) cautions against fixed significance thresholds. (LLM, conf=0.89)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Part of this paper’s reference list could not be split into single entries.
The ASA Statement on <i>p</i> -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · cited 4× · see citing works
“In response to the widespread expressions of concern over the misuse and misunderstanding of P-values, the ASA convened a group of experts to consider the issues and to collaborate on drafting an official statement on P-values [Wasserstein and Lazar, 2016].”
BackgroundMethodInfluentialRaise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 2× · see citing works
“…would indeed help to minimise the number of times that pharmacologists make fools of themselves by reducing the number of false positive results in pharmacological journals and, consequently, reduce the number of faulty leads that fail to translate into a therapeutic [Begley and Ellis, 2012].”
BackgroundCumming, G. (2008) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2008.00079.x · Cited by 451 · see citing works
Questionable research practices in ecology and evolution
Fraser, H., Parker, T., Nakagawa, S., Barnett, A., & Fidler, F. (2018) · PLOS ONE · doi:10.1371/journal.pone.0200303 · Cited by 346 · see citing works
When should we use one-tailed hypothesis testing?
Ruxton, G. D., & Neuhäuser, M. (2010) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2010.00014.x · Cited by 269 · see citing works
McCullagh, P. (2002) · The Annals of Statistics · doi:10.1214/aos/1035844977 · Cited by 248 · cited 2× · see citing works
“tion of errors in his observations, and about facts that are assumed to be known for the purposes of statistical calculations. |[Colquhoun, 1971, p. v] Statistical models can take a variety of forms [McCullagh, 2002], but the model for the familiar Student’s t-test for independent samples is reasonably representative.…”
Background
Flagged first, then by citations
Research works printed but not linked: 2
- [15]15. Student. The Probable Error of a Mean. Biometrika , 6(1):1-25, Mar. 1908. - B. Thompson. The nature of statistical evidence , volume 189 of Lecture notes in statistics . Springer, 2007. - D. Trafimow and M. Marks. Editorial. Basic and Applied Social Psychology , 37(1):1-2, 2015. doi: 10.1080/01973533.2015.1012991. URL https://doi. org/10.1080/01973533.2015.1012991 . - J. W. Tukey. The Philosophy of Multiple Comparisons. Statistical Science , 6 (1):100-116, Feb. 1991. - B. Voelkl, L. Vogt, E. S. Sena, and H. W¨ urbel. Reproducibility of preclinical animal research improves with heterogeneity of study samples. PLOS Biology , 16(2):e2003693-13, Feb. 2018. · DOI
- [4]4. E.-J. Wagenmakers. A practical solution to the pervasive problems of p values. Psychonomic bulletin & review , 14(5):779-804, Oct. 2007.
Cited by
Sorted by citations
Rubin, M. (2021) · Synthese · doi:10.1007/s11229-021-03276-4 · Cited by 371 · see citing works
Does preregistration improve the credibility of research findings?
Rubin, M. (2020) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.16.4.p376 · Cited by 61 · see citing works
“If the researcher only makes claims about specific colors of jelly bean (e.g., “eating green jelly beans is linked to acne, p = .023”), then they are not testing a joint hypothesis, and an unadjusted conventional alpha level of .050 is appropriate in each case (Lew, 2019, pp. 21-22).”
Background
- Authors
- Lew, M. J.
- Corresponding author
- Lew, Michael J.
- ORCID
- Lew, Michael J.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1910.02042
- OpenAlex
- W2978499195
- Cited by
- 2
- References
- 6
- Citations by year
- 2021: 1 · 2020: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- p-values, local evidence, hypothesis testing, significance testing, power analysis, multiple comparisons, HARKing, p-hacking, replication, experimental design, statistical inference, basic pharmacology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
