The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
The reign of the p -value is over: what alternative analyses could we employ to fill the power vacuum?
Open accessAbstract
The p -value has long been the figurehead of statistical analysis in biology, but its position is under threat. p is now widely recognized as providing quite limited information about our data, and as being easily misinterpreted. Many biologists are aware of p 's frailties, but less clear about how they might change the way they analyse their data in response. This article highlights and summarizes four broad statistical approaches that augment or replace the p -value, and that are relatively straightforward to apply. First, you can augment your p -value with information about how confident you are in it, how likely it is that you will get a similar p -value in a replicate study, or the probability that a statistically significant finding is in fact a false positive. Second, you can enhance the information provided by frequentist statistics with a focus on effect sizes and a quantified confidence that those effect sizes are accurate. Third, you can augment or substitute p -values with the Bayes factor to inform on the relative levels of evidence for the null and alternative hypotheses; this approach is particularly appropriate for studies where you wish to keep collecting data until clear evidence for or against your hypothesis has accrued. Finally, specifically where you are using multiple variables to predict an outcome through model building, Akaike information criteria can take the place of the p -value, providing quantified information on what model is best. Hopefully, this quick-and-easy guide to some simple yet powerful statistical options will support biologists in adopting new approaches where they feel that the p -value alone is not doing their data justice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
The ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306
Conjectures and Refutations: The Growth of Scientific Knowledge.
Hutchison, T. W., & Popper, K. R. (1964) · Economica · doi:10.2307/2551371 · Cited by 4,197
Burnham, K. P., Anderson, D. R., & Huyvaert, K. P. (2011) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-010-1029-6 · Cited by 4,104
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Effect size, confidence interval and statistical significance: a practical guide for biologists
Nakagawa, S., & Cuthill, I. C. (2007) · Biological Reviews · doi:10.1111/j.1469-185x.2007.00027.x · Cited by 3,835
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Scientists rise up against statistical significance
Amrhein, V., Greenland, S., & McShane, B. (2019) · Nature · doi:10.1038/d41586-019-00857-9 · Cited by 2,983
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Understanding predictive information criteria for Bayesian models
Gelman, A., Hwang, J., & Vehtari, A. (2014) · Statistics and Computing · doi:10.1007/s11222-013-9416-2 · Cited by 2,104
Things I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781
Moving beyond P values: data analysis with estimation graphics
Ho, J., Tumkaya, T., Aryal, S., Choi, H., & Claridge-Chang, A. (2019) · Nature Methods · doi:10.1038/s41592-019-0470-3 · Cited by 1,758
Scientific method: Statistical errors
Nuzzo, R. (2014) · Nature · doi:10.1038/506150a · Cited by 1,536
The Insignificance of Statistical Significance Testing
Johnson, D. H. (1999) · The Journal of Wildlife Management · doi:10.2307/3802789 · Cited by 1,104
Error bars in experimental biology
Cumming, G., Fidler, F., & Vaux, D. L. (2007) · The Journal of Cell Biology · doi:10.1083/jcb.200611141 · Cited by 940
Kullback-Leibler information as a basis for strong inference in ecological studies
Burnham, K. P., & Anderson, D. R. (2001) · Wildlife Research · doi:10.1071/wr99107 · Cited by 926
The test of significance in psychological research.
Bakan, D. (1966) · Psychological Bulletin · doi:10.1037/h0020412 · Cited by 911
An investigation of the false discovery rate and the misinterpretation of p -values
Colquhoun, D. (2014) · Royal Society Open Science · doi:10.1098/rsos.140216 · Cited by 792
The Philosophy of Multiple Comparisons
Tukey, J. W. (1991) · Statistical Science · doi:10.1214/ss/1177011945 · Cited by 773
The fickle P value generates irreproducible results
Halsey, L. G., Curran-Everett, D., Vowler, S. L., & Drummond, G. B. (2015) · Nature Methods · doi:10.1038/nmeth.3288 · Cited by 765
Self-citationUsing confidence intervals for graphically based data interpretation.
Masson, M. E. J., & Loftus, G. R. (2003) · Canadian Journal of Experimental Psychology / Revue canadienne de psychologie expérimentale · doi:10.1037/h0087426 · Cited by 730
Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.
Schönbrodt, F. D., Wagenmakers, E.-J., Zehetleitner, M., & Perugini, M. (2017) · Psychological Methods · doi:10.1037/met0000061 · Cited by 567
How meta-analysis increases statistical power.
Cohn, L. D., & Becker, B. J. (2003) · Psychological Methods · doi:10.1037/1082-989x.8.3.243 · Cited by 498
Cumming, G. (2008) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2008.00079.x · Cited by 451
Stepwise Model Fitting and Statistical Inference: Turning Noise into Signal Pollution
Mundry, R., & Nunn, C. L. (2009) · The American Naturalist · doi:10.1086/593303 · Cited by 440
The importance of proving the null
Gallistel, C. R. (2009) · Psychological Review · doi:10.1037/a0015251 · Cited by 392
Akaike Information Criterion Statistics.
Parzen, E., Sakamoto, Y., Ishiguro, M., Kitagawa, G., & Reidel, D. (1988) · Journal of the American Statistical Association · doi:10.2307/2289329 · Cited by 392
Statistical Methods and Scientific Inference
Snell, J. L., & Fisher, S. R. (1959) · Econometrica · doi:10.2307/1907790 · Cited by 336
Murtaugh, P. A. (2014) · Ecology · doi:10.1890/13-0590.1 · Cited by 317
Analyzing data: Sanctification or detective work?
Tukey, J. W. (1969) · American Psychologist · doi:10.1037/h0027108 · Cited by 313
The reproducibility of research and the misinterpretation of p -values
Colquhoun, D. (2017) · Royal Society Open Science · doi:10.1098/rsos.171085 · Cited by 256
Considering aspects of the 3Rs principles within experimental animal biology
Sneddon, L. U., Halsey, L. G., & Bury, N. R. (2017) · Journal of Experimental Biology · doi:10.1242/jeb.147058 · Cited by 254
Self-citation(2016) · Cited by 252
Bzdok, D., Krzywinski, M., & Altman, N. (2017) · Nature Methods · doi:10.1038/nmeth.4526 · Cited by 246
O'Keefe, D. J. (2007) · Communication Methods and Measures · doi:10.1080/19312450701641375 · Cited by 245
Introduction to the New Statistics
Cumming, G., & Calin-Jageman, R. (2016) · Routledge eBooks · doi:10.4324/9781315708607 · Cited by 201
Tests of Significance Considered as Evidence
Berkson, J. (1942) · Journal of the American Statistical Association · doi:10.2307/2279000 · Cited by 174
Loftus, G. R. (1993) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03204506 · Cited by 168
P-Value Precision and Reproducibility
Boos, D. D., & Stefanski, L. A. (2011) · The American Statistician · doi:10.1198/tas.2011.10129 · Cited by 166
The New Statistics for Better Science: Ask How Much, How Uncertain, and What Else Is Known
Calin-Jageman, R. J., & Cumming, G. (2019) · The American Statistician · doi:10.1080/00031305.2018.1518266 · Cited by 166
Of P-Values and Bayes: A Modest Proposal
Goodman, S. N. (2001) · Epidemiology · doi:10.1097/00001648-200105000-00006 · Cited by 164
Lane, A., Luminet, O., Nave, G., & Mikolajczak, M. (2016) · Journal of Neuroendocrinology · doi:10.1111/jne.12384 · Cited by 129
Sena, E. S., Briscoe, C. L., Howells, D. W., Donnan, G. A., Sandercock, P. A., & Macleod, M. R. (2010) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.2010.116 · Cited by 110
Krzywinski, M. I., & Altman, N. (2014) · Nature Methods · doi:10.1038/nmeth.2858 · Cited by 90
Krzywinski, M. I., & Altman, N. (2014) · Nature Methods · doi:10.1038/nmeth.2900 · Cited by 86
Altman, N., & Krzywinski, M. I. (2017) · Nature Methods · doi:10.1038/nmeth.4210 · Cited by 85
On the sensitivity of bayes factors to the prior distributions
Sinharay, S., & Stern, H. S. (2002) · The American Statistician · doi:10.1198/000313002137 · Cited by 72
A Proposed Hybrid Effect Size Plus p -Value Criterion: Empirical Evidence Supporting its Use
Goodman, W. M., Spruill, S. E., & Komaroff, E. (2019) · The American Statistician · doi:10.1080/00031305.2018.1564697 · Cited by 64
Lew, M. J. (2012) · British Journal of Pharmacology · doi:10.1111/j.1476-5381.2012.01931.x · Cited by 55
Recurring controversies aboutPvalues and confidence intervals revisited
Spanos, A. (2014) · Ecology · doi:10.1890/13-1291.1 · Cited by 47
Show the data, don't conceal them
Drummond, G. B., & Vowler, S. L. (2011) · The Journal of Physiology · doi:10.1113/jphysiol.2011.205062 · Cited by 44
P values, hypothesis testing, and model selection: it's déjà vu all over again1
Ellison, A. M., Gotelli, N. J., Inouye, B. D., & Strong, D. R. (2014) · Ecology · doi:10.1890/13-1911.1 · Cited by 41
Solutions for quantifying P-value uncertainty and replication power
Lazzeroni, L. C., Lü, Y., & Belitskaya‐Lévy, I. (2016) · Nature Methods · doi:10.1038/nmeth.3741 · Cited by 34
Model Selection, Hypothesis Testing, and Risks of Condemning Analytical Tools
Steidl, R. J. (2006) · Journal of Wildlife Management · doi:10.2193/0022-541x(2006)70[1497:mshtar]2.0.co;2 · Cited by 33
State of the Evidence: Current Status and Prospects of Meta-Analysisin Infectious Diseases
Ioannidis, J. P. A., & Lau, J. T. F. (1999) · Clinical Infectious Diseases · doi:10.1086/313443 · Cited by 26
To P or not to P: on the evidential nature of P-values and their place in scientific inferencePreprint
Lew, M. J. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1311.0081 · Cited by 16
Murtaugh, P. A. (2014) · Ecology · doi:10.1890/13-1858.1 · Cited by 10
Bayesian prediction intervals for assessing P-value variability in prospective replication studies
Vsevolozhskaya, O. A., Ruiz, G., & Zaykin, D. V. (2017) · Translational Psychiatry · doi:10.1038/s41398-017-0024-3 · Cited by 5
Cited by
Indices of Effect Existence and Significance in the Bayesian Framework
Makowski, D., Ben-Shachar, M. S., Chen, S. H. A., & Lüdecke, D. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02767 · Cited by 983
Rewriting results sections in the language of evidence
Muff, S., Nilsen, E. B., O’Hara, R. B., & Nater, C. R. (2022) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2021.10.009 · Cited by 636
Big-Data Science in Porous Materials: Materials Genomics and Machine Learning
Jablonka, K. M., Ongari, D., Moosavi, S. M., & Smit, B. (2020) · Chemical Reviews · doi:10.1021/acs.chemrev.0c00004 · Cited by 599
Pabst, A., Proksch, S., Médé, B., Comstock, D. C., Ross, J. M., & Balasubramaniam, R. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104587 · Cited by 68
Loss, C. M., Melleu, F. F., Domingues, K., Lino-de-Oliveira, C., & Viola, G. G. (2021) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2021.763428 · Cited by 44
Metadata
- Authors
- Halsey, L. G.
- Venue
- Biology Letters
- Publisher
- The Royal Society
- Published
- 2019-05-22
- Volume
- 15
- Issue
- 5
- Pages
- 20190174
- ISSN
- 1744-9561 (ISSN-L)
- 1744-957X
- OpenAlex
- W2945098413
- Cited by
- 357
- References
- 58
- FWCI
- 29.83
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.