Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn
Open accessAbstract
Permutation tests are amongst the most commonly used statistical tools in modern genomic research, a process by which p-values are attached to a test statistic by randomly permuting the sample or gene labels. Yet permutation p-values published in the genomic literature are often computed incorrectly, understated by about 1/m, where m is the number of permutations. The same is often true in the more general situation when Monte Carlo simulation is used to assign p-values. Although the p-value understatement is usually small in absolute terms, the implications can be serious in a multiple testing context. The understatement arises from the intuitive but mistaken idea of using permutation to estimate the tail probability of the test statistic. We argue instead that permutation should be viewed as generating an exact discrete null distribution. The relevant literature, some of which is likely to have been relatively inaccessible to the genomic community, is reviewed and summarized. A computation strategy is developed for exact p-values when permutations are randomly drawn. The strategy is valid for any number of permutations and samples. Some simple recommendations are made for the implementation of permutation tests in practice.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The permutation p-value is super-uniform under H0, giving finite-sample Type-I control for calibrated scores.
“The permutation p-value in Equation ( 10) is super-uniform under H 0 (i.e., P H0 (p ≤ α) ≤ α for all α ∈ [0, 1]), a standard consequence of randomization inference (Nichols & Holmes, 2002;Phipson & Smyth, 2010;Good, 2005)”
From Revisiting the Platonic Representation Hypothesis: An Aristotelian View (2026) · cites this paper as Phipson & Smyth, 2010
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Jackknife, the Bootstrap and Other Resampling Plans
Efron, B. (1982) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611970319 · Cited by 8,120 · see citing works
Bootstrap Methods and their Application
Davison, A. C., & Hinkley, D. V. (1997) · doi:10.1017/cbo9780511802843 · Cited by 7,350 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- Edgington, E.S. and Onghena, P. (2007) Randomization tests , Fourth Edition, Chapman & Hall/CRC, Boca Raton, FL.
- Fisher, R.A. (1935) The Design of Experiments , 3rd Edition, Oliver & Boyd, London.
- Good, P. (1994) Permutation tests: a practical guide to resampling methods for testing hypotheses , Springer-Verlag New York, New York.
- Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics , 6 , 65-70.
- Manly, B.F.J. (1997) Randomization, bootstrap and Monte Carlo methods in biology , Second Edition, Chapman & Hall, London.
Cited by
Sorted by citations
Permutation inference for the general linear model
Winkler, A. M., Ridgway, G. R., Webster, M. A., Smith, S. M., & Nichols, T. E. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.01.060 · Cited by 3,563 · see citing works
Functional and topological characteristics of mammalian regulatory domains
Symmons, O., Uslu, V. V., Tsujimura, T., Ruf, S., Nassari, S., Schwarzer, W., Ettwiller, L., & Spitz, F. (2014) · Genome Research · doi:10.1101/gr.163519.113 · Cited by 455 · see citing works
Batch effects and the effective design of single-cell gene expression studies
Tung, P.-Y., Blischak, J. D., Hsiao, C. J., Knowles, D. A., Burnett, J. E., Pritchard, J. K., & Gilad, Y. (2017) · Scientific Reports · doi:10.1038/srep39921 · Cited by 356 · see citing works
Cell type prioritization in single-cell data
Skinnider, M. A., Squair, J. W., Kathe, C., Anderson, M. A., Gautier, M., Matson, K. J. E., Milano, M., Hutson, T. H., Barraud, Q., Phillips, A. A., et al. (2021) · Nature Biotechnology · doi:10.1038/s41587-020-0605-1 · Cited by 274 · see citing works
A visual pathway for skylight polarization processing in Drosophila
Hardcastle, B. J., Omoto, J. J., Kandimalla, P., Nguyen, B.-C. M., Keleş, M. F., Boyd, N. K., Hartenstein, V., & Frye, M. A. (2021) · eLife · doi:10.7554/elife.63225 · Cited by 111 · see citing works
Prioritization of cell types responsive to biological perturbations in single-cell data with Augur
Squair, J. W., Skinnider, M. A., Gautier, M., Foster, L. J., & Courtine, G. (2021) · Nature Protocols · doi:10.1038/s41596-021-00561-x · Cited by 61 · see citing works
Enrichment pathway analysis. The inflammatory genetic background in Bipolar Disorder
Drago, A., Crisafulli, C., Calabrò, M., & Serretti, A. (2015) · Journal of Affective Disorders · doi:10.1016/j.jad.2015.03.032 · Cited by 47 · see citing works
Testing conditional independence in supervised learning algorithms
Watson, D. S., & Wright, M. N. (2021) · Machine Learning · doi:10.1007/s10994-021-06030-6 · Cited by 13 · see citing works
Genomewide interaction and enrichment analysis on antidepressant response
Antypa, N., Drago, A., & Serretti, A. (2014) · Psychological Medicine · doi:10.1017/s0033291713001554 · Cited by 9 · see citing works
Revisiting the Platonic Representation Hypothesis: An Aristotelian View
Gröger, F., Wen, S., & Brbić, M. (2026) · arXiv · doi:10.48550/arxiv.2602.14486
- Authors
- Phipson, B., & Smyth, G. K.
- Affiliations
- Walter and Eliza Hall Institute of Medical Research (AU)
- Venue
- Statistical Applications in Genetics and Molecular Biology
- Publisher
- Walter de Gruyter GmbH
- Published
- 2010-10-31
- Volume
- 9
- Issue
- 1
- Pages
- Article39-Article39
- ISSN
- 1544-6115 (ISSN-L)
- OpenAlex
- W2013223495
- Cited by
- 712
- References
- 2
- Citations by year
- 2026: 206 · 2025: 58 · 2024: 39 · 2023: 35 · 2022: 42 · 2021: 48 · 2020: 59 · 2019: 43 · 2018: 39 · 2017: 46 · 2016: 35 · 2015: 19 · 2014: 19 · 2013: 13 · 2012: 9
- FWCI
- 6.54
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Statistical Applications in Genetics and Molecular Biology (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Keywords
- permutation tests, exact p-values, multiple comparisons, genomic data analysis
- MeSH terms
- Probability, Models, Statistical, Gene Expression Profiling, Genomics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
