The harmonic mean p -value for combining dependent tests
Open accessPublished version of a preprint
The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
Significance: The widespread use of Bonferroni correction encumbers the scientific process and wastes opportunities for discovery presented by big data, because it discourages exploratory analyses by overpenalizing the total number of statistical tests performed. In this paper, I introduce the harmonic mean p -value (HMP), a simple to use and widely applicable alternative to Bonferroni correction motivated by Bayesian model averaging that greatly improves statistical power while maintaining control of the gold standard false positive rate. The HMP has a range of desirable properties and offers a different way to think about large-scale exploratory data analysis in classical statistics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“We then aggregated the values of porig using the harmonic mean p-value to test the global null hypothesis that none of the pairs were statistically inconsistent (Wilson, 2019), yielding an aggregated porig of 0.0005, which is strong evidence of some inconsistency in the tested pairs.”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Wilson, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · cited 3×
“Controlling the FDR guarantees that, among the significant tests, the proportion in which the null hypothesis is incorrectly rejected in favor of the alternative is limited (9).”
BackgroundMethodEstimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
Guindon, S., Dufayard, J.-F., Lefort, V., Anisimova, M., Hordijk, W., & Gascuel, O. (2010) · Systematic Biology · doi:10.1093/sysbio/syq010 · Cited by 18,376
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490 · cited 1×
“The widely used Benjamini–Hochberg (BH) procedure (9) for controlling the FDR shares with the Bonferroni method a robustness to positive correlation between tests (10) but is less conservative.”
MethodStatistical significance for genomewide studies
Storey, J. D., & Tibshirani, R. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1530509100 · Cited by 9,692
Handbook of Personality: Theory and Research
Krueger, R. F., & Johnson, W. (2008) · Cited by 7,884
A haplotype map of the human genome
The International HapMap Consortium (2005) · Nature · doi:10.1038/nature04226 · Cited by 5,858
A Direct Approach to False Discovery Rates
Storey, J. D. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00346 · Cited by 5,603
METAL: fast and efficient meta-analysis of genomewide association scans
Willer, C. J., Li, Y., & Abecasis, G. R. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq340 · Cited by 5,205
The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses
Wilks, S. S. (1938) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177732360 · Cited by 3,493 · cited 1×
“For nested hypotheses (ΘM0 ∈ΘMi ), Wilks’ theorem (14) approximates the null distribution of Ri as LogGamma(α= ν/2,β= 1) when there are ν degrees of freedom.”
An improved Bonferroni procedure for multiple tests of significance
SIMES, R. J. (1986) · Biometrika · doi:10.1093/biomet/73.3.751 · Cited by 2,396 · cited 7×
“Simes RJ (1986) An improved Bonferroni procedure for multiple tests of significance.”
MethodInfluentialEmpirical Bayes analysis of a microarray experiment
Efron, B., Tibshirani, R., Storey, J. D., & Tusher, V. (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753382129 · Cited by 1,733
On closed testing procedures with special reference to ordered analysis of variance
MARCUS, R., ERIC, P., & GABRIEL, K. R. (1976) · Biometrika · doi:10.1093/biomet/63.3.655 · Cited by 1,423 · cited 2×
“Controlling the FWER in the presence of some true positives is challenging and considered the strongest form of protection against false positives (4).”
BackgroundMethodStatistical Methods for Research Workers
(1933) · Nature · doi:10.1038/131383b0 · Cited by 1,092
The genetic architecture of type 2 diabetes
Fuchsberger, C., Flannick, J., Teslovich, T. M., Mahajan, A., Agarwala, V., Gaulton, K. J., Ma, C., Fontanillas, P., Moutsianas, L., McCarthy, D. J., et al. (2016) · Nature · doi:10.1038/nature18642 · Cited by 1,051 · cited 1×
“With the rapidly decreasing costs of DNA sequencing, whole-genome sequencing is becoming routine, raising the possibility of detecting associations at ever more variants (2, 3).”
BackgroundStudy, L. C., Okbay, A., Baselmans, B. M. L., De Neve, J.-E., Turley, P., Nivard, M. G., Fontana, M. A., Meddens, S. F. W., Linnér, R. K., Rietveld, C. A., et al. (2016) · Nature Genetics · doi:10.1038/ng.3552 · Cited by 1,013 · cited 4×
“4× 10−8, had not validated when tested in a quasireplication exercise that involved testing variants associated with neuroticism for association with subjective wellbeing or depressive symptoms (23).”
BackgroundMethodInfluentialClonalFrameML: Efficient Inference of Recombination in Whole Bacterial Genomes
Didelot, X., & Wilson, D. J. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004041 · Cited by 882
Self-citationZheng, J., Erzurumluoglu, A. M., Elsworth, B. L., Kemp, J. P., Howe, L., Haycock, P. C., Hemani, G., Tansey, K., Laurin, C., Early Genetics and Lifecourse Epidemiology (EAGLE) Eczema Consortium, et al. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btw613 · Cited by 869
Calibration of ρ Values for Testing Precise Null Hypotheses
Sellke, T., Bayarri, M. J., & Berger, J. O. (2001) · The American Statistician · doi:10.1198/000313001300339950 · Cited by 854 · cited 6×
“In this paper, I introduce the harmonic mean pvalue (HMP), a simple to use and widely applicable alternative to Bonferroni correction motivated by Bayesian model averaging that greatly improves statistical power while maintaining control of the gold standard false positive rate.”
BackgroundInfluentialPe'er, I., Yelensky, R., Altshuler, D., & Daly, M. J. (2008) · Genetic Epidemiology · doi:10.1002/gepi.20303 · Cited by 783
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478397 · Cited by 775
Estimation of significance thresholds for genomewide association scans
Dudbridge, F., & Gusnanto, A. (2008) · Genetic Epidemiology · doi:10.1002/gepi.20297 · Cited by 703
Approximate Bayes factors and accounting for model uncertainty in generalised linear models
Raftery, A. (1996) · Biometrika · doi:10.1093/biomet/83.2.251 · Cited by 633
PARALS Registry, van Rheenen, W., SLALOM Group, SLAP Registry, FALS Sequencing Consortium, SLAGEN Consortium, NNIPPS Study Group, Shatunov, A., Dekker, A. M., McLaughlin, R. L., et al. (2016) · Nature Genetics · doi:10.1038/ng.3622 · Cited by 571 · cited 1×
“With the rapidly decreasing costs of DNA sequencing, whole-genome sequencing is becoming routine, raising the possibility of detecting associations at ever more variants (2, 3).”
BackgroundTesting a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.2307/2289131 · Cited by 548
Chance and Stability: Stable Distributions and their Applications
Uchaikin, V. V., & Zolotarev, V. M. (1999) · De Gruyter eBooks · doi:10.1515/9783110935974 · Cited by 479 · cited 1×
“Instead, generalized central limit theorem can be used (15), which states that, for equal weights (wi = 1/L) and independent and identically distributed Ris,”
BackgroundBayesian statistical methods for genetic association studies
Stephens, M., & Balding, D. J. (2009) · Nature Reviews Genetics · doi:10.1038/nrg2615 · Cited by 468
Chance and Stability: Stable Distributions and Their Applications
Uchaikin, V. V., & Zolotarev, V. M. (1999) · CERN Document Server (European Organization for Nuclear Research) · Cited by 466
Reevaluation of SNP heritability in complex human traits
the UCLEB Consortium, Speed, D., Cai, N., Johnson, M. R., Nejentsev, S., & Balding, D. J. (2017) · Nature Genetics · doi:10.1038/ng.3865 · Cited by 456
A general framework for multiple testing dependence
Leek, J. T., & Storey, J. D. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0808709105 · Cited by 374 · cited 1×
“Other phenomena, such as unmeasured confounders, can induce dependence, even when alternative hypotheses are not mutually exclusive, such as in gene expression analyses (7).”
BackgroundThe (in)famous GWAS P-value threshold revisited and updated for low-frequency variants
Fadista, J., Manning, A. K., Florez, J. C., & Groop, L. (2016) · European Journal of Human Genetics · doi:10.1038/ejhg.2015.269 · Cited by 359
On the energy loss of fast particles by ionization
LANDAU, L. D. (1944) · Cited by 315
Earle, S. G., Wu, C.-H., Charlesworth, J., Stoesser, N., Gordon, N. C., Walker, T. M., Spencer, C. C. A., Iqbal, Z., Clifton, D. A., Hopkins, K. L., et al. (2016) · Nature Microbiology · doi:10.1038/nmicrobiol.2016.41 · Cited by 286
Self-citationAn Additive Method for Combining Probability Values from Independent Experiments
Edgington, E. S. (1972) · The Journal of Psychology · doi:10.1080/00223980.1972.9924813 · Cited by 270
Chewapreecha, C., Marttinen, P., Croucher, N. J., Salter, S. J., Harris, S. R., Mather, A. E., Hanage, W. P., Goldblatt, D., Nosten, F. H., Turner, C., et al. (2014) · PLoS Genetics · doi:10.1371/journal.pgen.1004547 · Cited by 218
A systematic comparison of methods for combining p-values from independent tests
Loughin, T. M. (2004) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2003.11.020 · Cited by 210
Sur un mode de croissance régulière. Théorèmes fondamentaux
Karamata, J. (1933) · Bulletin de la Société mathématique de France · doi:10.24033/bsmf.1196 · Cited by 210
Ansari, M. A., Pedergnana, V., L C Ip, C., Magri, A., Von Delft, A., Bonsall, D., Chaturvedi, N., Bartha, I., Smith, D., Nicholson, G., et al. (2017) · Nature Genetics · doi:10.1038/ng.3835 · Cited by 136 · cited 2×
“Joint human–pathogen GWAS reanalysis of viral load in 410 HCV genotype 3a-infected white Europeans (25).”
BackgroundMethodCrawford, L. A., Zeng, P., Mukherjee, S., & Zhou, X. (2017) · PLoS Genetics · doi:10.1371/journal.pgen.1006869 · Cited by 122 · cited 1×
“In the case of GWAS, even if no individual variant shows sufficient evidence of association in a region, the model-averaged signal across that region may yet achieve sufficiently strong posterior odds (12, 13).”
BackgroundTHE COMBINATION OF PROBABILITIES: AN APPLICATION OF ORTHONORMAL FUNCTIONS
Lancaster, H. O. (1961) · Australian Journal of Statistics · doi:10.1111/j.1467-842x.1961.tb00058.x · Cited by 120
Approximating the Distribution of Pareto Sums
Zaliapin, I., Kagan, Y. Y., & Schoenberg, F. P. (2005) · Pure and Applied Geophysics · doi:10.1007/s00024-004-2666-3 · Cited by 98
Huang, Y., Xu, H., Calian, V., & Hsu, J. C. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl383 · Cited by 84 · cited 1×
“However, simulating can be time consuming; model-based simulations require knowledge of the dependency structure, which may be limited; and permutation-based procedures are not always appropriate (8).”
BackgroundRegular variation, subexponentiality and their applications in probability theory
Mikosch, T. (1999) · TU/e Research Portal · Cited by 77
Stable limits for sums of dependent infinite variance random variables
Bartkiewicz, K., Jakubowski, A., Mikosch, T., & Wintenberger, O. (2010) · Probability Theory and Related Fields · doi:10.1007/s00440-010-0276-9 · Cited by 65
Hochberg, Y., & Liberman, U. (1994) · Statistics & Probability Letters · doi:10.1016/0167-7152(94)90216-x · Cited by 25
Modeling interactions with known risk loci-a Bayesian model averaging approach
Ferreira, T. C., & Marchini, J. (2010) · Annals of Human Genetics · doi:10.1111/j.1469-1809.2010.00618.x · Cited by 24 · cited 1×
“In the case of GWAS, even if no individual variant shows sufficient evidence of association in a region, the model-averaged signal across that region may yet achieve sufficiently strong posterior odds (12, 13).”
BackgroundA Decision-Theoretic Formulation of Fisher’s Approach to Testing
Rice, K. M. (2010) · The American Statistician · doi:10.1198/tast.2010.09060 · Cited by 18
Research works printed but not linked: 1
- The Royal Society 2017 Machine Learning: The Power and Promise of Computers That Learn by Example The Royal Society London
Cited by
Major Impacts of Widespread Structural Variation on Gene Expression and Crop Improvement in Tomato
Alonge, M., Wang, X., Benoit, M., Soyk, S., Pereira, L., Zhang, L., Suresh, H., Ramakrishnan, S., Maumus, F., Ciren, D., et al. (2020) · Cell · doi:10.1016/j.cell.2020.05.021 · Cited by 819
“The adjusted p values for each annotation and tissue type were aggregated using two methods: Fisher’s method and a harmonic mean estimate (Sitgreaves, 1960; Wilson, 2019), and are listed in Table S4A.”
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
Machine learning identifies candidates for drug repurposing in Alzheimer’s disease
Rodriguez, S., Hug, C., Todorov, P., Moret, N., Boswell, S. A., Evans, K., Zhou, G., Johnson, N. T., Hyman, B. T., Sorger, P. K., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-21330-0 · Cited by 299
Lee, H., Fenster, R. J., Pineda, S. S., Gibbs, W. S., Mohammadi, S., Davila-Velderrain, J., Garcia, F. J., Therrien, M., Novis, H. S., Gao, F., et al. (2020) · Neuron · doi:10.1016/j.neuron.2020.06.021 · Cited by 279
“We then combined individual p-value s (Wilson, 2019) for each transcription factor to estimate a meta p-value for its activity.”
MethodThe Global Flourishing Study: Study Profile and Initial Results on Flourishing
VanderWeele, T. J., Johnson, B. R., Białowolski, P., Bonhag, R., Bradshaw, M., Breedlove, T., Case, B. W., Chen, Y., Chen, Z. J., Counted, V., et al. (2025) · Nature Mental Health · doi:10.1038/s44220-025-00423-5 · Cited by 253
Metadata
- Authors
- Wilson, D. J.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Wilson, Daniel J.
- ORCID
- Wilson, Daniel J.
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2019-01-04
- Volume
- 116
- Issue
- 4
- Pages
- 1195-1200
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC6347718
- OpenAlex
- W2907099021
- Cited by
- 449
- References
- 48
- Citations by year
- 2026: 58 · 2025: 115 · 2024: 44 · 2023: 54 · 2022: 61 · 2021: 57 · 2020: 45 · 2019: 12
- FWCI
- 34.33
- Type
- journal article
- Language
- en
- Update notices
- correction (2019-10-07)
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Wellcome (101237/Z/13/Z)
- Royal Society (101237/Z/13/Z)
- RCUK | Medical Research Council (MR/K01532X/1)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genetic Associations and Epidemiology
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Keywords
- family-wise error rate, statistical power, generalized central limit theorem, group testing, genome-wide association study, neuroticism, Benjamini-Hochberg procedure, false discovery rate
- MeSH terms
- Hepatitis C, Humans, Models, Statistical, Hepacivirus, Viral Load, Genome-Wide Association Study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.