The harmonic mean p -value for combining dependent tests
Open accessPublished version of a preprint
The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
Significance: The widespread use of Bonferroni correction encumbers the scientific process and wastes opportunities for discovery presented by big data, because it discourages exploratory analyses by overpenalizing the total number of statistical tests performed. In this paper, I introduce the harmonic mean p -value (HMP), a simple to use and widely applicable alternative to Bonferroni correction motivated by Bayesian model averaging that greatly improves statistical power while maintaining control of the gold standard false positive rate. The HMP has a range of desirable properties and offers a different way to think about large-scale exploratory data analysis in classical statistics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“We then aggregated the values of porig using the harmonic mean p-value to test the global null hypothesis that none of the pairs were statistically inconsistent (Wilson, 2019), yielding an aggregated porig of 0.0005, which is strong evidence of some inconsistency in the tested pairs.”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Wilson, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · cited 3×
“Controlling the FDR guarantees that, among the significant tests, the proportion in which the null hypothesis is incorrectly rejected in favor of the alternative is limited (9).”
BackgroundMethodEstimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
Guindon, S., Dufayard, J.-F., Lefort, V., Anisimova, M., Hordijk, W., & Gascuel, O. (2010) · Systematic Biology · doi:10.1093/sysbio/syq010 · Cited by 18,376
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490 · cited 1×
“The widely used Benjamini–Hochberg (BH) procedure (9) for controlling the FDR shares with the Bonferroni method a robustness to positive correlation between tests (10) but is less conservative.”
MethodStatistical significance for genomewide studies
Storey, J. D., & Tibshirani, R. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1530509100 · Cited by 9,692
Handbook of Personality: Theory and Research
Krueger, R. F., & Johnson, W. (2008) · Cited by 7,884
A haplotype map of the human genome
The International HapMap Consortium (2005) · Nature · doi:10.1038/nature04226 · Cited by 5,858
A Direct Approach to False Discovery Rates
Storey, J. D. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00346 · Cited by 5,603
METAL: fast and efficient meta-analysis of genomewide association scans
Willer, C. J., Li, Y., & Abecasis, G. R. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq340 · Cited by 5,205
Research works printed but not linked: 1
- The Royal Society 2017 Machine Learning: The Power and Promise of Computers That Learn by Example The Royal Society London
Cited by
Items 401–460 of 460
Wang, S., & Rho, Y. (2025) · Statistical Methods & Applications · doi:10.1007/s10260-025-00789-x
“Another method for controlling the FWER is the harmonic mean p -value (HMP), which is robust to dependencies among individual tests (Good 1958; Wilson 2019).”
Can you trust your reconstructed lineage tree? A homoplasy-based approach for irreversible evolutionPreprint
Zilber, P., Prillo, S., Neumeier, Y., Yosef, N., & Nadler, B. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.27.667007
“We remark that there are other possible schemes to fuse the scores in (11) into a single test statistic, e.g. the harmonic mean [Wil19].”
Cardiorespiratory and Bicep Muscle Responses to Assembly Line Work Volume
Furk, D., Silva, L., Dias, M., Probst, P., & Gamboa, H. F. S. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-96899-0_6
Gou, J., Wu, K. C., & Chén, O. Y. (2025) · Pharmaceutical Statistics · doi:10.1002/pst.70040
“Alternatively, other methods can be used to combine the p - values; for instance, the harmonic mean method yields a p - value of 0.965 for the CoCo–1 test and 0.761 for the CoCo–2 test [60–63].”
InfluentialZheng, Y., Gao, F., & Hou, L. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf590
“The use of HMP to aggregate results across gene sets is statistically well-founded and particularly suitable for combining dependent tests (Wilson 2019).”
Melkani, D., Harnwal, N., Desai, S., Patel, D., & Melkani, G. C. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.10.31.685789
“Since multiple p-values are associated with each comparison genotype 𝐺 𝑚 (main effect + all interaction terms), the Harmonic Mean p-value (HMP) method [15] is used to combine them.”
Decoupling of visual feature selectivity in the retinocollicular pathway
Schwartz, O. S., Matsumoto, A., Yamamoto, H., & Yonehara, K. (2025) · Current Biology · doi:10.1016/j.cub.2025.11.050
Barthelson, K., Pederson, S., Newman, M., & Lardelli, M. T. (2020) · Figshare · doi:10.6084/m9.figshare.13115231
Barthelson, K., Pederson, S., Newman, M., & Lardelli, M. T. (2020) · Figshare · doi:10.6084/m9.figshare.13115255
Barthelson, K., Pederson, S., Newman, M., & Lardelli, M. T. (2020) · Figshare · doi:10.6084/m9.figshare.13115234
Qin, T., Lee, C., Li, S., Cavalcante, R. G., Orchard, P., Yao, H., Zhang, H., Wang, S., Patil, S., Boyle, A. P., et al. (2022) · Figshare · doi:10.6084/m9.figshare.19663875
Barthelson, K., Pederson, S., Newman, M., & Lardelli, M. T. (2020) · Figshare · doi:10.6084/m9.figshare.13115228
Barthelson, K., Pederson, S., Newman, M., & Lardelli, M. T. (2020) · Figshare · doi:10.6084/m9.figshare.13115240
Shao, Z., Wang, T., Qiao, J., Zhang, Y., Shuiping, H., & Zeng, P. (2022) · Figshare · doi:10.6084/m9.figshare.20745267
Shao, Z., Wang, T., Qiao, J., Zhang, Y., Shuiping, H., & Zeng, P. (2022) · Figshare · doi:10.6084/m9.figshare.20745264
Barthelson, K., Pederson, S., Newman, M., & Lardelli, M. T. (2020) · Figshare · doi:10.6084/m9.figshare.13115237
Barthelson, K., Pederson, S., Newman, M., & Lardelli, M. T. (2020) · Figshare · doi:10.6084/m9.figshare.13115252
Chen, H., Pelizzola, M., & Futschik, A. (2024) · Figshare · doi:10.6084/m9.figshare.26613140
Qin, T., Lee, C., Li, S., Cavalcante, R. G., Orchard, P., Yao, H., Zhang, H., Wang, S., Patil, S., Boyle, A. P., et al. (2022) · Figshare · doi:10.6084/m9.figshare.19663869
Barthelson, K., Pederson, S., Newman, M., & Lardelli, M. T. (2020) · Figshare · doi:10.6084/m9.figshare.13115246
Barthelson, K., Pederson, S., Newman, M., & Lardelli, M. T. (2020) · Figshare · doi:10.6084/m9.figshare.13115249
Zhang, S., Jiang, Z., & Zeng, P. (2024) · Figshare · doi:10.6084/m9.figshare.25376120
Additional file 1 of PhyloMed: a phylogeny-based test of mediation effect in microbiome
Hong, Q., Chen, G., & Tang, Z.-Z. (2023) · Figshare · doi:10.6084/m9.figshare.22621100
Additional file 2 of PhyloMed: a phylogeny-based test of mediation effect in microbiome
Hong, Q., Chen, G., & Tang, Z.-Z. (2023) · Figshare · doi:10.6084/m9.figshare.22621103
Ένα ευέλικτο πακέτο λογισμικού για το συνδυασμό εξαρτημένων ή ανεξάρτητων p-values
Νικολίτσα, Ε. Κ., & Νικολίτσα, Ε. Κ. (2024) · University of Thessaly Institutional Repository (University of Thessaly)
Umeton, R., Bellucci, G., Bigi, R., Romano, S., Buscarinu, M. C., Reniè, R., Rinaldi, V., Umeton, R. P., Morena, E., Romano, C., et al. (2021) · The Journal of the American Medical Association (JAMA) Network (American Medical Association)
Patop, I. L., Anduaga, A. M., Rashkover, T., Morales, J., Brownstein, N., Carmona, C. R., Bussi, I. L., Ceriani, M. F., Kurmangaliyev, Y. Z., & Kadener, S. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2025.12.23.696232
Oncological Care Needs of People With Mental Illness: A Single Institution Experience in Australia
Yeoh, H., Hong, W., Holmes, C., Leong, M. M. Y., Courts, K., Potasz, N., Lundy, J., Wong, Z. W., Newton, R. L., & Antill, Y. C. (2026) · Asia-Pacific Journal of Clinical Oncology · doi:10.1111/ajco.70086
The Generalized Harmonic Mean for p‐Values: Combining Dependent and Independent Tests
Zhengbang, L. I., & Zhou, X. (2026) · Statistics in Medicine · doi:10.1002/sim.70439
Testing for gene-environment (GxE) interaction using p-value aggregation identifies many GxE lociPreprint
Mishra, S., Patra, R. R., Reddy, A. S., Mandal, A., & Majumdar, A. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.02.24.707798
“Two well-known p-value combination approaches are the Harmonic mean aggregation [10], and the Cauchy aggregation [11, 12].”
Multiple testing in rolling-window analysis via p-value combination
Ling, X., Wu, Q., & Chen, K. (2026) · Economics Letters · doi:10.1016/j.econlet.2026.113198
A Framework for Optimizing Breeding Pairs Using Artificial Intelligence
Loh, Z. K., van der Werf, J. J. H., & Clark, S. (2022) · RUNE (Research UNE)
Reynes, L. (2021) · theses.fr (ABES) · doi:10.70675/623f1e9cz015dz4ae3zacefzb980ab2f491d · Cited by 2
Federated learning of Bayesian networks preserving privacy for personalised medical applications
Benikhlef, S. (2025) · theses.fr (ABES) · doi:10.70675/851ea73az7fedz485fza157z6aa9de5418fc
Childhood predictors of political voice across 22 countries in the global flourishing study
Ssozi, J., Kiweewa, J. M., Norman-Krause, N., Bradshaw, M., VanderWeele, T. J., & Johnson, B. R. (2026) · Scientific Reports · doi:10.1038/s41598-026-46513-x
Figley, T. D., Kornelsen, J., Uddin, M. N., Wong, K., Pirzada, S., Carter, S., Helmick, C. A., O’Grady, C., Mazerolle, E. L., Patel, R., et al. (2026) · Journal of Neuroscience · doi:10.1523/jneurosci.0278-25.2026
Sooknah, M., Srinivasan, R., Sankarapandian, S., Chen, Z., & Xu, J. (2026) · medRxiv · doi:10.64898/2026.04.10.26350624
He, M., & Zhao, N. (2026) · Biometrics · doi:10.1093/biomtc/ujag077
Riverine phosphorus gain and loss across the conterminous United States
Wang, Y., Zhang, X., Zhao, K., Sabo, R. D., Miao, Y., & Clark, C. M. (2026) · Earth system science data · doi:10.5194/essd-18-3355-2026
“To minimize random effects, the model was run 10 times, and we calculated the average importance value and harmonic mean p -value (Wilson, 2019).”
Schlaudraff, J., Turco, D. D., Key, J., Deller, T., & Auburger, G. (2026) · Frontiers in Molecular Neuroscience · doi:10.3389/fnmol.2026.1758390 · Cited by 1
A cross-national study of demographic variation and childhood predictors of traumatic distress
Sung, M., Mattson, S. A., Cowden, R. G., Wilkinson, R., Padgett, R. N., Denckla, C. A., Shiba, K., Johnson, B. R., & VanderWeele, T. J. (2026) · Communications Medicine · doi:10.1038/s43856-026-01657-9
“Within each country, a global test of variation of TD, acrosslevels ofeach sociodemographic characteristic was conducted, and a pooled global p -value 61 was reported across countries to evaluate if a given association for the sociodemographic characteristic with TD holds withinatleastonecountry.”
Yang, K., Hu, C., Zhao, H., Peng, X., Yan, Y., Wang, J., Jiang, W., Mu, D., Cao, Y., Li, X. Q., et al. (2026) · Advanced Science · doi:10.1002/advs.76250
“…of the five scores), MedianScore (median value, robust to outliers), RankAve [ 19 ] (average normalized rank across scoring methods), HarmonicMean [ 20 ] (harmonic mean emphasizing concordantly high scores), TrimmedMean [ 21 ] (mean after removing the upper and lower 10% of values), WeightedMean…”
gamdid: generalized additive models for differential distributions in single cell experimentsPreprint
Beerland, L., Vandenbulcke, S., Vanderaa, C., Martens, L., & Clement, L. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.18.733106
“…github (github.com/statOmics/gamdid) and will be submitted to Bioconductor. gamdid builds upon the CRAN packages mgcv [13] and harmonicmeanp [24] for fitting generalized additive models and p-value aggregation; and on the R/Bioconductor packages stageR [19] for stagewise testing and msqrob2…”
Hypothesis-Informed Feature Stability Scoring for High-Dimensional ETL Pipelines
Piryankov, K., Grigorova, I., Karamfilov, A., & Efremov, A. (2026) · Applied Sciences · doi:10.3390/app16136445
Network-based machine learning to identify biomarkers for systemic lupus erythematosus
Park, M., Kim, D., Lee, J., Noh, J., Kim, C. J., Oh, Y., Suh, C. S., Kim, J.-W., Im, S., Kim, S., et al. (2026) · BMC Biology · doi:10.1186/s12915-026-02681-w
“In cases where multiple algorithms produced identical proportions of enriched clusters, the best algorithm was selected based on combined p -values across clusters, calculated using the “har-monicmeanp” [97] R package.”
The causal effects of immune factors on sarcopenia-related phenotypes
Hou, J.-L., Chai, Y., Wu, J.-X., Niu, B., Deng, F.-Y., & Lei, S. (2026) · European Journal of Applied Physiology · doi:10.1007/s00421-026-06383-7 · Cited by 1
Kim, J., & Park, J. (2026) · Statistical Methods in Medical Research · doi:10.1177/09622802261480383
“7 This approach is designed to mitigate the effect of overall dependence among the p -values by allowing a small proportion of extreme terms in the summation of the transformed p -values to dominate.”
Mallawaarachchi, S., Tandon, K., Rajan, N., Marcelino, V. R., Sandhu, S. K., Bedoui, S., Ingle, D. J., Gunjur, A., & Tonkin‐Hill, G. Q. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.08.30.748153
“Importantly, in addition to standard multiple-testing correction, StrainSpy incorporates a phylogenetic-aware approach using the Harmonic mean p-value(33), increasing the statistical power of studies that consider large numbers of strains.”
Optimal Adjustment and Combination of Independent Discrete p -Values
Contador, G., & Wu, Z. (2026) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2026.2732157
Early-Life Predictors of Perceiving Divine Punishment in Adulthood Across 22 Countries
Cowden, R. G., Henry, J., Harris, C., Johnson, K. A., Case, B. W., Padgett, R. N., Johnson, B. R., & VanderWeele, T. J. (2026) · Applied Research in Quality of Life · doi:10.1007/s11482-026-10644-0
“A pooled p -value (Wilson, 2019) was used across countries to report whether there is evidence of an association in at least one country.”
Distributive Effects of AI Support: A Framework for Analyzing Performance Shifts Within Groups
Campagner, A., Natali, C., Fregosi, C., & Cabitza, F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37976-4_16
Zhou, M., Ma, S., & Wu, M. (2026) · Journal of the American Statistical Association · doi:10.1080/01621459.2026.2737387
Lasting aversive consequences of a single dose of 4-hydroxytamoxifenPreprint
Kyzar, E. J., Setara, R., Eisengart, M., Virkar, S., Rogerson, L., Ramirez, A., & Salzman, C. D. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.09.16.751757
“P -values were computed for each pair-wise comparison of mice (i.e., n=5 in each group yields 25 pairwise comparisons) and combined using a harmonic mean approach 5,42 .”
A TWAS Method Calibrated for the Uncertainty of Predicted Expression
Majumdar, A., & Haldar, T. (2026) · Genetic Epidemiology · doi:10.1002/gepi.70060 · Cited by 1
Denckla, C. A., Cowden, R. G., Wilkinson, R., Padgett, R. N., Sung, M., Johnson, B. R., & VanderWeele, T. J. (2026) · The Lancet Public Health · doi:10.1016/s2468-2667(26)00185-4
Yadav, V., Kelley, L., Young, E., Garcia-Garcia, C., Lane, M. A., Adams, C., Hollen, C., Fryman, A., Vong, C., Orban, A., et al. (2026) · Multiple Sclerosis and Related Disorders · doi:10.1016/j.msard.2026.108019
Lomas, T., Padgett, R. N., Ritchie‐Dunham, J. L., Lee, M. T., Pawelski, J. O., Johnson, B. R., & VanderWeele, T. J. (2026) · Journal of Happiness Studies · doi:10.1007/s10902-026-01127-y
Metadata
- Authors
- Wilson, D. J.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Wilson, Daniel J.
- ORCID
- Wilson, Daniel J.
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2019-01-04
- Volume
- 116
- Issue
- 4
- Pages
- 1195-1200
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC6347718
- OpenAlex
- W2907099021
- Cited by
- 449
- References
- 48
- Citations by year
- 2026: 58 · 2025: 115 · 2024: 44 · 2023: 54 · 2022: 61 · 2021: 57 · 2020: 45 · 2019: 12
- FWCI
- 34.33
- Type
- journal article
- Language
- en
- Update notices
- correction (2019-10-07)
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Wellcome (101237/Z/13/Z)
- Royal Society (101237/Z/13/Z)
- RCUK | Medical Research Council (MR/K01532X/1)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genetic Associations and Epidemiology
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Keywords
- family-wise error rate, statistical power, generalized central limit theorem, group testing, genome-wide association study, neuroticism, Benjamini-Hochberg procedure, false discovery rate
- MeSH terms
- Hepatitis C, Humans, Models, Statistical, Hepacivirus, Viral Load, Genome-Wide Association Study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.