The harmonic mean p -value for combining dependent tests
Open accessPublished version of a preprint
The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
Significance: The widespread use of Bonferroni correction encumbers the scientific process and wastes opportunities for discovery presented by big data, because it discourages exploratory analyses by overpenalizing the total number of statistical tests performed. In this paper, I introduce the harmonic mean p -value (HMP), a simple to use and widely applicable alternative to Bonferroni correction motivated by Bayesian model averaging that greatly improves statistical power while maintaining control of the gold standard false positive rate. The HMP has a range of desirable properties and offers a different way to think about large-scale exploratory data analysis in classical statistics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“We then aggregated the values of porig using the harmonic mean p-value to test the global null hypothesis that none of the pairs were statistically inconsistent (Wilson, 2019), yielding an aggregated porig of 0.0005, which is strong evidence of some inconsistency in the tested pairs.”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Wilson, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · cited 3×
“Controlling the FDR guarantees that, among the significant tests, the proportion in which the null hypothesis is incorrectly rejected in favor of the alternative is limited (9).”
BackgroundMethodEstimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
Guindon, S., Dufayard, J.-F., Lefort, V., Anisimova, M., Hordijk, W., & Gascuel, O. (2010) · Systematic Biology · doi:10.1093/sysbio/syq010 · Cited by 18,376
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490 · cited 1×
“The widely used Benjamini–Hochberg (BH) procedure (9) for controlling the FDR shares with the Bonferroni method a robustness to positive correlation between tests (10) but is less conservative.”
MethodStatistical significance for genomewide studies
Storey, J. D., & Tibshirani, R. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1530509100 · Cited by 9,692
Handbook of Personality: Theory and Research
Krueger, R. F., & Johnson, W. (2008) · Cited by 7,884
A haplotype map of the human genome
The International HapMap Consortium (2005) · Nature · doi:10.1038/nature04226 · Cited by 5,858
A Direct Approach to False Discovery Rates
Storey, J. D. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00346 · Cited by 5,603
METAL: fast and efficient meta-analysis of genomewide association scans
Willer, C. J., Li, Y., & Abecasis, G. R. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq340 · Cited by 5,205
Research works printed but not linked: 1
- The Royal Society 2017 Machine Learning: The Power and Promise of Computers That Learn by Example The Royal Society London
Cited by
Major Impacts of Widespread Structural Variation on Gene Expression and Crop Improvement in Tomato
Alonge, M., Wang, X., Benoit, M., Soyk, S., Pereira, L., Zhang, L., Suresh, H., Ramakrishnan, S., Maumus, F., Ciren, D., et al. (2020) · Cell · doi:10.1016/j.cell.2020.05.021 · Cited by 819
“The adjusted p values for each annotation and tissue type were aggregated using two methods: Fisher’s method and a harmonic mean estimate (Sitgreaves, 1960; Wilson, 2019), and are listed in Table S4A.”
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
Machine learning identifies candidates for drug repurposing in Alzheimer’s disease
Rodriguez, S., Hug, C., Todorov, P., Moret, N., Boswell, S. A., Evans, K., Zhou, G., Johnson, N. T., Hyman, B. T., Sorger, P. K., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-21330-0 · Cited by 299
Lee, H., Fenster, R. J., Pineda, S. S., Gibbs, W. S., Mohammadi, S., Davila-Velderrain, J., Garcia, F. J., Therrien, M., Novis, H. S., Gao, F., et al. (2020) · Neuron · doi:10.1016/j.neuron.2020.06.021 · Cited by 279
“We then combined individual p-value s (Wilson, 2019) for each transcription factor to estimate a meta p-value for its activity.”
MethodThe Global Flourishing Study: Study Profile and Initial Results on Flourishing
VanderWeele, T. J., Johnson, B. R., Białowolski, P., Bonhag, R., Bradshaw, M., Breedlove, T., Case, B. W., Chen, Y., Chen, Z. J., Counted, V., et al. (2025) · Nature Mental Health · doi:10.1038/s44220-025-00423-5 · Cited by 253
Combining p-values via averaging
Vovk, V., & Wang, R. (2020) · Biometrika · doi:10.1093/biomet/asaa027 · Cited by 266
Tumor microbiome links cellular programs and immunity in pancreatic cancer
Ghaddar, B., Biswas, A., Harris, C., Omary, M. B., Carpizo, D. R., Blaser, M. J., & De, S. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.009 · Cited by 181
Kogan, N. E., Clemente, L., Liautaud, P., Kaashoek, J., Link, N. B., Nguyen, A. T., Lu, F. S., Huybers, P., Resch, B., Havas, C., et al. (2021) · Science Advances · doi:10.1126/sciadv.abd6989 · Cited by 172
E-values: Calibration, combination and applications
Vovk, V., & Wang, R. (2021) · The Annals of Statistics · doi:10.1214/20-aos2020 · Cited by 172
Testing for outliers with conformal p-values
Bates, S., Candès, E. J., Lei, L., Romano, Y., & Sesia, M. (2023) · The Annals of Statistics · doi:10.1214/22-aos2244 · Cited by 323
Distinct longevity mechanisms across and within species and their association with aging
Tyshkovskiy, A., Ma, S., Shindyapina, A. V., Tikhonov, S., Lee, S.-G., Bozaykut, P., Castro, J. P., Seluanov, A., Schork, N. J., Gorbunova, V., et al. (2023) · Cell · doi:10.1016/j.cell.2023.05.002 · Cited by 162
Hypothesis Testing with E-values
Ramdas, A., & Wang, R. (2025) · Foundations and Trends® in Statistics · doi:10.1561/3600000002 · Cited by 236
The poolr Package for Combining Independent and Dependent p Values
Cinar, O., & Viechtbauer, W. (2022) · Journal of Statistical Software · doi:10.18637/jss.v101.i01 · Cited by 133
“Finally, a method for combining (possibly dependent) p values was recently described by Wilson (2019), which is implemented in the harmonicmeanp package.”
BackgroundMethodInfluentialOzaki, Y., Tierney, A. T., Pfordresher, P. Q., McBride, J. M., Benetos, E., Proutskova, P., Chiba, G., Liu, F., Jacoby, N., Purdy, S. C., et al. (2024) · Science Advances · doi:10.1126/sciadv.adm9797 · Cited by 112
Dataset Inference: Ownership Resolution in Machine LearningPreprint
Maini, P., Yaghini, M., & Papernot, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.10706 · Cited by 134
“To control for multiple testing, and account for the unknown dependence of the p-values thus generated, we aggregate these values using the harmonic mean (Wilson, 2018).”
MethodGraudenzi, A., Maspero, D., Angaroni, F., Piazza, R., & Ramazzotti, D. (2021) · iScience · doi:10.1016/j.isci.2021.102116 · Cited by 78
Analytic methodology for demographic variation analyses for wave 1 of the global flourishing study
Padgett, R. N., Bradshaw, M., Chen, Y., Cowden, R. G., Jang, S. J., Kim, E. S., Shiba, K., Johnson, B. R., & VanderWeele, T. J. (2025) · BMC Global and Public Health · doi:10.1186/s44263-025-00140-2 · Cited by 75
Shiraishi, Y., Koya, J., Chiba, K., Okada, A., Arai, Y., Saito, Y., Shibata, T., & Kataoka, K. (2023) · Nucleic Acids Research · doi:10.1093/nar/gkad526 · Cited by 72
Statistical power in network neuroscience
Helwegen, K., Libedinsky, I., & van den Heuvel, M. P. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.12.011 · Cited by 63
Virtual Populations for Quantitative Systems Pharmacology Models
Cheng, Y., Straube, R., Alnaif, A. E., Huang, L., Leil, T. A., & Schmidt, B. J. (2022) · Methods in Molecular Biology · doi:10.1007/978-1-0716-2265-0_8 · Cited by 54
Analytic methodology for childhood predictor analyses for wave 1 of the Global Flourishing Study
Padgett, R. N., Bradshaw, M., Chen, Y., Cowden, R. G., Jang, S. J., Kim, E. S., Shiba, K., Johnson, B. R., & VanderWeele, T. J. (2025) · BMC Global and Public Health · doi:10.1186/s44263-025-00142-0 · Cited by 62
Li, J. J., & Tong, X. (2020) · Patterns · doi:10.1016/j.patter.2020.100115 · Cited by 67
“Although there are methods for combining dependent p-values [26, 27, 28], we think that this hypothesis testing approach overly complicated the cancer driver gene problem, which could be better and more easily formulated as a binary classification task.”
Prenatal Exposure to General Anesthesia and Childhood Behavioral Deficit
Ing, C. H., Landau, R., DeStephano, D. M., Miles, C. H., von Ungern-Sternberg, B. S., Li, G., & Whitehouse, A. (2021) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000005389 · Cited by 59
Proteomic cardiovascular risk assessment in chronic kidney disease
Deo, R., Dubin, R. F., Ren, Y., Murthy, A. C., Wang, J., Zheng, H., Zheng, Z., Feldman, H. I., Shou, H., Coresh, J., et al. (2023) · European Heart Journal · doi:10.1093/eurheartj/ehad115 · Cited by 63
Campylobacter jejuni genotypes are associated with post-infection irritable bowel syndrome in humans
Peters, S., Pascoe, B., Wu, Z., Bayliss, S., Zeng, X., Edwinson, A. L., Gurunathan, S. V., Jawahir, S., Calland, J., Mourkas, E., et al. (2021) · Communications Biology · doi:10.1038/s42003-021-02554-8 · Cited by 59
Recent advances toward understanding the mysteries of the acute to chronic pain transition
Price, T. J., & Ray, P. R. (2019) · Current Opinion in Physiology · doi:10.1016/j.cophys.2019.05.015 · Cited by 59
Intestinal microbiota controls graft-versus-host disease independent of donor-host genetic disparity
Koyama, M., Hippe, D. S., Srinivasan, S., Proll, S. C., Miltiadous, O., Li, N., Zhang, P., Ensbey, K. S., Hoffman, N. G., Schmidt, C. R., et al. (2023) · Immunity · doi:10.1016/j.immuni.2023.06.024 · Cited by 54
Hu, Y., Satten, G. A., & Hu, Y. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2122788119 · Cited by 56
Dryland sensitivity to climate change and variability using nonlinear dynamics
Sasaki, T., Collins, S. L., Rudgers, J. A., Gantsetseg, B., Baasandai, E., & Kinugasa, T. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305050120 · Cited by 52
“The results were metasignificant for all climate variables, as determined using a recently proposed method to combine the P-values [harmonic mean P-values (37)].”
MethodInternational REACH forgiveness intervention: a multisite randomised controlled trial
Ho, M. Y., Worthington, E. L., Cowden, R. G., Bechara, A. O., Chen, Z. J., Gunatirin, E. Y., Joynt, S., Khalanskyi, V. V., Korzhov, H., Kurniati, N. M. T., et al. (2024) · BMJ Public Health · doi:10.1136/bmjph-2023-000072 · Cited by 52
Spitzer, K., Pelizzola, M., & Futschik, A. (2020) · The Annals of Applied Statistics · doi:10.1214/19-aoas1301 · Cited by 56
“An advantage of our approach is that the resulting proper p-values can be combined with a standard procedure such as harmonic mean p-values by Wilson (2019) or Benjamini-Hochberg (Benjamini and Hochberg, 1995) that control for multiple testing.”
MethodAdmissible ways of merging p-values under arbitrary dependence
Vovk, V., Wang, B., & Wang, R. (2022) · The Annals of Statistics · doi:10.1214/21-aos2109 · Cited by 80
“In particular, we do not discuss methods that are valid for specific classes of dependence structures; for the latter, see e.g., Sarkar (1998), Wilson (2019), and Liu and Xie (2020), as well as Chen et al. (2020) for a summary.”
InfluentialVlachos, C., Burny, C., Pelizzola, M., Borges, R., Futschik, A., Kofler, R., & Schlötterer, C. (2019) · Genome biology · doi:10.1186/s13059-019-1770-8 · Cited by 47
Karhunen, V., Bond, T. A., Zuber, V., Hurtig, T. M., Moilanen, I. K., Järvelin, M., Evangelou, M., & Rodriguez, A. (2021) · Translational Psychiatry · doi:10.1038/s41398-021-01584-4 · Cited by 50
Classification of non-TCGA cancer samples to TCGA molecular subtypes using compact feature sets
Ellrott, K., Wong, C. K., Yau, C. S. F., Castro, M. A. A., Lee, J., Karlberg, B., Grewal, J., Lagani, V., Tercan, B., Friedl, V., et al. (2025) · Cancer Cell · doi:10.1016/j.ccell.2024.12.002 · Cited by 45
GWAS of longitudinal trajectories at biobank scale
Ko, S., German, C., Jensen, A. E., Shen, J., Wang, A., Mehrotra, D. V., Sun, Y. V., Sinsheimer, J. S., Zhou, H., & Zhou, J. J. (2022) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2022.01.018 · Cited by 52
Cellular profiling of a recently-evolved social behavior in cichlid fishes
Johnson, Z. V., Hegarty, B. E., Gruenhagen, G. W., Lancaster, T. J., McGrath, P. T., & Streelman, J. T. (2023) · Nature Communications · doi:10.1038/s41467-023-40331-9 · Cited by 46
Fanidis, D., & Moulos, P. (2020) · Briefings in Bioinformatics · doi:10.1093/bib/bbaa156 · Cited by 43
A method for creating a burn severity atlas: an example from Alberta, Canada
Whitman, E., Parisien, M., Holsinger, L. M., Park, J., & Parks, S. A. (2020) · International Journal of Wildland Fire · doi:10.1071/wf19177 · Cited by 43
“To describe the relationships between severity maps generated using paired images and mean compositing in GEE, we randomly sampled 0.01% of pixels from each of 11 fires with field data and conducted CV Spearman correlation tests with 1000 repeats (P values combined using harmonicmeans;Wilson 2019).”
MethodInfluentialGenomic diversity landscapes in outcrossing and selfing Caenorhabditis nematodes
Teterina, A. A., Willis, J. H., Lukac, M., Jovelin, R., Cutter, A. D., & Phillips, P. C. (2023) · PLoS Genetics · doi:10.1371/journal.pgen.1010879 · Cited by 43
Barthelson, K., Pederson, S., Newman, M., & Lardelli, M. T. (2020) · Molecular Brain · doi:10.1186/s13041-020-00681-7 · Cited by 42
Integrating multi-omics summary data using a Mendelian randomization framework
Jin, C., Lee, B., Shen, L., Long, Q., & Initiative, A. D. N. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac376 · Cited by 42
“• Harmonic mean P-value (HMP): HMP is a P-value combination method originated from Bayesian model averaging offered as an alternative to Fisher’s method when the P-values are correlated [21]”
MethodYoung, J. C., Pettit, E. C., Arendt, A. A., Hood, E. W., Liston, G. E., & Beamer, J. P. (2021) · Water Resources Research · doi:10.1029/2020wr027404 · Cited by 39
“To identify the statistical significance 570 of each trend, we report a harmonic mean p-value, a formulation for combining p-values 571 from tests that cannot be guaranteed to be independent [Wilson, 2019], e.”
Zeng, P., Dai, J., Jin, S., & Zhou, X. (2021) · Human Molecular Genetics · doi:10.1093/hmg/ddab056 · Cited by 38
Melnychuk, M. C., Robertson, I. H., Plini, E. R. G., & Dockree, P. M. (2021) · Brain Sciences · doi:10.3390/brainsci11101324 · Cited by 38
“To assess significance at the group level, Wilson’s Harmonic p-value calculation [86] was used.”
MethodMulti-hallmark long noncoding RNA maps reveal non-small cell lung cancer vulnerabilities
Esposito, R., Polidori, T., Meise, D. F., Pulido-Quetglas, C., Chouvardas, P., Forster, S., Schaerer, P., Kobel, A., Schlatter, J., Kerkhof, E., et al. (2022) · Cell Genomics · doi:10.1016/j.xgen.2022.100171 · Cited by 37
“Subsequently, the harmonic mean P-value 847 (HMP)(107) was calculated using the two significance scores from RRA and EBM.”
MethodAdaptive group-regularized logistic elastic net regression
Münch, M., Peeters, C. F. W., van der Vaart, A., & van de Wiel, M. A. (2019) · Biostatistics · doi:10.1093/biostatistics/kxz062 · Cited by 38
“The two resulting FDRs were combined through the harmonic mean (Wilson, 2019) and discretized using the method described in Section 3.”
MethodSamakkarnthai, P., Saul, D., Zhang, L., Aversa, Z., Doolittle, M. L., Sfeir, J. G., Kaur, J., Atkinson, E. J., Edwards, J. R., Russell, G. G., et al. (2023) · Aging · doi:10.18632/aging.204701 · Cited by 34
“The harmonic mean test [74] was used to combine the p-values from individual markers.”
Multi-Decadal Glacier Area and Mass Balance Change in the Southern Peruvian Andes
Taylor, L., Quincey, D. J., Smith, M. W., Potter, E., Castro, J., & Fyffe, C. (2022) · Frontiers in Earth Science · doi:10.3389/feart.2022.863933 · Cited by 40
“Significance for these values was calculated using the Harmonic mean p-value, weighted to the size of the glacier, as a robust measure of significance in large datasets (Wilson, 2019).”
MethodProbabilistic forecasting of replication studies
Pawel, S., & Held, L. (2020) · PLoS ONE · doi:10.1371/journal.pone.0231416 · Cited by 46
Genomic signatures of clonality in the deep water kelp Laminaria rodriguezii
Reynes, L., Thibaut, T., Mauger, S., Blanfuné, A., Holon, F., Cruaud, C., Couloux, A., Valéro, M., & Aurelle, D. (2021) · Molecular Ecology · doi:10.1111/mec.15860 · Cited by 39
“The significance of the test was evaluated by the method of the harmonic mean p- values (HMP) (Wilson, 2019), implemented in the harmonicmeanp v3 R package.”
MethodInfluentialNo Evidence for Galactic Latitude Dependence of the Fast Radio Burst Sky Distribution
Josephy, A., Chawla, P., Curtin, A. P., Kaspi, V. M., Bhardwaj, M., Boyle, P. J., Brar, C., Cassanelli, T., Fonseca, E., Gaensler, B. M., et al. (2021) · The Astrophysical Journal · doi:10.3847/1538-4357/ac33ad · Cited by 34
“A relevant statistic is the harmonic mean p-value (HMP; Wilson 2019), which controls for familywise error rate and, importantly, is robust to inter-dependent samples of p-values.”
BackgroundBeaufils, V., & Tomin, J. (2020) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/5swba · Cited by 34
Alley, J. C., Gassen, J., & Slavich, G. M. (2024) · Brain Behavior and Immunity · doi:10.1016/j.bbi.2024.07.019 · Cited by 34
“Harmonic mean p analyses were used to control familywise error and false discovery rate in the context of dependent hypothesis tests (see Supplemental Materials 1 and 3) (Wilson, 2019).”
MethodRistl, R., Ballarini, N. M., Götte, H., Schüler, A., Posch, M., & König, F. (2020) · Pharmaceutical Statistics · doi:10.1002/pst.2062 · Cited by 33
“Other combination functions with similar properties may be explored though, for example, the combination of individual tests via the harmonic mean of their Pvalues.26 Group sequential methods have been proposed as another approach to tackle the planning uncertainties under unknown patterns of non-proportional hazards.”
BackgroundInferring differential subcellular localisation in comparative spatial proteomics using BANDLE
Crook, O. M., Davies, C. T. R., Breckels, L. M., Christopher, J. A., Gatto, L., Kirk, P., & Lilley, K. S. (2022) · Nature Communications · doi:10.1038/s41467-022-33570-9 · Cited by 33
Confidence and Discoveries with E-values
Vovk, V., & Wang, R. (2023) · Statistical Science · doi:10.1214/22-sts874 · Cited by 54
Association of the IceCube neutrinos with blazars in the CGRaBS sample
Kouch, P. M., Lindfors, E. J., Hovatta, T., Liodakis, I., Koljonen, K. I. I., Nilsson, K., Kiehlmann, S., Max-Moerbeck, W., Readhead, A. C. S., Reeves, R. A., et al. (2024) · Astronomy and Astrophysics · doi:10.1051/0004-6361/202347624 · Cited by 30
Chen, H., Zeng, Z., & Ma, J. (2020) · PLoS ONE · doi:10.1371/journal.pone.0229510 · Cited by 30
“For the multiple comparisons problem, we calculated the harmonic mean p-value that could be interpreted directly when the number of tests are small [19].”
MethodYoung, B., Wu, C., Charlesworth, J., Earle, S. G., Price, J. R., Gordon, N. C., Cole, K., Dunn, L. E. M., Liu, E., Oakley, S. J., et al. (2021) · Microbial Genomics · doi:10.1099/mgen.0.000700 · Cited by 33
“The harmonic mean pvalue (HMP) has recently been developed as a method to combine alternative hypotheses against the null hypothesis, without sacrificing power, even when the tests are not independent [69].”
BackgroundMethodSelf-citationCriticality supports cross-frequency cortical-thalamic information transfer during conscious states
Toker, D., Müller, E. J., Miyamoto, H., Riga, M. S., Lladó-Pelfort, L., Yamakawa, K., Artigas, F., Shine, J. M., Hudson, A. E., Pouratian, N., et al. (2024) · eLife · doi:10.7554/elife.86547 · Cited by 30
Viability of Preictal High-Frequency Oscillation Rates as a Biomarker for Seizure Prediction
Scott, J. M., Gliske, S., Kuhlmann, L., & Stacey, W. C. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2020.612899 · Cited by 28
A binary interaction map between turnip mosaic virus and Arabidopsis thaliana proteomes
Martínez, F., Carrasco, J. L., Toft, C., Hillung, J., Giménez‐Santamarina, S., Yenush, L., Rodrigo, G., & Elena, S. F. (2023) · Communications Biology · doi:10.1038/s42003-023-04427-8 · Cited by 31
The landscape of metabolic pathway dependencies in cancer cell lines
Joly, J. H., Chew, B. T. L., & Graham, N. A. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008942 · Cited by 31
“Since the underlying gene expression profiles were derived from the CCLE for both Sanger and DepMap Genetic PDEAs, we combined statistical tests using the harmonic mean p-value (HMP) which combines dependent statistical tests while controlling the family-wise error rate (FWER) [36].”
MethodEarle, S. G., Lobanovska, M., Lavender, H., Tang, C., Exley, R. M., Ramos‐Sevillano, E., Browning, D. F., Kostiou, V., Harrison, O. B., Bratcher, H. B., et al. (2021) · PLoS Pathogens · doi:10.1371/journal.ppat.1009992 · Cited by 26
“The harmonic mean p -value (HMP) method performs a combined test of the null hypothesis that no p -value is significant [81].”
Self-citationDystrophin deficiency affects human astrocyte properties and response to damage
Lange, J., Gillham, O., Alkharji, R. R., Eaton, S. D., Ferrari, G., Madej, M. J., Flower, M. D., Tedesco, F. S., Muntoni, F., & Ferretti, P. (2021) · Glia · doi:10.1002/glia.24116 · Cited by 26
“For comparison, p values were also combined using the Fisher's method (Dai et al., 2014) and the harmonic mean (Wilson, 2019).”
MethodPermutation-based true discovery guarantee by sum tests
Vesely, A., Finos, L., & Goeman, J. J. (2023) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1093/jrsssb/qkad019 · Cited by 40
Cancer drivers and clonal dynamics in acute lymphoblastic leukaemia subtypes
Studd, J. B., Cornish, A. J., Hoang, P. H., Law, P., Kinnersley, B., & Houlston, R. S. (2021) · Blood Cancer Journal · doi:10.1038/s41408-021-00570-9 · Cited by 25
“1 [30] and OncodriveFML [31] using Harmonic means [32] and Benjamini–Hochberg correction.”
MethodG4mismatch: Deep neural networks to predict G-quadruplex propensity based on G4-seq data
Barshai, M., Engel, B., Haim, I., & Orenstein, Y. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010948 · Cited by 25
“For each property, we combined the p-values of the Spearman correlations across the 11 other species by the harmonic mean p-value [39].”
MethodThe overlapping genetic architecture of psychiatric disorders and cortical brain structure
Sha, Z., Warrier, V., Bethlehem, R. A. I., Schultz, L. M., Merikangas, A. K., Sun, K. Y., Gur, R. C., Gur, R. E., Shinohara, R. T., Gandal, M. J., et al. (2025) · Nature Mental Health · doi:10.1038/s44220-025-00475-7 · Cited by 24
“For each disorder–CT or disorder–SA pair, the significance threshold of HMP 0.05 is represented by red lines.”
InfluentialMalik, P., Kumar, J., Singh, S., Sharma, S., Meher, P. K., Sharma, M. K., Roy, J. K., Sharma, P. K., Balyan, H. S., Gupta, P. K., et al. (2021) · Molecular Breeding · doi:10.1007/s11032-021-01240-1 · Cited by 30
The Mainstreaming of Marx: Measuring the Effect of the Russian Revolution on Karl Marx’s Influence
Magness, P., & Makovi, M. (2020) · Journal of Political Economy · doi:10.1086/722933 · Cited by 23
“However, depending on the form of dependence, the HMP method may not always have the desired strength; that is, it may have a higher type I (false-positive) rate than is desired (Goeman, Rosenblatt, and Nichols 2019; Wilson 2019d).”
InfluentialLange, J., Gillham, O., Flower, M. D., Ging, H., Eaton, S. D., Kapadia, S., Neueder, A., Duchen, M. R., Ferretti, P., & Tabrizi, S. J. (2023) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2023.102448 · Cited by 25
“For comparison, p values were also combined using the Fisher’s method (Dai et al., 2014) and the harmonic mean (Wilson, 2019).”
The impact of exercise on gene regulation in association with complex trait genetics
Vetr, N. G., Gay, N. R., MoTrPAC Study Group, Adkins, J., Albertson, B. G., Amar, D., Amper, M. A. S., Armenteros, J. J. A., Ashley, E. A., Ávila-Pacheco, J., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-45966-w · Cited by 25
Partitioned glioma heritability shows subtype-specific enrichment in immune cells
Ostrom, Q. T., Edelson, J. L., Byun, J., Han, Y., Kinnersley, B., Melin, B. S., Houlston, R. S., Monje, M., Amos, C. I., Barnholtz‐Sloan, J. S., et al. (2021) · Neuro-Oncology · doi:10.1093/neuonc/noab072 · Cited by 24
A General Framework For Detecting Anomalous Inputs to DNN ClassifiersPreprint
Raghuram, J., Chandrasekaran, V., Jha, S., & Banerjee, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.15147 · Cited by 34
“In our problem, we apply the HMP method with the weights all set to the same value, resulting in the p-value aggregation function r ( · ) defined in Eq.”
BackgroundMethodInfluentialYue, Y., & Hu, Y. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac310 · Cited by 23
Large‐scale impacts of selective logging on canopy tree beta‐diversity in the Brazilian Amazon
Bousfield, C. G., Massam, M. R., Peres, C. A., & Edwards, D. P. (2023) · Journal of Applied Ecology · doi:10.1111/1365-2664.14403 · Cited by 21
Białowolski, P., Makridis, C. A., Bradshaw, M., Węziak‐Białowolska, D., Gundersen, C., Pertel, N. L., Gibson, C. B., Jang, S. J., Padgett, R. N., Johnson, B. R., et al. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02207-4 · Cited by 24
“A pooled P value 87 in the two-tailed test was then calculated across countries to determine whether significant variation existed within at least one country.”
A cross-national analysis of sociodemographic variation in suffering across 22 countries
Cowden, R. G., Chen, Z. J., Wilkinson, R., Węziak‐Białowolska, D., Breedlove, T., Gundersen, C., Shiba, K., Padgett, R. N., Johnson, B. R., & VanderWeele, T. J. (2025) · Communications Medicine · doi:10.1038/s43856-025-00859-x · Cited by 20
Exploring the interactions between the human and viral genomes
Fellay, J., & Pedergnana, V. (2019) · Human Genetics · doi:10.1007/s00439-019-02089-3 · Cited by 20
“Recently, (Wilson 2019) introduced the harmonic mean p value using Bayesian model averaging as an alternative to Bonferroni correction.”
MethodLu, H., Zhang, J., Jiang, Z., Zhang, M., Wang, T., Zhao, H., & Zeng, P. (2021) · Frontiers in Genetics · doi:10.3389/fgene.2021.656545 · Cited by 21
Mathur, M. B., Peacock, J., Robinson, T. N., & Gardner, C. D. (2021) · Nutrients · doi:10.3390/nu13124555 · Cited by 23
“We tested the global null hypothesis that the intervention affected none of the secondary outcomes by calculating harmonic mean p-values [59] for all secondary outcomes considered together, for all secondary food outcomes considered together, and for all exploratory attitude outcomes considered together.”
MethodNakamura, J. S., Węziak‐Białowolska, D., Woodberry, R. D., Kubzansky, L. D., Shiba, K., Padgett, R. N., Johnson, B. R., & VanderWeele, T. J. (2025) · Scientific Reports · doi:10.1038/s41598-025-96009-3 · Cited by 22
Breen, C. E., Papale, L. A., Clark, L. R., Bergmann, P. E., Madrid, A., Asthana, S., Johnson, S. C., Keleş, S., Alisch, R. S., & Hogan, K. J. (2023) · Alzheimer s & Dementia · doi:10.1002/alz.13514 · Cited by 20
Qin, T., Lee, C., Li, S., Cavalcante, R. G., Orchard, P., Yao, H., Zhang, H., Wang, S., Patil, S. B., Boyle, A. P., et al. (2022) · Genome biology · doi:10.1186/s13059-022-02668-0 · Cited by 20
“The enrichment results were combined 640 across EnTDefs for each GO term by taking the Harmonic Mean (HM) p-values[75].”
MethodKim, Y.-I., VanderWeele, T. J., & Johnson, B. R. (2025) · Scientific Reports · doi:10.1038/s41598-024-78201-z · Cited by 21
“Within each country, a global test of association of each childhood predictor variable group with outcome was conducted, and a pooled p-value 60 across countries reported concerning evidence for association within any country.”
Torralvo, K., de Fraga, R., LIMA, A. P., Dayrell, J. S., & Magnusson, W. E. (2021) · Biotropica · doi:10.1111/btp.13053 · Cited by 18
“We also tested the original variables with the harmonicmeanp R- package, a statistical technique used for addressing multiple- comparison questions that controls the probability of making one or more false discoveries, or type I errors, when performing multiple hypotheses tests (Wilson, 2019).”
MethodIs There Evidence of P-Hacking in Imaging Research?
Rooprai, P., Islam, N., Salameh, J., Ebrahimzadeh, S., Kazi, A., Frank, R., Ramsay, T., Mathur, M. B., Absi, M., Khalil, A. A., et al. (2022) · Canadian Association of Radiologists Journal · doi:10.1177/08465371221139418 · Cited by 18
High-Altitude Andean H194R HIF2A Allele Is a Hypomorphic Allele
Jorgensen, K. C., Song, D., Weinstein, J. B., Garcia, O. A., Pearson, L. N., Inclán, M., Rivera, M. C., Lèon‐Velarde, F., Kiyamu, M., Brutsaert, T. D., et al. (2023) · Molecular Biology and Evolution · doi:10.1093/molbev/msad162 · Cited by 18
“We chose to use the HMP instead of Fisher’s method as it works to combine separate tests performed on dependent data sets whereas Fisher’s method is robust for combining separate tests completed on independent data sets.”
MethodInfluentialProjective independence tests in high dimensions: the curses and the cures
Yao-wu, Z., & Zhu, L. (2023) · Biometrika · doi:10.1093/biomet/asad070 · Cited by 18
Bradshaw, M., Kent, B. V., Levin, J., Wortham, J. S., Pertel, N. L., VanderWeele, T. J., & Johnson, B. R. (2025) · BMC Global and Public Health · doi:10.1186/s44263-025-00141-1 · Cited by 19
Lomas, T., Padgett, R. N., Ritchie‐Dunham, J. L., Lee, M. T., Pawelski, J. O., Shiba, K., Johnson, B. R., & VanderWeele, T. J. (2025) · Journal of Happiness Studies · doi:10.1007/s10902-024-00822-y · Cited by 21
“Within each country, a global test of variation of outcome across levels of each particular demographic variable was conducted, and a pooled p-value (Wilson, 2019 ) across countries reported concerning evidence for variation within any country.”
Nakamura, J. S., Gibson, C. B., Woodberry, R. D., Lee, M. T., Kim, Y.-I., Shiba, K., Padgett, R. N., Johnson, B. R., & VanderWeele, T. J. (2025) · Scientific Reports · doi:10.1038/s41598-025-05459-2 · Cited by 23
“This pooled p -value 46 across countries reported evidence for variation within any country.”
Lomas, T., Koga, H. K., Padgett, R. N., Pawelski, J. O., Kim, E. S., Makridis, C. A., Gundersen, C., Bradshaw, M., Pertel, N. L., Shiba, K., et al. (2026) · Scientific Reports · doi:10.1038/s41598-026-35777-y · Cited by 19
Mining GWAS and eQTL data for CF lung disease modifiers by gene expression imputation
Dang, H., Polineni, D., Pace, R. G., Stonebraker, J. R., Corvol, H., Cutting, G. R., Drumm, M. L., Strug, L. J., O’Neal, W. K., & Knowles, M. R. (2020) · PLoS ONE · doi:10.1371/journal.pone.0239189 · Cited by 18
“For significant modifier genes from each analysis platform, a p-value 0.01 from both the HMP, and correlation adjusted method (EBM for PrediXcan, or omnibus for TWAS) was chosen.”
MethodInfluentialQuick, C., Wen, X., Abecasis, G. R., Boehnke, M., & Kang, H. M. (2020) · PLoS Genetics · doi:10.1371/journal.pgen.1009060 · Cited by 20
“The unadjusted HMP p-value is defined While this statistic can be anti-conservative when directly interpreted as a p-value, Wilson (2019) showed that 1/ p HMP follows a Landau distribution (with scale and location parameters given in Table 1), which can be used to compute an asymptotically exact…”
MethodInfluentialGeneral-purpose Unsupervised Cyber Anomaly Detection via Non-negative Tensor Factorization
Eren, M. E., Moore, J. S., Skau, E. W., Moore, E., Bhattarai, M., Chennupati, G., & Alexandrov, B. S. (2022) · Digital Threats Research and Practice · doi:10.1145/3519602 · Cited by 23
“We achieve this summarization using p-value fusion of dependent p-values over the dimensions of the tensor [62].”
MethodCombining P-Values Via AveragingPreprint
Vovk, V., & Wang, R. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3166304 · Cited by 20
Cowden, R. G., Worthington, E. L., Padgett, R. N., Felton, C., Weziak-Bialowolska, D., Wilkinson, R., Jackson-Meyer, K., Chen, Z. J., Bradshaw, M., Johnson, B. R., et al. (2026) · npj Mental Health Research · doi:10.1038/s44184-026-00187-5 · Cited by 18
Metadata
- Authors
- Wilson, D. J.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Wilson, Daniel J.
- ORCID
- Wilson, Daniel J.
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2019-01-04
- Volume
- 116
- Issue
- 4
- Pages
- 1195-1200
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC6347718
- OpenAlex
- W2907099021
- Cited by
- 449
- References
- 48
- Citations by year
- 2026: 58 · 2025: 115 · 2024: 44 · 2023: 54 · 2022: 61 · 2021: 57 · 2020: 45 · 2019: 12
- FWCI
- 34.33
- Type
- journal article
- Language
- en
- Update notices
- correction (2019-10-07)
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Wellcome (101237/Z/13/Z)
- Royal Society (101237/Z/13/Z)
- RCUK | Medical Research Council (MR/K01532X/1)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genetic Associations and Epidemiology
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Keywords
- family-wise error rate, statistical power, generalized central limit theorem, group testing, genome-wide association study, neuroticism, Benjamini-Hochberg procedure, false discovery rate
- MeSH terms
- Hepatitis C, Humans, Models, Statistical, Hepacivirus, Viral Load, Genome-Wide Association Study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.