The harmonic mean p -value for combining dependent tests
Open accessPublished version of a preprint
The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
Significance: The widespread use of Bonferroni correction encumbers the scientific process and wastes opportunities for discovery presented by big data, because it discourages exploratory analyses by overpenalizing the total number of statistical tests performed. In this paper, I introduce the harmonic mean p -value (HMP), a simple to use and widely applicable alternative to Bonferroni correction motivated by Bayesian model averaging that greatly improves statistical power while maintaining control of the gold standard false positive rate. The HMP has a range of desirable properties and offers a different way to think about large-scale exploratory data analysis in classical statistics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“We then aggregated the values of porig using the harmonic mean p-value to test the global null hypothesis that none of the pairs were statistically inconsistent (Wilson, 2019), yielding an aggregated porig of 0.0005, which is strong evidence of some inconsistency in the tested pairs.”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Wilson, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · cited 3×
“Controlling the FDR guarantees that, among the significant tests, the proportion in which the null hypothesis is incorrectly rejected in favor of the alternative is limited (9).”
BackgroundMethodEstimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
Guindon, S., Dufayard, J.-F., Lefort, V., Anisimova, M., Hordijk, W., & Gascuel, O. (2010) · Systematic Biology · doi:10.1093/sysbio/syq010 · Cited by 18,376
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490 · cited 1×
“The widely used Benjamini–Hochberg (BH) procedure (9) for controlling the FDR shares with the Bonferroni method a robustness to positive correlation between tests (10) but is less conservative.”
MethodStatistical significance for genomewide studies
Storey, J. D., & Tibshirani, R. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1530509100 · Cited by 9,692
Handbook of Personality: Theory and Research
Krueger, R. F., & Johnson, W. (2008) · Cited by 7,884
A haplotype map of the human genome
The International HapMap Consortium (2005) · Nature · doi:10.1038/nature04226 · Cited by 5,858
A Direct Approach to False Discovery Rates
Storey, J. D. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00346 · Cited by 5,603
METAL: fast and efficient meta-analysis of genomewide association scans
Willer, C. J., Li, Y., & Abecasis, G. R. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq340 · Cited by 5,205
Research works printed but not linked: 1
- The Royal Society 2017 Machine Learning: The Power and Promise of Computers That Learn by Example The Royal Society London
Cited by
Items 201–300 of 460
Chen, Y., Węziak‐Białowolska, D., Kim, E. S., Nakamura, J. S., Hanson, J. A., Padgett, R. N., Johnson, B. R., & VanderWeele, T. J. (2025) · Scientific Reports · doi:10.1038/s41598-024-83539-5 · Cited by 9
Kim, E. S., Bradshaw, M., Chen, Y., Chopik, W. J., Okuzono, S. S., Wilkinson, R., Padgett, R. N., Lachman, M. E., Johnson, B. R., & VanderWeele, T. J. (2025) · Scientific Reports · doi:10.1038/s41598-025-03045-0 · Cited by 8
“Within each country, a Wald-type global test of association of each childhood predictor variable group with outcome was conducted, and a pooled p -value 47 across countries reported evidence for an association within any country.”
Childhood predictors of religious reading: a cross-national analysis in the Global Flourishing Study
Węziak‐Białowolska, D., Cowden, R. G., Białowolski, P., Bradshaw, M., Padgett, R. N., Johnson, B. R., & VanderWeele, T. J. (2025) · Scientific Reports · doi:10.1038/s41598-025-10142-7 · Cited by 8
A cross-national analysis of childhood predictors of physical pain
Macchia, L., Okafor, C. N., Breedlove, T., Shiba, K., Piper, A., Johnson, B. R., & VanderWeele, T. J. (2025) · Communications Medicine · doi:10.1038/s43856-025-00997-2 · Cited by 8
“Within each country, a global test of association of each childhood predictor variable group with physical pain was conducted, and a pooled p -value 37 across countries reported concerning evidence for association within any country.”
Causal Discovery of Stochastic Dynamical Systems: A Markov Chain Approach
Stippinger, M., Bencze, A., Zlatniczki, Á., Somogyvári, Z., & Telcs, A. (2023) · Mathematics · doi:10.3390/math11040852 · Cited by 8
Yang, Y., Kaur, R., Dutta, S., & Lee, I. (2024) · ACM Transactions on Cyber-Physical Systems · doi:10.1145/3643892 · Cited by 15
“With inputs coming in a time-series sequence to the CPS, we propose to use the Harmonic Mean p -value (HMP) method [63] to combine conformal predictions from individual inputs (in the sequence) for detection on the sequence.”
InfluentialSociodemographic variations of belief in life after death across 22 Countries
Chen, Z. J., Cowden, R. G., Moreira‐Almeida, A., Breedlove, T., Kent, B. V., Padgett, R. N., Johnson, B. R., & VanderWeele, T. J. (2025) · Scientific Reports · doi:10.1038/s41598-024-83541-x · Cited by 9
Treefrogs exploit temporal coherence to form perceptual objects of communication signals
Gupta, S., & Bee, M. A. (2020) · Biology Letters · doi:10.1098/rsbl.2020.0573 · Cited by 8
“However, because all 12 replicates tested the same null hypothesis, we correct for multiple testing by reporting results for each choice test as the harmonic mean p-value (HMP) [49] of the 12 separate binomial tests across the 12 dependent replicates.”
MethodTesting partial conjunction hypotheses under dependency, with applications to meta-analysis
Bogomolov, M. (2023) · Electronic Journal of Statistics · doi:10.1214/22-ejs2100 · Cited by 12
“, [38], [42], [56], [18], [58], and references therein.”
MethodForking Paths in Empirical StudiesPreprint
Coqueret, G. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.3999379 · Cited by 12
GUIDE deconstructs genetic architectures using association studiesPreprint
Lazarev, D., Chau, G., Bloemendal, A., Churchhouse, C. L., & Neale, B. M. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.05.03.592285 · Cited by 9
“1) can be combined using the harmonic mean ( cf. [68]).”
Lee, C. G., Kwon, E., Paltzer, J., Okafor, C. N., VanderWeele, T. J., Johnson, B. R., & Choi, J. H. (2025) · BMC Public Health · doi:10.1186/s12889-025-23594-3 · Cited by 7
“Within each country, a global test of variation of outcome across levels of each particular demographic variable was conducted, and a pooled p -value [32] across countries reported concerning evidence for variation within any country.”
Behavioral Analysis of Information Salience in Large Language Models
Trienes, J., Schlötterer, J., Li, J. J., & Seifert, C. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1204 · Cited by 7
Padgett, R. N., Felton, C., Bradshaw, M., Chen, Y. (., Cowden, R. G., Kim, E. S., Wilkinson, R., Johnson, B. R., & VanderWeele, T. J. (2026) · BMC Global and Public Health · doi:10.1186/s44263-026-00287-6 · Cited by 8
Liu, X., Cong, X. S., Li, G., Maas, K. R., & Chen, K. (2021) · Statistics in Medicine · doi:10.1002/sim.9273 · Cited by 8
Nguyen, T.-D., Itayama, T., Iwami, N., Shimizu, K., Dao, T., Pham, L. T., Tran, V. Q., & Maseda, H. (2023) · Drug and Chemical Toxicology · doi:10.1080/01480545.2023.2239524 · Cited by 7
“…the Bonferroni method is not recommended for hypotheses with positive dependence assumptions since it tends to yield over-conservative outcomes (Wilson 2019). conversely, the By correction becomes less conservative than the Bonferroni method at high-ranked raw p values, making it a more…”
On new omnibus tests of uniformity on the hypersphere
Fernández-de-Marcos, A., & García‐Portugués, E. (2023) · Test · doi:10.1007/s11749-023-00882-x · Cited by 14
“Recently, the Harmonic Mean P -value (HMP), a method that tests whether no p -value is significant in a set of p -values, has been proposed in Wilson (2019).”
MethodInfluentialTesting with p*-values: Between p-values, mid p-values, and e-values
Wang, R. (2024) · Bernoulli · doi:10.3150/23-bej1633 · Cited by 10
Chandurkar, M. K., Mittal, N., Royer, S. P., Lehmann, S. D., Michels, E. B., Haarman, S. E., Severance, S. A., Rho, Y., & Han, S. J. (2024) · American Journal of Physiology-Heart and Circulatory Physiology · doi:10.1152/ajpheart.00067.2024 · Cited by 7
Chen, Y., Case, B. W., Long, K. N. G., Woodberry, R. D., Kim, E. S., Padgett, R. N., Johnson, B. R., & VanderWeele, T. J. (2025) · Scientific Reports · doi:10.1038/s41598-025-24020-9 · Cited by 7
multiMiAT: an optimal microbiome-based association test for multicategory phenotypes
Sun, H. L., Wang, Y., Xiao, Z. Z., Huang, X., Wang, H., He, T., & Jiang, X. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad012 · Cited by 7
Kim, E. S., Padgett, R. N., Bradshaw, M., Chen, Y., Chopik, W. J., Okuzono, S. S., Wilkinson, R., Lachman, M. E., Johnson, B. R., & VanderWeele, T. J. (2025) · Scientific Reports · doi:10.1038/s41598-025-15304-1 · Cited by 7
“Within each country, a global test of variation of sense of mastery across categories of each particular demographic variable was conducted, and a pooled p value 45 across countries was also reported concerning evidence for variation within any country.”
cDNA-detector: detection and removal of cDNA contamination in DNA sequencing libraries
Qi, M., Nayar, U., Ludwig, L. S., Wagle, N., & Rheinbay, E. (2021) · BMC Bioinformatics · doi:10.1186/s12859-021-04529-2 · Cited by 7
Subuniformity of harmonic mean p‐values
Chen, Y., Wang, R., Wang, Y., & Zhu, W. (2025) · Canadian Journal of Statistics · doi:10.1002/cjs.70017 · Cited by 8
“Moreover, we have 𝑀 − 1 ≤ 𝑆 ; this inequality is in Theorem 3 of Chen et al. (2023) and was also discussed in section 4 of the online appendix of Wilson (2019), but one can also check it directly.”
InfluentialA semiparametric mixture method for local false discovery rate estimation from multiple studies
Jeong, S.-O., Choi, D., & Jang, W. (2020) · The Annals of Applied Statistics · doi:10.1214/20-aoas1341 · Cited by 8
Waves of Colonization and Gene Flow in a Great Speciator
Gyllenhaal, E. F., Brady, S. S., DeCicco, L. H., Naikatini, A. N., Hime, P. M., Manthey, J. D., Kelly, J. K., Moyle, R. G., & Andersen, M. J. (2025) · Systematic Biology · doi:10.1093/sysbio/syaf023 · Cited by 7
van Kruining, D., Losen, M. H., Dehairs, J., Swinnen, J. V., Waelkens, E., Honing, M., & Martínez‐Martínez, P. (2025) · Neurobiology of Disease · doi:10.1016/j.nbd.2025.106923 · Cited by 6
“The resulting p - values were aggregated using the Harmonic Mean P -value (HMP) method ( “ harmonicmeanp ” R package (Wilson, 2019)), which effectively accounts for dependent p-values and improves the detection of combined signals across lipid species.”
Cross-national variation in faith sharing across religious traditions
Woodberry, R. D., Bradshaw, M., Weele, T. J. V., & Johnson, B. R. (2025) · Scientific Reports · doi:10.1038/s41598-024-83531-z · Cited by 6
Childhood predictors of belief in life after death across 22 countries
Chen, Z. J., Cowden, R. G., Moreira‐Almeida, A., Breedlove, T., Kent, B. V., Padgett, R. N., Johnson, B. R., & VanderWeele, T. J. (2025) · Scientific Reports · doi:10.1038/s41598-025-91615-7 · Cited by 7
“Within each country, a global test of association of each childhood predictor variable group with outcome was conducted, and a pooled p-value 50 across countries reported concerning evidence for association within any country.”
KATP channel dependent heart multiome atlas
Arrell, D. K., Park, S., Yamada, S., Alekseev, A. E., Garmany, A., Jeon, R., Vučković, I., Lindor, J. Z., & Terzic, A. (2022) · Scientific Reports · doi:10.1038/s41598-022-11323-4 · Cited by 6
Intergenomic signatures of coevolution between Tasmanian devils and an infectious cancer
Gallinson, D. G., Kozakiewicz, C. P., Rautsaw, R. M., Beer, M. A., Ruiz‐Aravena, M., Comte, S., Hamilton, D. G., Kerlin, D. H., McCallum, H. I., Hamede, R., et al. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2307780121 · Cited by 7
Blood transcriptome bridges orthogonal immunotypes and gut microbiome enterotypes in humans
Affaticati, F., Hà, M., Gehrmann, T., Boeck, I. D., Kuznetsova, M., Vandoren, R., Deuren, V. M. L. V., Alcedo, S., Jansens, H., Reu, H. D., et al. (2026) · Cell Reports · doi:10.1016/j.celrep.2026.117752 · Cited by 6
Rethinking carcinogenesis: The detached pericyte hypothesis
Baker, S. G. (2020) · Medical Hypotheses · doi:10.1016/j.mehy.2020.110056 · Cited by 6
Avoiding Conflict: When Speaker Coordination Does Not Require Conceptual Agreement
Kabbach, A., & Herbelot, A. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.523920 · Cited by 9
Węziak‐Białowolska, D., Białowolski, P., Cowden, R. G., Jang, S. J., Bradshaw, M., Padgett, R. N., Johnson, B. R., & VanderWeele, T. J. (2025) · Journal of Research in Personality · doi:10.1016/j.jrp.2025.104627 · Cited by 7
Paltzer, J., Carter, E. W., Kwon, E., Okafor, C. N., Padgett, R. N., Benfer, J., VanderWeele, T. J., & Johnson, B. R. (2025) · BMC Global and Public Health · doi:10.1186/s44263-025-00190-6 · Cited by 6
“Within each country, a global test of variation of outcome across levels of each particular demographic variable was conducted, and a pooled p -value [36] across countries was reported concerning evidence for variation within any country.”
Wilkinson, R., Shiba, K., Gibson, C. B., Okafor, C. N., Chen, Y., Padgett, R. N., Johnson, B. R., & VanderWeele, T. J. (2025) · Social Indicators Research · doi:10.1007/s11205-025-03743-2 · Cited by 6
“Within each country, a global test of variation of social relationship quality across levels of each sociodemographic variable was conducted, and a pooled p-value (Wilson, 2019) across countries reported evidence for variation within any country.”
Kernel mean embedding based hypothesis tests for comparing spatial point patterns
Rustamov, R. M., & Klosowski, J. T. (2020) · Spatial Statistics · doi:10.1016/j.spasta.2020.100459 · Cited by 8
“We carry out the latter by applying t-tests on each coordinate of the embedding, and obtain an overall p-value using the harmonic mean p-value combination technique [17, 37], leading to well-calibrated and powerful tests as confirmed by the simulations.”
BackgroundMethodInfluentialCharacterization and clustering of kinase isoform expression in metastatic melanoma
Holland, D. O., Gotea, V., Fedkenheuer, K., Jaiswal, S. K., Baugher, C., Tan, H., Fedkenheuer, M., & Elnitski, L. L. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010065 · Cited by 6
“We combined the p-values from each principal component with both Fisher’s method (FM) and the asymptotically exact harmonic mean (HMP) from DJ Wilson [42].”
MethodReply to Held: When is a harmonic mean p -value a Bayes factor?
Wilson, D. J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1902157116 · Cited by 8
“5, [7]) and deriving an asymptotically exact test for general use (Eq.”
BackgroundMethodSelf-citationNovel probabilistic similarity scores for sets of replicate EI mass spectra
Krishnaswamy–Usha, A., Capistran, B. A., & Kearsley, A. J. (2025) · Analytica Chimica Acta · doi:10.1016/j.aca.2025.343858 · Cited by 5
Childhood predictors of adults’ belief in god, gods, and spiritual forces across 22 countries
Moon, J. W., Johnson, K. A., Case, B. W., Padgett, R. N., Johnson, B. R., & VanderWeele, T. J. (2025) · Scientific Reports · doi:10.1038/s41598-025-98796-1 · Cited by 5
Biological traits predict species’ time-varying responses to multiple global change drivers
Sasaki, T., Iwachido, Y., Lam‐Gordillo, O., Cook, K. M., Douglas, E. J., Gladstone‐Gallagher, R. V., Greenfield, B. L., Hailes, S. F., Carter, K., Ishii, N. I., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-70606-w · Cited by 5
“To determine the meta-signi fi cance of CCM tests across estuarine sites, we applied a recently proposed method to combine P -values 103 .”
Sukprasert, S., Thanasutives, P., Araki, S., Uranishi, K., Matsuo, T., & Shimadera, H. (2026) · Atmospheric Environment · doi:10.1016/j.atmosenv.2026.122006 · Cited by 5
The homogeneous MeerKAT and Swift /XRT X-ray binary radio:X-ray plane
Crook-Mansour, J., Fender, R. P., Hughes, A. K., Motta, S., Woudt, P. A., Bahramian, A., Santo, M. D., Zhang, Z., Russell, T. D., van den Eijnden, J., et al. (2026) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stag1022 · Cited by 5
Linville, R. M., James, B. T., Galani, K., Ho, L., Shin, J. H., Oliver, E., Fass, S. B., Cameron, J. C., Fitzwalter, B. E., Bock, R., et al. (2026) · Cell · doi:10.1016/j.cell.2026.08.006 · Cited by 5
Inter-regional correlation estimators for functional magnetic resonance imaging
Achard, S., Coeurjolly, J., de Micheaux, P. L., Lbath, H., & Richiardi, J. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.120388 · Cited by 6
“We then computed the average W value across runs, as well as the harmonic mean p-value (Wilson, 2019) across runs, a procedure with strong family-wise error rate (FWER) control even for positively dependent tests.”
Meta-analysis of RNA interaction profiles of RNA-binding protein using the RBPInper tool
Cogan, J. A., Benova, N., Kuklinkova, R., Boyne, J. R., & Anene, C. A. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbae127 · Cited by 5
“…optional arguments in the function call to specify whether the P -values should be integrated with alternative methods [i.e. cell group-specific—dependence assumed: Bonferroni (default) or Harmonic mean (Wilson 2019), and global—independence assumed: Fisher (default) or Binomial (Wilkinson 1951)].”
MethodYue, Y., Mao, Y., Read, T. D., Fedirko, V., Satten, G. A., Chen, X., Zhan, X., & Hu, Y. (2025) · Journal of the American Statistical Association · doi:10.1080/01621459.2025.2516205 · Cited by 6
Tomicki, S., Dieguez, G., DeStephano, D. M., Chang, M. M., & Cockrum, P. C. (2022) · ClinicoEconomics and Outcomes Research · doi:10.2147/ceor.s373316 · Cited by 5
Zaldo-Aubanell, Q., Mas-Ponce, A., Asbert, G., Clota, B., Isnard, M., Proia, L., Bach, A., & Mateo, S. S. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e34615 · Cited by 6
“Similar to other research, we partitioned tests into separate groups [43], in our case, considering each independent variable as separate families of comparisons with their own set of tests.”
ResultZhang, J., Yan, R., Chen, Y., Liu, Y., Wang, P., Chen, Z., & Sun, H. L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127514 · Cited by 6
Early life experiences and adult orientation to promote good in 22 countries
Chen, Y., Kim, E. S., Nakamura, J. S., Węziak‐Białowolska, D., Wilkinson, R., Padgett, R. N., Johnson, B. R., & VanderWeele, T. J. (2025) · Scientific Reports · doi:10.1038/s41598-025-03343-7 · Cited by 5
Childhood antecedents of adult place satisfaction in 22 countries
Counted, V., Węziak‐Białowolska, D., Cowden, R. G., Johnson, B. R., & VanderWeele, T. J. (2025) · Scientific Reports · doi:10.1038/s41598-025-00731-x · Cited by 5
A cross-national analysis of childhood predictors of daily smoking in adulthood
Jang, S. J., de la Rosa, P. A., Padgett, R. N., Bradshaw, M., VanderWeele, T. J., & Johnson, B. R. (2025) · Communications Medicine · doi:10.1038/s43856-025-01005-3 · Cited by 5
“Within each country, a global test of association of each childhood predictor group with outcome was conducted, and a pooled p -value 44 across countries reported concerning evidence for association within any country.”
Палионная, С. И., & Shestakov, O. V. (2020) · Mathematics · doi:10.3390/math8111913 · Cited by 5
“The papers [4,5] are devoted to the harmonic mean p-value (HMP) method.”
MethodZinc Transporter 9 (SLC30A9) Expression Is Decreased in the Vaginal Tissues of Menopausal Women
Csikós, A., Kozma, B., Pór, Á., Kovács, I., Lampé, R., Miklós, I., & Takács, P. (2021) · Biological Trace Element Research · doi:10.1007/s12011-020-02525-w · Cited by 5
Inferring differential subcellular localisation in comparative spatial proteomics using BANDLEPreprint
Crook, O. M., Davies, C. T. R., Breckels, L. M., Christopher, J. A., Gatto, L., Kirk, P., & Lilley, K. S. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.01.04.425239 · Cited by 6
“Thirdly, it is inappropriate to cube p -values: to combine p -value across experiments one could use Fisher’s method (Mosteller and Fisher, 1948; Brown, 1975; Kost and McDermott, 2002) or the Harmonic mean p -value (HMP) (Good, 1958; Wilson, 2019) depending on the context.”
MethodRech, E. V. F., de Oliveira, L. E. D. L., & Sant’Ana, W. C. (2025) · IEEE Access · doi:10.1109/access.2025.3615553 · Cited by 4
“The HMP is particularly useful when there is a mix of weak and strong signals across different tests [29].”
Chen, W., & Lai, J. H. K. (2025) · Building and Environment · doi:10.1016/j.buildenv.2025.113791 · Cited by 5
Confidence maps: statistical inference of cryo-EM maps
Beckers, M., Palmer, C. M., & Sachse, C. (2020) · Acta Crystallographica Section D Structural Biology · doi:10.1107/s2059798320002995 · Cited by 4
“Multiple testing is a major field of research in statistical inference (Wilson, 2019; Zhang et al. , 2019; Ignatiadis et al. , 2016), and several additional error rates and error-controlling procedures beyond Benjamini–Yekutieli FDR control have been proposed.”
BackgroundToward the fabric of the Milky Way: I. The density of disk streams from a local 2503 pc3 volume
Ratzenböck, S., Alves, J., Hunt, E. L., Miret-Roig, N., Meingast, S., & Möller, T. B. (2025) · Astronomy and Astrophysics · doi:10.1051/0004-6361/202451866 · Cited by 4
Furlan, C., Diez, M. S., & Saccenti, E. (2025) · Cancers · doi:10.3390/cancers17162601 · Cited by 4
“For each protein, the 1000 p -values were collected and pooled to a single p -value using the harmonic mean approach for combining the p -values of dependent tests [63].”
Węziak‐Białowolska, D., Cwynar, A., Białowolski, P., Cowden, R. G., Wilkinson, R., Padgett, R. N., Johnson, B. R., & VanderWeele, T. J. (2025) · Social Science & Medicine · doi:10.1016/j.socscimed.2025.118622 · Cited by 4
Chi, J. T., Ipsen, I. C. F., Hsiao, T., Lin, C., Wang, L., Lee, W., Lu, T., & Tzeng, J. (2021) · Frontiers in Genetics · doi:10.3389/fgene.2021.710055 · Cited by 4
“Then we combine these pj’s across annotation classes using the Cauchy Combination Test (Liu and Xie, 2020) or the Harmonic Mean p-value (Wilson, 2019), and obtain the final p-value of the variant set.”
MethodAll Burglaries Are Not the Same: Predicting Near-Repeat Burglaries in Cities Using Modus Operandi
Borg, A., & Svensson, M. G. A. (2022) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi11030160 · Cited by 4
Haplotype based testing for a better understanding of the selective architecture
Chen, H., Pelizzola, M., & Futschik, A. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05437-3 · Cited by 5
Zhang, S., Jiang, Z., & Zeng, P. (2024) · Journal of Translational Medicine · doi:10.1186/s12967-024-05053-6 · Cited by 4
“It has been demonstrated that HMP is robust against positive dependency among combined P values [30, 41].”
BackgroundMethodInfluentialWang, J., Wu, D., DeLorenzo, C., & Yang, J. (2024) · PLoS ONE · doi:10.1371/journal.pone.0299625 · Cited by 6
New statistical framework for interlaboratory evaluation of anti-doping testing results by WADA
Meija, J., Possolo, A. M., Garrido, B. C., Kisoona, S., & Barroso, O. (2024) · Accreditation and Quality Assurance · doi:10.1007/s00769-024-01595-w · Cited by 4
“While various methods have been used to assess the laboratory bias in such cases, including rank-sum analysis [35] or pooling the z -scores [39, 43], the use of a mixed effects model provides a direct assessment of the bias of each laboratory and its uncertainty without the need for further…”
Childhood Predictors of Weekly Alcohol Use in Adulthood: A Cross-National Analysis
de la Rosa, P. A., Jang, S. J., Paltzer, J., Okafor, C. N., Bradshaw, M., Padgett, R. N., Johnson, B. R., & VanderWeele, T. J. (2025) · Journal of Child and Family Studies · doi:10.1007/s10826-025-03126-8 · Cited by 4
“Within each country, a global test of association of each childhood predictor variable group with outcome was conducted, and a pooled p -value (Wilson, 2019) across countries reported concerning evidence for association within any country.”
Demographic variation in weekly alcohol use across countries in the Global Flourishing Study
de la Rosa, P. A., Jang, S. J., Paltzer, J., Okafor, C. N., Bradshaw, M., Padgett, R. N., Johnson, B. R., & VanderWeele, T. J. (2026) · Communications Medicine · doi:10.1038/s43856-025-01346-z · Cited by 4
Chica, N., Andersen, A. N., Orellana‐Muñoz, S., Garcia, I., Aurélie Nguéa, P., Nakken, S., Durán, P. A., Håkensbakken, L., Schultz, S. W., Rødningen, E., et al. (2026) · Nature Cell Biology · doi:10.1038/s41556-025-01837-0 · Cited by 4
Error control in tree structured hypothesis testing
Miecznikowski, J. C., & Wang, J. (2022) · Wiley Interdisciplinary Reviews Computational Statistics · doi:10.1002/wics.1603 · Cited by 5
Talkar, T., Low, D. M., Simpkin, A. J., Ghosh, S., O'Keeffe, D. T., & Quatieri, T. F. (2023) · Scientific Reports · doi:10.1038/s41598-023-27934-4 · Cited by 6
Decoded cardiopoietic cell secretome linkage to heart repair biosignature
Garmany, A., Arrell, D. K., Yamada, S., Jeon, R., Behfar, A., Park, S., & Terzic, A. (2024) · Stem Cells Translational Medicine · doi:10.1093/stcltm/szae067 · Cited by 4
“34,35 Ingenuity Pathway Analysis (v70750971) was applied to identify enriched biological functions and network interactions.”
McIlwain, S. J., Hoefges, A., Erbe, A. K., Sondel, P. M., & Ong, I. M. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae637 · Cited by 5
Paltzer, J., Kwon, E., Okafor, C. N., Padgett, R. N., Carter, E. W., Benfer, J., VanderWeele, T. J., & Johnson, B. R. (2025) · BMC Global and Public Health · doi:10.1186/s44263-025-00188-0 · Cited by 4
Reply to Goeman et al.: Trade-offs in model averaging using multilevel tests
Wilson, D. J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1910684116 · Cited by 4
“There were two errors in Wilson [1] which I have corrected [2], but I do not accept the four claimed by Goeman, Rosenblatt and Nichols [3]; I rebut them point-by-point in an appendix (https://doi.”
BackgroundSelf-citationProbabilistic forecasting of replication studiesPreprint
Pawel, S., & Held, L. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/fhwb7 · Cited by 4
Loh, Z. H. M., Clark, S., & van der Werf, J. H. J. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.02.19.481168 · Cited by 4
“…the Bonferoni method has also been criticized when applied to GWAS as with very large numbers of SNPs tested, it has been perceived as being overconservative, leading to reduced power in identifying causal variants (Wilson 2019; Gao et al. 2010; Huang et al. 2018; Llinares-López et al. 2015).”
MethodPredictive Data Calibration for Linear Correlation Significance TestingPreprint
Patil, K. R., Eickhoff, S. B., & Langner, R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.07081 · Cited by 5
Marshall, R. D., Ribbers, A., Freeman, I., Nguyen, S., Magers, J., Maitland, L., & Sheridan, D. C. (2024) · Psychiatric Services · doi:10.1176/appi.ps.20230445 · Cited by 3
Cognitive Control Network and Language Reorganization in Patients with Brain Tumors
Pasquini, L., Napolitano, A., Spitoni, L., Schmid, M., Dellepiane, F., Jenabi, M., Peck, K. K., & Holodny, A. I. (2024) · American Journal of Neuroradiology · doi:10.3174/ajnr.a8638 · Cited by 3
Training self-regulation to promote healthy eating: Evidence from a longitudinal intervention study
Lopez, R. B., Werner, K. M., Traub, G., Saunders, B., Cosme, D., & Hofmann, W. (2025) · Appetite · doi:10.1016/j.appet.2025.108310 · Cited by 3
Gene Flow Complicates Phylogenetic Inference in an Archipelago Radiation
Gyllenhaal, E. F., Klicka, L. B., DeCicco, L. H., Weeks, B. C., Moyle, R. G., & Andersen, M. J. (2025) · Systematic Biology · doi:10.1093/sysbio/syaf081 · Cited by 3
“We also calculated the harmonic mean P -value for the same categories ( Wilson 2019 ).”
Racek, D., Zhang, Q., Thurner, P. W., Zhu, X. X., & Kauermann, G. (2025) · PNAS Nexus · doi:10.1093/pnasnexus/pgaf367 · Cited by 5
“We do this by constructing the weighted harmonic mean P -value (HMP) (21) defined as where B refers to the building, t to the time period, N to the number of pixels that make up the building, p j , t the P -value of a pixel j that is part of building B , and w j its corresponding weight based on…”
InfluentialDesign and implementation of an automated Drosophila locomotor assay using computer vision tracking
Melkani, D., Harnwal, N., Desai, S., Patel, D., & Melkani, G. C. (2026) · Scientific Reports · doi:10.21203/rs.3.rs-8769384/v1 · Cited by 3
“(cid:1872) = (cid:1868) ≤ 0.001, (cid:1868) ≤ Since multiple p-values are associated with each comparison genotype (cid:3040) (main effect + all interaction terms), the Harmonic Mean p-value (HMP) method [15] is used to combine them.”
InfluentialTesting Uniform Random Samplers: Methods, Datasets, and Protocols
Zeyen, O., Cordy, M., Gubri, M., Perrouin, G., & Acher, M. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3797477 · Cited by 3
“We finish by introducing a test inspired by the birthday paradox and adapted from an existing test on PRNGs [32].”
Covariate balanced allocation of samples to batches to mitigate the impacts of technical variability
Mulvey, J. F., & Lundby, A. (2026) · Bioinformatics Advances · doi:10.1093/bioadv/vbag048 · Cited by 3
“Where multiple covariates are present, by default we define a ‘balance score’ as the harmonic mean of the P -val- ues of all of these tests: allowing their combination into a single metric without making strict assumptions of their independence (Wilson 2019).”
Probing the statistical correlation of optical tidal disruption events with high-energy neutrinos
Langis, D. A., Liodakis, I., Koljonen, K. I. I., & Kouch, P. M. (2026) · Astronomy and Astrophysics · doi:10.1051/0004-6361/202556514 · Cited by 3
Combining power of different methods to detect associations in large data sets
He, L., Zhang, H., & Jiang, H. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab488 · Cited by 3
Wyenberg-Henzler, T. (2022) · PeerJ · doi:10.7717/peerj.13174 · Cited by 3
“This process was repeated over 1,000 iterations and the resulting p -values were combined using the harmonic mean p method, which is designed to combine multiple p -values from non-independent tests into a single value ( Wilson, 2019 ).”
Jumanne, A. S., Njoroge, J., & Moi, E. J. (2023) · American Journal of Public Policy and Administration · doi:10.47672/ajppa.1716 · Cited by 3
“Hesitation to either agree or disagree to some extent justifies Razavi, Kapiriri, Abelson and Wilson (2019) argument that vulnerable groups were never involved in stakeholder participation and that conflicting priorities, values and interest were among the main challenges facing stakeholder…”
BackgroundMethodInfluentialUsher, C. T., Freeman, I., Wesemann, D. G., Ross, E., Fleisher, C., & Choi-Kain, L. W. (2024) · Academic Psychiatry · doi:10.1007/s40596-024-01984-w · Cited by 3
“Due to the moderate number (14) of correlated tests, the harmonic mean p -value was calculated using R-package “harmonicmeanp” to control for false positives as an alternative to the Bonferroni correction; a small harmonic mean p -value suggests adequate control of potential false positives [20].”
Du, C., & Chen, T. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3686673 · Cited by 6
Vriend, C., Fitzsimmons, S., Aarts, I., Broekhuizen, A., van der Werf, Y. D., Douw, L., Visser, H. A. D., Thomaes, K., & van den Heuvel, O. A. (2025) · NeuroImage Clinical · doi:10.1016/j.nicl.2025.103870 · Cited by 3
“For the global topology measures we calculated the harmonic mean p-value across folds which can handle dependencies between p-values and is less stringent than a Bonferroni correction (Wilson, 2019).”
REVA as A Well-Curated Database for Human Expression-Modulating Variants
Wang, Y., Shi, F., Liang, Y., & Gao, G. (2021) · Genomics Proteomics & Bioinformatics · doi:10.1016/j.gpb.2021.06.001 · Cited by 3
“The harmonic mean P value (HMP) method [25] was used in the meta-analysis, and the cutoff for the meta P value was set to 0.”
MethodInteractive Analysis, Exploration, and Visualization of RNA-Seq Data with SeqCVIBE
Bothos, E., Hatzis, P., & Moulos, P. (2022) · Methods and Protocols · doi:10.3390/mps5020027 · Cited by 3
“Furthermore, they come with a variety of underlying statistical models, often leading to disconnected outcomes as a result of the different initial assumptions and requiring algorithms such as PANDORA [8] or the Harmonic mean p-value combination [28] to construct summarized results.”
BackgroundThe Effect of Stress on Semantic Memory Retrieval: A Multiverse Analysis
Heyman, T., Boere, R., de Jong, S., Hoogeterp, L., Kraaijenbrink, J., Kuipers, C., van Dijk, M., van Rijn, L., & van Wijk, T. (2022) · Collabra Psychology · doi:10.1525/collabra.35745 · Cited by 3
Metadata
- Authors
- Wilson, D. J.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Wilson, Daniel J.
- ORCID
- Wilson, Daniel J.
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2019-01-04
- Volume
- 116
- Issue
- 4
- Pages
- 1195-1200
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC6347718
- OpenAlex
- W2907099021
- Cited by
- 449
- References
- 48
- Citations by year
- 2026: 58 · 2025: 115 · 2024: 44 · 2023: 54 · 2022: 61 · 2021: 57 · 2020: 45 · 2019: 12
- FWCI
- 34.33
- Type
- journal article
- Language
- en
- Update notices
- correction (2019-10-07)
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Wellcome (101237/Z/13/Z)
- Royal Society (101237/Z/13/Z)
- RCUK | Medical Research Council (MR/K01532X/1)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genetic Associations and Epidemiology
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Keywords
- family-wise error rate, statistical power, generalized central limit theorem, group testing, genome-wide association study, neuroticism, Benjamini-Hochberg procedure, false discovery rate
- MeSH terms
- Hepatitis C, Humans, Models, Statistical, Hepacivirus, Viral Load, Genome-Wide Association Study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.