MultipleTesting.com: A tool for life science researchers for multiple hypothesis testing correction
Open accessPublished version of a preprint
The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Abstract
Scientists from nearly all disciplines face the problem of simultaneously evaluating many hypotheses. Conducting multiple comparisons increases the likelihood that a non-negligible proportion of associations will be false positives, clouding real discoveries. Drawing valid conclusions require taking into account the number of performed statistical tests and adjusting the statistical confidence measures. Several strategies exist to overcome the problem of multiple hypothesis testing. We aim to summarize critical statistical concepts and widely used correction approaches while also draw attention to frequently misinterpreted notions of statistical inference. We provide a step-by-step description of each multiple-testing correction method with clear examples and present an easy-to-follow guide for selecting the most suitable correction technique. To facilitate multiple-testing corrections, we developed a fully automated solution not requiring programming skills or the use of a command line. Our registration free online tool is available at www.multipletesting.com and compiles the five most frequently used adjustment tools, including the Bonferroni, the Holm (step-down), the Hochberg (step-up) corrections, allows to calculate False Discovery Rates (FDR) and q-values. The current summary provides a much needed practical synthesis of basic statistical concepts regarding multiple hypothesis testing in a comprehensible language with well-illustrated examples. The web tool will fill the gap for life science researchers by providing a user-friendly substitute for command-line alternatives.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“To account for multiple comparisons, Holm-adjusted p-values were used as the primary criterion for interpretation [26].”
From Characterization of human-animal interactions among pet owners in Mexico from a One Health perspective: A cross-sectional study (2026) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490
Statistical significance for genomewide studies
Storey, J. D., & Tibshirani, R. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1530509100 · Cited by 9,692
A Direct Approach to False Discovery Rates
Storey, J. D. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00346 · Cited by 5,603
Thresholding of statistical maps in functional neuroimaging using the false discovery rate
Genovese, C. R., Lazar, N. A., & Nichols, T. (2002) · NeuroImage · doi:10.1006/nimg.2001.1037 · Cited by 4,923
A sharper Bonferroni procedure for multiple tests of significance
HOCHBERG, Y. (1988) · Biometrika · doi:10.1093/biomet/75.4.800 · Cited by 4,829
Multiple significance tests: the Bonferroni method
Bland, J. M., & Altman, D. G. (1995) · BMJ · doi:10.1136/bmj.310.6973.170 · Cited by 3,341
Scientists rise up against statistical significance
Amrhein, V., Greenland, S., & McShane, B. (2019) · Nature · doi:10.1038/d41586-019-00857-9 · Cited by 2,983
Rectangular Confidence Regions for the Means of Multivariate Normal Distributions
Šidák, Z. (1967) · Journal of the American Statistical Association · doi:10.1080/01621459.1967.10482935 · Cited by 2,639
The positive false discovery rate: a Bayesian interpretation and the q-value
Storey, J. D. (2003) · The Annals of Statistics · doi:10.1214/aos/1074290335 · Cited by 2,204
Statistical Methods for Research Workers.
I., L., & Fisher, R. A. (1926) · Journal of the Royal Statistical Society · doi:10.2307/2341488 · Cited by 2,128
Empirical Bayes analysis of a microarray experiment
Efron, B., Tibshirani, R., Storey, J. D., & Tusher, V. (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753382129 · Cited by 1,733
Teoria Statistica Delle Classi e Calcolo Delle Probabilità
(2010) · Encyclopedia of Research Design · doi:10.4135/9781412961288.n455 · Cited by 1,698
Rectangular Confidence Regions for the Means of Multivariate Normal Distributions
Šidák, Z. (1967) · Journal of the American Statistical Association · doi:10.2307/2283989 · Cited by 1,455
False discovery rates: a new deal
Stephens, M. (2016) · Biostatistics · doi:10.1093/biostatistics/kxw041 · Cited by 1,067
Implementing false discovery rate control: increasing your power
Verhoeven, K. J. F., Simonsen, K. L., & McIntyre, L. M. (2005) · Oikos · doi:10.1111/j.0030-1299.2005.13727.x · Cited by 974
Data-driven hypothesis weighting increases detection power in genome-scale multiple testing
Ignatiadis, N., Klaus, B., Zaugg, J. B., & Huber, W. (2016) · Nature Methods · doi:10.1038/nmeth.3885 · Cited by 649
A practical guide to methods controlling false discoveries in computational biology
Korthauer, K., Kimes, P. K., Duvallet, C., Reyes, A., Subramanian, A., Teng, M., Shukla, C., Alm, E. J., & Hicks, S. C. (2019) · Genome Biology · doi:10.1186/s13059-019-1716-1 · Cited by 436
Il calcolo delle assicurazioni su gruppi di teste
Bonferroni, C., & Bonferroni, C. (1935) · Medical Entomology and Zoology · Cited by 434
Multiple hypothesis testing in genomics
Goeman, J. J., & Solari, A. (2014) · Statistics in Medicine · doi:10.1002/sim.6082 · Cited by 375
False discovery control with p-value weighting
Genovese, C. R., Roeder, K., & Wasserman, L. (2006) · Biometrika · doi:10.1093/biomet/93.3.509 · Cited by 320
On multiple‐testing correction in genome‐wide association studies
Moskvina, V., & Schmidt, K. M. (2008) · Genetic Epidemiology · doi:10.1002/gepi.20331 · Cited by 264
Teoria Statistica Delle Classi e Calcolo Delle Probabilità
(2022) · The SAGE Encyclopedia of Research Design · doi:10.4135/9781071812082.n627 · Cited by 261
Technical trading revisited: False discoveries, persistence tests, and transaction costs
Bajgrowicz, P., & Scaillet, O. (2012) · Journal of Financial Economics · doi:10.1016/j.jfineco.2012.06.001 · Cited by 253
Capturing variation impact on molecular interactions in the IMEx Consortium mutations data set
The IMEx Consortium Curators, Khadake, J., Meldal, B., Panni, S., Thorneycroft, D., van Roey, K., Abbani, S., Salwinski, L., Pellegrini, M., Iannuccelli, M., et al. (2019) · Nature Communications · doi:10.1038/s41467-018-07709-6 · Cited by 247
Robust estimation of the false discovery rate
Pounds, S., & Cheng, C. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl328 · Cited by 230
False Discovery Rates and Related Statistical Concepts in Mass Spectrometry-Based Proteomics
Choi, H., & Nesvizhskii, A. I. (2008) · Journal of Proteome Research · doi:10.1021/pr700747q · Cited by 217
AdaPT: An Interactive Procedure for Multiple Testing with Side Information
Lei, L., & Fithian, W. (2018) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/rssb.12274 · Cited by 202
Bretz, F., Posch, M., Glimm, E., Klinglmueller, F., Maurer, W., & Rohmeyer, K. (2011) · Biometrical Journal · doi:10.1002/bimj.201000239 · Cited by 172
Controlling the Proportion of False Positives in Multiple Dependent Tests
Fernando, R. L., Nettleton, D., Southey, B. R., Dekkers, J. C. M., Rothschild, M. F., & Soller, M. (2004) · Genetics · doi:10.1534/genetics.166.1.611 · Cited by 165
Controlling the rate of Type I error over a large set of statistical tests
Keselman, H. J., Cribbie, R., & Holland, B. (2002) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711002159680 · Cited by 155
Simultaneous Testing of Grouped Hypotheses: Finding Needles in Multiple Haystacks
Cai, T., & Sun, W. (2009) · Journal of the American Statistical Association · doi:10.1198/jasa.2009.tm08415 · Cited by 135
A direct approach to estimating false discovery rates conditional on covariates
Boca, S. M., & Leek, J. T. (2018) · PeerJ · doi:10.7717/peerj.6035 · Cited by 109
Sidak, Z. (1971) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177693504 · Cited by 104
Scott, J. G., Kelly, R. C., Smith, M. A., Zhou, P., & Kass, R. E. (2015) · Journal of the American Statistical Association · doi:10.1080/01621459.2014.990973 · Cited by 102
Krzywinski, M. I., & Altman, N. (2014) · Nature Methods · doi:10.1038/nmeth.2858 · Cited by 90
Krzywinski, M. I., & Altman, N. (2014) · Nature Methods · doi:10.1038/nmeth.2900 · Cited by 86
Technical Trading Revisited: False Discoveries, Persistence Tests, and Transaction CostsPreprint
Bajgrowicz, P., & Olivier Scaillet, B. (2011) · SSRN Electronic Journal · doi:10.2139/ssrn.1095202 · Cited by 74
Comments on the analysis of unbalanced microarray data
Kerr, K. F. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp363 · Cited by 57
Gaus, W. (2015) · Clinical and Experimental Pharmacology · doi:10.4172/2161-1459.1000182 · Cited by 51
Retiring statistical significance would give bias a free pass
Ioannidis, J. P. A. (2019) · Nature · doi:10.1038/d41586-019-00969-2 · Cited by 47
Advances in p-Value Based Multiple Test Procedures
Tamhane, A. C., & Gou, J. (2017) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543406.2017.1378666 · Cited by 45
Zhang, M. J., Xia, F., & Zou, J. (2019) · Nature Communications · doi:10.1038/s41467-019-11247-0 · Cited by 45
Multiple comparisons in mass-spectrometry-based -omics technologies
Franceschi, P., Giordan, M., & Wehrens, R. (2013) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2013.04.011 · Cited by 32
Multiple Testing in Large-Scale Genetic Studies
Bouaziz, M., Jeanmougin, M., & Guedj, M. (2012) · Methods in molecular biology · doi:10.1007/978-1-61779-870-2_13 · Cited by 24
Estimating the proportion of true null hypotheses when the statistics are discrete
Dialsingh, I., Austin, S., & Altman, N. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv104 · Cited by 19
Yurko, R., G’Sell, M., Roeder, K., & Devlin, B. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1918862117 · Cited by 17
AdaPT: An interactive procedure for multiple testing with side informationPreprint
Lei, L., & Fithian, W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.06035 · Cited by 11
Research works printed but not linked: 2
- 42. KS P, S D, MJ vdL. Multiple Testing Procedures: R multtest Package and Applications to Genomics, in Bioinformatics and Computational Biology Solutions Using R and Bioconductor. Springer. 2005.
- 43. Storey JD BA, Dabney A, Robinson D. qvalue: Q-value estimation for false discovery rate control. R package version 2220. 2020.
Cited by
Győrffy, B. (2024) · The Innovation · doi:10.1016/j.xinn.2024.100625 · Cited by 464
Self-citationExtended difficulties following the use of psychedelic drugs: A mixed methods study
Evans, J., Robinson, O. C., Argyri, E. K., Suseelan, S., Murphy-Beiner, A., McAlpine, R., Luke, D., Michelle, K., & Prideaux, E. (2023) · PLOS ONE · doi:10.1371/journal.pone.0293349 · Cited by 185
Predictive biomarkers of immunotherapy response with pharmacological applications in solid tumors
Kovács, S. A., Fekete, J. T., & Győrffy, B. (2023) · Acta Pharmacologica Sinica · doi:10.1038/s41401-023-01079-6 · Cited by 195
Self-citationRosati, D., Palmieri, M., Brunelli, G., Morrione, A., Iannelli, F., Frullanti, E., & Giordano, A. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.02.018 · Cited by 175
DNA methylation-based diagnostic, prognostic, and predictive biomarkers in colorectal cancer
Müller, D., & Győrffy, B. (2022) · Biochimica et Biophysica Acta (BBA) - Reviews on Cancer · doi:10.1016/j.bbcan.2022.188722 · Cited by 174
Self-citation
Metadata
- Authors
- Menyhart, O., Weltz, B., & Győrffy, B.
- Affiliations
- Institute of Molecular Life Sciences (HU)
- Semmelweis University (HU)
- Corresponding author
- Győrffy, Balázs
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2021-06-09
- Volume
- 16
- Issue
- 6
- Pages
- e0245824
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC8189492
- OpenAlex
- W3168713078
- Cited by
- 216
- References
- 48
- Citations by year
- 2026: 48 · 2025: 51 · 2024: 56 · 2023: 29 · 2022: 23 · 2021: 9
- FWCI
- 21.81
- Type
- journal article
- Language
- en
- Update notices
- correction (2022-09-09)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Nemzeti Kutatási Fejlesztési és Innovációs Hivatal (2018-2.1.17-TET-KR-00001, 2020-1.1.6-JÖVŐ-2021-00013, 2018-1.3.1-VKE-2018-00032, 2020-4.1.1.-TKP2020)
- ELIXIR Hungary
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Keywords
- multiple comparisons, multiple testing correction, false positives, Bonferroni correction, false discovery rate, Q-value, statistical inference, life science research
- MeSH terms
- Biological Science Disciplines, Data Interpretation, Statistical, Humans, Models, Theoretical, Software, Statistics as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.