Online control of the familywise error rate
Open accessAbstract
Biological research often involves testing a growing number of null hypotheses as new data is accumulated over time. We study the problem of online control of the familywise error rate (FWER), that is testing an apriori unbounded sequence of hypotheses (p-values) one by one over time without knowing the future, such that with high probability there are no false discoveries in the entire sequence. This paper unifies algorithmic concepts developed for offline (single batch) FWER control and online false discovery rate control to develop novel online FWER control methods. Though many offline FWER methods (e.g. Bonferroni, fallback procedures and Sidak's method) can trivially be extended to the online setting, our main contribution is the design of new, powerful, adaptive online algorithms that control the FWER when the p-values are independent or locally dependent in time. Our experiments demonstrate substantial gains in power, that are also formally proved in a Gaussian sequence model. Multiple testing, FWER control, online setting.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Online family-wise error control handles potentially unbounded hypothesis sequences; confidence sequences and betting methods support optional stopping.
“Online family-wise error control handles potentially unbounded hypothesis sequences (Tian & Ramdas, 2019); confidence sequences and betting methods support optional stopping (Howard et al., 2021;Waudby-Smith & Ramdas, 2024).”
From SGM: A Statistical Godel Machine for Risk-Controlled Recursive Self-Modification (2025) · cites this paper as Tian & Ramdas, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
More powerful procedures for multiple significance testing
Hochberg, Y., & Benjamini, Y. (1990) · Statistics in Medicine · doi:10.1002/sim.4780090710 · Cited by 2,715 · see citing works
Association of Random Variables, with Applications
Esary, J. D., Proschan, F., & Walkup, D. W. (1967) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177698701 · Cited by 963
Plots of P-values to evaluate many tests simultaneously
Schweder, T., & Spjøtvoll, E. (1982) · Biometrika · doi:10.1093/biomet/69.3.493 · Cited by 440 · see citing works
A graphical approach to sequentially rejective multiple test procedures
Bretz, F., Maurer, W., Brannath, W., & Posch, M. (2009) · Statistics in Medicine · doi:10.1002/sim.3495 · Cited by 420 · see citing works
Prevalence of sexual dimorphism in mammalian phenotypic traits
Karp, N. A., Mason, J., Beaudet, A. L., Benjamini, Y., Bower, L., Braun, R. E., Brown, S. D. M., Chesler, E. J., Dickinson, M. E., Flenniken, A. M., et al. (2017) · Nature Communications · doi:10.1038/ncomms15475 · Cited by 257 · see citing works
A recycling framework for the construction of Bonferroni‐based multiple tests
Burman, C., Sonesson, C., & Guilbaud, O. J. M. (2009) · Statistics in Medicine · doi:10.1002/sim.3513 · Cited by 136 · see citing works
The Fallback Procedure for Evaluating a Single Family of Hypotheses
Wiens, B. L., & Dmitrienko, A. A. (2005) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543400500265660 · Cited by 127 · see citing works
The sequential rejection principle of familywise error control
Goeman, J. J., & Solari, A. (2010) · The Annals of Statistics · doi:10.1214/10-aos829 · Cited by 113 · see citing works
α-Investing: a Procedure for Sequential Control of Expected False Discoveries
Foster, D. P., & Stine, R. A. (2008) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2007.00643.x · Cited by 111 · see citing works
Online rules for control of false discovery rate and false discovery exceedance
Javanmard, A., & Andrea Montanari, A. (2018) · The Annals of Statistics · doi:10.1214/17-aos1559 · Cited by 97 · see citing works
Finner, H., & Gontscharuk, V. (2009) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2009.00719.x · Cited by 69 · see citing works
Generalized α-investing: Definitions, Optimality Results and Application to Public Databases
Aharoni, E., & Rosset, S. (2013) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/rssb.12048 · Cited by 49 · see citing works
On adaptive procedures controlling the familywise error rate
Sarkar, S. K., Guo, W., & Finner, H. (2011) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2011.06.022 · Cited by 32 · see citing works
Zhao, Q., Small, D. S., & Su, W. (2018) · Journal of the American Statistical Association · doi:10.1080/01621459.2018.1497499 · Cited by 23 · see citing works
Asynchronous Online Testing of Multiple HypothesesPreprint
Zrnic, T., Ramdas, A., & Jordan, M. I. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.05068 · Cited by 18 · see citing works
Self-citationGaining power in multiple testing of interval hypotheses via conditionalization
Ellis, J. L., Pecanka, J., & Goeman, J. (2018) · arXiv · doi:10.48550/arxiv.1801.00141
Flagged first, then by citations
Research works printed but not linked: 9
- [3][3] John Storey. A direct approach to false discovery rates. Journal of the Royal Statistical Society, Series B (Statistical Methodology) , 64:479-498, 2002. -
- [8][8] Aaditya Ramdas, Fanny Yang, Martin Wainwright, and Michael Jordan. Online control of the false discovery rate with decaying memory. In Advances In Neural Information Processing Systems , pages 5655-5664, 2017. -
- [9][9] Aaditya Ramdas, Tijana Zrnic, Martin Wainwright, and Michael Jordan. SAFFRON: an adaptive algorithm for online control of the false discovery rate. In Proceedings of the 35th International Conference on Machine Learning , volume 80, pages 4286-4294, 2018. -
- [10][10] Jinjin Tian and Aaditya Ramdas. ADDIS: adaptive algorithms for online FDR control with conservative nulls. In Proceedings of the 33rd Conference on Neural Information Processing Systems , 2019. -
- [11][11] Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics , pages 65-70, 1979. -
- [12][12] Yosef Hochberg. A sharper bonferroni procedure for multiple tests of significance. Biometrika , 75(4):800-802, 1988. -
- [14][14] Zbynˇ ek Šidák. Rectangular confidence regions for the means of multivariate normal distributions. Journal of the American Statistical Association , 62(318):626-633, 1967. -
- [22][22] Ward Whitt. Uniform conditional stochastic order. Journal of Applied Probability , 17(1):112-123, 1980. -
- [24][24] Wenge Guo. A note on adaptive bonferroni and holm procedures under dependence. Biometrika , 96(4):10121018, 2009. -
Cited by
Sorted by citations
SGM: A Statistical Godel Machine for Risk-Controlled Recursive Self-Modification
Wu, X., Yin, S., Kang, Y., Zhang, X., Xu, Q., Chen, Z., & Zhang, W. (2025) · arXiv · doi:10.48550/arxiv.2510.10232
- Authors
- Tian, J., & Ramdas, A.
- Affiliations
- Carnegie Mellon University (US)
- Corresponding author
- Tian, Jinjin
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- Volume
- 30
- Issue
- 4
- Pages
- 976-993
- ISSN
- 0962-2802 (ISSN-L)
- arXiv
- 1910.04900
- OpenAlex
- W3120054632
- Cited by
- 18
- References
- 17
- Citations by year
- 2025: 3 · 2024: 3 · 2023: 7 · 2022: 2 · 2021: 1 · 2020: 2
- FWCI
- 2.40
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Statistical Methods in Medical Research (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- family-wise error rate, false discovery rate, sequential testing, p-values, Bonferroni procedure, local dependence, statistical power, International Mouse Phenotyping Consortium
- MeSH terms
- Algorithms, Animals, Research Design, Normal Distribution, Mice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
