The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
SOME DESIRABLE PROPERTIES OF THE BONFERRONI CORRECTION: IS THE BONFERRONI CORRECTION REALLY SO BAD?
Open accessAbstract
The Bonferroni correction (1) for multiple testing is sometimes criticized as being overly conservative. The correction is indeed conservative, and there are uniformly more powerful approaches that preserve type I error of the global null hypothesis (2) (see Appendix). However, there are considerations concerning the Bonferroni correction which suggest that its use, and reporting by its standard—perhaps accompanied by other metrics as well (3)—may not be entirely undesirable. First, use of the Bonferroni correction is often motivated by the desire to preserve the type I error of the global null hypothesis of all tested associations’ being in fact null. By dividing the nominal significance level of the α test (e.g., α = 0.05) by the number of tests, one is guaranteed, within a hypothesis-testing framework, to reject the global null hypothesis of no association no more than the proportion α (e.g., α = 5%) of the time when the global null does in fact hold. While this is often the motivation presented for use of the Bonferroni correction, the correction itself does have a much stronger property. Suppose one were testing some number K of potential associations, and after Bonferroni correction, J associations were rejected at the α/K significance level. The standard property of the Bonferroni correction that is often pointed out is (as above) that no more than 5% of the time will one incorrectly conclude, “There is at least 1 true association.” However, with J rejections at an α/K significance level, one can in fact also consider the much stronger conclusion that “there are at least J true associations,” and one will draw this conclusion, when it is false, at most 5% of the time. This is because even if there were in fact only J − 1 true associations, the probability of rejecting J or more would still be less than [K − (J − 1)] × α/K < K × α/K = α. The fact that this much stronger statement, like the rejection of the global null hypothesis, also has only a 5% error rate gives the Bonferroni correction a much stronger interpretation when results surpass this more conservative threshold. Second, while the Bonferroni correction can impose a fairly severe penalty when sample sizes are small or when many tests are being conducted, in settings in which sample sizes are very large (such as many major epidemiologic cohort studies) and when only a moderate number of tests are being carried out, use of the Bonferroni correction will in fact often make relatively little difference in the magnitude of effect sizes that can generally be detected. Suppose one were examining a single exposure and its associations with a number of subsequent outcomes using data from a large cohort study, recently referred to as an outcome-wide epidemiologic study (4). Consider a recent study setting (5) with sample size n = 3,929 and with K = 24 outcomes, with a mean linear and logistic regression coefficient standard error of 0.031 across the various outcomes. In this context, for an outcome with a standard error of 0.031, an effect estimate above 0.061 would suffice to pass the nominal α = 0.05 significance level threshold and an effect size above 0.095 would suffice to pass the Bonferroni-corrected significance level threshold, of α = 0.05/24 = 0.0021. There is a relatively modest range of effect sizes, 0.061–0.095, for which the nominal significance level would be passed but the Bonferroni-corrected threshold would not be. If variability of the outcomes were similar but the sample size were n = 10,000, an effect estimate above 0.038 (e.g., an odds ratio of 1.039) would suffice to pass the nominal α = 0.05 significance level and an effect size above 0.060 (e.g., an odds ratio of 1.062) would suffice to pass the Bonferroni-corrected significance level, of α = 0.05/24 = 0.0021. Here the range of effect estimates for which the nominal significance threshold is passed but the Bonferroni-corrected one is not is even narrower, and arguably in many cases that effect size range is sufficiently narrow to often not be of much scientific or public health importance (e.g., if the odds ratio is not even 1.062, the effect size may be too small to be of much importance). Thus, with large sample sizes, in many settings, if the effect size estimate is sufficient to surpass the nominal threshold of α = 0.05, then it will very often be sufficient to pass the Bonferroni-corrected threshold as well. Indeed, in the study referred to above, with the actual sample size of n = 3,929, of the 20 P values that surpassed the nominal α = 0.05 significance level, 17 of those also surpassed the Bonferroni-corrected significance level of α = 0.05/24 = 0.0021. Moreover, as per the first point above, one could then make the statement “There are at least 17 true associations,” and, under repeated sampling, statements similarly constructed by reporting the number of Bonferroni-corrected rejections would be false less than 5% of the time. Of course, just because the Bonferroni c
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526
Sensitivity analysis in observational research: Introducing the E-value
VanderWeele, T. J., & Ding, P. (2017) · Annals of Internal Medicine · doi:10.7326/m16-2607 · Cited by 6,350
Self-citationNo adjustments are needed for multiple comparisons.
Rothman, K. J. (1990) · Epidemiology · doi:10.1097/00001648-199001000-00010 · Cited by 5,965
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
Multiple comparisons among means
Dunn, O. J. (1961) · Journal of the American Statistical Association · doi:10.1080/01621459.1961.10482090 · Cited by 5,006
(2009) · doi:10.1007/978-0-387-87959-8 · Cited by 554
Applying Quantitative Bias Analysis to Epidemiologic Data
Fink, A. K., Fox, M. P., & Lash, T. L. (2009) · Statistics for Biology and Health · doi:10.1007/b97920 · Cited by 343
VanderWeele, T. J. (2017) · Epidemiology · doi:10.1097/ede.0000000000000641 · Cited by 228
Self-citationControl of the mean number of false discoveries, Bonferroni and stability of multiple testing
Gordon, A., Glazko, G., Qiu, X., & Yakovlev, A. (2007) · The Annals of Applied Statistics · doi:10.1214/07-aoas102 · Cited by 176
Cited by
Outcome-Wide Longitudinal Designs for Causal Inference: A New Template for Empirical Studies
VanderWeele, T. J., Mathur, M. B., & Chen, Y. (2020) · Statistical Science · doi:10.1214/19-sts728 · Cited by 310
Self-citationCourtney, D. B., Watson, P., Krause, K. R., Chan, B. W. C., Bennett, K., Gunlicks-Stoessel, M., Rodak, T., Neprily, K., Zentner, T., & Szatmari, P. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.46331 · Cited by 63
New role of fat-free mass in cancer risk linked with genetic predisposition
Harris, B. H. L., Di Giovannantonio, M., Zhang, P., Harris, D. A., Lord, S. R., Allen, N. E., Maughan, T. S., Bryant, R. J., Harris, A. L., Bond, G. L., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-54291-7 · Cited by 24
The socioeconomic landscape of the exposome during pregnancy
Sum, K. K., Tint, M. T., Aguilera, R., Dickens, B. S. L., Choo, S., Ang, L. T., Phua, D., Law, E. C., Ng, S., Tan, K. M.-L., et al. (2022) · Environment International · doi:10.1016/j.envint.2022.107205 · Cited by 18
Mähs, M., Pithan, J. S., Bergmann, I., Gabrys, L., Graf, J., Hölzemann, A., Van Laerhoven, K., Otto-Hagemann, S., Popescu, M. L., Schwermann, L., et al. (2022) · Trials · doi:10.1186/s13063-022-06550-z · Cited by 4
Metadata
- Authors
- VanderWeele, T. J., & Mathur, M. B.
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-11-19
- Volume
- 188
- Issue
- 3
- Pages
- 617-618
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- OpenAlex
- W2901814704
- Cited by
- 424
- References
- 16
- FWCI
- 9.88
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: SOME DESIRABLE PROPERTIES OF THE BONFERRONI CORRECTION: IS THE BONFERRONI CORRECTION REALLY SO BAD? https://app.cochat.ai/papers/10.1093/aje/kwy250
- The paper
- VanderWeele, T. J., & Mathur, M. B. (2019). SOME DESIRABLE PROPERTIES OF THE BONFERRONI CORRECTION: IS THE BONFERRONI CORRECTION REALLY SO BAD? American Journal of Epidemiology. https://doi.org/10.1093/aje/kwy250
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.