Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 424

The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.

SOME DESIRABLE PROPERTIES OF THE BONFERRONI CORRECTION: IS THE BONFERRONI CORRECTION REALLY SO BAD?

Open access
VanderWeele, Tyler J, Mathur, Maya B · 2019 · American Journal of Epidemiology

Abstract

The Bonferroni correction (1) for multiple testing is sometimes criticized as being overly conservative. The correction is indeed conservative, and there are uniformly more powerful approaches that preserve type I error of the global null hypothesis (2) (see Appendix). However, there are considerations concerning the Bonferroni correction which suggest that its use, and reporting by its standard—perhaps accompanied by other metrics as well (3)—may not be entirely undesirable. First, use of the Bonferroni correction is often motivated by the desire to preserve the type I error of the global null hypothesis of all tested associations’ being in fact null. By dividing the nominal significance level of the α test (e.g., α = 0.05) by the number of tests, one is guaranteed, within a hypothesis-testing framework, to reject the global null hypothesis of no association no more than the proportion α (e.g., α = 5%) of the time when the global null does in fact hold. While this is often the motivation presented for use of the Bonferroni correction, the correction itself does have a much stronger property. Suppose one were testing some number K of potential associations, and after Bonferroni correction, J associations were rejected at the α/K significance level. The standard property of the Bonferroni correction that is often pointed out is (as above) that no more than 5% of the time will one incorrectly conclude, “There is at least 1 true association.” However, with J rejections at an α/K significance level, one can in fact also consider the much stronger conclusion that “there are at least J true associations,” and one will draw this conclusion, when it is false, at most 5% of the time. This is because even if there were in fact only J − 1 true associations, the probability of rejecting J or more would still be less than [K − (J − 1)] × α/K < K × α/K = α. The fact that this much stronger statement, like the rejection of the global null hypothesis, also has only a 5% error rate gives the Bonferroni correction a much stronger interpretation when results surpass this more conservative threshold. Second, while the Bonferroni correction can impose a fairly severe penalty when sample sizes are small or when many tests are being conducted, in settings in which sample sizes are very large (such as many major epidemiologic cohort studies) and when only a moderate number of tests are being carried out, use of the Bonferroni correction will in fact often make relatively little difference in the magnitude of effect sizes that can generally be detected. Suppose one were examining a single exposure and its associations with a number of subsequent outcomes using data from a large cohort study, recently referred to as an outcome-wide epidemiologic study (4). Consider a recent study setting (5) with sample size n = 3,929 and with K = 24 outcomes, with a mean linear and logistic regression coefficient standard error of 0.031 across the various outcomes. In this context, for an outcome with a standard error of 0.031, an effect estimate above 0.061 would suffice to pass the nominal α = 0.05 significance level threshold and an effect size above 0.095 would suffice to pass the Bonferroni-corrected significance level threshold, of α = 0.05/24 = 0.0021. There is a relatively modest range of effect sizes, 0.061–0.095, for which the nominal significance level would be passed but the Bonferroni-corrected threshold would not be. If variability of the outcomes were similar but the sample size were n = 10,000, an effect estimate above 0.038 (e.g., an odds ratio of 1.039) would suffice to pass the nominal α = 0.05 significance level and an effect size above 0.060 (e.g., an odds ratio of 1.062) would suffice to pass the Bonferroni-corrected significance level, of α = 0.05/24 = 0.0021. Here the range of effect estimates for which the nominal significance threshold is passed but the Bonferroni-corrected one is not is even narrower, and arguably in many cases that effect size range is sufficiently narrow to often not be of much scientific or public health importance (e.g., if the odds ratio is not even 1.062, the effect size may be too small to be of much importance). Thus, with large sample sizes, in many settings, if the effect size estimate is sufficient to surpass the nominal threshold of α = 0.05, then it will very often be sufficient to pass the Bonferroni-corrected threshold as well. Indeed, in the study referred to above, with the actual sample size of n = 3,929, of the 20 P values that surpassed the nominal α = 0.05 significance level, 17 of those also surpassed the Bonferroni-corrected significance level of α = 0.05/24 = 0.0021. Moreover, as per the first point above, one could then make the statement “There are at least 17 true associations,” and, under repeated sampling, statements similarly constructed by reporting the number of Bonferroni-corrected rejections would be false less than 5% of the time. Of course, just because the Bonferroni c

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/aje/kwy250

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 16 references

Cited by

See all 424

Metadata

Authors
VanderWeele, T. J., & Mathur, M. B.
Venue
American Journal of Epidemiology
Published
2018-11-19
Volume
188
Issue
3
Pages
617-618
ISSN
  • 0002-9262 (ISSN-L)
  • 1476-6256
OpenAlex
W2901814704
Cited by
424
References
16
FWCI
9.88
Type
journal article
Language
en
Open access
bronze

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: SOME DESIRABLE PROPERTIES OF THE BONFERRONI CORRECTION: IS THE BONFERRONI CORRECTION REALLY SO BAD? https://app.cochat.ai/papers/10.1093/aje/kwy250
The paper
VanderWeele, T. J., & Mathur, M. B. (2019). SOME DESIRABLE PROPERTIES OF THE BONFERRONI CORRECTION: IS THE BONFERRONI CORRECTION REALLY SO BAD? American Journal of Epidemiology. https://doi.org/10.1093/aje/kwy250

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.