The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing
Open accessAbstract
Consider the problem of testing k hypotheses simultaneously. In this article we discuss finite- and large-sample theory of stepdown methods that provide control of the familywise error rate (FWE). To improve on the Bonferroni method or on Holm's stepdown method, Westfall and Young made effective use of resampling to construct stepdown methods that implicitly estimate the dependence structure of the test statistics. However, their methods depend on an assumption known as “subset pivotality.” Our goal here is to construct general stepdown methods that do not require such an assumption. To accomplish this, we take a close look at what makes stepdown procedures work; a key component is a monotonicity requirement of critical values. By imposing monotonicity on estimated critical values (which is not an assumption on the model but rather is an assumption on the method), we show how to construct stepdown tests that can be applied in a stagewise fashion so that at most k tests need to be computed. Moreover, at each stage, an intersection test that controls the usual probability of a type 1 error is calculated, which allows us to draw on an enormous resampling literature as a general means of test construction. In addition, it is possible to carry out this method using the same set of resamples (or subsamples) for each of the intersection tests.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Similar to traditional multiple-hypothesis testing procedures, such as the Bonferroni or Holm procedures (see, e.g., Lehmann and Romano (2005), for a discussion of these procedures), the stepdown algorithm of Romano and Wolf (2005) exhibits strong FWER control, in contrast with the classical tests like the F or χ2.”
From Analyzing social experiments as implemented: A reexamination of the evidence from the HighScope Perry Preschool Program (2010) · cites this paper as Romano and Wolf (2005)
The cited source says what the claim attributes to it.
In the citing paper
“The validity of this procedure follows from the analysis of Romano and Wolf (2005), who provide general results on the use of stepdown multiple-hypothesis testing procedures.”
From Analyzing social experiments as implemented: A reexamination of the evidence from the HighScope Perry Preschool Program (2010) · cites this paper as Romano and Wolf (2005)
The cited source says what the claim attributes to it.
In the citing paper
“Romano and Wolf (2005) show that their monotonicity assumption is weaker than the subset pivotality assumption.”
From Analyzing social experiments as implemented: A reexamination of the evidence from the HighScope Perry Preschool Program (2010) · cites this paper as Romano and Wolf (2005)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490
Multiple Comparison Procedures
Hochberg, Y., & Tamhane, A. C. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316672 · Cited by 2,389
The Bootstrap and Edgeworth Expansion
Hall, P. (1992) · Springer series in statistics · doi:10.1007/978-1-4612-4384-7 · Cited by 2,102
On closed testing procedures with special reference to ordered analysis of variance
Marcus, R., Peritz, E., & Gabriel, K. R. (1976) · Biometrika · doi:10.1093/biomet/63.3.655 · Cited by 1,417
Multiple Hypothesis Testing in Microarray Experiments
Dudoit, S., Shaffer, J. P., & Boldrick, J. C. (2003) · Statistical Science · doi:10.1214/ss/1056397487 · Cited by 1,054
Politis, D. N., Romano, J. P., & Wolf, M. (1999) · Springer Series in Statistics · doi:10.1007/978-1-4612-1554-7 · Cited by 760
Self-citationJeankyung, K., & Pollard, D. (1990) · The Annals of Statistics · doi:10.1214/aos/1176347498 · Cited by 640
Two Guidelines for Bootstrap Hypothesis Testing
Hall, P., & Wilson, S. R. (1991) · Biometrics · doi:10.2307/2532163 · Cited by 471
The Large-Sample Power of Tests Based on Permutations of Observations
Hoeffding, W. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729436 · Cited by 379
Bootstrap and Randomization Tests of some Nonparametric Hypotheses
Romano, J. P. (1989) · The Annals of Statistics · doi:10.1214/aos/1176347007 · Cited by 199
Self-citationOn the Behavior of Randomization Tests Without a Group Invariance Assumption
Romano, J. P. (1990) · Journal of the American Statistical Association · doi:10.1080/01621459.1990.10474928 · Cited by 178
Self-citationBeran, R. J. (1986) · The Annals of Statistics · doi:10.1214/aos/1176349847 · Cited by 119
A Bootstrap Revival of Some Nonparametric Distance Tests
Romano, J. P. (1988) · Journal of the American Statistical Association · doi:10.1080/01621459.1988.10478650 · Cited by 110
Self-citationDelgado, M. A., Rodrı́guez-Póo, J. M., & Wolf, M. (2001) · Economics Letters · doi:10.1016/s0165-1765(01)00494-3 · Cited by 98
Self-citationA Stepwise Resampling Method of Multiple Hypothesis Testing
Troendle, J. F. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476522 · Cited by 93
Multiple test procedures for arbitrary dependence structures
Hommel, G. (1986) · Metrika · doi:10.1007/bf01894765 · Cited by 80
On Subsampling Estimators with Unknown Rate of Convergence
Bertail, P., Politis, D. N., & Romano, J. P. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10474151 · Cited by 75
Self-citationMultiple Comparisons by Rerandomization Tests
Petrondas, D. A., & Gabriel, K. R. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477045 · Cited by 64
Finner, H., & Roters, M. (1998) · The Annals of Statistics · doi:10.1214/aos/1028144847 · Cited by 35
Some tests for correlation matrices
Aitkin, M. A. (1969) · Biometrika · doi:10.1093/biomet/56.2.443 · Cited by 28
Correction: Some Tests for Correlation Matrices
Aitkin, M. A. (1971) · Biometrika · doi:10.2307/2334343 · Cited by 4
Flagged first, then by citations
Research works printed but not linked: 9
- Ser. B, 57, 289
- Association, 94, 569
- The Annals, Statistics, 26, 505
- Scandinavian Journal, Statistics, 6, 65
- Statistics, 18, 191
- Tests, American Statistical Association, 78, 949
- Journal, 43, 617
- Tests, American Statistical Association, 83, 698
- Distributions, The American Statistician, 51, 3
Cited by
Sorted by citations
Anderson, M. L. (2008) · Journal of the American Statistical Association · doi:10.1198/016214508000000841 · Cited by 2,410
Early Childhood Investments Substantially Boost Adult Health
Campbell, F., Conti, G., Heckman, J. J., Moon, S. H., Pinto, R., Pungello, E., & Pan, Y. (2014) · Science · doi:10.1126/science.1248429 · Cited by 1,106
Labor market returns to an early childhood stimulation intervention in Jamaica
Gertler, P., Heckman, J., Pinto, R., Zanolini, A., Vermeersch, C., Walker, S., Chang, S. M., & Grantham-McGregor, S. (2014) · Science · doi:10.1126/science.1251178 · Cited by 776
Behavior-dependent short-term assembly dynamics in the medial prefrontal cortex
Fujisawa, S., Amarasingham, A., Harrison, M. T., & Buzsáki, G. (2008) · Nature Neuroscience · doi:10.1038/nn.2134 · Cited by 701
Young, A. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjy029 · Cited by 637
Metadata
- Authors
- Romano, J. P., & Wolf, M.
- Affiliations
- Stanford University (US)
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2005-03
- Volume
- 100
- Issue
- 469
- Pages
- 94-108
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W3124231044
- Cited by
- 609
- References
- 23
- Citations by year
- 2026: 44 · 2025: 63 · 2024: 51 · 2023: 52 · 2022: 60 · 2021: 47 · 2020: 38 · 2019: 33 · 2018: 38 · 2017: 29 · 2016: 25 · 2015: 15 · 2014: 22 · 2013: 11 · 2012: 14
- FWCI
- 10.03
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- Journal of the American Statistical Association (publishedVersion)
- SSRN Electronic Journal (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Repositori digital de la UPF (Universitat Pompeu Fabra) (submittedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, other-oa)
- Funders
- National Science Foundation
- Ministerio de Ciencia y Tecnología
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- multiple comparisons, family-wise error rate, resampling, Bonferroni correction, intersection tests, asymptotic theory
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
