Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing
Open accessAbstract
Consider the problem of testing k hypotheses simultaneously. In this article we discuss finite- and large-sample theory of stepdown methods that provide control of the familywise error rate (FWE). To improve on the Bonferroni method or on Holm's stepdown method, Westfall and Young made effective use of resampling to construct stepdown methods that implicitly estimate the dependence structure of the test statistics. However, their methods depend on an assumption known as “subset pivotality.” Our goal here is to construct general stepdown methods that do not require such an assumption. To accomplish this, we take a close look at what makes stepdown procedures work; a key component is a monotonicity requirement of critical values. By imposing monotonicity on estimated critical values (which is not an assumption on the model but rather is an assumption on the method), we show how to construct stepdown tests that can be applied in a stagewise fashion so that at most k tests need to be computed. Moreover, at each stage, an intersection test that controls the usual probability of a type 1 error is calculated, which allows us to draw on an enormous resampling literature as a general means of test construction. In addition, it is possible to carry out this method using the same set of resamples (or subsamples) for each of the intersection tests.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The stepdown algorithm of Romano and Wolf (2005) exhibits strong family-wise error rate control.
“Similar to traditional multiple-hypothesis testing procedures, such as the Bonferroni or Holm procedures (see, e.g., Lehmann and Romano (2005), for a discussion of these procedures), the stepdown algorithm of Romano and Wolf (2005) exhibits strong FWER control, in contrast with the classical tests like the F or χ2.”
From Analyzing social experiments as implemented: A reexamination of the evidence from the HighScope Perry Preschool Program (2010) · cites this paper as Romano and Wolf (2005)
The validity of the stepdown algorithm for multiple-hypothesis testing follows from the analysis of Romano and Wolf (2005).
“The validity of this procedure follows from the analysis of Romano and Wolf (2005), who provide general results on the use of stepdown multiple-hypothesis testing procedures.”
From Analyzing social experiments as implemented: A reexamination of the evidence from the HighScope Perry Preschool Program (2010) · cites this paper as Romano and Wolf (2005)
Romano and Wolf (2005) show that their monotonicity assumption is weaker than the subset pivotality assumption.
“Romano and Wolf (2005) show that their monotonicity assumption is weaker than the subset pivotality assumption.”
From Analyzing social experiments as implemented: A reexamination of the evidence from the HighScope Perry Preschool Program (2010) · cites this paper as Romano and Wolf (2005)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526 · see citing works
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490 · see citing works
Multiple Comparison Procedures
Hochberg, Y., & Tamhane, A. C. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316672 · Cited by 2,389 · see citing works
The Bootstrap and Edgeworth Expansion
Hall, P. V. (1992) · Springer series in statistics · doi:10.1007/978-1-4612-4384-7 · Cited by 2,067 · see citing works
On closed testing procedures with special reference to ordered analysis of variance
Marcus, R., Peritz, E., & Gabriel, K. R. (1976) · Biometrika · doi:10.1093/biomet/63.3.655 · Cited by 1,417 · see citing works
Multiple Hypothesis Testing in Microarray Experiments
Dudoit, S., Shaffer, J. P., & Boldrick, J. C. (2003) · Statistical Science · doi:10.1214/ss/1056397487 · Cited by 1,054 · see citing works
Politis, D. N., Romano, J. P., & Wolf, M. (1999) · Springer Series in Statistics · doi:10.1007/978-1-4612-1554-7 · Cited by 760 · see citing works
Self-citationJeankyung, K., & Pollard, D. (1990) · The Annals of Statistics · doi:10.1214/aos/1176347498 · Cited by 640 · see citing works
Two Guidelines for Bootstrap Hypothesis Testing
Hall, P., & Wilson, S. R. (1991) · Biometrics · doi:10.2307/2532163 · Cited by 471 · see citing works
The Large-Sample Power of Tests Based on Permutations of Observations
Hoeffding, W. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729436 · Cited by 379 · see citing works
Bootstrap and Randomization Tests of some Nonparametric Hypotheses
Romano, J. P. (1989) · The Annals of Statistics · doi:10.1214/aos/1176347007 · Cited by 204 · see citing works
Self-citationOn the Behavior of Randomization Tests Without a Group Invariance Assumption
Romano, J. P. (1990) · Journal of the American Statistical Association · doi:10.1080/01621459.1990.10474928 · Cited by 178 · see citing works
Self-citationBeran, R. J. (1986) · The Annals of Statistics · doi:10.1214/aos/1176349847 · Cited by 119 · see citing works
A Bootstrap Revival of Some Nonparametric Distance Tests
Romano, J. P. (1988) · Journal of the American Statistical Association · doi:10.1080/01621459.1988.10478650 · Cited by 110 · see citing works
Self-citationDelgado, M. A., Rodrı́guez-Póo, J. M., & Wolf, M. (2001) · Economics Letters · doi:10.1016/s0165-1765(01)00494-3 · Cited by 98 · see citing works
Self-citationA Stepwise Resampling Method of Multiple Hypothesis Testing
Troendle, J. F. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476522 · Cited by 93 · see citing works
Multiple test procedures for arbitrary dependence structures
Hommel, G. (1986) · Metrika · doi:10.1007/bf01894765 · Cited by 80 · see citing works
On Subsampling Estimators with Unknown Rate of Convergence
Bertail, P., Politis, D. N., & Romano, J. P. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10474151 · Cited by 75 · see citing works
Self-citationMultiple Comparisons by Rerandomization Tests
Petrondas, D. A., & Gabriel, K. R. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477045 · Cited by 64 · see citing works
Finner, H., & Roters, M. (1998) · The Annals of Statistics · doi:10.1214/aos/1028144847 · Cited by 35 · see citing works
Some tests for correlation matrices
Aitkin, M. A. (1969) · Biometrika · doi:10.1093/biomet/56.2.443 · Cited by 28 · see citing works
Correction: Some Tests for Correlation Matrices
Aitkin, M. A. (1971) · Biometrika · doi:10.2307/2334343 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Hochberg, Y. and Tamhane, A. (1987). Multple Comparison Procedures. Wiley, New York.
- Politis, D. N., Romano, J. P., and Wolf, M. (1999). Subsampling. Springer, New York.
- Westfall, P. H. and Young, S. S. (1993). Resampling-Based Multiple Testing: Examples and Methods for P-Value Adjustment. John Wiley, New York.
Cited by
Sorted by citations
Anderson, M. L. (2008) · Journal of the American Statistical Association · doi:10.1198/016214508000000841 · Cited by 2,410 · see citing works
Early Childhood Investments Substantially Boost Adult Health
Campbell, F., Conti, G., Heckman, J. J., Moon, S. H., Pinto, R., Pungello, E., & Pan, Y. (2014) · Science · doi:10.1126/science.1248429 · Cited by 1,106 · see citing works
Labor market returns to an early childhood stimulation intervention in Jamaica
Gertler, P., Heckman, J., Pinto, R., Zanolini, A., Vermeersch, C., Walker, S., Chang, S. M., & Grantham-McGregor, S. (2014) · Science · doi:10.1126/science.1251178 · Cited by 776 · see citing works
Behavior-dependent short-term assembly dynamics in the medial prefrontal cortex
Fujisawa, S., Amarasingham, A., Harrison, M. T., & Buzsáki, G. (2008) · Nature Neuroscience · doi:10.1038/nn.2134 · Cited by 679 · see citing works
Young, A. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjy029 · Cited by 637 · see citing works
Estimating Marginal Returns to Education
Carneiro, P., Heckman, J. J., & Vytlacil, E. J. (2011) · American Economic Review · doi:10.1257/aer.101.6.2754 · Cited by 612 · see citing works
Multiple hypothesis testing in experimental economics
List, J. A., Shaikh, A. M., & Xu, Y. (2019) · Experimental Economics · doi:10.1007/s10683-018-09597-5 · Cited by 465 · see citing works
The Formation of Prosociality: Causal Evidence on the Role of Social Environment
Kosse, F., Deckers, T., Pinger, P., Schildberg-Hörisch, H., & Falk, A. (2016) · Journal of Political Economy · doi:10.1086/704386 · Cited by 396 · see citing works
Attanasio, O. P., Fernandez, C., Fitzsimons, E., Grantham‐McGregor, S., Meghir, C., & Rubio‐Codina, M. (2014) · BMJ · doi:10.1136/bmj.g5785 · Cited by 340 · see citing works
Chernozhukov, V., Chetverikov, D., & Kato, K. (2013) · The Annals of Statistics · doi:10.1214/13-aos1161 · Cited by 325 · see citing works
Randomization Inference with Stata: A Guide and Software
Heß, S. (2017) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x1701700306 · Cited by 320 · see citing works
Efficient computation of adjusted p -values for resampling-based stepdown multiple testing
Romano, J. P., & Wolf, M. (2016) · Statistics & Probability Letters · doi:10.1016/j.spl.2016.02.012 · Cited by 307 · see citing works
Self-citationChernozhukov, V., Chetverikov, D., & Kato, K. (2013) · doi:10.1920/wp.cem.2013.7613 · Cited by 312 · see citing works
(2010) · Quantitative Economics · doi:10.3982/qe8 · Cited by 304 · see citing works
“Lehmann and Romano (2005) and Romano and Wolf (2005) discuss the stepdown procedure in depth.”
InfluentialOn the External Validity of Social Preference Games: A Systematic Lab-Field Study
Galizzi, M. M., & Navarro‐Martinez, D. (2018) · Management Science · doi:10.1287/mnsc.2017.2908 · Cited by 267 · see citing works
The Romano–Wolf multiple-hypothesis correction in Stata
Clarke, D., Romano, J. P., & Wolf, M. (2020) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x20976314 · Cited by 269 · see citing works
Self-citationInference for the Identified Set in Partially Identified Econometric Models
Romano, J. P., & Shaikh, A. M. (2010) · Econometrica · doi:10.3982/ecta6706 · Cited by 263 · see citing works
Self-citationTranscriptional modulation of the developing immune system by early life social adversity
Cole, S. W., Conti, G., Arevalo, J. M. G., Ruggiero, A. M., Heckman, J. J., & Suomi, S. J. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1218253109 · Cited by 207 · see citing works
Parent-Child Information Frictions and Human Capital Investment: Evidence from a Field Experiment
Bergman, P. (2020) · Journal of Political Economy · doi:10.1086/711410 · Cited by 219 · see citing works
Facts, alternative facts, and fact checking in times of post-truth politics
Barrera, Ó., Guriev, S. M., Henry, E., & Zhuravskaya, E. V. (2020) · Journal of Public Economics · doi:10.1016/j.jpubeco.2019.104123 · Cited by 206 · see citing works
Facts, Alternative Facts, and Fact Checking in Times of Post-Truth PoliticsPreprint
Barrera, Ó., Guriev, S. M., Henry, E., & Zhuravskaya, E. V. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3004631 · Cited by 197 · see citing works
Multiple Hypothesis Testing in Experimental Economics
List, J. A., Shaikh, A. M., & Xu, Y. (2016) · National Bureau of Economic Research · doi:10.3386/w21875 · Cited by 188 · see citing works
FORMALIZED DATA SNOOPING BASED ON GENERALIZED ERROR RATES
Romano, J. P., Shaikh, A. M., & Wolf, M. (2007) · Econometric Theory · doi:10.1017/s0266466608080171 · Cited by 185 · see citing works
Self-citationControl of generalized error rates in multiple testing
Romano, J. P., & Wolf, M. (2007) · The Annals of Statistics · doi:10.1214/009053606000001622 · Cited by 181 · see citing works
Self-citationHaushofer, J., & Shapiro, J. (2013) · Cited by 181 · see citing works
Long-Term Propranolol Use in Severely Burned Pediatric Patients
Herndon, D. N., Rodriguez, N. A., Diaz, E. C., Hegde, S. D., Jennings, K., Mlcak, R. P., Suri, J. S., Lee, J. O., Williams, F. N., Meyer, W. J., et al. (2012) · Annals of Surgery · doi:10.1097/sla.0b013e318265427e · Cited by 154 · see citing works
Heckman, J. J., Moon, S. H., Pinto, R. R., Savelyev, P. A., & Yavitz, A. (2010) · National Bureau of Economic Research · doi:10.3386/w16238 · Cited by 145 · see citing works
Porro, L. J., Herndon, D. N., Rodriguez, N. A., Jennings, K., Klein, G. L., Mlcak, R. P., Meyer, W. J., Lee, J. O., Suman, O. E., & Finnerty, C. C. (2012) · Journal of the American College of Surgeons · doi:10.1016/j.jamcollsurg.2011.12.038 · Cited by 143 · see citing works
Journal De La Société Française De Statistique
Caussinus, H., & de Falguerolles, A. (2005) · Wiley eBooks · doi:10.1002/0471667196.ess7017 · Cited by 141 · see citing works
Simultaneous Inference for High-Dimensional Linear Models
Zhang, X., & Cheng, G. (2016) · Journal of the American Statistical Association · doi:10.1080/01621459.2016.1166114 · Cited by 139 · see citing works
Primate evidence on the late health effects of early-life adversity
Conti, G., Hansman, C., Heckman, J. J., Novak, M. F. X., Ruggiero, A. M., & Suomi, S. J. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1205340109 · Cited by 135 · see citing works
Hypothesis Testing in Econometrics
Romano, J. P., Shaikh, A. M., & Wolf, M. (2010) · Annual Review of Economics · doi:10.1146/annurev.economics.102308.124342
Self-citationIntergenerational and Intragenerational Externalities of the Perry Preschool Project
Heckman, J. J., & Karapakula, G. (2019) · National Bureau of Economic Research · doi:10.3386/w25889 · Cited by 117 · see citing works
The sequential rejection principle of familywise error control
Goeman, J. J., & Solari, A. (2010) · The Annals of Statistics · doi:10.1214/10-aos829 · Cited by 113 · see citing works
Living Up to Expectations: How Job Training Made Women Better Off and Men Worse Off
Acevedo, P., Cruces, G., Gertler, P., & Martínez, S. (2017) · National Bureau of Economic Research · doi:10.3386/w23264 · Cited by 102 · see citing works
On combinatorial testing problems
Addario‐Berry, L., Broutin, N., Devroye, L., & Lugosi, G. (2010) · The Annals of Statistics · doi:10.1214/10-aos817 · Cited by 97 · see citing works
Group Sessions or Home Visits for Early Childhood Development in India: A Cluster RCT
Grantham-McGregor, S., Adya, A., Attanasio, O., Augsburg, B., Behrman, J., Caeyers, B., Day, M., Jervis, P., Kochar, R., Makkar, P., et al. (2020) · Pediatrics · doi:10.1542/peds.2020-002725 · Cited by 93 · see citing works
Is the Social Safety Net a Long-Term Investment? Large-Scale Evidence From the Food Stamps Program
Bailey, M. J., Hoynes, H. W., Rossin‐Slater, M., & Walker, R. (2023) · The Review of Economic Studies · doi:10.1093/restud/rdad063 · Cited by 107 · see citing works
Short-Run Effects of Parental Job Loss on Child Health
Schaller, J., & Zerpa, M. (2017) · American Journal of Health Economics · doi:10.1162/ajhe_a_00106 · Cited by 89 · see citing works
Breaking the Cycle? Intergenerational Effects of an Antipoverty Program in Early Childhood
Barr, A., & Gibbs, C. R. (2022) · Journal of Political Economy · doi:10.1086/720764 · Cited by 90 · see citing works
On the uniform asymptotic validity of subsampling and the bootstrap
Romano, J. P., & Shaikh, A. M. (2010) · Cited by 87 · see citing works
Self-citationAndrew, A., Attanasio, O. P., Augsburg, B., Day, M., Grantham‐McGregor, S., Meghir, C., Mehrin, F., Pahwa, S., & Rubio‐Codina, M. (2019) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13171 · Cited by 90 · see citing works
Powerful short‐cuts for multiple testing procedures with special reference to gatekeeping strategies
Hommel, G., Bretz, F., & Maurer, W. (2007) · Statistics in Medicine · doi:10.1002/sim.2873 · Cited by 86 · see citing works
Solidarity and fairness in times of crisis
Cappelen, A. W., Falch, R., Sørensen, E. Ø., & Tungodden, B. (2021) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2021.03.017 · Cited by 87 · see citing works
Relief from usury: Impact of a self-help group lending program in rural India
Hoffmann, V., Rao, V., Surendra, V., & Datta, U. (2020) · Journal of Development Economics · doi:10.1016/j.jdeveco.2020.102567 · Cited by 84 · see citing works
Estimating marginal returns to education
Carneiro, P., Heckman, J. J., & Vytlacil, E. (2010) · doi:10.1920/wp.cem.2010.2910 · Cited by 87 · see citing works
Balanced control of generalized error rates
Romano, J. P., & Wolf, M. (2009) · The Annals of Statistics · doi:10.1214/09-aos734 · Cited by 74 · see citing works
Self-citationLee, S., & Shaikh, A. M. (2013) · Journal of Applied Econometrics · doi:10.1002/jae.2327 · Cited by 73 · see citing works
Black, S. E., Bütikofer, A., Devereux, P. J., & Salvanes, K. G. (2018) · The Review of Economics and Statistics · doi:10.1162/rest_a_00815 · Cited by 73 · see citing works
Matsunaga, M. (2007) · Communication Methods and Measures · doi:10.1080/19312450701641409 · Cited by 72 · see citing works
High-dimensional econometrics and regularized GMM
Belloni, A., Chernozhukov, V., Chetverikov, D., Hansen, C., & Kato, K. (2019) · doi:10.1920/wp.cem.2018.3518 · Cited by 73 · see citing works
Dorner, Z. (2019) · Journal of Environmental Economics and Management · doi:10.1016/j.jeem.2019.102257 · Cited by 72 · see citing works
Westfall, P. H., Troendle, J. F., & Pennello, G. A. (2010) · Biometrics · doi:10.1111/j.1541-0420.2010.01408.x · Cited by 69 · see citing works
Can educational interventions reduce susceptibility to financial fraud?
Burke, J., Kieffer, C. N., Mottola, G. R., & Pérez‐Arce, F. (2022) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2022.03.028 · Cited by 74 · see citing works
Testing the correlated random coefficient model
Heckman, J. J., Schmierer, D., & Urzúa, S. (2010) · Journal of Econometrics · doi:10.1016/j.jeconom.2010.01.005 · Cited by 68 · see citing works
DoubleML: An Object-Oriented Implementation of Double Machine Learning in R
Bach, P., Kurz, M. S., Chernozhukov, V., Spindler, M., & Klaaßen, S. (2024) · Journal of Statistical Software · doi:10.18637/jss.v108.i03 · Cited by 67 · see citing works
A multiple testing approach to the regularisation of large sample correlation matrices
Bailey, N. T. J., Pesaran, M. H., & Smith, L. V. (2018) · Journal of Econometrics · doi:10.1016/j.jeconom.2018.10.006 · Cited by 66 · see citing works
LASSO-driven inference in time and space
Chernozhukov, V., Härdle, W. K., Huang, C., & Wang, N. (2021) · The Annals of Statistics · doi:10.1214/20-aos2019 · Cited by 61 · see citing works
Bertoni, M., Brunello, G., & Mazzarella, G. (2018) · Journal of Health Economics · doi:10.1016/j.jhealeco.2018.02.011 · Cited by 63 · see citing works
The Predictive Power of Self-Control for Life Outcomes
Cobb-Clark, D., Dahmann, S. C., Kamhöfer, D. A., & Schildberg-Hörisch, H. (2022) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2022.02.028 · Cited by 50 · see citing works
Martin, M. A. (2007) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2007.01.020 · Cited by 60 · see citing works
Marriage Age, Social Status, and Intergenerational Effects in Uganda
Sunder, N. (2019) · Demography · doi:10.1007/s13524-019-00829-8 · Cited by 60 · see citing works
The First 2,000 Days and Child Skills
Doyle, O. (2019) · Journal of Political Economy · doi:10.1086/705707 · Cited by 64 · see citing works
What makes a productive Ph.D. student?
Corsini, A., Pezzoni, M., & Visentin, F. (2022) · Research Policy · doi:10.1016/j.respol.2022.104561 · Cited by 53 · see citing works
Meinshausen, N., Maathuis, M. H., & Bühlmann, P. (2016) · Cited by 51 · see citing works
TESTING REGRESSION MONOTONICITY IN ECONOMETRIC MODELS
Chetverikov, D. (2018) · Econometric Theory · doi:10.1017/s0266466618000282 · Cited by 49 · see citing works
IPAD: Stable Interpretable Forecasting with Knockoffs Inference
Fan, Y., Lv, J., Sharifvaghefi, M., & Uematsu, Y. (2019) · Journal of the American Statistical Association · doi:10.1080/01621459.2019.1654878 · Cited by 49 · see citing works
Gaussian approximation for high dimensional vector under physical dependence
Zhang, X., & Cheng, G. (2018) · Bernoulli · doi:10.3150/17-bej939 · Cited by 48 · see citing works
Post hoc confidence bounds on false positives using reference families
Blanchard, G., Neuvial, P., & Roquain, É. (2020) · The Annals of Statistics · doi:10.1214/19-aos1847 · Cited by 46 · see citing works
Doyle, O., Harmon, C. P., Heckman, J. J., Logue, C., & Moon, S. H. (2016) · Labour Economics · doi:10.1016/j.labeco.2016.11.002 · Cited by 46 · see citing works
Frisancho, V. (2022) · The Economic Journal · doi:10.1093/ej/ueac084 · Cited by 58 · see citing works
Minimax optimality of permutation tests
Kim, I., Balakrishnan, S., & Wasserman, L. (2022) · The Annals of Statistics · doi:10.1214/21-aos2103 · Cited by 45 · see citing works
SAT optional policies: Do they influence graduate quality, selectivity or diversity?
Saboe, M., & Terrizzi, S. (2018) · Economics Letters · doi:10.1016/j.econlet.2018.10.017 · Cited by 44 · see citing works
Testing regression monotonicity in econometric models
Chetverikov, D. (2012) · doi:10.1920/wp.cem.2012.3512 · Cited by 44 · see citing works
An Analysis of the Memphis Nurse-Family Partnership Program
Heckman, J. J., Holland, M. L., Makino, K. K., Pinto, R. R., & Rosales-Rueda, M. F. (2017) · National Bureau of Economic Research · doi:10.3386/w23610 · Cited by 44 · see citing works
Evaluating the effects of a targeted home visiting program on maternal and child health outcomes
Sandner, M., Cornelißen, T., Jungmann, T., & Herrmann, P. (2018) · Journal of Health Economics · doi:10.1016/j.jhealeco.2018.02.008 · Cited by 43 · see citing works
Unconditional cash transfers, child labour and education: theory and evidence
Churchill, S. A., Iqbal, N., Nawaz, S., & Yew, S. L. (2021) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2021.04.012 · Cited by 43 · see citing works
The Impacts of GM Foods: Results from a Randomized Controlled Trial of Bt Eggplant in Bangladesh
Ahmed, A. U., Hoddinott, J., Abedin, N., & Hossain, N. (2021) · American Journal of Agricultural Economics · doi:10.1111/ajae.12162 · Cited by 42 · see citing works
Meinshausen, N., Maathuis, M. H., & Bühlmann, P. (2011) · The Annals of Statistics · doi:10.1214/11-aos946 · Cited by 42 · see citing works
Jessoe, K., Lade, G., Loge, F. J., & Spang, E. S. (2021) · Journal of Environmental Economics and Management · doi:10.1016/j.jeem.2021.102519 · Cited by 42 · see citing works
Arlot, S., Blanchard, G., & Roquain, É. (2007) · arXiv (Cornell University) · doi:10.1214/08-aos667 · Cited by 4 · see citing works
Improving Workplace Climate in Large Corporations: A Clustered Randomized Intervention
Alan, Ş., Çörekçioğlu, G., & Sutter, M. (2022) · The Quarterly Journal of Economics · doi:10.1093/qje/qjac034 · Cited by 39 · see citing works
Fogarty, C. B., & Small, D. S. (2015) · Journal of the American Statistical Association · doi:10.1080/01621459.2015.1120675 · Cited by 38 · see citing works
Confidence sets for model selection by F -testing
Ferrari, D., & Yang, Y. (2014) · Statistica Sinica · doi:10.5705/ss.2014.110 · Cited by 38 · see citing works
Early Stimulation and Nutrition: The Impacts of a Scalable Intervention
Attanasio, O. P., Baker‐Henningham, H., Bernal, R., Meghir, C., Pineda, D., & Rubio‐Codina, M. (2018) · National Bureau of Economic Research · doi:10.3386/w25059 · Cited by 38 · see citing works
Efficient Computation of Adjusted P-Values for Resampling-Based Stepdown Multiple TestingPreprint
Romano, J. P., & Wolf, M. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2737429 · Cited by 39 · see citing works
Self-citationAdaptive test of independence based on HSIC measures
Albert, M., Laurent, B., Marrel, A., & Meynaoui, A. (2022) · The Annals of Statistics · doi:10.1214/21-aos2129 · Cited by 37 · see citing works
How vocational education made women better off but left men behind
Acevedo, P., Cruces, G., Gertler, P., & Martínez, S. (2020) · Labour Economics · doi:10.1016/j.labeco.2020.101824 · Cited by 40 · see citing works
Can Early Intervention Improve Maternal Well-Being? Evidence from a Randomized Controlled Trial
Doyle, O., Delaney, L. D., O’Farrelly, C., Fitzpatrick, N., & Daly, M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0169829 · Cited by 34 · see citing works
Helwig, N. E. (2019) · Wiley Interdisciplinary Reviews Computational Statistics · doi:10.1002/wics.1457 · Cited by 34 · see citing works
When good fences aren’t enough: The impact of neighboring air pollution on infant health
DeCicca, P., & Malak, N. (2020) · Journal of Environmental Economics and Management · doi:10.1016/j.jeem.2020.102324 · Cited by 34 · see citing works
Shortcuts for Locally Consonant Closed Test Procedures
Brannath, W., & Bretz, F. (2010) · Journal of the American Statistical Association · doi:10.1198/jasa.2010.tm08127 · Cited by 34 · see citing works
Bootstrapping Upper Confidence BoundPreprint
Hao, B., Abbasi-Yadkori, Y., Wen, Z., & Cheng, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05247 · Cited by 34 · see citing works
The effect of relationship status on health with dynamic health and persistent relationships
Kohn, J. L., & Averett, S. L. (2014) · Journal of Health Economics · doi:10.1016/j.jhealeco.2014.03.010 · Cited by 31 · see citing works
Kim, M., Schwartz, G. L., White, J. S., Glymour, M., Reardon, S. F., Kershaw, K. N., Gomez, S., Collin, D. F., Inamdar, P. P., Wang, G.-Y., et al. (2022) · PLoS Medicine · doi:10.1371/journal.pmed.1004031 · Cited by 26 · see citing works
High-Dimensional Data Bootstrap
Chernozhukov, V., Chetverikov, D., Kato, K., & Koike, Y. (2023) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-040120-022239 · Cited by 31 · see citing works
Multiple testing and its applications to microarrays
Ge, Y., Sealfon, S. C., & Speed, T. P. (2009) · Statistical Methods in Medical Research · doi:10.1177/0962280209351899 · Cited by 30 · see citing works
How Rational and Competitive Is the Market for Mutual Funds?
Leippold, M., & Rueegg, R. (2019) · European Finance Review · doi:10.1093/rof/rfz011 · Cited by 30 · see citing works
Li, J., Song, W., & Shi, J. (2015) · Statistics & Probability Letters · doi:10.1016/j.spl.2015.07.002 · Cited by 29 · see citing works
- Authors
- Romano, J. P., & Wolf, M.
- Affiliations
- Stanford University (US)
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2005-03
- Volume
- 100
- Issue
- 469
- Pages
- 94-108
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W3124231044
- Cited by
- 609
- References
- 23
- Citations by year
- 2026: 44 · 2025: 63 · 2024: 51 · 2023: 52 · 2022: 60 · 2021: 47 · 2020: 38 · 2019: 33 · 2018: 38 · 2017: 29 · 2016: 25 · 2015: 15 · 2014: 22 · 2013: 11 · 2012: 14
- FWCI
- 10.03
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- Journal of the American Statistical Association (publishedVersion)
- SSRN Electronic Journal (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Repositori digital de la UPF (Universitat Pompeu Fabra) (submittedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, other-oa)
- Funders
- National Science Foundation
- Ministerio de Ciencia y Tecnología
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- multiple comparisons, family-wise error rate, resampling, Bonferroni correction, intersection tests, asymptotic theory
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
