Stepwise Multiple Testing as Formalized Data Snooping
Open accessAbstract
In econometric applications, often several hypothesis tests are carried out at once. The problem then becomes how to decide which hypotheses to reject, accounting for the multitude of tests. This paper suggests a stepwise multiple testing procedure that asymptotically controls the familywise error rate. Compared to related single-step methods, the procedure is more powerful and often will reject more false hypotheses. In addition, we advocate the use of studentization when feasible. Unlike some stepwise methods, the method implicitly captures the joint dependence structure of the test statistics, which results in increased ability to detect false hypotheses. The methodology is presented in the context of comparing several strategies to a common benchmark. However, our ideas can easily be extended to other contexts where multiple tests occur. Some simulation studies show the improvements of our methods over previous proposals. We also provide an application to a set of real data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The study adopts a stepwise test based on White's reality check and methodologies from Hansen (2005), Romano and Wolf (2005), and Hsu, Hsu, and Kuan (2010).
“We thus adopt a stepwise test that is based on a series of methodologies based on White's reality check test, including Hansen (2005); Romano and Wolf (2005); Hsu, Hsu, and Kuan (2010).”
From Profitability and Data-Snooping Tests of Four Technical Trade Strategies for Cryptocurrency Pair BTC/USDT and ETH/USDT in Cryptocurrency Markets During 2022–2023 (2023) · cites this paper as Romano and Wolf (2005)
The study uses a stepwise test to control the family-wise error rate and mitigate data snooping bias.
“To address and mitigate the issue of data snooping bias, a stepwise test is specified with a certain Type I error level within a defined period (𝑡 = 1, ⋯ , 𝑇).The main objective of this test is to effectively control the family-wise error, which represents the probability of rejecting at least one correct null hypothesis.”
From Profitability and Data-Snooping Tests of Four Technical Trade Strategies for Cryptocurrency Pair BTC/USDT and ETH/USDT in Cryptocurrency Markets During 2022–2023 (2023) · cites this paper as Romano and Wolf (2005)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
Politis, D. N., & Romano, J. P. (1994) · Journal of the American Statistical Association · doi:10.1080/01621459.1994.10476870 · Cited by 2,649 · see citing works
Self-citationGeneralizations of the familywise error rate
Lehmann, E. L., & Romano, J. P. (2005) · The Annals of Statistics · doi:10.1214/009053605000000084 · Cited by 320 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 20
- Beran, R. (1988). Balanced simultaneous confidence sets. Journal of the American Statistical Association, 83:679-686.
- Diebold, F. X. (2000). Elements of Forecasting. South-Western College Publishing, Cincinnati, Ohio, second edition.
- Dunnett, C. W. and Tamhane, A. C. (1992). A step-up multiple test procedure. Journal of the American Statistical Association, 87:162-170.
- Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Annals of Statistics, 7:1-26.
- Götze, F. and Künsch, H. R. (1996). Second order correctness of the blockwise bootstrap for stationary observations. Annals of Statistics, 24:1914-1933.
- Hall, P. (1992). The Bootstrap and Edgeworth Expansion. Springer, New York.
- Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6:65-70.
- Jorion, P. (2000). Value at Risk: The New Benchmark for Managing Financial Risk. McGraw- Hill, second edition.
- Künsch, H. R. (1989). The jackknife and the bootstrap for general stationary observations. Annals of Statistics, 17:1217-1241.
- Lahiri, S. N. (1992). Edgeworth correction by 'moving block' bootstrap for stationary and nonstationary data. In LePage, R. and Billard, L., editors, Exploring the Limits of Bootstrap, pages 183-214.
- Leamer, E. (1983). Let's take the con out of econometrics. American Economic Review, 73:31-43.
- Lo, A. and MacKinley, C. (1990). Data snooping biases in tests of financial asset pricing models. Review of Financial Studies, 3:431-468.
- Loh, W. Y. (1987). Calibrating confidence coefficients. Journal of the American Statistical Association, 82:155-162.
- Politis, D. N. and Romano, J. P. (1992). A circular block-resampling procedure for stationary data. In LePage, R. and Billard, L., editors, Exploring the Limits of Bootstrap, pages 263- 270.
- Politis, D. N., Romano, J. P., and Wolf, M. (1999). Subsampling. Springer, New York.
- Rodríguez-Poo, J., Delgado, M., and Wolf, M. (2001). Subsampling inference in cube root asymptotics with an application to Manski's maximum score estimator. Economics Letters, 73:241-250.
- Shao, J. and Tu, D. (1995). The Jackknife and the Bootstrap. Springer, New York.
- Westfall, P. H. and Young, S. S. (1993). Resampling-Based Multiple Testing: Examples and Methods for P-Value Adjustment. John Wiley, New York.
- White, H. L. (2000). A reality check for data snooping. Econometrica, 68(5):1097-1126.
- White, H. L. (2001). Asymptotic Theory for Econometricians. Academic Press, New York, second edition.
Cited by
Sorted by citations
Anderson, M. L. (2008) · Journal of the American Statistical Association · doi:10.1198/016214508000000841 · Cited by 2,410 · see citing works
Hein, G., Silani, G., Preuschoff, K., Batson, C. D., & Singer, T. (2010) · Neuron · doi:10.1016/j.neuron.2010.09.003 · Cited by 821 · see citing works
Technical trading revisited: False discoveries, persistence tests, and transaction costs
Bajgrowicz, P., & Scaillet, O. (2012) · Journal of Financial Economics · doi:10.1016/j.jfineco.2012.06.001 · Cited by 253 · see citing works
Bailey, D. H., Borwein, J. M., López de Prado, M., & Zhu, Q. J. (2014) · Notices of the American Mathematical Society · doi:10.1090/noti1105 · Cited by 221 · see citing works
The Role of Parallel Trends in Event Study Settings: An Application to Environmental Economics
Marcus, M., & Sant’Anna, P. H. C. (2021) · Journal of the Association of Environmental and Resource Economists · doi:10.1086/711509 · Cited by 192 · see citing works
Performance of technical analysis in growth and small cap segments of the US equity market
Shynkevich, A. (2012) · Journal of Banking & Finance · doi:10.1016/j.jbankfin.2011.07.001 · Cited by 108 · see citing works
Technical trading: is it still beating the foreign exchange market?
Hsu, P.-H., Taylor, M. P., & Wang, Z. (2016) · Journal of International Economics · doi:10.1016/j.jinteco.2016.03.012 · Cited by 113 · see citing works
A Bayesian Approach to the Correction for Multiplicity
de Jong, T. (2019) · doi:10.31234/osf.io/s56mk · Cited by 35 · see citing works
FX technical trading rules can be profitable sometimes!
Zarrabi, N., Snaith, S., & Coakley, J. (2017) · International Review of Financial Analysis · doi:10.1016/j.irfa.2016.12.010 · Cited by 32 · see citing works
Performance of technical trading rules: evidence from the crude oil market
Psaradellis, I., Laws, J., Pantelous, A. A., & Sermpinis, G. (2019) · The European Journal of Finance · doi:10.1080/1351847x.2018.1552172 · Cited by 27 · see citing works
Roads to Development: Experimental Evidence from Urban Road Pavement
Gonzalez-Navarro, M., & Quintana-Domeque, C. (2010) · doi:10.2139/ssrn.1558631 · Cited by 11 · see citing works
Li, W., Wang, X., Wang, Z., & Yang, M. (2023) · Journal of Accounting and Finance · doi:10.33423/jaf.v23i4.6453 · Cited by 2 · see citing works
- Authors
- Romano, J. P., & Wolf, M.
- Affiliations
- Stanford University (US)
- Universitat Pompeu Fabra (ES)
- Venue
- Econometrica
- Publisher
- The Econometric Society
- Published
- 2005-07
- Volume
- 73
- Issue
- 4
- Pages
- 1237-1282
- ISSN
- 0012-9682 (ISSN-L)
- 1468-0262
- OpenAlex
- W3123939061
- Cited by
- 1,019
- References
- 3
- Citations by year
- 2026: 89 · 2025: 98 · 2024: 71 · 2023: 80 · 2022: 79 · 2021: 76 · 2020: 77 · 2019: 69 · 2018: 74 · 2017: 34 · 2016: 31 · 2015: 41 · 2014: 42 · 2013: 33 · 2012: 34
- FWCI
- 9.69
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Econometrica (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, other-oa)
- Repositori digital de la UPF (Universitat Pompeu Fabra) (submittedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Keywords
- family-wise error rate, data snooping, studentization, hypothesis testing, benchmark comparison, multiple comparisons
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
