A Reality Check for Data Snooping
Abstract
Data snooping occurs when a given set of data is used more than once for purposes of inference or model selection. When such data reuse occurs, there is always the possibility that any satisfactory results obtained may simply be due to chance rather than to any merit inherent in the method yielding the results. This problem is practically unavoidable in the analysis of time-series data, as typically only a single history measuring a given phenomenon of interest is available for analysis. It is widely acknowledged by empirical researchers that data snooping is a dangerous practice to be avoided, but in fact it is endemic. The main problem has been a lack of sufficiently simple practical methods capable of assessing the potential dangers of data snooping in a given situation. Our purpose here is to provide such methods by specifying a straightforward procedure for testing the null hypothesis that the best model encountered in a specification search has no predictive superiority over a given benchmark model. This permits data snooping to be undertaken with some degree of confidence that one will not mistake results that could have been generated by chance for genuinely good results.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The paper applies White's reality check and a stepwise test to address multiple testing and data snooping.
“We utilize the bootstrap reality check implementation, which was introduced by White (2000), to compute the p-value for the reality check”
From Profitability and Data-Snooping Tests of Four Technical Trade Strategies for Cryptocurrency Pair BTC/USDT and ETH/USDT in Cryptocurrency Markets During 2022–2023 (2023) · cites this paper as White (2000)
The study adopts a stepwise test based on White's reality check and methodologies from Hansen (2005), Romano and Wolf (2005), and Hsu, Hsu, and Kuan (2010).
“We thus adopt a stepwise test that is based on a series of methodologies based on White's reality check test, including Hansen (2005); Romano and Wolf (2005); Hsu, Hsu, and Kuan (2010).”
From Profitability and Data-Snooping Tests of Four Technical Trade Strategies for Cryptocurrency Pair BTC/USDT and ETH/USDT in Cryptocurrency Markets During 2022–2023 (2023) · cites this paper as White (2000)
Individual testing is inadequate for assessing the profitability of technical trading strategies because it fails to account for the possibility that the tested strategy is already the best among a group, leading to overestimated statistical significance and a higher chance of Type I error.
“Nevertheless, the individual testing is inadequate for assessing the profitability of technical trading strategies.This is because it fails to account for the possibility that the strategy being tested is already the best among a group of strategies.Consequently, the statistical significance is overestimated, and there is a higher chance of Type I error.”
From Profitability and Data-Snooping Tests of Four Technical Trade Strategies for Cryptocurrency Pair BTC/USDT and ETH/USDT in Cryptocurrency Markets During 2022–2023 (2023) · cites this paper as White (2000)
The study uses a stepwise test to control the family-wise error rate and mitigate data snooping bias.
“To address and mitigate the issue of data snooping bias, a stepwise test is specified with a certain Type I error level within a defined period (𝑡 = 1, ⋯ , 𝑇).The main objective of this test is to effectively control the family-wise error, which represents the probability of rejecting at least one correct null hypothesis.”
From Profitability and Data-Snooping Tests of Four Technical Trade Strategies for Cryptocurrency Pair BTC/USDT and ETH/USDT in Cryptocurrency Markets During 2022–2023 (2023) · cites this paper as White (2000)
Sullivan et al. (1999) and White (2000) show that the findings of Brock et al. (1992) are not due to data-snooping.
“while Sullivan et al. (1999) and White (2000) show that the findings of Brock et al. (1992) are not due to data-snooping.”
From Technical trading and cryptocurrencies (2021) · cites this paper as White (2000)
Legitimacy Theory and Stakeholder Theory explain the importance of firms publicly reporting inclusion practices to generate legitimacy and satisfy stakeholder expectations.
“From a macro-institutional lens, it draws from Legitimacy Theory and Stakeholder Theory that explain the importance of Nirms publicly reporting inclusion practices to generate legitimacy in the eyes of society and to satisfy stakeholder expectations (26,27).”
From Studying Workforce Diversity, Inclusion and Engagement Under Universal Reporting for India's IT Sector (2026) · cites this paper as [27]
Cites
Co-integration and error correction: representation, estimation and testing
Engle, R. F., & Granger, C. W. J. (1987) · Econometrica · doi:10.2307/1913236 · Cited by 32,007 · see citing works
System Identification: Theory for the User
Lennart Ljung, A. (1987) · Cited by 17,690 · see citing works
Convergence of Probability Measures
Billingsley, P. (1999) · Wiley series in probability and statistics · doi:10.1002/9780470316962 · Cited by 14,153 · see citing works
(2012) · American Economic Review · doi:10.1257/aer.102.2.i · Cited by 14,047 · see citing works
Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881 · see citing works
Linear Statistical Inference and its Applications
Rao, C. R. (1973) · Wiley series in probability and statistics · doi:10.1002/9780470316436 · Cited by 10,567 · see citing works
Convergence of Probability Measures
Kingmán, J. F. C., & Billingsley, P. (1969) · Revue de l Institut International de Statistique / Review of the International Statistical Institute · doi:10.2307/1402124 · Cited by 6,930 · see citing works
Simultaneous Statistical Inference
Miller, R. G. (1981) · Springer series in statistics · doi:10.1007/978-1-4613-8122-8 · Cited by 5,608 · see citing works
Diebold, F. X., & Mariano, R. S. (1994) · National Bureau of Economic Research · doi:10.3386/t0169 · Cited by 5,498 · see citing works
Diebold, F. X., & Mariano, R. S. (1995) · Journal of Business and Economic Statistics · doi:10.1080/07350015.1995.10524599 · Cited by 5,240 · see citing works
A sharper Bonferroni procedure for multiple tests of significance
HOCHBERG, Y. (1988) · Biometrika · doi:10.1093/biomet/75.4.800 · Cited by 4,829 · see citing works
Tests for parameter instability and structural change with unknown change point.
Andrews, D. W. K. (1993) · Econometrica · doi:10.2307/2951764 · Cited by 4,334 · see citing works
(1947) · The American Statistician · doi:10.1080/00031305.1947.10501534 · Cited by 3,745 · see citing works
Politis, D. N., & Romano, J. P. (1994) · Journal of the American Statistical Association · doi:10.1080/01621459.1994.10476870 · Cited by 2,649 · see citing works
The Jackknife and the Bootstrap for General Stationary Observations
Künsch, H. R. (1989) · The Annals of Statistics · doi:10.1214/aos/1176347265 · Cited by 2,454 · see citing works
Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment
Martín, M. A., Westfall, P. H., & Young, S. (1993) · Biometrics · doi:10.2307/2533464 · Cited by 2,308 · see citing works
Simple technical trading rules and the stochastic properties of stock returns
BROCK, W., LAKONISHOK, J., & LeBARON, B. (1992) · The Journal of Finance · doi:10.1111/j.1540-6261.1992.tb04681.x · Cited by 2,301 · see citing works
Let's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,201 · see citing works
Hawkes, J., & Billingsley, P. (1987) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2982048 · Cited by 1,671 · see citing works
Journal of the Royal Statistical Society.
(1921) · The Economic Journal · doi:10.1093/ej/31.121.133 · Cited by 1,503 · see citing works
Asymptotic Inference about Predictive Ability
West, K. D. (1996) · Econometrica · doi:10.2307/2171956 · Cited by 1,469 · see citing works
Asymptotic Theory for Econometricians.
Robinson, P. M., & White, H. (1988) · Economica · doi:10.2307/2554258 · Cited by 1,333 · see citing works
Self-citationData-Snooping Biases in Tests of Financial Asset Pricing Models
Lo, A. W., & MacKinlay, A. C. (1990) · Review of Financial Studies · doi:10.1093/rfs/3.3.431 · Cited by 1,134 · see citing works
Specification Searches: Ad Hoc Inference with Non-Experimental Data.
Richard, J., & Leamer, E. E. (1981) · Economica · doi:10.2307/2553491 · Cited by 1,129 · see citing works
Model uncertainty, data mining and statistical inference
Chatfield, C. (1995) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2983440 · Cited by 1,103 · see citing works
Data-snooping, technical trading rule performance and the Bootstrap
Sullivan, R., Timmermann, A., & White, H. (1999) · The Journal of Finance · doi:10.1111/0022-1082.00163 · Cited by 1,028 · see citing works
Self-citationResampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment
Chaubey, Y. P., Westfall, P. H., & Young, S. S. (1993) · Technometrics · doi:10.2307/1270279 · Cited by 943 · see citing works
Large Sample Confidence Regions Based on Subsamples under Minimal Assumptions
Politis, D. N., & Romano, J. P. (1994) · The Annals of Statistics · doi:10.1214/aos/1176325770 · Cited by 748 · see citing works
Can Stock Market Forecasters Forecast?
Cowles, A. (1933) · Econometrica · doi:10.2307/1907042 · Cited by 739 · see citing works
Cross-Validatory Choice and Assessment of Statistical Predictions (With Discussion)
Stone, M. (1976) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1976.tb01573.x · Cited by 696 · see citing works
On Blocking Rules for the Bootstrap with Dependent Data
HALL, P., Horowitz, J. L., & Jing, B. (1995) · Biometrika · doi:10.1093/biomet/82.3.561 · Cited by 647 · see citing works
On a Heuristic Method of Test Construction and its use in Multivariate Analysis
Roy, S. (1953) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729029 · Cited by 631 · see citing works
Asymptotic Theory for Econometricians
(1984) · Elsevier eBooks · doi:10.1016/c2009-0-21869-1 · Cited by 573 · see citing works
Moving blocks jackknife and bootstrap capture weak dependence
Liu, R. (1992) · Medical Entomology and Zoology · Cited by 503 · see citing works
Sharper Bounds for Gaussian and Empirical Processes
Talagrand, M. (1994) · The Annals of Probability · doi:10.1214/aop/1176988847 · Cited by 496 · see citing works
Data mining reconsidered: encompassing and the general‐to‐specific approach to specification search
Hoover, K. D., & Perez, S. J. (1999) · Econometrics Journal · doi:10.1111/1368-423x.00025 · Cited by 488 · see citing works
Journal of Business and Economic Statistics
Fildes, R. A. (1994) · International Journal of Forecasting · doi:10.1016/0169-2070(94)90039-6 · Cited by 416 · see citing works
Data Mining: Statistics and More?
Hand, D. J. (1998) · The American Statistician · doi:10.1080/00031305.1998.10480549 · Cited by 380 · see citing works
Lovell, M. C. (1983) · The Review of Economics and Statistics · doi:10.2307/1924403 · Cited by 355 · see citing works
Statistical significance tests.
Cox, D. (1982) · British Journal of Clinical Pharmacology · doi:10.1111/j.1365-2125.1982.tb01987.x · Cited by 335 · see citing works
Specification Searches: Ad Hoc Inference With Nonexperimental Data.
Hill, B. M., & Leamer, E. E. (1980) · Journal of the American Statistical Association · doi:10.2307/2287437 · Cited by 325 · see citing works
Information criteria for selecting possibly misspecified parametric models
Sin, C.-Y., & White, H. (1996) · Journal of Econometrics · doi:10.1016/0304-4076(94)01701-8 · Cited by 305 · see citing works
Self-citationExploring the Limits of Bootstrap.
Davison, A. C., LePage, R. D., & Billard, L. (1993) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.2307/2982877 · Cited by 253 · see citing works
Multiple Comparisons in Induction Algorithms
Jensen, D. D., & Cohen, P. R. (2000) · Machine Learning · doi:10.1023/a:1007631014630 · Cited by 241 · see citing works
Effects of Model Selection on Inference
Pötscher, B. M. (1991) · Econometric Theory · doi:10.1017/s0266466600004382 · Cited by 231 · see citing works
Stone, M. (1978) · Series Statistics · doi:10.1080/02331887808801414 · Cited by 221 · see citing works
The moving blocks bootstrap and robust inference for linear least squares and quantile regressions
Fitzenberger, B. (1998) · Journal of Econometrics · doi:10.1016/s0304-4076(97)00058-4 · Cited by 200 · see citing works
Politis, D. N., & Romano, J. P. (1992) · The Annals of Statistics · doi:10.1214/aos/1176348899 · Cited by 178 · see citing works
Paparoditis, E., & Politis, D. N. (2001) · Biometrika · doi:10.1093/biomet/88.4.1105 · Cited by 161 · see citing works
A Comparison of Two Modified Bonferroni Procedures
Hommel, G. (1989) · Biometrika · doi:10.1093/biomet/76.3.624 · Cited by 151 · see citing works
ECONOMICS AS A HARD SCIENCE: REALISTIC GOAL OR WISHFUL THINKING?
Mayer, T. F. (1980) · Economic Inquiry · doi:10.1111/j.1465-7295.1980.tb00568.x · Cited by 110 · see citing works
The Effect of Model Selection on Confidence Regions and Prediction Regions
Kabaila, P. V. (1995) · Econometric Theory · doi:10.1017/s0266466600009403 · Cited by 107 · see citing works
The Bonferroni and the Scheffe Multiple Comparison Procedures
Savin, N. E. (1980) · The Review of Economic Studies · doi:10.2307/2297112 · Cited by 90 · see citing works
THE BOOTSTRAP OF THE MEAN FOR DEPENDENT HETEROGENEOUS ARRAYS
Gonçalves, Ś., & White, H. (2002) · Econometric Theory · doi:10.1017/s0266466602186051 · Cited by 88 · see citing works
Self-citationPredictive ability with cointegrated variables
Corradi, V., Swanson, N. R., & Claudia Olivetti, Z. (2001) · Journal of Econometrics · doi:10.1016/s0304-4076(01)00086-0 · Cited by 80 · see citing works
Generalized Predictive Tests and Structural Change Analysis in Econometrics
Dufour, J., Ghysels, É., & Hall, A. (1994) · International Economic Review · doi:10.2307/2527098 · Cited by 53 · see citing works
Dangers of Data-Driven Inference: The Case of Calendar Effects in Stock ReturnsPreprint
Sullivan, R. J., Timmermann, A., & White, H. (1998) · RePEc: Research Papers in Economics · Cited by 52 · see citing works
Self-citationNOTE ON A LARGE-SAMPLE RESULT IN SPECIFICATION ANALYSISPreprint
Kloek, T., & Kloek, T. (1973) · RePEc: Research Papers in Economics · doi:10.22004/ag.econ.272063 · Cited by 10 · see citing works
Hand, D. J. (1999) · Econometrics Journal · doi:10.1111/1368-423x.00030 · Cited by 9 · see citing works
The Theory of Household Saving
Miller, R. F. (1963) · The Review of Economics and Statistics · doi:10.2307/1924136 · Cited by 7 · see citing works
Note on a Large-Sample Result in Specification Analysis
Kloek, T. (1975) · Econometrica · doi:10.2307/1911335 · Cited by 7 · see citing works
Estimating Input Demand Equations by Direct and Indirect Methods
Wills, H. R. (1981) · The Review of Economic Studies · doi:10.2307/2296883 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
… and the Cross-Section of Expected Returns
Harvey, C. R., Liu, Y., & Zhu, H. (2016) · Review of Financial Studies · doi:10.1093/rfs/hhv059 · Cited by 2,119 · see citing works
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · see citing works
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · see citing works
Data-snooping, technical trading rule performance and the Bootstrap
Sullivan, R., Timmermann, A., & White, H. (1999) · The Journal of Finance · doi:10.1111/0022-1082.00163 · Cited by 1,028 · see citing works
Self-citationForecasting the equity risk premium: the role of technical indicators
Neely, C. J., Rapach, D. E., Tu, J., & Zhou, G. (2014) · Management Science · doi:10.1287/mnsc.2013.1838 · Cited by 1,003 · see citing works
Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0912-z · Cited by 651 · see citing works
Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach
Coppock, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2018.10 · Cited by 515 · see citing works
Diebold, F. X. (2015) · Journal of Business & Economic Statistics · doi:10.1080/07350015.2014.983236 · Cited by 496 · see citing works
Technical trading revisited: False discoveries, persistence tests, and transaction costs
Bajgrowicz, P., & Scaillet, O. (2012) · Journal of Financial Economics · doi:10.1016/j.jfineco.2012.06.001 · Cited by 253 · see citing works
p-Curve and p-Hacking in Observational Research
Bruns, S. B., & Ioannidis, J. P. A. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149144 · Cited by 210 · see citing works
Bailey, D. H., Borwein, J. M., López de Prado, M., & Zhu, Q. J. (2014) · Notices of the American Mathematical Society · doi:10.1090/noti1105 · Cited by 221 · see citing works
Factor Models, Machine Learning, and Asset Pricing
Giglio, S., Kelly, B., & Xiu, D. (2022) · Annual Review of Financial Economics · doi:10.1146/annurev-financial-101521-104735 · Cited by 187 · see citing works
Can commodity futures be profitably traded with quantitative market timing strategies?
Marshall, B. R., Cahan, R. H., & Cahan, J. M. (2008) · Journal of Banking & Finance · doi:10.1016/j.jbankfin.2007.12.011 · Cited by 118 · see citing works
Performance of technical analysis in growth and small cap segments of the US equity market
Shynkevich, A. (2012) · Journal of Banking & Finance · doi:10.1016/j.jbankfin.2011.07.001 · Cited by 108 · see citing works
Technical trading: is it still beating the foreign exchange market?
Hsu, P.-H., Taylor, M. P., & Wang, Z. (2016) · Journal of International Economics · doi:10.1016/j.jinteco.2016.03.012 · Cited by 113 · see citing works
Technical trading and cryptocurrencies
Hudson, R., & Urquhart, A. (2021) · Annals of Operations Research · doi:10.1007/s10479-019-03357-1 · Cited by 90 · see citing works
“In equity markets, Brock et al. (1992) show that technical trading provides significant predictability over 90 years for theDow Jones Industrial Average, while Sullivan et al. (1999) and White (2000) show that the findings of Brock et al. (1992) are not due to data-snooping.”
BackgroundThe Obstinate Passion of Foreign Exchange Professionals: Technical Analysis
Menkhoff, L., & Taylor, M. P. (2007) · Journal of Economic Literature · doi:10.1257/jel.45.4.936 · Cited by 78 · see citing works
“White (2000) develops a bootstrap simulation technique—the " reality check " —for examining whether it is inherent skill or pure chance that leads to the best rule being chosen out of any given universe of rules.”
BackgroundFX technical trading rules can be profitable sometimes!
Zarrabi, N., Snaith, S., & Coakley, J. (2017) · International Review of Financial Analysis · doi:10.1016/j.irfa.2016.12.010 · Cited by 32 · see citing works
Performance of technical trading rules: evidence from the crude oil market
Psaradellis, I., Laws, J., Pantelous, A. A., & Sermpinis, G. (2019) · The European Journal of Finance · doi:10.1080/1351847x.2018.1552172 · Cited by 27 · see citing works
Technical Analysis and Cryptocurrencies
Hudson, R., & Urquhart, A. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3387950 · Cited by 7 · see citing works
Li, W., Wang, X., Wang, Z., & Yang, M. (2023) · Journal of Accounting and Finance · doi:10.33423/jaf.v23i4.6453 · Cited by 2 · see citing works
Nagpal, A., Rana, S., & Vyas, B. (2026) · International Research Journal of Multidisciplinary Scope · doi:10.47857/irjms.2026.v07i02.011034
“From a macro-institutional lens, it draws from Legitimacy Theory and Stakeholder Theory that explain the importance of Nirms publicly reporting inclusion practices to generate legitimacy in the eyes of society and to satisfy stakeholder expectations (26, 27).”
- Authors
- White, H.
- Affiliations
- University of California System (US)
- Corresponding author
- White, Halbert
- Venue
- Econometrica
- Publisher
- The Econometric Society
- Published
- 2000-09
- Volume
- 68
- Issue
- 5
- Pages
- 1097-1126
- ISSN
- 0012-9682 (ISSN-L)
- 1468-0262
- OpenAlex
- W2135606128
- Cited by
- 1,851
- References
- 62
- Citations by year
- 2026: 113 · 2025: 53 · 2024: 58 · 2023: 45 · 2022: 51 · 2021: 74 · 2020: 88 · 2019: 104 · 2018: 89 · 2017: 108 · 2016: 88 · 2015: 124 · 2014: 94 · 2013: 98 · 2012: 101
- FWCI
- 33.61
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Econometrica (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Keywords
- data snooping, data reuse, model selection, specification search, time series forecasting, benchmark models, null hypothesis testing, overfitting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
