CoChat

Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance

Open access
Bailey, David H., Borwein, Jonathan M., López de Prado, Marcos, Zhu, Qiji Jim · 2014 · Notices of the American Mathematical Society

The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index; none of the 6 sources its claims cite is checked for support yet.

Abstract

Another thing I must point out is that you cannot prove a vague theory wrong.[…] Also, if the process of computing the consequences is indefinite, then with a little skill any experimental result can be made to look like the expected consequences.

Overview

CoChat Summary

TL;DR

The paper argues that backtest overfitting makes it easy to manufacture high in-sample Sharpe ratios, while out-of-sample performance can be zero or negative, especially when performance series have memory. It derives a Minimum Backtest Length and uses simulations plus a seasonal-strategy example to show that high in-sample Sharpe ratios can arise from skill-less random data.

Key findings

  • With only \(N=10\) independent strategy configurations, the expected maximum in-sample Sharpe ratio is 1.57 even though all strategies have expected out-of-sample Sharpe ratio of zero; with \(N=128\) trials, the expected maximum Sharpe ratio is above 2.6.
  • If only five years of data are available, no more than 45 independent model configurations should be tried to keep the expected maximum in-sample Sharpe ratio at 1 while expected out-of-sample Sharpe ratio is 0; after only seven independent configurations, the expected maximum in-sample Sharpe ratio is 1 for a two-year backtest, with expected out-of-sample Sharpe ratio 0.
  • Adding a global constraint or serial dependence creates compensation effects: recentering makes out-of-sample performance negative even for a trendless process, with the p-value for the IS-performance relation equal to 0; for an AR(1) process with \(\phi=0.995\), the half-life is about 138 observations and the p-value for the IS-performance relation is again 0.
  • In a synthetic seasonal-strategy example using 1,000 daily prices and 8,800 parameter combinations, the optimal rule has an annualized Sharpe ratio of 1.27 and a PSR-Stat of 2.83, implying less than 1 percent probability that the true Sharpe ratio is below 0, despite no true seasonal effect.

Methods

  • Theoretical derivation: assumes IID Normal excess returns and uses the Sharpe ratio, extreme-value/Gumbel approximations, and propositions to derive the expected maximum Sharpe ratio and Minimum Backtest Length (MinBTL).
  • Monte Carlo simulations: generates Gaussian random walks of size \(T=1000\), splits them into 500 in-sample and 500 out-of-sample observations, and studies selection of the best in-sample Sharpe ratio; also runs recentered and first-order autoregressive (\(\phi=0.995\)) variants.
  • Practical application: performs a grid search over Entry day (1–22), Holding period (1–20), Stop loss (0–10), and Side (\(\pm1\)), forming 8,800 parameter combinations on a 1,000-day random-walk price series, and reports the optimal Sharpe ratio and PSR-Stat.

Limitations

  • The paper states that MinBTL is a necessary but not sufficient condition to avoid overfitting, and that Proposition 1 assumes independent trials, which is a conservative estimate; if trials are not independent, the number of independent trials could be derived using a dimension-reduction procedure such as Principal Component Analysis.
  • The paper states that the hold-out method cannot assess the representativeness of a backtest because it does not account for the number of trials attempted, and it leaves a more precise measure of backtest overfitting to Bailey et al. [1].
  • Evident from the text: the simulations and practical example rely on synthetic IID Normal/Gaussian random walks and specific parameter choices rather than real market data, so the numerical illustrations are model-based demonstrations rather than empirical estimates.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1090/noti1105

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Overfitting a backtest's results as the number of trials grows.
Figure 1 Overfitting a backtest's results as the number of trials grows.From the PDF
Figure 2 Minimum Backtest Length needed to avoid overfitting, as a function of the number of trials.
Figure 2 Minimum Backtest Length needed to avoid overfitting, as a function of the number of trials.From the PDF
Figure 3 Performance IS vs. OOS before introducing strategy selection.
Figure 3 Performance IS vs. OOS before introducing strategy selection.From the PDF
Figure 4 Performance IS vs. performance OOS for one path after introducing strategy selection.
Figure 4 Performance IS vs. performance OOS for one path after introducing strategy selection.From the PDF
Figure 5 Performance degradation after introducing strategy selection in absence of compensation effects.
Figure 5 Performance degradation after introducing strategy selection in absence of compensation effects.From the PDF
Figure 6 Performance degradation as a result of strategy selection under compensation effects (global constraint).
Figure 6 Performance degradation as a result of strategy selection under compensation effects (global constraint).From the PDF
Figure 7 Performance degradation as a result of strategy selection under compensation effects (first-order serial correlation).
Figure 7 Performance degradation as a result of strategy selection under compensation effects (first-order serial correlation).From the PDF
Figure 8 Backtested performance of a seasonal strategy (Example 6).
Figure 8 Backtested performance of a seasonal strategy (Example 6).From the PDF

Verification

Claims the paper makes

0 Valid · 8 Not valid · 6 Not checked · 68 Own results and interpretations

45 more claims

Create a free account to see all 82 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 21
  • Probably not a substantial claim. 16
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 28 references

Research works printed but not linked: 4

  1. [3][3] , Drawdown-based stop-outs and the triple penance rule, 2013, working paper. Available at http://ssrn.com/abstract=2201302 . -
  2. [9][9] C. Harvey and Y. Liu , Backtesting, working paper, SSRN, 2013, http://ssrn.com/abstract=2345489 . -
  3. [10][10] C. Harvey, Y. Liu , and H. Zhu , : : : and the crosssection of expected returns, working paper, SSRN, 2013, http://ssrn.com/abstract=2249314 . -
  4. [12][12] Y. Hirsch , Don't Sell Stocks on Monday , Penguin Books, 1st edition, 1987. -

Cited by

Metadata

Authors
Bailey, D. H., Borwein, J. M., López de Prado, M., & Zhu, Q. J.
Affiliations
  • Lawrence Berkeley National Laboratory (US)
  • Western Michigan University (US)
Venue
Notices of the American Mathematical Society
Published
2014-05-01
Volume
61
Issue
5
Pages
458
ISSN
  • 0002-9920 (ISSN-L)
  • 1088-9477
OpenAlex
W3125595766
Cited by
221
References
25
Citations by year
2026: 43 · 2025: 18 · 2024: 12 · 2023: 16 · 2022: 6 · 2021: 23 · 2020: 15 · 2019: 16 · 2018: 16 · 2017: 11 · 2016: 15 · 2015: 20 · 2014: 9
FWCI
33.55
Type
journal article
Language
en
Open access
Locations
  • Notices of the American Mathematical Society (publishedVersion)
  • NOVA (University of Newcastle Australia) (submittedVersion)
Funders
  • American Mathematical Society
  • University of Oxford
  • Eidgenössische Technische Hochschule Zürich
  • University of California, Irvine
  • Lawrence Berkeley National Laboratory
Topics
  • Social Sciences › Economics, Econometrics and Finance › Finance › Financial Markets and Investment Strategies
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Stock Market Forecasting Methods
  • Social Sciences › Economics, Econometrics and Finance › Finance › Stochastic processes and financial applications
Keywords
backtest overfitting, out-of-sample performance, financial forecasting, model selection, time series forecasting

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.