Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index; none of the 6 sources its claims cite is checked for support yet.
Abstract
Another thing I must point out is that you cannot prove a vague theory wrong.[…] Also, if the process of computing the consequences is indefinite, then with a little skill any experimental result can be made to look like the expected consequences.
Overview
CoChat Summary
TL;DR
The paper argues that backtest overfitting makes it easy to manufacture high in-sample Sharpe ratios, while out-of-sample performance can be zero or negative, especially when performance series have memory. It derives a Minimum Backtest Length and uses simulations plus a seasonal-strategy example to show that high in-sample Sharpe ratios can arise from skill-less random data.
Key findings
- With only \(N=10\) independent strategy configurations, the expected maximum in-sample Sharpe ratio is 1.57 even though all strategies have expected out-of-sample Sharpe ratio of zero; with \(N=128\) trials, the expected maximum Sharpe ratio is above 2.6.
- If only five years of data are available, no more than 45 independent model configurations should be tried to keep the expected maximum in-sample Sharpe ratio at 1 while expected out-of-sample Sharpe ratio is 0; after only seven independent configurations, the expected maximum in-sample Sharpe ratio is 1 for a two-year backtest, with expected out-of-sample Sharpe ratio 0.
- Adding a global constraint or serial dependence creates compensation effects: recentering makes out-of-sample performance negative even for a trendless process, with the p-value for the IS-performance relation equal to 0; for an AR(1) process with \(\phi=0.995\), the half-life is about 138 observations and the p-value for the IS-performance relation is again 0.
- In a synthetic seasonal-strategy example using 1,000 daily prices and 8,800 parameter combinations, the optimal rule has an annualized Sharpe ratio of 1.27 and a PSR-Stat of 2.83, implying less than 1 percent probability that the true Sharpe ratio is below 0, despite no true seasonal effect.
Methods
- Theoretical derivation: assumes IID Normal excess returns and uses the Sharpe ratio, extreme-value/Gumbel approximations, and propositions to derive the expected maximum Sharpe ratio and Minimum Backtest Length (MinBTL).
- Monte Carlo simulations: generates Gaussian random walks of size \(T=1000\), splits them into 500 in-sample and 500 out-of-sample observations, and studies selection of the best in-sample Sharpe ratio; also runs recentered and first-order autoregressive (\(\phi=0.995\)) variants.
- Practical application: performs a grid search over Entry day (1–22), Holding period (1–20), Stop loss (0–10), and Side (\(\pm1\)), forming 8,800 parameter combinations on a 1,000-day random-walk price series, and reports the optimal Sharpe ratio and PSR-Stat.
Limitations
- The paper states that MinBTL is a necessary but not sufficient condition to avoid overfitting, and that Proposition 1 assumes independent trials, which is a conservative estimate; if trials are not independent, the number of independent trials could be derived using a dimension-reduction procedure such as Principal Component Analysis.
- The paper states that the hold-out method cannot assess the representativeness of a backtest because it does not account for the number of trials attempted, and it leaves a more precise measure of backtest overfitting to Bailey et al. [1].
- Evident from the text: the simulations and practical example rely on synthetic IID Normal/Gaussian random walks and specific parameter choices rather than real market data, so the numerical illustrations are model-based demonstrations rather than empirical estimates.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
0 Valid · 8 Not valid · 6 Not checked · 68 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 6 shows that adding a single global constraint causes the OOS performance to be negative even though the underlying process was trendless.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This illustrates the fact that, in absence of compensation effects, overfitting the IS performance ( x -axis) has no bearing on the OOS performance ( y -axis), which remains around zero.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Proposition 1. Given a sample of IID random variables, xn Z , n 1 ; : : : ; N , where Z is the CDF of the Standard Normal distribution, the expected maximum of that sample, E max N E max φ xn γ , can be approximated for a large N as”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- Rests on 3 assumptions not shown here.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Theorem 2. The Minimum Backtest Length (MinBTL, in years) needed to avoid selecting a strategy with an IS Sharpe ratio of E max N among N independent strategies with an expected OOS Sharpe ratio of zero is (6)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 3 assumptions not shown here.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This has allowed us to determine the Minimum Backtest Length (MinBTL) needed to avoid selecting a strategy with a given IS Sharpe ratio among N trials with an expected OOS Sharpe ratio of zero.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
45 more claims
Create a free account to see all 82 claims and every source verdict.
- It may state two separate claims as one. 21
- Probably not a substantial claim. 16
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“For example, we could explore all nodes that span the set φ 1 ; : : : ; 22 γ for Entry day , the set φ 1 ; : : : ; 20 γ for Holding period , the set φ 0 ; : : : ; 10 γ for Stop loss , and φ 1 ; 1 γ for Side . The parameter combinations involved form a four-dimensional mesh of 8,800 elements.”
Reported 8800 · recomputed 9680
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Is Backtest Overfitting a Fraud? · p. 9
“Consider an investment manager who emails his stock market forecast for the next month to 2 n x prospective investors, where x and n are positive integers.”
§ Is Backtest Overfitting a Fraud? · p. 9
“The investment manager is hiding the fact that for every one of the x successful witnesses, he has tried 2 n unsuccessful ones”
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Embrechts, P., Klüppelberg, C., & Mikosch, T. (1997) · doi:10.1007/978-3-642-33483-2 · Cited by 4,402
Extreme Values, Regular Variation and Point Processes
Resnick, S. I. (1987) · Springer Series in Operations Research and Financial Engineering · doi:10.1007/978-0-387-75953-1 · Cited by 3,158
Hawkins, D. M. (2004) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci0342472 · Cited by 2,522
Feynman, R. (2017) · The MIT Press eBooks · doi:10.7551/mitpress/11068.001.0001 · Cited by 1,881
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851 · cited 1×
“—Richard Feynman [1964]”
Rules for Ordering Uncertain Prospects
Hadar, J., & Russell, W. R. (1969) · American Economic Review · Cited by 1,720
Stepwise Multiple Testing as Formalized Data Snooping
Romano, J. P., & Wolf, M. (2005) · Econometrica · doi:10.1111/j.1468-0262.2005.00615.x · Cited by 1,019
Weiss, S. M., & Kulikowski, C. A. (1991) · Morgan Kaufmann Publishers Inc. eBooks · Cited by 898
The Statistics of Sharpe Ratios
Lo, A. W. (2002) · Financial Analysts Journal · doi:10.2469/faj.v58.n4.2453 · Cited by 843
Research works printed but not linked: 4
- [3][3] , Drawdown-based stop-outs and the triple penance rule, 2013, working paper. Available at http://ssrn.com/abstract=2201302 . -
- [9][9] C. Harvey and Y. Liu , Backtesting, working paper, SSRN, 2013, http://ssrn.com/abstract=2345489 . -
- [10][10] C. Harvey, Y. Liu , and H. Zhu , : : : and the crosssection of expected returns, working paper, SSRN, 2013, http://ssrn.com/abstract=2249314 . -
- [12][12] Y. Hirsch , Don't Sell Stocks on Monday , Penguin Books, 1st edition, 1987. -
Cited by
Items 201–208 of 208
Machine learning in stock market forecasting: a comprehensive review
Khemka, M., Haddad, K., Redwan, T., & Ahmed, M. (2026) · Discover Computing · doi:10.1007/s10791-026-10491-5 · Cited by 4
“These approaches require rigorous safeguards against overfitting, data leakage, and unrealistic backtest assumptions [78].”
Vu, M.-S., Pham, T.-T., & Hong-Viet, T. (2026) · Machine Learning with Applications · doi:10.1016/j.mlwa.2026.100987 · Cited by 1
Boukeffa, H., Djeffal, S., & Ghoul, A. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1953907
Large Language Quant: Cross-Domain Mathematical Trading Strategy Generation with an Agentic LLM
Clayman, A., Kampouridis, M., & Papastylianou, T. (2026) · IEEE Conference on Computational Intelligence for Financial Engineering & Economics · doi:10.1109/cifer67845.2026.11692403
“Backtest overfitting is a well-documented risk [14], [15], motivating our use of distributions of OOS performance from repeated runs rather than single-point estimates in Stage 2.”
Execution-Aware Alpha Mining: Teaching LLM Factor Agents to Account for Trading CostsPreprint
Nagireddy, R. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7498983
Zhang, Y.-Q., & Zhang, D. (2026) · 2026 7th International Conference on Pattern Recognition and Machine Learning (PRML) · doi:10.1109/prml71097.2026.11687268
Macias, R. C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7525882
Metadata
- Authors
- Bailey, D. H., Borwein, J. M., López de Prado, M., & Zhu, Q. J.
- Affiliations
- Lawrence Berkeley National Laboratory (US)
- Western Michigan University (US)
- Venue
- Notices of the American Mathematical Society
- Publisher
- American Mathematical Society (AMS)
- Published
- 2014-05-01
- Volume
- 61
- Issue
- 5
- Pages
- 458
- ISSN
- 0002-9920 (ISSN-L)
- 1088-9477
- OpenAlex
- W3125595766
- Cited by
- 221
- References
- 25
- Citations by year
- 2026: 43 · 2025: 18 · 2024: 12 · 2023: 16 · 2022: 6 · 2021: 23 · 2020: 15 · 2019: 16 · 2018: 16 · 2017: 11 · 2016: 15 · 2015: 20 · 2014: 9
- FWCI
- 33.55
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Notices of the American Mathematical Society (publishedVersion)
- NOVA (University of Newcastle Australia) (submittedVersion)
- Funders
- American Mathematical Society
- University of Oxford
- Eidgenössische Technische Hochschule Zürich
- University of California, Irvine
- Lawrence Berkeley National Laboratory
- Topics
- Social Sciences › Economics, Econometrics and Finance › Finance › Financial Markets and Investment Strategies
- Social Sciences › Decision Sciences › Management Science and Operations Research › Stock Market Forecasting Methods
- Social Sciences › Economics, Econometrics and Finance › Finance › Stochastic processes and financial applications
- Keywords
- backtest overfitting, out-of-sample performance, financial forecasting, model selection, time series forecasting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.