CoChat
Sign Up
Cited by 118Financial Markets and Investment StrategiesFinancial Risk and Volatility ModelingStatistical Methods in Clinical Trials

False (and Missed) Discoveries in Financial Economics

HARVEY, CAMPBELL R., LIU, YAN·2020·The Journal of Finance
DOI 10.1111/jofi.12951
Open this paper in CoChat — free account

Abstract

Multiple testing plagues many important questions in finance such as fund and factor selection. We propose a new way to calibrate both Type I and Type II errors. Next, using a double‐bootstrap method, we establish a t ‐statistic hurdle that is associated with a specific false discovery rate (e.g., 5%). We also establish a hurdle that is associated with a certain acceptable ratio of misses to false discoveries (Type II error scaled by Type I error), which effectively allows for differential costs of the two types of mistakes. Evaluating current methods, we find that they lack power to detect outperforming managers.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1111/jofi.12951

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Only about 18% of studies impose a price filter to exclude penny stocks.

    “Harvey and Liu (2020) show that only about 18% of the studies impose a price filter to exclude penny stocks.”

    From Empirical Asset Pricing via Machine Learning: The Role of Research Design Choices (2026) · cites this paper as Harvey and Liu (2020)

    The data description (A.1, pp. 19–20) discusses investment strategies and anomaly data, but does not report how often studies exclude penny stocks. Searches for “penny stocks,” “price filter,” “18% of studies,” and minimum stock-price restrictions found no passage addressing the claimed percentage. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. Andrikogiannopoulou, Angie, and Filippos Papakonstantinou, 2019, Reassessing false discoveries in mutual fund performance: Skill, luck, or lack of power? Journal of Finance 74, 2667-2688.
  2. Avramov, Doron, Robert Kosowski, Narayan Y. Naik, and Melvyn Teo, 2011, Hedge funds, managerial skill, and macroeconomic variables, Journal of Financial Economics 99, 672-692.
  3. Ayadi, Mohamed A., and Lawrence Kryzanowski, 2011, Fixed-income fund performance: Role of luck and ability in tail membership, Journal of Empirical Finance 18, 379-392.
  4. Bajgrowicz, Pierre, and Olivier Scaillet, 2012, Technical trading revisited: False discoveries, persistence tests, and transaction costs, Journal of Financial Economics 106, 473-491.
  5. Barras, Laurent, 2019, A large-scale approach for evaluating asset pricing models, Journal of Financial Economics 134, 549-569.
  6. Barras, Laurent, Olivier Scaillet, and Russ Wermers, 2010, False discoveries in mutual fund performance: Measuring luck in estimated alphas, Journal of Finance 65, 179-216.
  7. Beneish, Messod D., 1997, Detecting GAAP violation: Implications for assessing earnings management among firms with extreme financial performance, Journal of Accounting and Public Policy , Fall 1997, 271-309.
  8. Beneish, Messod D., 1999, The detection of earnings manipulation, Financial Analysts' Journal 55, 24-36.
  9. Benjamini, Yoav, and Yosef Hochberg, 1995, Controlling for the false discovery rate: A practical and powerful approach to multiple testing, Journal of the Royal Statistical Society , Series B 57, 289-300.
  10. Blake, David, Alberto Rossi, Allan Timmermann, Ian Tonks, and Russ Wermers, 2013, Decentralized investment management: Evidence from the pension fund industry, Journal of Finance 68, 1133-1178.
  11. Brown, S. J., W. Goetzmann, R. G. Ibbotson, and S. A. Ross, 1992, Survivorship bias in performance studies, Review of Financial Studies 5, 553-580.
  12. Busse, Jeffrey A., Amit Goyal, and Sunil Wahal, 2014, Investing in a global world, Review of Finance 18, 561-590.
  13. Carhart, Mark M., 1997, On persistence in mutual fund performance, Journal of Finance 52, 57-82.
  14. Carhart, M. M., J. N. Carpenter, A. W. Lynch, and D. K. Musto, 2002, Mutual fund survivorship, Review of financial studies 15, 1439-1463.
  15. Cao, Charles, Yong Chen, Bing Liang, and Andrew W. Lo, 2013, Can hedge funds time market liquidity? Journal of Financial Economics 109, 493-516.
  16. Chen, Yong, and Bing Liang, 2007, Do market timing hedge funds time the market? Journal of Financial and Quantitative Analysis 42, 827-856.
  17. Chordia, Tarun, Amit Goyal, and Alessio Saretto, 2020, Anomalies and false rejections. Review of Financial Studies 33, 2134-2179.
  18. D'Agostino, Antonello, Kieran McQuinn, and Karl Whelan, 2012, Are some forecasters really better than others? Journal of Money, Credit, and Banking 44, 715-732.
  19. De Long, J. Bradford, and Kevin Lang, 1992, Are all economic hypotheses false? Journal of Political Economy 100, 1257-1272.
  20. DeGroot, Morris, 1975, Probability and Statistics (Addison-Wesley, Reading, MA).
  21. DeGroot, Morris, and Mark J. Schervish, 2011, Probability and Statistics, 4th edition (Pearson Education Limited, Harlow, UK)
  22. Del Guercio, Diane, and Paula A. Tkac, 2008, Star power: The effect of Morningstar ratings on mutual fund flow, Journal of Financial and Quantitative Analysis 43, 907-936.
  23. Elton, Edwin J., Martin J. Gruber, and Christopher R. Blake, 1996, Survivor bias and mutual fund performance, Review of Financial Studies 9, 1097-1120.
  24. Elton, Edwin J., Martin J. Gruber, and Christopher R. Blake, 2001, A first look at the accuracy of the CRSP mutual fund database and a comparison of the CRSP and Morningstar mutual fund database, Journal of Finance 56, 2415-2430.
  25. Fama, Eugene F., and Kenneth R. French, 1993, Common risk factors in the returns on stocks and bonds, Journal of Financial Economics 33, 3-56.
  26. Fama, Eugene F., and Kenneth R. French, 2010, Luck versus skill in the cross-section of mutual fund returns, Journal of Finance 65, 1915-1947.
  27. Fawcett, T., 2006, An introduction to ROC analysis, Pattern Recognition Letters 27, 861874.
  28. Genovese, Christopher, and Larry Wasserman, 2002, Operating charateristics and extensions of the false discovery rate procedure, Journal of the Royal Statistical Society , Series B 64, 499-517.
  29. Hastie, T., Robert Tibshirani, and Jerome Friedman, 2009, The elements of statistical learning: data mining, inference, and prediction, Springer Science & Business Media New York, New York.
  30. Harvey, Campbell R., 2017, Presidential address: The scientific outlook in financial economics, Journal of Finance 72, 1399-1440.
  31. Harvey, Campbell R., and Yan Liu, 2017, Luck vs. skill and factor selection, in John Cochrane and Tobias J. Moskowitz, eds.: The Fama Portfolio (University of Chicago Press, Chicago).
  32. Harvey, Campbell R., and Yan Liu, 2018, Detecting repeatable performance, Review of Financial Studies 31, 2499-2552.
  33. Hau, Harald, and Sandy Lai, 2013, Real effects of stock underpricing, Journal of Financial Economics 108, 392-408.
  34. Horowitz, Joel L., 2001, The bootstrap, Handbook of Econometrics 5 (Elsevier, Amsterdam, Netherlands).
  35. Hou, Kewei, Chen Xue, and Lu Zhang, 2020, Replicating anomalies, Review of Financial Studies 33, 2019-2133.
  36. Ioannidis, John P. A., and Chris H. Doucouliagos, 2013, What's to know about the credibility of empirical economics, Journal of Economic Surveys 27, 997-1004.
  37. Ioannidis, John P. A., Tom D. Stanley, and Hristos Doucouliagos, 2017, The power of bias in economics research, Economic Journal 127, 236-265.
  38. Jiang, George J., Tong Yao, and Tong Yu, 2007, Do mutual funds time the market? Evidence from portfolio holdings, Journal of Financial Economics 86, 724-758.
  39. Jones, Christopher S., and Jay Shanken, 2005, Mutual fund performance with learning across funds, Journal of Financial Economics 78, 507-552.
  40. Kandel, Shmuel, and Robert F. Stambaugh, 1996, On the predictability of stock returns: An asset-allocation perspective, Journal of Finance 51, 385-424.
  41. Kosowski, Robert, Allan Timmermann, Russ Wermers, and Hal White, 2006, Can mutual fund 'stars' really pick stocks? New evidence from a bootstrap analysis, Journal of Finance 61, 2551-2595.
  42. Leamer, Edward E., 1983, Let's take the con out of econometrics, American Economic Review 73, 31-43.
  43. Nanda, Vikram, Z. Jay Wang, and Lu Zheng, 2004, Family values and the star phenomenon: Strategies of mutual fund families, Review of Financial Studies 17, 667-698.
  44. Storey, John D., 2002, A direct approach to false discovery rates, Journal of the Royal Statistical Society , Series B 64, 479-498.
  45. Storey, John D., 2003, The positive false discovery rate: A Bayesian interpretation and the q-value. Annals of Statistics 31, 2013-2035.
  46. Yan, Xuemin, and Lingling Zheng, 2017, Fundamental analysis and the cross-section of stocks returns: A data-mining approach, Review of Financial Studies 30, 1382-1423.
  47. Ziliak, Stephen T., and Deirdre N. McCloskey, 2004, Size matters: The standard error of regressions in the American Economic Review 33, 527-546.

Cited by

Sorted by citations

Authors
HARVEY, C. R., & LIU, Y.
Affiliations
  • National Bureau of Economic Research (US)
  • University of Southern California (US)
Corresponding author
HARVEY, CAMPBELL R.
ORCID
HARVEY, CAMPBELL R.
Venue
The Journal of Finance
Publisher
Wiley
Published
2020-06-16
Volume
75
Issue
5
Pages
2503-2553
ISSN
  • 0022-1082 (ISSN-L)
  • 1540-6261
DOI
10.1111/jofi.12951
OpenAlex
W3121153477
Cited by
118
References
5
Citations by year
2026: 22 · 2025: 22 · 2024: 13 · 2023: 11 · 2022: 16 · 2021: 16 · 2020: 9 · 2019: 8 · 2018: 1
FWCI
13.52
Type
journal article
Language
en
Open access
closed
Licence
onlinelibrary.wiley.com/termsAndConditions#vor (vor)
Locations
  • The Journal of Finance (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Topics
  • Social Sciences › Economics, Econometrics and Finance › Finance › Financial Markets and Investment Strategies
  • Social Sciences › Economics, Econometrics and Finance › Finance › Financial Risk and Volatility Modeling
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords
multiple comparisons, false discovery rate, type I and type II errors, double bootstrap, fund selection, factor selection, manager performance evaluation, statistical power

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review