… and the Cross-Section of Expected Returns

Probably not a substantial claim: may not pass S3 (load-bearing).

Barras, Scaillet, and Wermers (2010) argue for a correlation of zero among fund returns.

“Barras, Scaillet, and Wermers (2010) argue for a correlation of zero among fund returns (i.e., excess returns against benchmark factors)”

• cites Barras, Scaillet, and Wermers (2010) sourceDisputed

  Section I.C, “Cross-Sectional Dependence among Funds,” reports an average pairwise correlation of 0.08 between four-factor model residuals, not zero. The authors describe the dependencies as sufficiently low for their estimators and explicitly incorporate the observed correlations in simulations; they do not argue that fund residuals are uncorrelated. (LLM, conf=0.88)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Lehmann and Romano (2005) develop methods to directly control the realized FDP and propose a step-down method to control the probability of FDP exceeding a threshold value.

“Instead of FDR, recent research by Lehmann and Romano (2005) develops methods to directly control the realized FDP.In particular, they propose a step-down method to control for the probability of FDP exceeding a threshold value.”

• cites Lehmann and Romano (2005) sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

The Fama and French (1992) HML and SMB factors are correlated.

“For example, it is well known that two of the most popular factors, the Fama and French (1992) HML and SMB, are correlated.”

• cites Fama and French (1992) sourceCould not verify

  The discussion in “A. Rational Asset-Pricing Stories” reports a −0.26 correlation between firm size and book-to-market equity, not between returns on the HML and SMB factors. Searches for “HML SMB factors correlated” and “high minus low small minus big factor returns correlation” found no passage in our copy reporting the claimed factor correlation. (LLM, conf=0.77)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

The search identified 63 working papers.

“We end with 63 working papers.”

Probably not a substantial claim: may not pass S4 (based).

Hundreds of papers and factors attempt to explain the cross-section of expected returns.

“Hundreds of papers and factors attempt to explain the cross-section of expected returns.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The search identified 313 articles, of which 250 were published.

“In total, we focus on 313 articles, among which are 250 published articles.”

May combine two claims.

For single tests at the usual 5% cutoff, 10 out of 10 factors are discovered; under multiple testing the cutoffs are BHY 0.85%, Holm 0.60%, and Bonferroni 0.5%.

“In summary, for single tests, using the usual 5% cutoff, 10 out of 10 are discovered.Allowing for multiple tests, the cutoffs are far smaller, with BHY at 0.85%, Holm at 0.60%, and Bonferroni at 0.5%.”

May combine two claims.

Using the M > R simulation, the new benchmark t-statistics for Bonferroni and Holm are estimated to be 4.01 and 3.96, respectively, both slightly higher than when M = R.

“Using this information, the new benchmark t-statistics for Bonferroni and Holm are estimated to be 4.01 and 3.96, respectively, both slightly higher than when M = R.”

• AssumptionThe authors state this as an assumption: M factors are tried, but only factors that exceed a certain t-statistic threshold are published; the threshold is set at 1.96.
• AssumptionThe authors state this as an assumption: The sample covers a fraction r of t-statistics between 1.96 and 2.57, and all t-statistics above 2.57 are covered.

Classified as probably not a substantial claim.

Based on the sample of observed t-statistics of published factors, three benchmark t-statistics are obtained.

“Based on our sample of observed t-statistics of published factors, 30 we obtain three benchmark t-statistics.”

May combine two claims.

Other specifications of N change the estimate of λ but leave the other parameters almost intact.

“Other specifications of N change the estimate of λ but leave the other parameters almost intact.”

May combine two claims.

Under M > R, the BHY implied t-statistic increases from 3.39 to 3.68 at 1% significance and from 2.78 to 3.18 at 5% significance.

“The BHY implied t-statistic increases from 3.39 to 3.68 at 1% significance and from 2.78 to 3.18 at 5% significance.”

• AssumptionThe authors state this as an assumption: M factors are tried, but only factors that exceed a certain t-statistic threshold are published; the threshold is set at 1.96.
• AssumptionThe authors state this as an assumption: The sample covers a fraction r of t-statistics between 1.96 and 2.57, and all t-statistics above 2.57 are covered.

May combine two claims.

All ten factors would be discovered under single hypothesis testing, but only three to six factors would be discovered under a multiple hypothesis test.

“While all ten factors would be discovered under single hypothesis testing, only three to six factors would be discovered under a multiple hypothesis test.”

May combine two claims.

Both Bonferroni and Holm adjusted benchmark t-statistics are monotonically increasing in the number of discoveries.

“Both Bonferroni and Holm adjusted benchmark t-statistics are monotonically increasing in the number of discoveries.”

Probably not a substantial claim: may not pass S3 (load-bearing).

These threshold t-statistics are not far from previous estimates of 3.78 (Holm adjustment controlling FWER at 5%) and 3.38 (BHY adjustment controlling FDR at 1%).

“Notice that these numbers are not far away from our previous estimates of 3.78 (Holm adjustment that controls FWER at 5%) and 3.38 (BHY adjustment that controls FDR at 1%).”

May combine two claims.

These c(M) = 1 thresholds are smaller than those under the default c(M) specification but are above 3.0 and consistent with the overall message.

“Indeed, these numbers are smaller than the numbers under our default specification of c(M) (i.e., c(M)= M j =1 1 j ).However, they are above 3.0 and therefore are consistent with our overall message.”

Probably not a substantial claim: may not pass S4 (based).

At least 316 factors have been tested to explain the cross-section of expected returns.

“At least 316 factors have been tested to explain the cross-section of expected returns.”

Classified as probably not a substantial claim. May combine two claims.

For the baseline sample (r = 1/2), the matched sample statistics are: total number of discoveries T = 353, 20th percentile Q1 = 2.39, 50th percentile Q2 = 3.16, and 90th percentile Q3 = 6.34.

“In particular, the quantities we choose to match and their values for the baseline sample (i.e., r = 1/2) are given by: ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ T = Total number of discoveries = 353, Q 1 = The 20th percentile of the sample of t-statistics = 2.39, Q 2 = The 50th percentile of the sample of t-statistics = 3.16, Q 3 = The 90th percentile of the sample of t-statistics = 6.34.”

• AssumptionThe authors state this as an assumption: The sample covers a fraction r of t-statistics between 1.96 and 2.57, and all t-statistics above 2.57 are covered.

May combine two claims.

Both p0 and M are increasing in ρ.

“For the other parameter estimates, both p 0 and M are increasing in ρ.”

May combine two claims.

BHY implied benchmarks are not monotonic; they fluctuate before 2000 and stabilize at 3.39 (p-value = 0.07%) after 2010.

“BHY implied benchmarks, on the other hand, are not monotonic.They fluctuate before year 2000 and stabilize at 3.39 (pvalue = 0.07%) after 2010.”

May combine two claims.

When γ = 0.10 and α = 0.05, the benchmark t-statistic is 2.70 (p-value = 0.69%), much higher than the conventional cutoff of 1.96.

“For instance, when γ = 0.10 and α = 0.05, the benchmark t-statistic is 2.70 (p-value = 0.69%), which is much higher than the conventional cutoff of 1.96.”

May combine two claims.

Using 113 common factors instead of 316 does not materially change the main conclusions; for example, Holm at 113 factors is 3.29 (p-value = 0.10%), while Holm at 316 factors is 3.64 (p-value = 0.03%).

“Figure 3 shows that our main conclusions do not materially change.For example, the Holm at 113 factors is 3.29 (p-value = 0.10%), while Holm at 316 factors is 3.64 (p-value = 0.03%).”

May combine two claims.

If α d is changed to 5%, the BHY implied benchmark t-statistic is 2.78 (p-value = 0.54%) in 2012 and 2.81 (p-value = 0.50%) in 2032, still much higher than 1.96.

“If we change α d to 5%, the corresponding BHY implied benchmark t-statistic is 2.78 (p-value = 0.54%) in 2012 and 2.81 (p-value = 0.50%) in 2032, still much higher than the starting value of 1.96.”

May combine two claims.

BHY leads to six significant factors (tests 8, 7, 4, 9, 6, and 2), three more than Bonferroni and two more than Holm.

“In the end, BHY leads to six significant factors (tests 8, 7, 4, 9, 6, and 2), three more than Bonferroni and two more than Holm.”

May combine two claims.

In the BHY example, M×c(M) α d is 6, so the first six ordered tests are rejected and the remaining four are not.

“M×c(M) α d is 6, and we reject the null hypothesis for the first six ordered tests and fail to reject for the remaining four tests.”

May combine two claims.

For the factor group published no earlier than 2000, using Fama-MacBeth tests, covering at least 1970-1995, and controlling for the Fama-French three factors (124 factors), the Bonferroni and Holm implied threshold t-statistics are 3.54 and 3.20 at 5% significance, and the BHY implied thresholds are 3.23 at 1% and 2.67 at 5% by 2012.

“Based on this factor group, the Bonferroni and Holm implied threshold t-statistics are 3.54 and 3.20 (5% significance), respectively, and the BHY implied thresholds are 3.23 (1% significance) and 2.67 (5% significance) by 2012.”

May combine two claims.

When ρ is increased to 0.60, a total of M = 1,775 factors have been tried and around 39.9% are true factors.

“When ρ is increased to 0.60, we estimate that a total of M = 1,775 factors have been tried and around 39.9% (= 1-0.601) are true factors.”

May combine two claims.

With c(M) = 1 and 71% of tried factors missing simulated as in Appendix A, the BHY implied threshold equals 3.05 at 5% significance and 3.17 at 1% significance.

“Based on our estimate that 71% of tried factors are missing and by simulating the missing tests as in Appendix A, we find that the BHY implied threshold equals 3.05 at 5% significance and 3.17 at 1% significance.”

• AssumptionThe authors state this as an assumption: M factors are tried, but only factors that exceed a certain t-statistic threshold are published; the threshold is set at 1.96.
• AssumptionThe authors state this as an assumption: The sample covers a fraction r of t-statistics between 1.96 and 2.57, and all t-statistics above 2.57 are covered.

May combine two claims.

Applying Holm's adjustment to the Table 4 example, the smallest b satisfying p(b) > α w M+1-b is 5, so the first four ordered tests are rejected and the remaining six are not.

“Starting from the smallest p-value and going up, we see that p (b) is below α w M+1-b until b = 5, at which point p (5) is above α w 10+1-5 .Therefore, the smallest b that satisfies p (b) > α w M+1-b is 5 and we reject the null hypothesis for the first four ordered tests (we discover four factors) and fail to reject the null for the remaining six tests.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Most of these factors have been proposed over the last ten years.

“Most of these factors have been proposed over the last ten years.”

May combine two claims.

Among prominent factors, HML, MOM, DCG, SRV, and MRT are significant across all types of t-statistic adjustments; EP, LIQ, and CVOL are sometimes significant; the rest are never significant.

“Among these factors, HML, MOM, DCG, SRV, and MRT are significant across all types of t-statistic adjustments, EP, LIQ, and CVOL are sometimes significant, and the rest are never significant.”

May combine two claims.

For Bonferroni, the benchmark t-statistic starts at 1.96 and increases to 3.78 by 2012; it reaches 4.00 in 2032.

“For Bonferroni, the benchmark t-statistic starts at 1.96 and increases to 3.78 by 2012.It reaches 4.00 in 2032.”

May combine two claims.

Holm implied t-statistics always fall below Bonferroni t-statistics, consistent with Bonferroni resulting in fewer discoveries than Holm, but Holm tracks Bonferroni closely and their differences are small.

“Holm implied t-statistics always fall below Bonferroni t-statistics, consistent with the fact that Bonferroni always results in fewer discoveries than Holm.However, Holm tracks Bonferroni closely and their differences are small.”

May combine two claims.

In total, 164 factors were discovered in the past nine years, roughly doubling the 84 factors discovered in all previous years.

“In total, 164 factors were discovered in the past nine years, roughly doubling the 84 factors discovered in all previous years.”

May combine two claims.

Across various correlation specifications, a t-statistic of 3.9 is needed to control FWER at 5%, and a t-statistic of 3.0 is needed to control FDR at 1%.

“Across various correlation specifications, our estimates show that in general a t-statistic of 3.9 and 3.0 is needed to control FWER at 5% and FDR at 1%, respectively.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The Holm implied t-statistic is 3.96 when r=1, higher than 3.64 before.

“The Holm implied t-statistic (3.96), while not necessarily increasing in the number of trials, is also higher than before (3.64).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

When r=1, the median estimate for total number of trials is 817, almost the same as the previous estimate of 820.

“When r = 1, the median estimate for the total number of trials is 817, 54 almost the same as our previous estimate of 820 based on the original sample.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The three conventional multiple testing frameworks differ in their assumptions but are consistent in their conclusions.

“While these frameworks differ in their assumptions, they are consistent in their conclusions.”

May combine two claims.

The Bonferroni implied benchmark t-statistic is 4.01 when r=1, larger than 3.78 obtained when ignoring unpublished works.

“Unsurprisingly, the Bonferroni implied benchmark t-statistic (4.01) is larger than 3.78, which is what we obtain when ignoring unpublished works.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The fitted density closely tracks the histogram and has a population mean of 2.07.

“The fitted density closely tracks the histogram and has a population mean of 2.07.”

May combine two claims.

The BHY implied t-statistic increases from 3.39 to 3.68 at 1% significance and from 2.78 to 3.18 at 5% significance.

“The BHY implied t-statistic increases from 3.39 to 3.68 at 1% significance and from 2.78 to 3.18 at 5% significance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The population mean estimate is 2.22 in panel [B].

“The population mean estimate is 2.22 in panel”

• AssumptionThe authors state this as an assumption: Our sample covers only half of all factors with t-statistics between 1.96 and 2.57.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

We only count factors that have t-statistics or equivalent statistics reported; roughly twenty new factors fail to satisfy this requirement.

“To be clear, we only count those that have t-statistics or equivalent statistics reported.Roughly twenty new factors fail to satisfy this requirement.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some factors are highly correlated; we include four versions of idiosyncratic volatility.

“As already mentioned, some of these factors are highly correlated.For example, we include four versions of idiosyncratic volatility, that is,Fama and MacBeth (1973),Ali, Hwang, and Trombley (2003),Ang et al. (2006), andFu (2009).”

• cites Fama and MacBeth (1973) sourceNot checked — no full text of the source
• cites Ali, Hwang, and Trombley (2003) sourceNot checked — no full text of the source
• cites Ang et al. (2006)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

There are a few very large t-statistics in our sample; we fit the truncated exponential model without dropping any large t-statistics.

“There are a few very large t-statistics in our sample.We fit the truncated exponential model without dropping any large t-statistics.”

May combine two claims.

When r doubles, the estimate of M approximately doubles and Bonferroni and Holm implied t-statistics increase by about 0.2, whereas BHY implied t-statistics increase by around 0.03.

“When r doubles, the estimate of M also approximately doubles and the Bonferroni and Holm implied t-statistics increase by about 0.2, whereas the BHY implied t-statistics increase by around 0.03 (under both significance levels).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Our previous estimate of 820 is a one-shot estimate based on the truncated sample; results in Table A.1 are based on repeated estimates based on resampled data, and 817 is the median of all these estimates.

“Our previous estimate of 820 is a one-shot estimate based on the truncated sample.The results in Table A.1 are based on repeated estimates based on resampled data: we resample many times, and 817 is the median of all these estimates.”

May combine two claims.

Under the stated assumptions (independence, N(0,1) for false factors, N(2,1) for true factors), simulations find FWER is 100% and FDR is 26%, both exceeding 5%.

“Under these assumptions about the joint distribution of the test statistics, we find via simulations that FWER is 100% and FDR is 26%, both exceeding 5%.”

• AssumptionThe authors state this as an assumption: For the simulation example, the t-statistics for the 700 hypotheses are independent; a false factor's t-statistic follows N(0,1), and a true factor's t-statistic follows N(2,1).

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

There are many forces that make the paper's guidance lenient; a credible case can be made for an even higher threshold for discovery.

“We want to emphasize that there are many forces that make our guidance lenient; that is, a credible case can be made for an even higher threshold for discovery.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Whether suggested by theory or empirical work, a t-statistic of 2.0 is too low.

“Nevertheless, whether suggested by theory or empirical work, a t-statistic of 2.0 is too low.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The multiple testing framework yields immediate guidance on whether a discovered factor is real.

“We pursue the multiple testing framework because it yields immediate guidance on whether a discovered factor is real.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The paper's message applies to many different areas of finance.

“While our focus is on the cross-section of equity returns, our message applies to many different areas of finance.”

Probably not a substantial claim: may not pass S2 (checkable).

Given extensive data mining, the usual criteria for establishing significance should not be used.

“Given this extensive data mining, it does not make sense to use the usual criteria for establishing significance.”

Probably not a substantial claim: may not pass S4 (based).

In finance and economics, it is very difficult to publish replication studies.

“In finance and economics, it is very difficult to publish replication studies.”

Probably not a substantial claim: may not pass S2 (checkable).

A factor derived from a theory should have a lower hurdle than a factor discovered from a purely empirical exercise.

“A factor derived from a theory should have a lower hurdle than a factor discovered from a purely empirical exercise.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the extreme case in which all tests are the same (correlation = 1.0), no adjustment is needed because FWER is the same as the type I error rate for single tests.

“Intuitively, in the extreme case in which all tests are the same (i.e., correlation = 1.0), we do not need to adjust at all: FWER is the same as the type I error rate for single tests.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This evidence on ρ is only suggestive given the weak identification of ρ in the model.

“However, this evidence is only suggestive given the weak identification of ρ in our model.”

Classified as probably not a substantial claim.

Information about the dependence structure, when appropriately incorporated, can help make the multiple-testing method more accurate (less stringent).

“Such information, when appropriately incorporated, can be helpful in making the method more accurate (i.e., less stringent).”

May combine two claims.

The benchmark t-statistic based on the incomplete sample provides a lower bound of the true t-statistic benchmark; if M > R, fewer factors would be expected than when M = R, so future t-statistics need to at least surpass the benchmark to claim significance.

“First, the benchmark t-statistic based on our incomplete sample provides a lower bound of the true t-statistic benchmark.In other words, if M (total number of tests) > R (total number of discoveries), then we would expect fewer factors than when M = R,foot_22 so future t-statistics need to at least surpass our benchmark to claim significance.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

A decrease in the type I error rate often leads to an increase in the type II error rate and vice versa.

“as in single hypothesis testing, a decrease in the type I error rate often leads to an increase in the type II error rate and vice versa.”

Probably not a substantial claim: may not pass S4 (based).

Multiple testing has not attracted much attention in the finance literature.

“Despite the rapid development of multiple testing methods, they have not attracted much attention in the finance literature.”

Probably not a substantial claim: may not pass S3 (load-bearing).

When the number of tests is very large, FWER controlling procedures often lead to a very limited number of discoveries, if any.

“As a result, they often lead to a very limited number of discoveries, if any.”

Probably not a substantial claim: may not pass S4 (based).

Hundreds or even thousands of factors could have been constructed and tested in the finance literature.

“Although there are around 300 published ones, hundreds or even thousands of factors could have been constructed and tested.”

May combine two claims.

The nonsignificant t-statistics typically belong to papers proposing a number of factors and likely represent only a small subsample of nonsignificant t-statistics for all tried factors.

“The nonsignificant ones typically belong to papers that propose a number of factors.These likely represent only a small subsample of nonsignificant t-statistics for all tried factors.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Ferson and Chen (2013) calibrate the correlation among fund returns to be between 0.04 and 0.09.

“while Ferson and Chen (2013) calibrate this number to be between 0.04 and 0.09.”

• cites Ferson and Chen (2013)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

FDR is less stringent than FWER and usually much less so when many tests are performed.

“It is less stringent (i.e., leads to more discoveries) than FWER and usually much less so when many tests are performed.”

May combine two claims.

Holm is uniformly more powerful than Bonferroni: every test rejected under Bonferroni is rejected under Holm, but not vice versa.

“It can also be shown that Holm is uniformly more powerful than Bonferroni in that tests rejected (factors discovered) under Bonferroni are always rejected under Holm, but not vice versa.”

• Also stated in: § Example 4.4.1 · plain

  “In general, Holm's approach leads to more discoveries and all discoveries under Bonferroni are also discoveries under Holm's criteria.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Green, Hand, and Zhang (2013a) find the average correlation among accounting-based factors to be between 0.06 and 0.20.

“Green, Hand, and Zhang (2013a) focus on accountingbased factors and find the average correlation to be between 0.06 and 0.20.”

• cites Green, Hand, and Zhang (2013a)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

BHY is robust to arbitrary dependence among test statistics but does not use any information about the dependence structure.

“Although the BHY method is robust to arbitrary dependence among test statistics, it does not use any information about the dependence structure.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The sample has limitations in making a direct inference on the level of correlation.

“Our sample has limitations in making a direct inference on the level of correlation.”

May combine two claims.

Because correlation is likely weakly identified in this framework and sample periods are not heterogeneous, the results are best interpreted as estimated t-statistic thresholds for a hypothetical level of correlation.

“As a result, our results are best interpreted as the estimated t-statistic thresholds for a hypothetical level of correlation.”

May combine two claims.

The nonmonotonic pattern for threshold t-statistics under FWER results from two opposing forces: increasing p0 and M raise the required threshold, while higher correlation reduces it.

“These two forces work against each other and result in the nonmonotonic pattern for the threshold t-statistics under FWER.”

Probably not a substantial claim: may not pass S3 (load-bearing).

BHY may generate too few discoveries when tests are independent or positively correlated.

“Similarly, BHY may generate too few discoveries when tests are independent or positively correlated.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Multiple testing adjustments will usually generate different results than single tests.

“Multiple testing adjustments will usually generate different results.”

May combine two claims.

The three types of adjustments are robust to heterogeneity in the time of discovery and testing methods; despite varying sample overlap and testing-method differences, the adjustment procedures guarantee that type I errors are controlled under their prespecified levels, so neither nonsimultaneity nor differences in testing methods invalidate the results.

“We want to stress that the three types of adjustments in our paper are robust to the heterogeneity in the time of discovery and testing methods among individual studies.That is, despite the varying degrees of sample overlap and the differences in the testing methods, our adjustment procedures guarantee that the type I errors (however they are defined) are controlled under their prespecified levels”

May combine two claims.

The time series of factor returns or cross-sectional regression slopes are not always observed; often all that is available is the single t-statistic summarizing a factor's significance.

“Unfortunately, we do not always observe the time series of factor returns (when a t-statistic is based on long-short strategy returns) or the time series of slopes in cross-sectional regressions (when a t-statistic is based on the slope coefficients in cross-sectional regressions).Because few researchers post their original data, often all we have is the single t-statistic that summarizes the sign”

Probably not a substantial claim: may not pass S3 (load-bearing).

Holm restricts FWER at α w without any requirement on the dependence structure of p-values.

“Like Bonferroni, Holm also restricts FWER at α w without any requirement on the dependence structure of p-values.”

May combine two claims.

Across various scenarios, the minimum threshold t-statistic is 3.18, corresponding to BHY's adjustment for M > R at 5% significance; alternative cases result in even higher benchmark t-statistics.

“In sum, across various scenarios, we think the minimum threshold t-statistic is 3.18, corresponding to BHY's adjustment for M >R at 5% significance.Alternative cases all result in even higher benchmark t-statistics.”

• AssumptionThe authors state this as an assumption: M factors are tried, but only factors that exceed a certain t-statistic threshold are published; the threshold is set at 1.96.
• AssumptionThe authors state this as an assumption: The sample covers a fraction r of t-statistics between 1.96 and 2.57, and all t-statistics above 2.57 are covered.

Probably not a substantial claim: may not pass S3 (load-bearing).

McLean and Pontiff (2015) estimate the correlation among anomaly returns to be around 0.05.

“McLean and Pontiff (2015) estimate the correlation among anomaly returns to be around 0.05.”

• cites McLean and Pontiff (2015) sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based).

At any fixed significance level α, the law of large numbers forces the false discovery rate to converge to a constant.

“Intuitively, at any fixed significance level α, the law of large numbers forces the false discovery rate (FDR) to converge to a constant.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Any type of multiple-testing adjustment procedure can become too stringent when there is a certain dependence structure in the data, penalizing type I errors too harshly and leading to a high type II error rate.

“In the context of multiple testing, any type of adjustment procedure can become too stringent when there is a certain dependence structure in the data.This is because these procedures are primarily designed to guard against type I errors.Under a certain correlation structure, they may penalize type I errors too harshly and lead to a high type II error rate.”

Classified as probably not a substantial claim.

When we have multiple tests, the exact type II error rate typically depends on a set of unknown parameters and is therefore difficult to assess.

“However, when we have multiple tests, the exact type II error rate typically depends on a set of unknown parameters and is therefore difficult to assess.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

If positive dependence exists among test statistics, all three methods would likely generate fewer significant factors than desired.

“If this type of positive dependence exists among test statistics, all three methods would likely generate fewer significant factors than desired.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Under independence, Bonferroni and Holm approximately achieve the prespecified significance level α when the number of tests is large.

“In theory, under independence, Bonferroni and Holm approximately achieve the prespecified significance level α when the number of tests is large.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Bonferroni operates as a single-step procedure that restricts FWER at levels less than or equal to (M0 times alpha_w)/M, without any assumption on the dependence structure of the p-values.

“Bonferroni operates as a single-step procedure that can be shown to restrict FWER at levels less than or equal to (M 0 ×α w )/M, without any assumption on the dependence structure of the p-values.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Many tests have been tried in the finance literature.

“In the context of our paper, we are sure that many tests have been tried in the finance literature.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Holm's adjustment is less stringent than Bonferroni's, generating more discoveries.

“Since less stringent hurdles are applied to the second to the (k -1)th p-values, more discoveries are generated under Holm's than Bonferroni's adjustment.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Most research in finance that does involve multiple testing focuses on the Bonferroni adjustment, which is known to be too stringent.

“Moreover, most of the research that does involve multiple testing focuses on the Bonferroni adjustment,foot_8 which is known to be too stringent.”

May combine two claims.

Sample quantiles are less susceptible to outliers than means and other moment-related statistics, and simulation studies show that quantiles in the model are more sensitive to parameter changes than other statistics.

“First, sample quantiles are less susceptible to outliers compared to means and other moment-related sample statistics.Our t-statistic sample does have a few influential observations, and we expect quantiles to be more useful descriptive statistics than the mean and the standard deviation.Second, simulation studies show that quantiles in our model are more sensitive to changes in parameters than”

May combine two claims.

After taking testing multiplicity into account, the minimum threshold t-statistic for 5% significance is about 2.8, corresponding to a p-value of 0.5% if a single test.

“In sum, after taking testing multiplicity into account, we believe the minimum threshold t-statistic for 5% significance is about 2.8, which corresponds to a p-value (if a single test) of 0.5%.”

• AssumptionThe authors state this as an assumption: The test results of all tried factors are available (M = R).

Probably not a substantial claim: may not pass S3 (load-bearing).

Since M0 is less than or equal to M, Bonferroni also controls FWER at level alpha_w.

“Since M 0 ≤ M, Bonferroni also controls FWER at level α w .”

Classified as probably not a substantial claim. May combine two claims.

An overparameterized distribution for the continuous distribution in the mixture model may result in imprecise estimates, so we use a simple one-parameter exponential distribution family.

“An overparameterized distribution for the continuous distribution in the mixture model, albeit flexible, may result in imprecise estimates.We therefore use the simple one-parameter exponential distribution family.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The Bonferroni adjusted t-statistic is monotonically increasing in the number of trials.

“The Bonferroni adjusted t-statistic is monotonically increasing in the number of trials, so the t-statistic benchmark”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The normality assumption for the μ_i's requires the reported X_i's to randomly represent either long/short or short/long strategy returns.

“The normality assumption for the μ i 's requires the reported X i 's to randomly represent either long/short or short/long strategy returns.”

• AssumptionThe authors state this as an assumption: The first-step population parameters μ_i's are independently generated from a normal distribution.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

In the quantitative asset management industry, significant results are not published-they are considered trade secrets.

“In the quantitative asset management industry, significant results are not published-they are considered”

Probably not a substantial claim: may not pass S1 (asserted), S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

We attempt to navigate the zoo and establish new benchmarks to guide empirical asset pricing tests.

“We attempt to navigate the zoo and establish new benchmarks to guide empirical asset pricing tests.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Average returns of different strategies can be correlated even when population means are known.

“We can easily think of scenarios in which average returns of different strategies are correlated, even when population means are known.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Simulation studies in Scott and Berger (2006) show that discovery probabilities for a few initial signals increase when more noise is added to the original sample.

“Simulation studies in Scott and Berger (2006) show how the discovery probabilities for a few initial signals increase when more noise is added to the original sample.”

• cites Scott and Berger (2006) sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

In our context, independence is likely to be violated.

“Independence, as we previously discussed, is likely to be violated in our context.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In astronomy and physics, even higher threshold t-statistics are often used to control for testing multiplicity; e.g., the Higgs Boson discovery has a t-statistic of more than 5.

“In astronomy and physics, even higher threshold t-statistics are often used to control for testing multiplicity.For instance, the high profile discovery of Higgs Boson has a t-statistic of more than 5 (p-value less than 0.0001%).”

• cites ATLAS Collaboration (2012)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites CMS Collaboration (2012)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Harvey and Liu (2014b)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

For the academic literature, factors with borderline t-statistics are difficult to get published.

“For the academic literature, factors with”

May combine two claims.

The baseline model is the best among all three models as it overcomes the missing data problem for the original sample and avoids guessing the fraction of missing observations in the 1.96-2.57 range.

“We think the baseline model is the best among all three models as it not only overcomes the missing data problem for the original sample, but also avoids guessing the fraction of missing observations in the 1.96-2.57range.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Truncated distributions have been used to study hidden tests (publication bias) in medical research.

“Truncated distributions have been used to study hidden tests (i.e., publication bias) in medical research.”

• cites Begg and Berlin (1988) sourceNot checked — no full text of the source
• cites Thornton and Lee (2000)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

A certain form of conditional independence is unavoidable for Bayesian hierarchical modeling.

“A certain form of conditional independence is unavoidable for Bayesian hierarchical modeling 57”

• cites [57] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The number of true nulls M0 is unknown, so we usually cannot make Bonferroni more powerful by increasing αw to α = Mαw/M0.

“The number of true nulls M 0 is unknown, so we usually cannot make Bonferroni more powerful by increasing α w to α = Mα w /M 0 (note that M 0 α/M = α w ).”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Extreme values are pivotal statistics for heavy-tailed distributions and are key for model estimation.

“In addition, extreme values are pivotal statistics for heavy-tailed distributions and are key for model estimation.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The multiple testing framework is robust to outliers.

“The multiple testing framework is robust to outliers.The procedures are based on either the total number of tests (Bonferroni) or the order statistics of t-statistics(Holm and BHY).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The normality assumptions in both H1 and H2 are necessary to guarantee the properness of the posterior distributions.

“The normality assumptions in both H1 and H2 are important as they are necessary to guarantee the properness of the posterior distributions.”

Probably not a substantial claim: may not pass S2 (checkable).

The optimal amount of data mining is not zero since some data mining produces knowledge.

“Importantly, the optimal amount of data mining is not zero since some data mining produces knowledge.”

May combine two claims.

The t-statistic sample may underrepresent all factors with t-statistics exceeding 2.57, so estimates of total trials are biased too low, affecting benchmark calculation.

“First, our t-statistic sample may underrepresent all factors with t-statistics exceeding 2.57.foot_33Hence, our estimates of the total trials are biased (too low), which affects our calculation of the benchmarks.”

May combine two claims.

Intuitively, t-statistics based on similar sample periods are more correlated than those based on distinct sample periods; therefore degree of overlap in sample period helps identify the correlation coefficient.

“Intuitively, t-statistics that are based on similar sample periods are more correlated than t-statistics that are based on distinct sample periods.Therefore, the degree of overlap in sample period helps identify the correlation coefficient.”

Probably not a substantial claim: may not pass S4 (based).

In medical research, insignificant results are still viewed as an indispensable part of overall statistical evidence and are given much more prominence than in financial economics research.

“However, in medical research, insignificant results are still viewed as an indispensable part of the overall statistical evidence and are given much more prominence than in the financial economics research.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Our research quantifies the warnings of both Fama (1991) and Schwert (2003).

“Our research quantifies the warnings of both Fama (1991) and Schwert (2003).”

• cites Fama (1991) sourceNot checked — no full text of the source
• cites Schwert (2003)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

The truncated distributional framework suits our study better because 1.96 is the obvious hurdle that research needs to overcome to be published.

“In this sense, the truncated distributional framework suits our study better-1.96is the obvious hurdle that research needs to overcome to be published.”

Classified as probably not a substantial claim. May combine two claims.

We do not pursue likelihood-based estimation because the likelihood function involves high-dimensional integrals and is difficult to optimize, leading to a GMM-based approach.

“We do not pursue a likelihood-based estimation.Given that we have more than a thousand factors and each of them is associated with an indicator variable that is missing, the likelihood function involves high-dimensional integrals and is difficult to optimize.This leads us to a GMM-based approach.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

More sophisticated generalizations modify the basic model but are unlikely to change the fundamental hierarchical testing structure.

“More sophisticated generalizations modify the basic model but are unlikely to change the fundamental hierarchical testing structure.”

• cites [56] Benjamini, 1995, Controlling the false discovery rate: A practical and powerful approach to multiple testing, Journal of the Royal Statistical Society Series B sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The common variance assumption may look restrictive, but factor returns can always be scaled by changing the dollar investment in the long-short strategy.

“The common variance assumption may look restrictive, but we can always scale factor returns by changing the dollar investment in the long-short strategy.”

Probably not a substantial claim: may not pass S4 (based).

The exponential distribution is better at modeling extreme observations than the usual normal density.

“In contrast to the usual normal density, exponential distribution is better at modeling extreme observations.”