The file drawer problem and tolerance for null results.

Rosenthal's fail-safe N indicated that 2,153,789 null effects would be needed to reduce the summary effect's significance to .05.

“We then calculated Rosenthal’s fail-safe N (Rosenthal, 1979), and determined that 2,153,789 effects evidencing no relationship between SA and psychopathology (g = 0) would be needed to reduce the significance level of the summary effect size to .05.”

The file drawer effect is a long-standing problem in academic literature.

“The file drawer effect is a perennial problem in the academic literature (Rosenthal, 1979)”

The fail-safe N was 4,274, the theoretical number of unpublished studies with effect sizes averaging zero that would be needed to render negligible the omnibus results.

“First, we calculated the fail-safe N [177] to be 4,274, which is the theoretical number of unpublished studies with effect sizes averaging zero (no effect) that would be needed to render negligible the omnibus results.”

Publication bias was assessed with funnel plots and Fail-safe N.

“As we did not include unpublished work in our meta-analyses we tested for publication bias in the following ways.Firstly, funnel plots were examined.demonstrating no systematic publication biases (tables available from first author on request).Secondly..Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”

Fail-safe N values for the primary aims ranged from k = 2048 to k = 6708, suggesting no publication bias.

“Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”

Publication bias was assessed with funnel plots, Rosenthal's fail-safe N, and Egger's regression intercept.

“Presence of publication bias was assessed with funnel plots, 22 Rosenthal's fail-safe N , 23 and Egger's regression intercept. 24”

There was some indication of publication bias whereby some associations between racism and health outcomes were stronger in published versus unpublished studies.

“In terms of study quality, there was some indication of publication bias whereby some associations between racism and health outcomes were stronger in published versus unpublished studies, consistent with the tendency for ‘null’ studies to remain unpublished (i.e., the ‘file drawer’ problem [418]).”

A significant amount of unpublished negative or inconclusive results (publication bias/file-drawer effect) may explain the excess of positive findings.

“An alternative, perhaps more plausible, explanation is that there could be a significant amount of unpublished negative or inconclusive results, a phenomenon referred to as the “file-drawer effect” or publication bias (23).”

If either of the two last hypotheses is true, there should exist many unpublished studies with null results in laboratories' drawers.

“If either of the two last hypotheses is true, there should exist many unpublished studies with null results lying in laboratories' drawers (32).”

REM sleep can occur rapidly in some individuals, so the first REM episode is not always about 70 minutes after falling asleep.

“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”

Publication bias was evaluated using Rosenthal's file drawer analysis, Egger's linear regression, and ANOVA comparing published versus unpublished studies.

“Publication bias was evaluated using Rosenthal's file drawer analysis (Rosenthal, 1979) and Egger's linear regression (Card, 2012), and applying ANOVA to compare the mean effect size of the published versus unpublished studies.”

Rosenthal suggested a fail-safe N value above 5k + 10 reflects results that are tolerant to contradicting studies.

“Rosenthal (1979) suggested a fail-safe N value above 5; k + 10 reflects results that are tolerant to contradicting studies”

Random-effects models were applied to fit the relations between collective effervescence and the criterion variables, following the guidelines proposed by Rosenthal (1979), Hunter and Schmidt (2004) and Cumming (2013).

“We applied random-effects models to fit the relations between CE and the criterion variables, following the guidelines proposed by Rosenthal (1979), Hunter and Schmidt (2004) and Cumming (2013).”

The classic fail-safe N was 3001, exceeding 5(36)+10, indicating no publication bias.

“The fail-safe value for this meta-analysis was computed to be 3001, which is more than 5(36) + 10.”

Our results may look like a lot of studies that ended up in the so-called "file drawer."

“Moreover, we suggest our results may look like a lot of studies that ended up in the so-called “file drawer” [57].”

Publication bias was addressed by computing the fail-safe N for the correlation coefficients.

“Finally, publication bias was addressed by computing the fail-safe N for the correlation coefficients (Hunter & Schmidt, 2004;Rosenthal, 1979), which reports the number of field effect sizes or the tolerance for future null results, to check whether a publication bias affected our results.”

Using Rosenthal's (1979) fail-safe N criterion, the fail-safe N for all outcome groups were at least eight times larger than the criterion value, suggesting that the effect sizes are unlikely to be an artifact of bias.

“Using Rosenthal’s (1979) fail-safe N criterion that the value should be over five times greater than the number of studies included in the meta-analysis, the fail-safe N for all outcome groups were at least eight times larger than the criterion value, suggesting that the effect sizes are unlikely to be an artifact of bias.”

A possible interpretation is that missing effect sizes are mainly a file-drawer problem, consistent with evidence that the file-drawer problem is the main culprit behind publication bias in the social sciences.

“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”

A Fail-safe N exceeding the number of studies by fivefold would provide strong support for the robustness of significant effects.

“Moreover, Rosenthal (1979) opined that a Fail-safe N exceeding the number of studies by fivefold would provide strong support for the robustness of the significant effects.”

The trim-and-fill method tends to adjust too little when bias is strong, and Fail-Safe N neither tests nor adjusts for the presence of bias.

“The trim-and-fill method (Duval & Tweedie, 2000) tends to adjust too little when bias is strong (Carter et al., 2019), and Fail-Safe N (Rosenthal, 1979) neither tests nor adjusts for the presence of bias.”

Not including unpublished or non-significant studies may lead to an overestimation of the true effect size, a phenomenon known as publication bias.

“Not including these studies may lead to an overestimation of the true effect size, a phenomenon known as ' publication bias ' (Rosenthal 1979).”

A common problem is that results may not have been reported in dissertations or research reports, or published in scientific journals, due to non-significant or unfavorable findings, and therefore are difficult to locate.

“However, a common problem is that results may not have been reported in dissertations or research reports, or published in scienti fi c (peer-reviewed) journals, due to non-signi fi cant or unfavorable fi ndings and, therefore, are dif fi cult to locate.”

Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results.

“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 12­23 ).”

Publication bias is insignificant and the study's results are reliable, based on a fail-safe N of 1904 exceeding the threshold of 100.

“The traditional fail-safe value is 1904, which is higher than the threshold value of 100, i.e., [5(18) + 10], indicating that publication bias is insignificant, and the study's results are reliable.”

The Q statistic was used to assess heterogeneity, and publication bias was estimated using Rosenthal's fail-safe number with α = 0.05.

“The Q statistic was used to assess heterogeneity30, and publication bias was estimated using Rosenthal’s fail-safe number with α = 0.0531.”

Rosenthal's fail-safe number was calculated, estimating the number of unpublished studies with non-significant results needed to bring an observed result to non-significance.

“Rosenthal's fail-safe number (NR) was calculated, estimating the number of unpublished studies with non-significant results needed to bring an observed result to non-significance (NR ⩾ 5k + 10 to rule out a file drawer problem) (Rosenthal, 1979).”

The study planned to examine the robustness of statistically significant effect sizes by calculating the Failsafe N.

“In case of statistically significant effect sizes, it was planned to examine the robustness of findings by calculating the Failsafe N (number of non‐significant studies that would bring the p‐value to non‐significant (p > 0.05)) 40.”

With negative results buried in file drawers, conclusions drawn from the published record can be misleading, and 'false facts' can become canonized in the literature.

“With negative results buried in file drawers (55), conclusions drawn from the published record can be misleading, and 'false facts' can become canonized in the literature (56).”

The fail-safe N was used to determine how many additional studies would be needed to make a significant effect size non-significant.

“the fail-safe N (i.e. the file drawer statistic) was used to determine how many additional studies would be necessary to turn a significant ES non-significant (32).”

File drawer analysis revealed that 1919 unpublished, filed, or unretrieved studies averaging null results would be required to bring the average unweighted effect size to nonsignificance.

“File drawer analysis (Rosenthal, 1979) revealed that 1919 unpublished, filed, or unretrieved studies averaging null results would be required to bring the average unweighted effect size to nonsignificance.”

Rosenthal's fail-safe N analysis reveals 186 unpublished null papers are needed to invalidate the population effect size of .235.

“Rosenthal's (1979) fail-safe N analysis reveals 186 unpublished papers (i.e., null effect) are needed to be included in the meta-analysis in order to invalidate the population effect size of small to medium (i.e., .235)found in this review.”

Not including unpublished studies may lead to an overestimation of the true effect size, the so-called publication bias.

“Not including these studies may lead to an overestimation of the true effect size, the so called ' publication bias ' (Rosenthal, 1979).”

Rosenthal's fail-safe N indicated 115 missing studies would be required to increase the two-tailed p-value above .05.

“Rosenthal's fail-safe N 37 calculated 115 missing studies would be required to increase the two-tailed p-value to above .05.”

Publication bias was investigated with Begg's rank correlation test and Rosenthal's file drawer analysis.

“Publication bias was investigated with Begg's rank correlation test for funnel plot asymmetry (Begg and Mazumdar, 1994) and where significant effects were present Rosenthal's file drawer analysis was also conducted (Rosenthal, 1979).”

This practice artificially increases Type I error and can make relationships appear to exist when they do not.

“Over time, this practice artificially inflates Type I error and may make it appear that relationships exist when in reality they do not (Rosenthal et al., 1979).”

A possible problem for meta-analysis is the file drawer problem, in which studies with nonsignificant results remain unpublished.

“However, a potential problem with meta-analysis is the file drawer problem, where studies are “filed away” unpublished if they fail to find statistically significant relationships.”

Meta-analyses are necessarily based on the available literature and hence may be affected by the potential publication-bias disfavoring null results.

“Meta-analyses are necessarily based on the available literature and hence may be affected by the potential publication-bias disfavoring null results (Rosenthal 1979).”

Journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results.

“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”