The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
The file drawer problem and tolerance for null results.
Abstract
For any given research area, one cannot tell how many studies have been con-ducted but never reported. The extreme view of the "file drawer problem " is that journals are filled with the 5 % of the studies that show Type I errors, while the file drawers are filled with the 95 % of the studies that show non-significant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed. Both behavioral researchers and statisti-cians have long suspected that the studies published in the behavioral sciences are a biased sample of the studies that are actually carried out (Bakan, 1967; McNemar, 1960; Smart, 1964; Sterling, 1959). The extreme view of this problem, the "file drawer prob-lem, " is that the journals are filled with the 5 % of the studies that show Type I errors, while the file drawers back at the lab are filled with the 95 % of the studies that show nonsignificant (e.g., p>.05) results. In the past there was very little one could do to assess the net effect of studies, tucked away in file drawers, that did not make the magic.05 level (Rosenthal & Gaito, 1963, 1964). Now, however, although no definitive solution to the problem is available, one can establish reasonable boundaries on the prob-lem and estimate the degree of damage to any research conclusion that could be done by the file drawer problem. This advance in our ability to cope with the file drawer is an outgrowth of the in-creasing interest of behavioral scientists in summarizing bodies of research literature sys-Preparation of this article was supported in part by the National Science Foundation. I would like to thank Judith A. Hall and Donald B. Rubin for their valuable improvements of an earlier version of this article. Requests for reprints should be sent to Robert
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We then calculated Rosenthal’s fail-safe N (Rosenthal, 1979), and determined that 2,153,789 effects evidencing no relationship between SA and psychopathology (g = 0) would be needed to reduce the significance level of the summary effect size to .05.”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“The file drawer effect is a perennial problem in the academic literature (Rosenthal, 1979)”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“First, we calculated the fail-safe N [177] to be 4,274, which is the theoretical number of unpublished studies with effect sizes averaging zero (no effect) that would be needed to render negligible the omnibus results.”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [177]
In the citing paper
“As we did not include unpublished work in our meta-analyses we tested for publication bias in the following ways.Firstly, funnel plots were examined.demonstrating no systematic publication biases (tables available from first author on request).Secondly..Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Presence of publication bias was assessed with funnel plots, 22 Rosenthal's fail-safe N , 23 and Egger's regression intercept. 24”
From Light Therapy for Patients With Bipolar Depression: Systematic Review and Meta-Analysis of Randomized Controlled Trials (2020) · cites this paper as [23]
In the citing paper
“In terms of study quality, there was some indication of publication bias whereby some associations between racism and health outcomes were stronger in published versus unpublished studies, consistent with the tendency for ‘null’ studies to remain unpublished (i.e., the ‘file drawer’ problem [418]).”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as [418]
In the citing paper
“An alternative, perhaps more plausible, explanation is that there could be a significant amount of unpublished negative or inconclusive results, a phenomenon referred to as the “file-drawer effect” or publication bias (23).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [23]
In the citing paper
“If either of the two last hypotheses is true, there should exist many unpublished studies with null results lying in laboratories' drawers (32).”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Publication bias was evaluated using Rosenthal's file drawer analysis (Rosenthal, 1979) and Egger's linear regression (Card, 2012), and applying ANOVA to compare the mean effect size of the published versus unpublished studies.”
From Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal (1979) suggested a fail-safe N value above 5; k + 10 reflects results that are tolerant to contradicting studies”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“We applied random-effects models to fit the relations between CE and the criterion variables, following the guidelines proposed by Rosenthal (1979), Hunter and Schmidt (2004) and Cumming (2013).”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal (1979)
In the citing paper
“The fail-safe value for this meta-analysis was computed to be 3001, which is more than 5(36) + 10.”
From Effects of digital game-based STEM education on students’ learning achievement: a meta-analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, we suggest our results may look like a lot of studies that ended up in the so-called “file drawer” [57].”
From Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory (2023) · cites this paper as [57]
In the citing paper
“Finally, publication bias was addressed by computing the fail-safe N for the correlation coefficients (Hunter & Schmidt, 2004;Rosenthal, 1979), which reports the number of field effect sizes or the tolerance for future null results, to check whether a publication bias affected our results.”
From Drivers of Consumer Adoption of e-Commerce: a Meta-Analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Using Rosenthal’s (1979) fail-safe N criterion that the value should be over five times greater than the number of studies included in the meta-analysis, the fail-safe N for all outcome groups were at least eight times larger than the criterion value, suggesting that the effect sizes are unlikely to be an artifact of bias.”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as Rosenthal (1979)
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, Rosenthal (1979) opined that a Fail-safe N exceeding the number of studies by fivefold would provide strong support for the robustness of the significant effects.”
From Extending the Theory of Planned Behavior in the Social Commerce Context: A Meta-Analytic SEM (MASEM) Approach (2023) · cites this paper as Rosenthal (1979)
In the citing paper
“The trim-and-fill method (Duval & Tweedie, 2000) tends to adjust too little when bias is strong (Carter et al., 2019), and Fail-Safe N (Rosenthal, 1979) neither tests nor adjusts for the presence of bias.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Rosenthal, 1979
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, a phenomenon known as ' publication bias ' (Rosenthal 1979).”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“However, a common problem is that results may not have been reported in dissertations or research reports, or published in scienti fi c (peer-reviewed) journals, due to non-signi fi cant or unfavorable fi ndings and, therefore, are dif fi cult to locate.”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [21]
In the citing paper
“The traditional fail-safe value is 1904, which is higher than the threshold value of 100, i.e., [5(18) + 10], indicating that publication bias is insignificant, and the study's results are reliable.”
From A meta-analysis to explore the role of design thinking in enhancing creativity as learning outcomes in STEM education (2026) · cites this paper as Rosenthal, 1979
In the citing paper
“The Q statistic was used to assess heterogeneity30, and publication bias was estimated using Rosenthal’s fail-safe number with α = 0.0531.”
From The limited prosocial effects of meditation: A systematic review and meta-analysis (2018) · cites this paper as [31]
In the citing paper
“Rosenthal's fail-safe number (NR) was calculated, estimating the number of unpublished studies with non-significant results needed to bring an observed result to non-significance (NR ⩾ 5k + 10 to rule out a file drawer problem) (Rosenthal, 1979).”
From Efficacy of non-invasive brain stimulation on cognitive functioning in brain disorders: a meta-analysis (2020) · cites this paper as Rosenthal, 1979
In the citing paper
“In case of statistically significant effect sizes, it was planned to examine the robustness of findings by calculating the Failsafe N (number of non‐significant studies that would bring the p‐value to non‐significant (p > 0.05)) 40.”
From Cognitive behavioral therapies for informal caregivers of patients with cancer and cancer survivors: a systematic review and meta-analysis (2017) · cites this paper as [40]
In the citing paper
“With negative results buried in file drawers (55), conclusions drawn from the published record can be misleading, and 'false facts' can become canonized in the literature (56).”
From Misinformation in and about science (2021) · cites this paper as [55]
In the citing paper
“the fail-safe N (i.e. the file drawer statistic) was used to determine how many additional studies would be necessary to turn a significant ES non-significant (32).”
From Peripheral cytokine and chemokine alterations in depression: a meta‐analysis of 82 studies (2017) · cites this paper as [32]
In the citing paper
“File drawer analysis (Rosenthal, 1979) revealed that 1919 unpublished, filed, or unretrieved studies averaging null results would be required to bring the average unweighted effect size to nonsignificance.”
From Happy to help? A systematic review and meta-analysis of the effects of performing acts of kindness on the well-being of the actor (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's (1979) fail-safe N analysis reveals 186 unpublished papers (i.e., null effect) are needed to be included in the meta-analysis in order to invalidate the population effect size of small to medium (i.e., .235)found in this review.”
From A meta-analytic review of the efficacy of physical exercise interventions on cognition in individuals with autism spectrum disorder and ADHD (2016) · cites this paper as Rosenthal's (1979)
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, the so called ' publication bias ' (Rosenthal, 1979).”
From The effects of physical activity interventions on psychosocial outcomes in adolescents: a meta-analytic review (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's fail-safe N 37 calculated 115 missing studies would be required to increase the two-tailed p-value to above .05.”
From Anxiety in Informal Dementia Carers: A Meta-Analysis of Prevalence (2020) · cites this paper as [37]
In the citing paper
“Publication bias was investigated with Begg's rank correlation test for funnel plot asymmetry (Begg and Mazumdar, 1994) and where significant effects were present Rosenthal's file drawer analysis was also conducted (Rosenthal, 1979).”
From Meta-analysis of the effects of intranasal oxytocin on interpretation and expression of emotions (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“Over time, this practice artificially inflates Type I error and may make it appear that relationships exist when in reality they do not (Rosenthal et al., 1979).”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“However, a potential problem with meta-analysis is the file drawer problem, where studies are “filed away” unpublished if they fail to find statistically significant relationships.”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“Meta-analyses are necessarily based on the available literature and hence may be affected by the potential publication-bias disfavoring null results (Rosenthal 1979).”
From Parsing the neural correlates of moral cognition: ALE meta-analysis on morality, theory of mind, and empathy (2012) · cites this paper as Rosenthal 1979
In the citing paper
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
From The Extent and Consequences of P-Hacking in Science (2015) · cites this paper as [7]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G. E., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,518
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718 · cited 1×
“Perhaps the simplest, most useful way of computing the overall p of a, set of research studies is the method of adding Zs (Cochran, 1954; Mosteller & Bush, 1954; Rosenthal, 1978).”
MethodMeta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154
Experimenter Effects in Behavioral Research
McGuigan, F. J., & Rosenthal, R. (1968) · The American Journal of Psychology · doi:10.2307/1420816 · Cited by 1,633
Self-citationArtifact in behavioral research
Rosenthal, R., & Rosnow, R. L. (1969) · Medical Entomology and Zoology · Cited by 1,186
Self-citationSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964
Combining results of independent studies.
Rosenthal, R. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.1.185 · Cited by 910
Self-citationGender effects in decoding nonverbal cues.
Hall, J. A. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.4.845 · Cited by 766
Wortman, C. B., Rosenthal, R., & Rosnow, R. L. (1976) · The American Journal of Psychology · doi:10.2307/1421418 · Cited by 428
Self-citation
Cited by
Sorted by citations
Items 1,201–1,300 of 1,374
Grist, R., & Cavanagh, K. (2013) · Journal of Contemporary Psychotherapy · doi:10.1007/s10879-013-9243-y · Cited by 100
Sirois, F. M., Molnar, D. S., & Hirsch, J. K. (2017) · European Journal of Personality · doi:10.1002/per.2098 · Cited by 108
“05) that would need to be included in the meta-analysis to threaten the conclusion of a significant association (Rosenthal, 1979).”
BackgroundMethodResultInfluentialZimmermann, T., Heinrichs, N., & Baucom, D. H. (2007) · Annals of Behavioral Medicine · doi:10.1007/bf02874548 · Cited by 107
Hopwood, T. L., & Schutte, N. S. (2017) · Psychology of Violence · doi:10.1037/vio0000056 · Cited by 106
“To check for publication bias, Duval and Tweedie’s (2000) trim and fill procedure with funnel plot and a classic fail-safe N test (Rosenthal, 1979) were conducted using CMA.”
BackgroundMethodImpact of invasions by alien plants on soil seed bank communities: Emerging patterns
Gioria, M., Jarošík, V., & Pyšek, P. (2014) · Perspectives in Plant Ecology, Evolution and Systematics · doi:10.1016/j.ppees.2014.03.003 · Cited by 105
Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., Corker, K. S., Corley, M., Hartshorne, J. K., IJzerman, H., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920958687 · Cited by 105
Taylor, J. J., Rytwinski, T., Bennett, J. R., Smokorowski, K. E., Lapointe, N. W. R., Janusz, R., Clarke, K., Tonn, B., Walsh, J. C., & Cooke, S. J. (2019) · Environmental Evidence · doi:10.1186/s13750-019-0162-6 · Cited by 105
“A fail-safe number estimates the number of non-significant unpublished studies required to eliminate a significant (weighted) overall effect size [49, 50].”
BackgroundFrench, K., Golijani-Moghaddam, N., & Schröder, T. (2017) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2017.08.002 · Cited by 100
Modelling the effects of subjective and objective decision making in scientific peer review
Park, I.-U., Peacey, M. W., & Munafò, M. R. (2014) · Nature · doi:10.1038/nature12786 · Cited by 103
Barrera, T. L., Mott, J. M., Hofstein, R. F., & Teng, E. J. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.09.005 · Cited by 103
Euser, A. S., Arends, L. R., Evans, B. E., Greaves-Lord, K., Huizink, A. C., & Franken, I. H. A. (2012) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2011.09.002 · Cited by 100
Hartley, M., Dorstyn, D., & Due, C. (2019) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-019-04145-3 · Cited by 99
“A recommended minimum Nfs was additionally computed by the formula 5 k + 10; where k is the number of studies included in the meta-analysis (Ellis 2010; Rosenthal 1979).”
MethodDurán, J. I., & Fernández-Dols, J.-M. (2021) · Emotion · doi:10.1037/emo0001015 · Cited by 101
Anxiety in Informal Dementia Carers: A Meta-Analysis of Prevalence
Kaddour, L., & Kishita, N. (2020) · Journal of Geriatric Psychiatry and Neurology · doi:10.1177/0891988719868313 · Cited by 98
Collins, R. N., & Kishita, N. (2019) · The Gerontologist · doi:10.1093/geront/gny024 · Cited by 101
Taylor, H., Strauss, C., & Cavanagh, K. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2021.102078 · Cited by 102
Boggero, I. A., Hostinar, C. E., Haak, E. A., Murphy, M. L. M., & Segerstrom, S. C. (2017) · Biological Psychology · doi:10.1016/j.biopsycho.2017.08.058 · Cited by 97
Conn, V. S., Koopman, R. J., Ruppar, T. M., Phillips, L. J., Mehr, D. R., & Hafdahl, A. R. (2014) · Journal of Primary Care & Community Health · doi:10.1177/2150131913520328 · Cited by 99
Augedal, A. W., Hansen, K. S., Kronhaug, C. R., Harvey, A. G., & Pallesen, S. (2013) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2012.06.001 · Cited by 98
Czosnek, L., Lederman, O., Cormie, P., Zopf, E., Stubbs, B., & Rosenbaum, S. (2019) · Mental Health and Physical Activity · doi:10.1016/j.mhpa.2018.11.001 · Cited by 94
Meditation interventions efficiently reduce cortisol levels of at-risk samples: A meta-analysis
Koncz, A., Demetrovics, Z., & Takacs, Z. K. (2021) · Health Psychology Review · doi:10.1080/17437199.2020.1760727 · Cited by 94
“Rosenthal’s fail-safe N method (Rosenthal, 1979) estimates the confidence of the results by calculating the number of studies with non-significant results that would be needed to turn the average effect non-significant.”
MethodThe effect of trauma on the severity of obsessive-compulsive spectrum symptoms: a meta-analysis
Miller, M. L., & Brock, R. L. (2017) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2017.02.005 · Cited by 93
Hallford, D. J., & Sharma, M. K. (2019) · British Journal of Clinical Psychology · doi:10.1111/bjc.12218 · Cited by 84
Work addiction and personality: A meta-analytic study
Kun, B., Takacs, Z. K., Richman, M. J., Griffiths, M. D., & Demetrovics, Z. (2021) · Journal of Behavioral Addictions · doi:10.1556/2006.2020.00097 · Cited by 88
“Additionally, Rosenthal’s fail-safe number (Rosenthal, 1979) was 952 Journal of Behavioral Addictions 9 (2020) 4, 945–966 Unauthenticated | Downloaded 01/15/22 07:11 AM UTC Table 2.”
Chi, T., Cheng, L., & Zhang, Z. (2023) · Brain and Behavior · doi:10.1002/brb3.2909 · Cited by 92
Does reactivation trigger episodic memory change? a meta-analysis
Scully, I. D., Napper, L. E., & Hupbach, A. (2017) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2016.12.012 · Cited by 90
“This bias may inflate effect size estimates in favor of significance (Rosenthal, 1979).”
BackgroundCollins, H., Booth, J. N., Duncan, A., Fawkner, S., & Niven, A. (2019) · Sports Medicine - Open · doi:10.1186/s40798-019-0205-0 · Cited by 90
“Publication bias was assessed by calculating Egger bias statistics [28] and Rosenthal’s Fail-safe N [29].”
BackgroundMethodThe registration continuum in clinical science: A guide toward transparent practices
Benning, S. D., Bachrach, R. L., Smith, E. A., Freeman, A. J., & Wright, A. G. C. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000451 · Cited by 77
“In a reversal of longstanding problems in the field (Ferguson & Heene, 2012; Rosenthal, 1979), fixing such problems prior to a study makes null results more publishable, as they are less likely to result from an uninterpretable failure to detect meaningful effects.”
BackgroundStatus of psychological trials in breast cancer patients: a report of three meta-analyses
Naaman, S. C., Radwan, K., Fergusson, D., & Johnson, S. (2009) · Psychiatry: Interpersonal and Biological Processes · doi:10.1521/psyc.2009.72.1.50 · Cited by 87
Zachariae, R., & O'Toole, M. S. (2015) · Psycho-Oncology · doi:10.1002/pon.3802 · Cited by 88
O'Toole, M. S., Zachariae, R., Renna, M. E., Mennin, D. S., & Applebaum, A. (2017) · Psycho-Oncology · doi:10.1002/pon.4144 · Cited by 86
“The failsafe number refers to the number of unidentified or unpublished studies with null findings that will reduce the pooled result to statistical non-significance [40].”
BackgroundJin, H. K., Lee, H. E., & Kim, E. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05944-8 · Cited by 85
“5n + 10 [51], where n represents the number of studies included in the meta-analysis.”
BackgroundUlrichsen, K. M., Kaufmann, T., Dørum, E. S., Kolskår, K. K., Richard, G., Alnæs, D., Arneberg, T. J., Westlye, L. T., & Nordvik, J. E. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.00912 · Cited by 86
“…note, analyses based on pooled results from published studies will always be vulnerable to systematic biases like the file-drawer publication bias, suggesting that positive results are published in a greater extent than their non-significant or negative counterparts (Rosenthal, 1979; Felson, 1992).”
BackgroundKarg, K., Burmeister, M., Shedden, K., & Sen, S. (2011) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2010.189 · Cited by 87
Efficacy of Smoking-Cessation Interventions for Young Adults
Suls, J. M., Luger, T. M., Curry, S. J., Mermelstein, R. J., Sporer, A. K., & An, L. C. (2012) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2012.02.013 · Cited by 87
The extent of dissociation in borderline personality disorder: A meta-analytic review
Scalabrini, A., Cavicchioli, M., Fossati, A., & Maffei, C. (2017) · Journal of Trauma & Dissociation · doi:10.1080/15299732.2016.1240738 · Cited by 86
Wilson, D. B. (2009) · Journal of Experimental Criminology · doi:10.1007/s11292-009-9085-5 · Cited by 86
Pizarro, J. J., Zumeta, L. N., Bouchat, P., Włodarczyk, A., Rimé, B., Basabe, N., Amutio, A., & Páez, D. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.974683 · Cited by 88
“Regarding robustness of the effects, we used fail-safe N tests (Rosenthal, 1979) (see Rubio-Aparicio et al., 2018) which represent how many new—or missing—studies with a zeroeffect size would be needed to transform a significant p-value into a non-significant one.”
BackgroundMethodEvaluating efficacy and validating games for health
Kato, P. M. (2012) · Games for Health Journal · doi:10.1089/g4h.2012.1017 · Cited by 79
Down-Regulating Narcissistic Tendencies: Communal Focus Reduces State Narcissism
Giacomin, M., & Jordan, C. H. (2014) · Personality and Social Psychology Bulletin · doi:10.1177/0146167213516635 · Cited by 85
Meta-Analysis of the English Version of the Beck Depression Inventory–Second Edition
Erford, B. T., Johnson, E., & Bardoshi, G. (2016) · Measurement and Evaluation in Counseling and Development · doi:10.1177/0748175615596783 · Cited by 85
Oldham, M., Kellett, S., Miles, E., & Sheeran, P. (2012) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0029630 · Cited by 84
“Moreover, the fail-safe N (Rosenthal, 1979) indicated that 1,326 unpublished studies with zero effect sizes would need to exist in order to invalidate the conclusion that the interventions had no significant effect on attendance behavior (p .05).”
Meta-analytic insights into evolutionary ecology: an introduction and synthesis
Nakagawa, S., & Poulin, R. (2012) · Evolutionary Ecology · doi:10.1007/s10682-012-9593-z · Cited by 77
Guidelines for Meta-Analyses of Counseling Psychology Research
Quintana, S. M., & Minami, T. (2006) · The Counseling Psychologist · doi:10.1177/0011000006286991 · Cited by 83
A meta-analysis of the association between birth weight and attention deficit hyperactivity disorder
Momany, A. M., Kamradt, J. M., & Nikolas, M. A. (2018) · Journal of Abnormal Child Psychology · doi:10.1007/s10802-017-0371-9 · Cited by 82
“Publication Bias The trim-and-fill method (Duval and Tweedie 2000) was used to evaluate the impact of publication bias and to address the Bfile drawer problem^ (Rosenthal 1979).”
MethodYin, X.-Q., de Vries, D. A., Gentile, D. A., & Wang, J.-L. (2019) · Social Science Computer Review · doi:10.1177/0894439318784908 · Cited by 82
“Moreover, as Rosenthal (1979) recommended, if Nfs exceeded the tolerance value of 5k þ 10 (where k is the number of effect sizes), no bias publication existed.”
BackgroundMacDonald, L., Chapman, S., Syrett, M., Bowskill, R., & Horne, R. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.01.002 · Cited by 80
“, 2008), we conducted additional checks of bias by applying a trim and fill method (Duval and Tweedie, 2000), running a classic fail safe analysis to compute the number of studies needed to produce a null result (Rosenthal, 1979) and Egger's test to measure funnel plot asymmetry (Egger et al.”
MethodUsing Animated Videos to Increase Patient Knowledge: A Meta-Analytic Review.
Feeley, T. H., Keller, M., & Kayler, L. (2023) · Health Education & Behavior · doi:10.1177/10901981221116791 · Cited by 85
“The file drawer problem in meta-analysis is not including all studies in a given area of investigation and the consequence can be greater concern for Type I error or finding an illusory significant effect (Rosenthal, 1979).”
BackgroundInzlicht, M., Gervais, W., & Berkman, E. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2659409 · Cited by 82
“attenuating some of the serious problems associated with the file drawer (Rosenthal, 1979).”
BackgroundMaking sense of learning gain in higher education
Evans, C., Kandiko Howson, C., & Forsythe, A. (2018) · Higher Education Pedagogies · doi:10.1080/23752696.2018.1508360 · Cited by 81
“This publication bias, also known as the file-drawer problem (Rosenthal, 1979), requires the calculation of a fail-safe number which estimates the number of unpublished studies in meta-analyses required to bring the meta-analytic mean effect down to a statistically insignificant level.”
BackgroundAnorexia nervosa and bulimia nervosa: A meta-analysis of executive functioning
Hirst, R. B., Beard, C. L., Colby, K. A., Quittner, Z., Mills, B. M., & Lavender, J. M. (2017) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.08.011 · Cited by 81
“4 and 5), and Rosenthal’s Fail-safe N (Rosenthal, 1979) was calculated in order to determine the number of null studies necessary to “nullify” the overall effect (Borenstein, 2005) and therefore render findings non-significant, resulting in 839 studies and 54 studies for anorexia and bulimia nervosa, respectively.”
BackgroundMethodTamminga, H. G. H., Reneman, L., Huizenga, H. M., & Geurts, H. M. (2016) · Psychological Medicine · doi:10.1017/s0033291716000350 · Cited by 82
“Robustness of the significant effects was demonstrated with Rosenthal’s fail-safe n calculation (Rosenthal, 1979), showing a high number of null findings needed to nullify the effects (overall n = 1091, p < 0.”
BackgroundResultInfluentialLeong, L.-Y., Hew, T.-S., Ooi, K.-B., Metri, B., & Dwivedi, Y. K. (2023) · Information Systems Frontiers · doi:10.1007/s10796-022-10337-7 · Cited by 80
“Moreover, Rosenthal (1979) opined that a Fail-safe N exceeding the number of studies by fivefold would provide strong support for the robustness of the significant effects.”
BackgroundNaeem, F., Khoury, B., Munshi, T., Ayub, M., Lecomte, T., Kingdon, D., & Farooq, S. (2016) · International Journal of Cognitive Therapy · doi:10.1521/ijct_2016_09_04 · Cited by 80
“RISK OF BIAS ACROSS STUDIES To assess publication bias, we computed the fail-safe N (Rosenthal, 1979) and constructed a funnel plot.”
MethodChow, J. C., & Ekholm, E. (2018) · Educational Psychology Review · doi:10.1007/s10648-018-9437-7 · Cited by 80
Rhee, E. S., & Jung, W. S. (2019) · Journal of Marketing Communications · doi:10.1080/13527266.2018.1461124 · Cited by 80
“Nonetheless, it is still worth examining the situations in which AAd has a stronger influence on AB because of the file drawer problem (also known as publication bias) (Rosenthal 1979).”
BackgroundCustomer participation in service recovery: a meta-analysis
Van Vaerenbergh, Y., Hazée, S., & Costers, A. (2018) · Marketing Letters · doi:10.1007/s11002-018-9470-9 · Cited by 80
“The average fail-safe N exceeds the recommended 5k + 10 threshold for all relationships (Rosenthal 1979), which indicates the significant relationships are relatively robust against publication bias.”
Telepsychology for Posttraumatic Stress Disorder: A systematic review
Bolton, A., & Dorstyn, D. (2015) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x15571996 · Cited by 74
Richard-Devantoy, S., Ding, Y., Turecki, G., & Jollant, F. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.02.046 · Cited by 79
Contribution of spontaneous improvement to placebo response in depression: A meta-analytic review
Rutherford, B. R., Mori, S., Sneed, J. R., Pimontel, M. A., & Roose, S. P. (2012) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2012.02.008 · Cited by 79
Liu, X., & Zhong, B. (2025) · Educational Research Review · doi:10.1016/j.edurev.2025.100741 · Cited by 78
Reproducing Statistical Results
Stodden, V. (2015) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-010814-020127 · Cited by 77
“Steps taken by the National Science Foundation (NSF) and the National Institutes of Health (NIH) to recognize software and data set contributions in the biographical sketch of a grant applicant encourage greater data and code production and disclosure (NSF 2013, Rockey 2014).”
BackgroundContingency management for treatment attendance: A meta-analysis
Pfund, R. A., Ginley, M. K., Rash, C. J., & Zajac, K. (2022) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2021.108556 · Cited by 75
Estimating Population Mean Power Under Conditions of Heterogeneity and Selection for Significance
Brunner, J., & Schimmack, U. (2020) · Meta-Psychology · doi:10.15626/mp.2018.874 · Cited by 76
“First, often non-significant results are not available because journals selectively publish significant results (Rosenthal, 1979; Sterling, 1959; Sterling et al., 1995).”
BackgroundResultTotura, C. M. W., Fields, S. A., & Karver, M. S. (2018) · Psychiatric Services · doi:10.1176/appi.ps.201700114 · Cited by 76
The role of technology-facilitated peer assessment and supporting strategies: a meta-analysis
Zheng, L., Zhang, X., & Cui, P. (2020) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2019.1644603 · Cited by 75
“The present study adopted the classic fail-safe N proposed by Rosenthal (1979) and Orwin ’ s fail-safe N (Orwin 1983) to evaluate publication bias.”
MethodPsychological treatments in adult ADHD: a systematic review
Fullen, T., Jones, S. L., Emerson, L. M., & Adamou, M. (2020) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-020-09794-8 · Cited by 73
A meta-analysis of dropout rates in acceptance and commitment therapy
Ong, C. W., Lee, E. B., & Twohig, M. P. (2018) · Behaviour Research and Therapy · doi:10.1016/j.brat.2018.02.004 · Cited by 69
Psychological Effects of Meditation for Healthy Practitioners: An Update
Sedlmeier, P., Loße, C., & Quasten, L. C. (2018) · Mindfulness · doi:10.1007/s12671-017-0780-4 · Cited by 74
Lam, R. W., Teng, M. Y., Jung, Y.-E., Evans, V. C., Gottlieb, J. F., Chakrabarty, T., Michalak, E. E., Murphy, J. K., Yatham, L. N., & Sit, D. K. (2020) · The Canadian Journal of Psychiatry · doi:10.1177/0706743719892471 · Cited by 70
Delay and Death-Thought Accessibility: A Meta-Analysis
Steinman, C. T., & Updegraff, J. A. (2015) · Personality and Social Psychology Bulletin · doi:10.1177/0146167215607843 · Cited by 72
Standard Operating Procedures: A Safety Net for Pre-Analysis Plans
Lin, W., & Green, D. P. (2016) · Political Science and Politics · doi:10.1017/s1049096516000810 · Cited by 71
Lan, M., Law, N., & Pan, Q. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107200 · Cited by 70
Jebeile, H., Gow, M. L., Baur, L. A., Garnett, S. P., Paxton, S. J., & Lister, N. B. (2019) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2019.2841 · Cited by 70
The Trauma and Mental Health Impacts of Coercive Control: A Systematic Review and Meta-Analysis
Lohmann, S., Cowlishaw, S., Ney, L., O’Donnell, M., & Felmingham, K. (2024) · Trauma, Violence, & Abuse · doi:10.1177/15248380231162972 · Cited by 107
“The results of the Duval and Tweedie’s (2000) trim and fill test, the classic fail-safe N test (Rosenthal, 1979), and Orwin’s (1983) fail-safe N test for each meta-analysis are summarized in Supplemental Appendix G.”
Mindfulness and Athlete Burnout: A Systematic Review and Meta-Analysis
Li, C., Zhu, Y., Zhang, M., Gustafsson, H., & Chen, T. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16030449 · Cited by 69
“Publication bias was addressed using the analysis of “fail-safe number” for effect size [32,33].”
MethodDrivers of Consumer Adoption of e-Commerce: a Meta-Analysis
Zerbini, C., Bijmolt, T. H. A., Maestripieri, S., & Luceri, B. (2022) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2022.04.003 · Cited by 68
“Such sources of grey literature were included to increase the sample size and attenuate publication bias (Rosenthal, 1979, 1995).”
BackgroundMethodBaer, R., Gu, J., Cavanagh, K., & Strauss, C. (2019) · Psychological Assessment · doi:10.1037/pas0000744 · Cited by 66
Giluk (2009) · doi:10.1017/s003329172000389x · Cited by 68
“In addition, we calculated the fail-safe Ns to represent the number of non-significant results that would need to exist to nullify an observed effect (Rosenthal, 1979).”
Sugar rush or sugar crash? A meta-analysis of carbohydrate effects on mood
Mantantzis, K., Schlaghecken, F., Sünram-Lea, S. I., & Maylor, E. A. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.03.016 · Cited by 67
“To assess the impact of publication bias in our analysis (the 'file drawer problem'; Rosenthal, 1979), we created funnel graphs by plotting effect sizes against the standard error”
MethodChen, A., Zhang, Y., Jia, J., Liang, M., Cha, Y., & Lim, C. P. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13064 · Cited by 67
Stubbs, B., Vancampfort, D., Rosenbaum, S., Ward, P. B., Richards, J., Ussher, M., & Schuch, F. B. (2016) · Sports Medicine · doi:10.1007/s40279-015-0441-5 · Cited by 66
A framework for learning from null results
Jacob, R. T., Doolittle, F., Kemple, J., & Somers, M.-A. (2019) · Educational Researcher · doi:10.3102/0013189x19891955 · Cited by 65
“This is both because studies of interventions that do not work are less likely to get published and program developers are more likely to try to publish evaluations that show that their program is effective (Franco et al., 2014; Rosenthal, 1979).”
BackgroundThe Life of p : “Just Significant” Results are on the Rise
Leggett, N. C., Thomas, N. A., Loetscher, T., & Nicholls, M. E. R. (2013) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2013.863371 · Cited by 64
Zhang, Y., Xue, J., & Huang, Y. (2020) · Medicine · doi:10.1097/md.0000000000020493 · Cited by 63
“A fail-safe N test estimated the number of unpublished studies needed to lower the effect size below significance.([22,23]) Duval and Tweedle’s trim and fill method was used to calculate an adjusted effect size by accounting for the impact of these estimated missing studies on overall effect size.”
MethodYury, C. A., & Fisher, J. E. (2007) · Psychotherapy and Psychosomatics · doi:10.1159/000101499 · Cited by 63
O'Toole, M. S., Bovbjerg, D. H., Renna, M. E., Lekander, M., Mennin, D. S., & Zachariae, R. (2018) · Brain, Behavior, and Immunity · doi:10.1016/j.bbi.2018.04.005 · Cited by 62
Shi, Y., Yang, H., MacLeod, J., Zhang, J., & Yang, H. H. (2020) · Journal of Educational Computing Research · doi:10.1177/0735633119881477 · Cited by 62
“…the fail-safe number (Nfs), which estimates the number of unretrieved studies averaging null results needed to bring the overall combined effect size at a nonsignificant level (Rosenthal, 1979), was calculated at the significant level of .05 by using the formula: Nfs 0 : 05 ¼ P Z 1 : 645 (cid:2) (cid”
BackgroundMethodNovelty versus Replicability: Virtues and Vices in the Reward System of Science
Romero, F. (2017) · Philosophy of Science · doi:10.1086/694005 · Cited by 61
“…metascientific evidence in various fields suggests that many successful published findings result from questionable practices: selective reporting (Rosenthal 1979), exclusion of data points (Bones 2012; John, Loewenstein, and Prelec 2012), p-hacking (Simmons, Nelson, and Simonsohn 2011), HARKing…”
BackgroundPlant, R. R., & Quinlan, P. T. (2013) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-013-0166-6 · Cited by 61
Hackländer, R. P. M., Janssen, S. M. J., & Bermeitinger, C. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-018-1545-3 · Cited by 61
“Bias Although we have done our best to avoid bias in our analyses, such as by estimating effect sizes for nonreported, nonsignificant findings, there is still one potential source of bias that may affect our interpretations: the file-drawer problem (Rosenthal, 1979).”
BackgroundBernard, R., Hofslundsengen, H., & Frazier Norbury, C. (2022) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2021_jslhr-21-00236 · Cited by 61
Efficacy of face-to-face versus self-guided treatments for disordered gambling: a meta-analysis
Goslar, M., Leibetseder, M., Muench, H. M., Hofmann, S. G., & Laireiter, A.-R. (2017) · Journal of Behavioral Addictions · doi:10.1556/2006.6.2017.034 · Cited by 58
“Risk of bias across studies To control for publication bias, we conducted a thorough literature search and computed Rosenthal’s fail-safe N (FS N; Rosenthal, 1979).”
MethodWhich preventive interventions effectively enhance depressed mothers' sensitivity? A meta-analysis
Kersten‐Alvarez, L. E., Hosman, C. M. H., Riksen‐Walraven, J. M., Van Doesum, K. T. M., & Hoefnagels, C. (2011) · Infant Mental Health Journal · doi:10.1002/imhj.20301 · Cited by 58
Tello, N., Harika-Germaneau, G., Serra, W., Jaafari, N., & Chatard, A. (2020) · Psychological Science · doi:10.1177/0956797619893062 · Cited by 54
“Recently, concerns related to publication bias (Rosenthal, 1979), questionable research practices (such as p-hacking; John, Loewenstein, & Prelec, 2012), and failures to replicate the findings of previous studies (Maxwell, Lau, & Howard, 2015; Open Science Collaboration, 2015) have led to a confidence crisis in scienc…”
BackgroundTutal, Ö., & Yazar, T. (2021) · Asia Pacific Education Review · doi:10.1007/s12564-021-09706-9 · Cited by 57
“The number of additional studies resulting from this calculation exceeded the “5k + 10” limit (Rosenthal, 1979).”
BackgroundMethodInfluentialDietrichson, J., Filges, T., Seerup, J. K., Klokker, R. H., Viinholt, B. C. A., Bøg, M., & Eiberg, M. (2021) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.1152 · Cited by 54
Dose‐response relationship of phototherapy for seasonal affective disorder: a meta‐analysis
Lee, T. M. C., & Chan, C. C. H. (1999) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.1999.tb07236.x · Cited by 52
Thibaudeau, É., Cellard, C., Turcotte, M., & Achim, A. M. (2021) · Schizophrenia Bulletin · doi:10.1093/schbul/sbaa182 · Cited by 51
Metadata
- Authors
- Rosenthal, R.
- Affiliations
- Harvard University (US)
- Corresponding author
- Rosenthal, Robert
- Venue
- Psychological Bulletin
- Published
- 1979-05
- Volume
- 86
- Issue
- 3
- Pages
- 638-641
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2006546769
- Cited by
- 9,068
- References
- 16
- Citations by year
- 2026: 421 · 2025: 521 · 2024: 509 · 2023: 526 · 2022: 525 · 2021: 536 · 2020: 612 · 2019: 607 · 2018: 492 · 2017: 436 · 2016: 417 · 2015: 392 · 2014: 337 · 2013: 289 · 2012: 252
- FWCI
- 26.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- file drawer problem, negative results, publication bias, non-significant results, type I error, selective reporting, behavioral science, evidence synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
