The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
The file drawer problem and tolerance for null results.
Abstract
For any given research area, one cannot tell how many studies have been con-ducted but never reported. The extreme view of the "file drawer problem " is that journals are filled with the 5 % of the studies that show Type I errors, while the file drawers are filled with the 95 % of the studies that show non-significant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed. Both behavioral researchers and statisti-cians have long suspected that the studies published in the behavioral sciences are a biased sample of the studies that are actually carried out (Bakan, 1967; McNemar, 1960; Smart, 1964; Sterling, 1959). The extreme view of this problem, the "file drawer prob-lem, " is that the journals are filled with the 5 % of the studies that show Type I errors, while the file drawers back at the lab are filled with the 95 % of the studies that show nonsignificant (e.g., p>.05) results. In the past there was very little one could do to assess the net effect of studies, tucked away in file drawers, that did not make the magic.05 level (Rosenthal & Gaito, 1963, 1964). Now, however, although no definitive solution to the problem is available, one can establish reasonable boundaries on the prob-lem and estimate the degree of damage to any research conclusion that could be done by the file drawer problem. This advance in our ability to cope with the file drawer is an outgrowth of the in-creasing interest of behavioral scientists in summarizing bodies of research literature sys-Preparation of this article was supported in part by the National Science Foundation. I would like to thank Judith A. Hall and Donald B. Rubin for their valuable improvements of an earlier version of this article. Requests for reprints should be sent to Robert
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We then calculated Rosenthal’s fail-safe N (Rosenthal, 1979), and determined that 2,153,789 effects evidencing no relationship between SA and psychopathology (g = 0) would be needed to reduce the significance level of the summary effect size to .05.”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“The file drawer effect is a perennial problem in the academic literature (Rosenthal, 1979)”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“First, we calculated the fail-safe N [177] to be 4,274, which is the theoretical number of unpublished studies with effect sizes averaging zero (no effect) that would be needed to render negligible the omnibus results.”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [177]
In the citing paper
“As we did not include unpublished work in our meta-analyses we tested for publication bias in the following ways.Firstly, funnel plots were examined.demonstrating no systematic publication biases (tables available from first author on request).Secondly..Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Presence of publication bias was assessed with funnel plots, 22 Rosenthal's fail-safe N , 23 and Egger's regression intercept. 24”
From Light Therapy for Patients With Bipolar Depression: Systematic Review and Meta-Analysis of Randomized Controlled Trials (2020) · cites this paper as [23]
In the citing paper
“In terms of study quality, there was some indication of publication bias whereby some associations between racism and health outcomes were stronger in published versus unpublished studies, consistent with the tendency for ‘null’ studies to remain unpublished (i.e., the ‘file drawer’ problem [418]).”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as [418]
In the citing paper
“An alternative, perhaps more plausible, explanation is that there could be a significant amount of unpublished negative or inconclusive results, a phenomenon referred to as the “file-drawer effect” or publication bias (23).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [23]
In the citing paper
“If either of the two last hypotheses is true, there should exist many unpublished studies with null results lying in laboratories' drawers (32).”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Publication bias was evaluated using Rosenthal's file drawer analysis (Rosenthal, 1979) and Egger's linear regression (Card, 2012), and applying ANOVA to compare the mean effect size of the published versus unpublished studies.”
From Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal (1979) suggested a fail-safe N value above 5; k + 10 reflects results that are tolerant to contradicting studies”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“We applied random-effects models to fit the relations between CE and the criterion variables, following the guidelines proposed by Rosenthal (1979), Hunter and Schmidt (2004) and Cumming (2013).”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal (1979)
In the citing paper
“The fail-safe value for this meta-analysis was computed to be 3001, which is more than 5(36) + 10.”
From Effects of digital game-based STEM education on students’ learning achievement: a meta-analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, we suggest our results may look like a lot of studies that ended up in the so-called “file drawer” [57].”
From Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory (2023) · cites this paper as [57]
In the citing paper
“Finally, publication bias was addressed by computing the fail-safe N for the correlation coefficients (Hunter & Schmidt, 2004;Rosenthal, 1979), which reports the number of field effect sizes or the tolerance for future null results, to check whether a publication bias affected our results.”
From Drivers of Consumer Adoption of e-Commerce: a Meta-Analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Using Rosenthal’s (1979) fail-safe N criterion that the value should be over five times greater than the number of studies included in the meta-analysis, the fail-safe N for all outcome groups were at least eight times larger than the criterion value, suggesting that the effect sizes are unlikely to be an artifact of bias.”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as Rosenthal (1979)
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, Rosenthal (1979) opined that a Fail-safe N exceeding the number of studies by fivefold would provide strong support for the robustness of the significant effects.”
From Extending the Theory of Planned Behavior in the Social Commerce Context: A Meta-Analytic SEM (MASEM) Approach (2023) · cites this paper as Rosenthal (1979)
In the citing paper
“The trim-and-fill method (Duval & Tweedie, 2000) tends to adjust too little when bias is strong (Carter et al., 2019), and Fail-Safe N (Rosenthal, 1979) neither tests nor adjusts for the presence of bias.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Rosenthal, 1979
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, a phenomenon known as ' publication bias ' (Rosenthal 1979).”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“However, a common problem is that results may not have been reported in dissertations or research reports, or published in scienti fi c (peer-reviewed) journals, due to non-signi fi cant or unfavorable fi ndings and, therefore, are dif fi cult to locate.”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [21]
In the citing paper
“The traditional fail-safe value is 1904, which is higher than the threshold value of 100, i.e., [5(18) + 10], indicating that publication bias is insignificant, and the study's results are reliable.”
From A meta-analysis to explore the role of design thinking in enhancing creativity as learning outcomes in STEM education (2026) · cites this paper as Rosenthal, 1979
In the citing paper
“The Q statistic was used to assess heterogeneity30, and publication bias was estimated using Rosenthal’s fail-safe number with α = 0.0531.”
From The limited prosocial effects of meditation: A systematic review and meta-analysis (2018) · cites this paper as [31]
In the citing paper
“Rosenthal's fail-safe number (NR) was calculated, estimating the number of unpublished studies with non-significant results needed to bring an observed result to non-significance (NR ⩾ 5k + 10 to rule out a file drawer problem) (Rosenthal, 1979).”
From Efficacy of non-invasive brain stimulation on cognitive functioning in brain disorders: a meta-analysis (2020) · cites this paper as Rosenthal, 1979
In the citing paper
“In case of statistically significant effect sizes, it was planned to examine the robustness of findings by calculating the Failsafe N (number of non‐significant studies that would bring the p‐value to non‐significant (p > 0.05)) 40.”
From Cognitive behavioral therapies for informal caregivers of patients with cancer and cancer survivors: a systematic review and meta-analysis (2017) · cites this paper as [40]
In the citing paper
“With negative results buried in file drawers (55), conclusions drawn from the published record can be misleading, and 'false facts' can become canonized in the literature (56).”
From Misinformation in and about science (2021) · cites this paper as [55]
In the citing paper
“the fail-safe N (i.e. the file drawer statistic) was used to determine how many additional studies would be necessary to turn a significant ES non-significant (32).”
From Peripheral cytokine and chemokine alterations in depression: a meta‐analysis of 82 studies (2017) · cites this paper as [32]
In the citing paper
“File drawer analysis (Rosenthal, 1979) revealed that 1919 unpublished, filed, or unretrieved studies averaging null results would be required to bring the average unweighted effect size to nonsignificance.”
From Happy to help? A systematic review and meta-analysis of the effects of performing acts of kindness on the well-being of the actor (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's (1979) fail-safe N analysis reveals 186 unpublished papers (i.e., null effect) are needed to be included in the meta-analysis in order to invalidate the population effect size of small to medium (i.e., .235)found in this review.”
From A meta-analytic review of the efficacy of physical exercise interventions on cognition in individuals with autism spectrum disorder and ADHD (2016) · cites this paper as Rosenthal's (1979)
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, the so called ' publication bias ' (Rosenthal, 1979).”
From The effects of physical activity interventions on psychosocial outcomes in adolescents: a meta-analytic review (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's fail-safe N 37 calculated 115 missing studies would be required to increase the two-tailed p-value to above .05.”
From Anxiety in Informal Dementia Carers: A Meta-Analysis of Prevalence (2020) · cites this paper as [37]
In the citing paper
“Publication bias was investigated with Begg's rank correlation test for funnel plot asymmetry (Begg and Mazumdar, 1994) and where significant effects were present Rosenthal's file drawer analysis was also conducted (Rosenthal, 1979).”
From Meta-analysis of the effects of intranasal oxytocin on interpretation and expression of emotions (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“Over time, this practice artificially inflates Type I error and may make it appear that relationships exist when in reality they do not (Rosenthal et al., 1979).”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“However, a potential problem with meta-analysis is the file drawer problem, where studies are “filed away” unpublished if they fail to find statistically significant relationships.”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“Meta-analyses are necessarily based on the available literature and hence may be affected by the potential publication-bias disfavoring null results (Rosenthal 1979).”
From Parsing the neural correlates of moral cognition: ALE meta-analysis on morality, theory of mind, and empathy (2012) · cites this paper as Rosenthal 1979
In the citing paper
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
From The Extent and Consequences of P-Hacking in Science (2015) · cites this paper as [7]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G. E., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,518
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718 · cited 1×
“Perhaps the simplest, most useful way of computing the overall p of a, set of research studies is the method of adding Zs (Cochran, 1954; Mosteller & Bush, 1954; Rosenthal, 1978).”
MethodMeta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154
Experimenter Effects in Behavioral Research
McGuigan, F. J., & Rosenthal, R. (1968) · The American Journal of Psychology · doi:10.2307/1420816 · Cited by 1,633
Self-citationArtifact in behavioral research
Rosenthal, R., & Rosnow, R. L. (1969) · Medical Entomology and Zoology · Cited by 1,186
Self-citationSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964
Combining results of independent studies.
Rosenthal, R. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.1.185 · Cited by 910
Self-citationGender effects in decoding nonverbal cues.
Hall, J. A. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.4.845 · Cited by 766
Wortman, C. B., Rosenthal, R., & Rosnow, R. L. (1976) · The American Journal of Psychology · doi:10.2307/1421418 · Cited by 428
Self-citation
Cited by
Sorted by citations
Items 1,101–1,200 of 1,374
Pratt, T. C., Cullen, F. T., Blevins, K. R., Daigle, L., & Unnever, J. D. (2002) · International Journal of Police Science & Management · doi:10.1350/ijps.4.4.344.10873 · Cited by 173
Kemp, A. H., Gordon, E., Rush, A. J., & Williams, L. M. (2008) · CNS Spectrums · doi:10.1017/s1092852900017120 · Cited by 170
Anxiety in youth with asthma: a meta-analysis
Dudeney, J., Sharpe, L., Jaffe, A., Jones, E. B., & Hunt, C. (2017) · Pediatric Pulmonology · doi:10.1002/ppul.23689 · Cited by 169
Demand for gasoline is more price-inelastic than commonly thought
Havranek, T., Irsova, Z., & Janda, K. (2012) · Energy Economics · doi:10.1016/j.eneco.2011.09.003 · Cited by 164
Tsai, Y.-L., & Tsai, C.-C. (2018) · Computers & Education · doi:10.1016/j.compedu.2018.06.020 · Cited by 164
A trans-diagnostic perspective on obsessive-compulsive disorder
Gillan, C. M., Fineberg, N. A., & Robbins, T. W. (2017) · Psychological Medicine · doi:10.1017/s0033291716002786 · Cited by 158
“Publication bias is an obstacle here as studies showing no difference between patient cohorts and controls, which are needed to make a case for specificity, are less likely to be published or even submitted for publication (Rosenthal, 1979).”
Apps As Learning Tools: A Systematic Review
Griffith, S. F., Hagan, M. B., Heymann, P., Heflin, B. H., & Bagner, D. M. (2020) · Pediatrics · doi:10.1542/peds.2019-1579 · Cited by 168
“84 Third, because of the nascent state of the literature, the summarized studies were highly heterogeneous.”
BackgroundPreti, A., & Cella, M. (2010) · Schizophrenia Research · doi:10.1016/j.schres.2010.07.026 · Cited by 167
Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168
“…i n a t e d b r o a d l y . positive or statistically significant findings, leaving negative and null results underreported (Greenwald, 1975; Ioannidis, 2005; Io-annidis & Trikalinos, 2007; Pfeiffer, Bertram, & Ioannidis, 2011; Rosenthal, 1979; Schimmack, 2012; Simonsohn, Nelson, & Sim-mons, 2014).”
Jackson, J. N. S., & MacKillop, J. (2016) · Biological Psychiatry: Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2016.01.007 · Cited by 167
“In this case, the bias indicators suggest a large number of null findings would need to be in the ‘file-drawer’ (41) to render the overall conclusion null and that the interrelationships between effect size magnitude and variance or year of publication are not suggestive of small study bias.”
BackgroundOlatunji, B. O., Kauffman, B. Y., Meltzer, S., Davis, M. L., Smits, J. A. J., & Powers, M. B. (2014) · Behaviour Research and Therapy · doi:10.1016/j.brat.2014.05.002 · Cited by 167
A systematic review and meta-analysis of interpersonal psychotherapy for perinatal women
Sockol, L. E. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.01.018 · Cited by 158
Peer attachment and self-esteem: A meta-analytic review
Gorrese, A., & Ruggieri, R. (2013) · Personality and Individual Differences · doi:10.1016/j.paid.2013.04.025 · Cited by 164
A historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
Systematic review and meta-analysis of ex-post evaluations on the effectiveness of carbon pricing
Döbbeling-Hildebrandt, N., Miersch, K., Khanna, T. M., Bachelet, M., Bruns, S. B., Callaghan, M., Edenhofer, O., Flachsland, C., Forster, P. M., Kalkuhl, M., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-48512-w · Cited by 164
Transdiagnostic approaches to the treatment of anxiety disorders: A quantitative review.
Norton, P. J., & Philipp, L. M. (2008) · Psychotherapy: Theory, Research, Practice, Training · doi:10.1037/0033-3204.45.2.214 · Cited by 159
Louise, S., Fitzpatrick, M., Strauss, C., Rossell, S. L., & Thomas, N. (2018) · Schizophrenia Research · doi:10.1016/j.schres.2017.05.023 · Cited by 160
IQ of Children Growing Up in Children's Homes: A Meta-Analysis on IQ Delays in Orphanages
van IJzendoorn, M. H., Luijk, M. P. C. M., & Juffer, F. (2008) · Merrill-Palmer Quarterly · doi:10.1353/mpq.0.0002 · Cited by 160
Bulimia nervosa: A meta-analysis of psychosocial and pharmacological treatments
Whittal, M. L., Agras, W. S., & Gould, R. A. (1999) · Behavior Therapy · doi:10.1016/s0005-7894(99)80049-5 · Cited by 159
Li, D., Zhao, H., & Gelernter, J. (2012) · Human Genetics · doi:10.1007/s00439-011-1116-4 · Cited by 158
Tan, B. W. Z., Pooley, J. A., & Speelman, C. P. (2016) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-016-2854-x · Cited by 157
A meta-analysis of signaling principle in multimedia learning environments
Alpizar, D., Adesope, O. O., & Wong, R. M. (2020) · Educational Technology Research and Development · doi:10.1007/s11423-020-09748-7 · Cited by 157
“Dissertations were included to minimize the potential for publication bias (Orwin 1983; Rosenthal 1979; Slavin 1995).”
BackgroundBowers, J. M., & Moyer, A. (2017) · Sleep Health · doi:10.1016/j.sleh.2017.08.004 · Cited by 156
Etiology of obsessions and compulsions: A meta-analysis and narrative review of twin studies
Taylor, S. (2011) · Clinical Psychology Review · doi:10.1016/j.cpr.2011.09.008 · Cited by 156
Jebeile, H., Gow, M. L., Baur, L. A., Garnett, S. P., Paxton, S. J., & Lister, N. B. (2019) · Obesity Reviews · doi:10.1111/obr.12866 · Cited by 160
“Publication bias was assessed by visual inspection of a funnel plot and use of the classic fail‐safe N statistic(16) with interpretation based on the tolerance level suggested by Rosenthal.(17)”
MethodSuicidal behaviour and memory: A systematic review and meta-analysis
Richard-Devantoy, S., Berlim, M. T., & Jollant, F. (2015) · The World Journal of Biological Psychiatry · doi:10.3109/15622975.2014.925584 · Cited by 153
“Finally, we used Funnel plots, Rosenthal’s Fail-Safe n (Rosenthal 1979) and egger’s Regression intercept (egger et al. 1997) to test for the presence of publication bias (cooper et al. 2009).”
MethodPublication Bias in Meta-Analysis: Confidence Intervals for Rosenthal’s Fail-Safe Number
Fragkos, K. C., Tsagris, M., & Frangos, C. C. (2014) · International Scholarly Research Notices · doi:10.1155/2014/825383 · Cited by 159
The influence of deliberate practice on musical achievement: a meta-analysis.
Platz, F., Kopiez, R., Lehmann, A. C., & Wolf, A. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00646 · Cited by 152
Richardson, M., McEwan, K., Maratos, F., & Sheffield, D. (2016) · Evolutionary Psychological Science · doi:10.1007/s40806-016-0065-5 · Cited by 155
“Publication bias analyses were undertaken first by calculating fail safe N (Rosenthal 1979).”
MethodEffect of Obstructive Sleep Apnea Treatment on Atrial Fibrillation Recurrence
Shukla, A., Aizer, A., Holmes, D., Fowler, S., Park, D. S., Bernstein, S., Bernstein, N., & Chinitz, L. (2015) · JACC: Clinical Electrophysiology · doi:10.1016/j.jacep.2015.02.014 · Cited by 155
Acar, F., Seurinck, R., Eickhoff, S. B., & Moerkerke, B. (2018) · PLOS ONE · doi:10.1371/journal.pone.0208177 · Cited by 154
“The original FSN [21] represents the required number of findings with an effect of zero to change the conclusions of the meta-analysis under the assumption that these findings are still in the “file drawer”.”
BackgroundInfluentialIntergroup contact and mental health stigma: a comparative effectiveness meta-analysis
Maunder, R. D., & White, F. A. (2019) · Clinical Psychology Review · doi:10.1016/j.cpr.2019.101749 · Cited by 152
Tonetti, L., Natale, V., & Randler, C. (2015) · Chronobiology International · doi:10.3109/07420528.2015.1049271 · Cited by 156
Sleep disorders in African Americans and Caucasian Americans: a meta-analysis
Ruiter, M. E., DeCoster, J., Jacobs, L., & Lichstein, K. L. (2010) · Behavioral Sleep Medicine · doi:10.1080/15402002.2010.509251 · Cited by 152
The expectation effect and cardiac pacing for refractory vasovagal syncope
Sud, S., Massel, D., Klein, G. J., Leong-Sit, P., Yee, R., Skanes, A. C., Gula, L. J., & Krahn, A. D. (2007) · The American Journal of Medicine · doi:10.1016/j.amjmed.2006.05.046 · Cited by 151
Beauchaine, T. P., Bell, Z., Knapton, E., McDonough‐Caplan, H., Shader, T., & Zisner, A. (2019) · Psychophysiology · doi:10.1111/psyp.13329 · Cited by 144
Pearson, N., Braithwaite, R., & Biddle, S. J. H. (2015) · Academic Pediatrics · doi:10.1016/j.acap.2014.08.009 · Cited by 149
Generative Learning: Which Strategies for What Age?
Brod, G. (2021) · Educational Psychology Review · doi:10.1007/s10648-020-09571-9 · Cited by 145
“Because studies finding no benefit of a particular strategy are less likely to be published (the well-known file-drawer problem; Rosenthal 1979), the table likely overestimates effectiveness.”
BackgroundJansen, J. E., Gleeson, J., Bendall, S., Rice, S., & Alvarez-Jimenez, M. (2020) · Schizophrenia Research · doi:10.1016/j.schres.2019.11.016 · Cited by 144
Meta-analysis of HIV risk-reduction interventions within drug abuse treatment programs
Prendergast, M. L., Urada, D., & Podus, D. (2001) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.69.3.389 · Cited by 148
Salehinejad, M. A., Wischnewski, M., Nejati, V., Vicario, C. M., & Nitsche, M. A. (2019) · PLOS ONE · doi:10.1371/journal.pone.0215095 · Cited by 126
“05), was calculated, which gives an indication of the amount of null findings that are needed to render the cumulative effect non-significant [66].”
BackgroundFuchs, C., Lee, J. K., Roemer, L., & Orsillo, S. M. (2013) · Cognitive and Behavioral Practice · doi:10.1016/j.cbpra.2011.12.004 · Cited by 139
Kraemer, H. C., Gardner, C., Brooks, J. O., & Yesavage, J. A. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.1.23 · Cited by 141
Retrieval potentiates new learning: A theoretical and meta-analytic review.
Chan, J. C. K., Meissner, C. A., & Davis, S. D. (2018) · Psychological Bulletin · doi:10.1037/bul0000166 · Cited by 143
A Selective Synthesis of the Experimental Literature on Dynamic Assessment
Swanson, H. L., & Lussier, C. M. (2001) · Review of Educational Research · doi:10.3102/00346543071002321 · Cited by 142
The relationship between dissociation and voices: A systematic literature review and meta-analysis
Pilton, M., Varese, F., Berry, K., & Bucci, S. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.06.004 · Cited by 141
Koesters, M., Becker, T., Kilian, R., Fegert, J., & Weinmann, S. (2009) · Journal of Psychopharmacology · doi:10.1177/0269881108092338 · Cited by 142
Haug, T., Nordgreen, T., Öst, L. G., & Havik, O. E. (2012) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.04.002 · Cited by 128
Worsening file-drawer problem in the abstracts of natural, medical and social science databases
Pautasso, M. (2010) · Scientometrics · doi:10.1007/s11192-010-0233-5 · Cited by 136
Bridgett, D. J., & Walker, M. E. (2006) · Psychological Assessment · doi:10.1037/1040-3590.18.1.1 · Cited by 134
Lead and attention-deficit/hyperactivity disorder (ADHD) symptoms: a meta-analysis
Goodlad, J. K., Marcus, D. K., & Fulton, J. J. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2013.01.009 · Cited by 137
Physical activity and body image among men and boys: A meta-analysis
Bassett-Gunter, R., McEwan, D., & Kamarhie, A. (2017) · Body Image · doi:10.1016/j.bodyim.2017.06.007 · Cited by 137
Roberts, P. D., Stewart, G. B., & Pullin, A. S. (2006) · Biological Conservation · doi:10.1016/j.biocon.2006.04.034 · Cited by 137
Guo, L. (2022) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12650 · Cited by 141
Slotema, C. W., Blom, J. D., de Weijer, A. D., Diederen, K. M., Goekoop, R., Looijestijn, J., Daalman, K., Rijkaart, A.-M., Kahn, R. S., Hoek, H. W., et al. (2011) · Biological Psychiatry · doi:10.1016/j.biopsych.2010.09.051 · Cited by 137
Toli, A., Webb, T. L., & Hardy, G. E. (2016) · British Journal of Clinical Psychology · doi:10.1111/bjc.12086 · Cited by 137
Bronkhorst, B., Tummers, L., Steijn, B., & Vijverberg, D. (2015) · Health Care Management Review · doi:10.1097/hmr.0000000000000026 · Cited by 137
“At the same time, we acknowledge this could potentially lead to a publication bias also known as the file-drawer problem (Rosenthal, 1979).”
Xu, Q., Anderson, D., & Lurie-Beck, J. (2011) · Obesity Research & Clinical Practice · doi:10.1016/j.orcp.2011.04.007 · Cited by 136
Hacker, T., Stone, P., & MacBeth, A. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2015.10.053 · Cited by 130
“Secondly..Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
BackgroundIs the Putative Mirror Neuron System Associated with Empathy? A Systematic Review and Meta-Analysis
Bekkali, S., Youssef, G. J., Donaldson, P. H., Albein-Urios, N., Hyde, C., & Enticott, P. G. (2021) · Neuropsychology Review · doi:10.1007/s11065-020-09452-6 · Cited by 128
Lane, A., Luminet, O., Nave, G., & Mikolajczak, M. (2016) · Journal of Neuroendocrinology · doi:10.1111/jne.12384 · Cited by 129
Pinquart, M. (2013) · Body Image · doi:10.1016/j.bodyim.2012.10.008 · Cited by 134
Troeung, L., Egan, S. J., & Gasson, N. (2013) · PLoS ONE · doi:10.1371/journal.pone.0079510 · Cited by 132
“Publication bias in the current meta-analysis was assessed using Egger’s [69] regression asymmetry test and Rosenthal’s [70] Fail-Safe N method.”
MethodFinding the power to reduce publication bias
Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. A. (2017) · Statistics in Medicine · doi:10.1002/sim.7228 · Cited by 131
Shadish, W. R., Zelinsky, N. A. M., Vevea, J. L., & Kratochwill, T. R. (2016) · Journal of Applied Behavior Analysis · doi:10.1002/jaba.308 · Cited by 130
Papalia, N., Spivak, B., Daffern, M., & Ogloff, J. R. P. (2019) · Clinical Psychology: Science and Practice · doi:10.1111/cpsp.12282 · Cited by 130
van Loon, A. W. G., Creemers, H. E., Beumer, W. Y., Okorn, A., Vogelaar, S., Saab, N., Miers, A. C., Westenberg, P. M., & Asscher, J. J. (2020) · Journal of Youth and Adolescence · doi:10.1007/s10964-020-01201-5 · Cited by 128
“First, Rosenthal’s failsafe test was performed, indicating no publication bias when the fail-safe N exceeds the critical value (derived by the formula 5 × k+ 10, where k is the number of studies) (Rosenthal 1979).”
MethodDissociation and emotion regulation strategies: A meta-analytic review
Cavicchioli, M., Scalabrini, A., Northoff, G., Mucci, C., Ogliari, A., & Maffei, C. (2021) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2021.09.011 · Cited by 146
Contextual interference: a meta-analytic study
Brady, F. (2004) · Perceptual and Motor Skills · doi:10.2466/pms.99.1.116-126 · Cited by 129
At Issue: Schizophrenia and Rheumatoid Arthritis: The Negative Association Revisited
Oken, R. J., & Schulzer, M. (1999) · Schizophrenia Bulletin · doi:10.1093/oxfordjournals.schbul.a033407 · Cited by 129
Martin, F., Wu, T., Wan, L., & Xie, K. (2022) · Online Learning · doi:10.24059/olj.v26i1.2604 · Cited by 132
“Rosenthal (1979) used a term “file drawer problem” to describe the fact that journals are filled with the 5% of studies showing Type I errors whereas the rest of the studies with nonsignificant results are left in the drawers.”
BackgroundMethodStrøm, H. K., Adolfsen, F., Fossum, S., Kaiser, S., & Martinussen, M. (2014) · Substance Abuse Treatment, Prevention, and Policy · doi:10.1186/1747-597x-9-48 · Cited by 128
Using Dissertations to Examine Potential Bias in Child and Adolescent Clinical Trials.
McLeod, B. D., & Weisz, J. R. (2004) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.72.2.235 · Cited by 125
Negt, P., Brakemeier, E., Michalak, J., Winter, L., Bleich, S., & Kahl, K. G. (2016) · Brain and Behavior · doi:10.1002/brb3.486 · Cited by 126
“To assess the impact of this bias on the combined overall effect sizes, fail safe N procedures (Rosenthal 1979; Becker 2005) as well as trim-and-fill analyses (Duval and Tweedie 2000a,b) were conducted.”
Agreeableness and Its Consequences: A Quantitative Review of Meta-Analytic Findings
Wilmot, M. P., & Ones, D. S. (2022) · Personality and Social Psychology Review · doi:10.1177/10888683211073007 · Cited by 130
Gomez, A. F., Barthel, A. L., & Hofmann, S. G. (2018) · Expert Opinion on Pharmacotherapy · doi:10.1080/14656566.2018.1472767 · Cited by 125
“To address the file drawer problem, a classic fail-safe N analysis was conducted [33,34].”
MethodSolmi, M., Veronese, N., Correll, C. U., Favaro, A., Santonastaso, P., Caregaro, L., Vancampfort, D., Luchini, C., De Hert, M., & Stubbs, B. (2016) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.12556 · Cited by 124
Jokela, M., & Hanin, Y. L. (1999) · Journal of Sports Sciences · doi:10.1080/026404199365434 · Cited by 123
A systematic review and meta-analysis of low intensity CBT for psychosis
Hazell, C. M., Hayward, M., Cavanagh, K., & Strauss, C. (2016) · Clinical Psychology Review · doi:10.1016/j.cpr.2016.03.004 · Cited by 120
Does the Job Satisfaction—Job Performance Relationship Vary Across Cultures?
Ng, T. W. H., Sorensen, K. L., & Yim, F. H. K. (2009) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022109339208 · Cited by 123
JOURNAL QUALITY, EFFECT SIZE, AND PUBLICATION BIAS IN META-ANALYSIS
Murtaugh, P. A. (2002) · Ecology · doi:10.1890/0012-9658(2002)083[1162:jqesap]2.0.co;2 · Cited by 126
Schindler, H. S., Kholoptseva, J., Oh, S. S., Yoshikawa, H., Duncan, G. J., Magnuson, K. A., & Shonkoff, J. P. (2015) · Journal of School Psychology · doi:10.1016/j.jsp.2015.04.001 · Cited by 121
“Second, we computed fail-safe Ns for our two primary signi fi cant fi ndings (Rosenthal, 1979).”
The relationship between mindfulness and objective measures of body awareness: A meta-analysis
Treves, I. N., Tello, L. Y., Davidson, R. J., & Goldberg, S. B. (2019) · Scientific Reports · doi:10.1038/s41598-019-53978-6 · Cited by 114
Long-Term Memory Performance in Adult ADHD: A Meta-Analysis
Skodzik, T., Holling, H., & Pedersen, A. (2017) · Journal of Attention Disorders · doi:10.1177/1087054713510561 · Cited by 120
“In addition, with respect to publication bias, we should keep in mind that the small number of included studies and the anticipated small effect led to a small Fail-safe N (cf. Rosenthal, 1979).”
BackgroundAmygdala Volume in Adults with Posttraumatic Stress Disorder: A Meta-Analysis
Woon, F. L., & Hedges, D. W. (2009) · The Journal of Neuropsychiatry and Clinical Neurosciences · doi:10.1176/jnp.2009.21.1.5 · Cited by 120
Outcomes associated with virtual reality in psychological interventions: where are we now?
Turner, W. A., & Casey, L. M. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.10.003 · Cited by 116
“Publication bias was assessed via funnel plot (Figure 2) and Rosenthal (1979) Fail- Safe N value.”
BackgroundMethodZhao, L., Mao, H., Compton, B. J., Peng, J., Fu, G., Fang, F., Heyman, G. D., & Lee, K. (2022) · Educational Research Review · doi:10.1016/j.edurev.2022.100455 · Cited by 120
“The inclusion of published and unpublished papers served to address, to some extent, the file drawer problem (Rosenthal, 1979).”
BackgroundMethodResultAmphetamine modulates brain signal variability and working memory in younger and older adults
Garrett, D. D., Nagel, I. E., Preuschhof, C., Burzynska, A. Z., Marchner, J., Wiegert, S., Jungehülsing, G. J., Nyberg, L., Villringer, A., Li, S.-C., et al. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1504090112 · Cited by 103
“To the extent that pharmacological DA studies suffer from the file drawer problem (43), drug administration order may be one culprit in zeroing out inverse behavioral/brain effects resulting from session order.”
BackgroundRamos-Sanchez, C. P., Schuch, F. B., Seedat, S., Louw, Q. A., Stubbs, B., Rosenbaum, S., Firth, J., van Winkel, R., & Vancampfort, D. (2021) · Psychiatry Research · doi:10.1016/j.psychres.2021.114046 · Cited by 118
Potential negative consequences of mindfulness in the moral domain
Schindler, S., Pfattheicher, S., & Reinhard, M. (2019) · European Journal of Social Psychology · doi:10.1002/ejsp.2570 · Cited by 116
Replication, Communication, and the Population Dynamics of Scientific Discovery
McElreath, R., & Smaldino, P. E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0136088 · Cited by 117
The Effects of Spaced Practice on Second Language Learning: A Meta‐Analysis
Kim, S. K., & Webb, S. (2022) · Language Learning · doi:10.1111/lang.12479 · Cited by 119
Johannsen, M., Farver, I., Beck, N., & Zachariae, R. (2013) · Breast Cancer Research and Treatment · doi:10.1007/s10549-013-2503-4 · Cited by 113
Cui, N., & Liu, J. (2020) · Trauma, Violence, & Abuse · doi:10.1177/1524838018757750 · Cited by 113
“A filedrawer effect, selective publication bias, exists when the failsafe N is less than 5 times the number of published studies plus 10 (Rosenthal, 1979).”
BackgroundIntimate partner violence in male survivors of child maltreatment: A meta-analysis
Godbout, N., Vaillancourt-Morel, M.-P., Bigras, N., Briere, J., Hébert, M., Runtz, M., & Sabourin, S. (2019) · Trauma, Violence, & Abuse · doi:10.1177/1524838017692382 · Cited by 111
Spatial Abilities and Aging: A Meta-Analysis
Techentin, C., Voyer, D., & Voyer, S. D. (2014) · Experimental Aging Research · doi:10.1080/0361073x.2014.926773 · Cited by 112
Witteman, J., van IJzendoorn, M. H., van de Velde, D., van Heuven, V. J. J. P., & Schiller, N. O. (2011) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2011.09.028 · Cited by 112
Kurz, M., Rosendahl, J., Rodeck, J., Muehleck, J., & Berger, U. (2022) · Journal of Prevention · doi:10.1007/s10935-021-00660-1 · Cited by 110
Asthma and Swimming: A Meta-Analysis
Goodman, M., & Hays, S. (2008) · Journal of Asthma · doi:10.1080/02770900802165980 · Cited by 109
Reproducibility and replicability: opportunities and challenges for geospatial research
Kedron, P., Li, W., Fotheringham, S., & Goodchild, M. (2021) · International Journal of Geographical Information Science · doi:10.1080/13658816.2020.1802032 · Cited by 108
Metadata
- Authors
- Rosenthal, R.
- Affiliations
- Harvard University (US)
- Corresponding author
- Rosenthal, Robert
- Venue
- Psychological Bulletin
- Published
- 1979-05
- Volume
- 86
- Issue
- 3
- Pages
- 638-641
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2006546769
- Cited by
- 9,068
- References
- 16
- Citations by year
- 2026: 421 · 2025: 521 · 2024: 509 · 2023: 526 · 2022: 525 · 2021: 536 · 2020: 612 · 2019: 607 · 2018: 492 · 2017: 436 · 2016: 417 · 2015: 392 · 2014: 337 · 2013: 289 · 2012: 252
- FWCI
- 26.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- file drawer problem, negative results, publication bias, non-significant results, type I error, selective reporting, behavioral science, evidence synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
