The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
The file drawer problem and tolerance for null results.
Abstract
For any given research area, one cannot tell how many studies have been con-ducted but never reported. The extreme view of the "file drawer problem " is that journals are filled with the 5 % of the studies that show Type I errors, while the file drawers are filled with the 95 % of the studies that show non-significant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed. Both behavioral researchers and statisti-cians have long suspected that the studies published in the behavioral sciences are a biased sample of the studies that are actually carried out (Bakan, 1967; McNemar, 1960; Smart, 1964; Sterling, 1959). The extreme view of this problem, the "file drawer prob-lem, " is that the journals are filled with the 5 % of the studies that show Type I errors, while the file drawers back at the lab are filled with the 95 % of the studies that show nonsignificant (e.g., p>.05) results. In the past there was very little one could do to assess the net effect of studies, tucked away in file drawers, that did not make the magic.05 level (Rosenthal & Gaito, 1963, 1964). Now, however, although no definitive solution to the problem is available, one can establish reasonable boundaries on the prob-lem and estimate the degree of damage to any research conclusion that could be done by the file drawer problem. This advance in our ability to cope with the file drawer is an outgrowth of the in-creasing interest of behavioral scientists in summarizing bodies of research literature sys-Preparation of this article was supported in part by the National Science Foundation. I would like to thank Judith A. Hall and Donald B. Rubin for their valuable improvements of an earlier version of this article. Requests for reprints should be sent to Robert
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We then calculated Rosenthal’s fail-safe N (Rosenthal, 1979), and determined that 2,153,789 effects evidencing no relationship between SA and psychopathology (g = 0) would be needed to reduce the significance level of the summary effect size to .05.”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“The file drawer effect is a perennial problem in the academic literature (Rosenthal, 1979)”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“First, we calculated the fail-safe N [177] to be 4,274, which is the theoretical number of unpublished studies with effect sizes averaging zero (no effect) that would be needed to render negligible the omnibus results.”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [177]
In the citing paper
“As we did not include unpublished work in our meta-analyses we tested for publication bias in the following ways.Firstly, funnel plots were examined.demonstrating no systematic publication biases (tables available from first author on request).Secondly..Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Presence of publication bias was assessed with funnel plots, 22 Rosenthal's fail-safe N , 23 and Egger's regression intercept. 24”
From Light Therapy for Patients With Bipolar Depression: Systematic Review and Meta-Analysis of Randomized Controlled Trials (2020) · cites this paper as [23]
In the citing paper
“In terms of study quality, there was some indication of publication bias whereby some associations between racism and health outcomes were stronger in published versus unpublished studies, consistent with the tendency for ‘null’ studies to remain unpublished (i.e., the ‘file drawer’ problem [418]).”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as [418]
In the citing paper
“An alternative, perhaps more plausible, explanation is that there could be a significant amount of unpublished negative or inconclusive results, a phenomenon referred to as the “file-drawer effect” or publication bias (23).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [23]
In the citing paper
“If either of the two last hypotheses is true, there should exist many unpublished studies with null results lying in laboratories' drawers (32).”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Publication bias was evaluated using Rosenthal's file drawer analysis (Rosenthal, 1979) and Egger's linear regression (Card, 2012), and applying ANOVA to compare the mean effect size of the published versus unpublished studies.”
From Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal (1979) suggested a fail-safe N value above 5; k + 10 reflects results that are tolerant to contradicting studies”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“We applied random-effects models to fit the relations between CE and the criterion variables, following the guidelines proposed by Rosenthal (1979), Hunter and Schmidt (2004) and Cumming (2013).”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal (1979)
In the citing paper
“The fail-safe value for this meta-analysis was computed to be 3001, which is more than 5(36) + 10.”
From Effects of digital game-based STEM education on students’ learning achievement: a meta-analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, we suggest our results may look like a lot of studies that ended up in the so-called “file drawer” [57].”
From Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory (2023) · cites this paper as [57]
In the citing paper
“Finally, publication bias was addressed by computing the fail-safe N for the correlation coefficients (Hunter & Schmidt, 2004;Rosenthal, 1979), which reports the number of field effect sizes or the tolerance for future null results, to check whether a publication bias affected our results.”
From Drivers of Consumer Adoption of e-Commerce: a Meta-Analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Using Rosenthal’s (1979) fail-safe N criterion that the value should be over five times greater than the number of studies included in the meta-analysis, the fail-safe N for all outcome groups were at least eight times larger than the criterion value, suggesting that the effect sizes are unlikely to be an artifact of bias.”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as Rosenthal (1979)
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, Rosenthal (1979) opined that a Fail-safe N exceeding the number of studies by fivefold would provide strong support for the robustness of the significant effects.”
From Extending the Theory of Planned Behavior in the Social Commerce Context: A Meta-Analytic SEM (MASEM) Approach (2023) · cites this paper as Rosenthal (1979)
In the citing paper
“The trim-and-fill method (Duval & Tweedie, 2000) tends to adjust too little when bias is strong (Carter et al., 2019), and Fail-Safe N (Rosenthal, 1979) neither tests nor adjusts for the presence of bias.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Rosenthal, 1979
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, a phenomenon known as ' publication bias ' (Rosenthal 1979).”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“However, a common problem is that results may not have been reported in dissertations or research reports, or published in scienti fi c (peer-reviewed) journals, due to non-signi fi cant or unfavorable fi ndings and, therefore, are dif fi cult to locate.”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [21]
In the citing paper
“The traditional fail-safe value is 1904, which is higher than the threshold value of 100, i.e., [5(18) + 10], indicating that publication bias is insignificant, and the study's results are reliable.”
From A meta-analysis to explore the role of design thinking in enhancing creativity as learning outcomes in STEM education (2026) · cites this paper as Rosenthal, 1979
In the citing paper
“The Q statistic was used to assess heterogeneity30, and publication bias was estimated using Rosenthal’s fail-safe number with α = 0.0531.”
From The limited prosocial effects of meditation: A systematic review and meta-analysis (2018) · cites this paper as [31]
In the citing paper
“Rosenthal's fail-safe number (NR) was calculated, estimating the number of unpublished studies with non-significant results needed to bring an observed result to non-significance (NR ⩾ 5k + 10 to rule out a file drawer problem) (Rosenthal, 1979).”
From Efficacy of non-invasive brain stimulation on cognitive functioning in brain disorders: a meta-analysis (2020) · cites this paper as Rosenthal, 1979
In the citing paper
“In case of statistically significant effect sizes, it was planned to examine the robustness of findings by calculating the Failsafe N (number of non‐significant studies that would bring the p‐value to non‐significant (p > 0.05)) 40.”
From Cognitive behavioral therapies for informal caregivers of patients with cancer and cancer survivors: a systematic review and meta-analysis (2017) · cites this paper as [40]
In the citing paper
“With negative results buried in file drawers (55), conclusions drawn from the published record can be misleading, and 'false facts' can become canonized in the literature (56).”
From Misinformation in and about science (2021) · cites this paper as [55]
In the citing paper
“the fail-safe N (i.e. the file drawer statistic) was used to determine how many additional studies would be necessary to turn a significant ES non-significant (32).”
From Peripheral cytokine and chemokine alterations in depression: a meta‐analysis of 82 studies (2017) · cites this paper as [32]
In the citing paper
“File drawer analysis (Rosenthal, 1979) revealed that 1919 unpublished, filed, or unretrieved studies averaging null results would be required to bring the average unweighted effect size to nonsignificance.”
From Happy to help? A systematic review and meta-analysis of the effects of performing acts of kindness on the well-being of the actor (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's (1979) fail-safe N analysis reveals 186 unpublished papers (i.e., null effect) are needed to be included in the meta-analysis in order to invalidate the population effect size of small to medium (i.e., .235)found in this review.”
From A meta-analytic review of the efficacy of physical exercise interventions on cognition in individuals with autism spectrum disorder and ADHD (2016) · cites this paper as Rosenthal's (1979)
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, the so called ' publication bias ' (Rosenthal, 1979).”
From The effects of physical activity interventions on psychosocial outcomes in adolescents: a meta-analytic review (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's fail-safe N 37 calculated 115 missing studies would be required to increase the two-tailed p-value to above .05.”
From Anxiety in Informal Dementia Carers: A Meta-Analysis of Prevalence (2020) · cites this paper as [37]
In the citing paper
“Publication bias was investigated with Begg's rank correlation test for funnel plot asymmetry (Begg and Mazumdar, 1994) and where significant effects were present Rosenthal's file drawer analysis was also conducted (Rosenthal, 1979).”
From Meta-analysis of the effects of intranasal oxytocin on interpretation and expression of emotions (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“Over time, this practice artificially inflates Type I error and may make it appear that relationships exist when in reality they do not (Rosenthal et al., 1979).”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“However, a potential problem with meta-analysis is the file drawer problem, where studies are “filed away” unpublished if they fail to find statistically significant relationships.”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“Meta-analyses are necessarily based on the available literature and hence may be affected by the potential publication-bias disfavoring null results (Rosenthal 1979).”
From Parsing the neural correlates of moral cognition: ALE meta-analysis on morality, theory of mind, and empathy (2012) · cites this paper as Rosenthal 1979
In the citing paper
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
From The Extent and Consequences of P-Hacking in Science (2015) · cites this paper as [7]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G. E., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,518
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718 · cited 1×
“Perhaps the simplest, most useful way of computing the overall p of a, set of research studies is the method of adding Zs (Cochran, 1954; Mosteller & Bush, 1954; Rosenthal, 1978).”
MethodMeta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154
Experimenter Effects in Behavioral Research
McGuigan, F. J., & Rosenthal, R. (1968) · The American Journal of Psychology · doi:10.2307/1420816 · Cited by 1,633
Self-citationArtifact in behavioral research
Rosenthal, R., & Rosnow, R. L. (1969) · Medical Entomology and Zoology · Cited by 1,186
Self-citationSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964
Combining results of independent studies.
Rosenthal, R. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.1.185 · Cited by 910
Self-citationGender effects in decoding nonverbal cues.
Hall, J. A. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.4.845 · Cited by 766
Wortman, C. B., Rosenthal, R., & Rosnow, R. L. (1976) · The American Journal of Psychology · doi:10.2307/1421418 · Cited by 428
Self-citation
Cited by
Sorted by citations
Items 1,001–1,100 of 1,374
Wu, J., & Lu, X. (2013) · Journal of the Association for Information Systems · doi:10.17705/1jais.00325 · Cited by 255
Normal sleep in African-Americans and Caucasian-Americans: a meta-analysis
Ruiter, M. E., DeCoster, J., Jacobs, L., & Lichstein, K. L. (2011) · Sleep Medicine · doi:10.1016/j.sleep.2010.12.010 · Cited by 255
Glanz, M., Klawansky, S., Stason, W. B., Berkey, C. S., & Chalmers, T. C. (1996) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/s0003-9993(96)90293-2 · Cited by 254
The Big Five and tertiary academic performance: A systematic review and meta-analysis
Vedel, A. (2014) · Personality and Individual Differences · doi:10.1016/j.paid.2014.07.011 · Cited by 263
Excessive reassurance seeking, depression, and interpersonal rejection: A meta-analytic review.
Starr, L. R., & Davila, J. (2008) · Journal of Abnormal Psychology · doi:10.1037/a0013866 · Cited by 251
de Vibe, M., Bjørndal, A., Tipton, E., Hammerstrøm, K., & Kowalski, K. (2012) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.4073/csr.2012.3 · Cited by 253
Veronese, N., Facchini, S., Stubbs, B., Luchini, C., Solmi, M., Manzato, E., Sergi, G., Maggi, S., Cosco, T., & Fontana, L. (2017) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.11.017 · Cited by 251
EEG power spectral measures of cognitive workload: A meta‐analysis
Chikhi, S., Matton, N., & Blanchet, S. (2022) · Psychophysiology · doi:10.1111/psyp.14009 · Cited by 258
Sharpe, D. (1997) · Clinical Psychology Review · doi:10.1016/s0272-7358(97)00056-1 · Cited by 334
Leisure reading and social cognition: A meta-analysis.
Mumper, M. L., & Gerrig, R. J. (2017) · Psychology of Aesthetics, Creativity, and the Arts · doi:10.1037/aca0000089 · Cited by 249
A Short (Personal) Future History of Revolution 2.0
Spellman, B. A. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615609918 · Cited by 213
“Relatedly, the “file drawer problem” is a well-known major flaw in all meta-analysis (Rosenthal, 1979).”
Attentional bias towards threatening stimuli in children with anxiety: A meta-analysis
Dudeney, J., Sharpe, L., & Hunt, C. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.05.007 · Cited by 247
A meta-analysis of the UTAUT model: Eleven years later
Khechine, H., Lakhal, S., & Ndjambou, P. (2016) · Canadian Journal of Administrative Sciences / Revue Canadienne des Sciences de l'Administration · doi:10.1002/cjas.1381 · Cited by 246
Burnout and depression in nurses: a systematic review and meta-analysis
Chen, C., & Meier, S. T. (2021) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2021.104099 · Cited by 246
Decision Making in Health and Medicine: Integrating Evidence and Values
Hunink, M. G. M., Weinstein, M. C., Wittenberg, E., Drummond, M. F., Pliskin, J. S., Wong, J. B., & Glasziou, P. P. (2014) · doi:10.1017/cbo9781139506779 · Cited by 425
Signaling text-picture relations in multimedia learning: A comprehensive meta-analysis
Richter, J., Scheiter, K., & Eitel, A. (2016) · Educational Research Review · doi:10.1016/j.edurev.2015.12.003 · Cited by 242
Collins, R. N., & Kishita, N. (2020) · Ageing and Society · doi:10.1017/s0144686x19000527 · Cited by 244
“Risk of bias across studies Publication bias was assessed by constructing funnel plots, and conducting the trim and fill method (Duval and Tweedie, 2000) and Rosenthal’s Fail Safe N (Rosenthal, 1979).”
MethodSchofield, T. J., Lee, R. D., & Merrick, M. T. (2013) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2013.05.004 · Cited by 242
Dauwan, M., Begemann, M. J. H., Slot, M. I. E., Lee, E. H. M., Scheltens, P., & Sommer, I. E. C. (2021) · Journal of Neurology · doi:10.1007/s00415-019-09493-9 · Cited by 239
“1 3 number of unpublished studies with non-significant results needed to bring the observed result to non-significance [60].”
BackgroundEmotion Regulation and Substance Use: A Meta-Analysis
Weiss, N. H., Kiefer, R., Goncharenko, S., Raudales, A. M., Forkus, S. R., Schick, M. R., & Contractor, A. A. (2022) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2021.109131 · Cited by 235
Schuch, F. B., Vancampfort, D., Rosenbaum, S., Richards, J., Ward, P. B., Veronese, N., Solmi, M., Cadore, E. L., & Stubbs, B. (2016) · Revista Brasileira de Psiquiatria · doi:10.1590/1516-4446-2016-1915 · Cited by 226
Luger, T. M., Suls, J., & Vander Weg, M. W. (2014) · Addictive Behaviors · doi:10.1016/j.addbeh.2014.05.011 · Cited by 231
Aminov, A., Rogers, J. M., Middleton, S., Caeyenberghs, K., & Wilson, P. H. (2018) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-018-0370-2 · Cited by 236
“The current review specifically focused on published, peer-review articles to ensure the high quality of included data, but performed a fail-safe N [119] calculation to account for missing studies and grey literature.”
MethodRigorous Large-Scale Educational RCTs Are Often Uninformative: Should We Be Concerned?
Lortie-Forgues, H., & Inglis, M. (2019) · Educational Researcher · doi:10.3102/0013189x19832850 · Cited by 234
“Unfavorable findings from traditional research are less likely to be published (Rosenthal, 1979), and many researchers selectively report analyses and conduct unplanned analyses (John, Lowenstein, & Prelec, 2012).”
BackgroundResultInfluentialA meta-analytic investigation of virtuality and information sharing in teams
Mesmer-Magnus, J. R., DeChurch, L. A., Jimenez-Rodriguez, M., Wildman, J., & Shuffler, M. (2011) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2011.03.002 · Cited by 234
The relationship of trait emotional intelligence with academic performance: A meta-analytic review
Perera, H. N., & DiGiacomo, M. (2013) · Learning and Individual Differences · doi:10.1016/j.lindif.2013.08.002 · Cited by 234
Promoting memory consolidation during sleep: A meta-analysis of targeted memory reactivation
Hu, X., Cheng, L. Y., Chiu, M. H., & Paller, K. A. (2020) · Psychological Bulletin · doi:10.1037/bul0000223 · Cited by 232
“…vi du al us er an d is no t to be di ss em in at ed br oa dl y. Publication bias can arise when significant findings consistent with researchers’ hypotheses are more likely to be published than nonsignificant findings, which poses threats for accurately estimating effect sizes (Rosenthal, 1979).”
Shaffer, J. A., Edmondson, D., Wasson, L. T., Falzon, L., Homma, K., Ezeokoli, N., Li, P., & Davidson, K. W. (2014) · Psychosomatic Medicine · doi:10.1097/psy.0000000000000044 · Cited by 227
Tsai, A. C., & Burns, B. F. O. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.06.024 · Cited by 227
“…in our review in that all demonstrated statistically significant associations between one or more psychosocial problems and the outcomes of interest, our summary conclusions could have been driven by non-publication of non-statistically significant findings (Rosenthal, 1979; Sterling, 1959).”
ResultThe influence of delaying judgments of learning on metacognitive accuracy: a meta-analytic review
Rhodes, M. G., & Tauber, S. K. (2011) · Psychological Bulletin · doi:10.1037/a0021705 · Cited by 230
Pillinger, T., D’Ambrosio, E., McCutcheon, R., & Howes, O. D. (2019) · Molecular Psychiatry · doi:10.1038/s41380-018-0058-9 · Cited by 230
“Thus, for each meta-analysis, Rosenthal’s “fail-safe N” [38] was used to calculate the number of additional null studies (i.”
The five factor model of personality and emotion regulation: A meta-analysis
Barańczuk, U. (2019) · Personality and Individual Differences · doi:10.1016/j.paid.2018.11.025 · Cited by 225
Braun, S. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00773 · Cited by 229
“We acknowledge that on the downside, this decision can bias conclusions drawn from this review in the light of the file drawer problem (Rosenthal, 1979; McDaniel et al., 2006; Howard et al., 2009).”
BackgroundLimitations of PET-PEESE and Other Meta-Analysis Methods
Stanley, T. D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617693062 · Cited by 229
“For decades, publication bias has been widely acknowledged to be a serious, yet intractable, problem for empirical science (Begg & Berlin, 1988; Glass et al., 1981; Hedges & Oklin, 1985; Rosenthal, 1979; Schmidt & Hunter, 2014; Sterling, 1959).”
InfluentialMock juror sampling issues in jury simulation research: a meta-analysis.
Bornstein, B. H., Golding, J. M., Neuschatz, J., Kimbrough, C., Reed, K., Magyarics, C., & Luecht, K. (2017) · Law and Human Behavior · doi:10.1037/lhb0000223 · Cited by 227
Ferguson, C. J. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.11.002 · Cited by 295
Johannsen, M., Damholdt, M. F., Zachariae, R., Lundorff, M., Farver-Vestergaard, I., & O'Connor, M. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.04.065 · Cited by 214
Aday, J. S., Davis, A. K., Mitzkovitz, C. M., Bloesch, E. K., & Davoli, C. C. (2021) · ACS Pharmacology & Translational Science · doi:10.1021/acsptsci.1c00014 · Cited by 224
Renna, M. E., O'Toole, M. S., Spaeth, P. E., Lekander, M., & Mennin, D. S. (2018) · Depression and Anxiety · doi:10.1002/da.22790 · Cited by 220
“problem, defined as the possibility that unpublished studies with null findings may alter any significant findings, was evaluated by Rosenthal's fail-safe number (Rosenthal, 1979).”
BackgroundInfluentialLong-term effects of parental divorce on mental health – A meta-analysis
Auersperg, F., Vlasak, T., Ponocny, I., & Barth, A. (2019) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2019.09.011 · Cited by 223
Bias in grading: A meta-analysis of experimental research findings
Malouff, J. M., & Thorsteinsson, E. B. (2016) · Australian Journal of Education · doi:10.1177/0004944116664618 · Cited by 222
Seligman, L. D., Ollendick, T. H., Langley, A. K., & Baldacci, H. B. (2004) · Journal of Clinical Child & Adolescent Psychology · doi:10.1207/s15374424jccp3303_13 · Cited by 220
How to conduct a meta-analysis in eight steps: A practical guide
Hansen, C., Steinmetz, H., & Block, J. (2022) · Management Review Quarterly · doi:10.1007/s11301-021-00247-4 · Cited by 218
“This so-called publication bias or “file drawer problem” (Rosenthal 1979) results from the preference of academia to publish more statistically significant and less statistically insignificant study results.”
BackgroundMethodDykiert, D., Der, G., Starr, J. M., & Deary, I. J. (2012) · PLoS ONE · doi:10.1371/journal.pone.0045759 · Cited by 218
A century of research on conscientiousness at work
Wilmot, M. P., & Ones, D. S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1908430116 · Cited by 215
Publication bias as a threat to the validity of meta-analytic results
Rothstein, H. R. (2008) · Journal of Experimental Criminology · doi:10.1007/s11292-007-9046-9 · Cited by 216
Driessen, E., Hollon, S. D., Bockting, C. L. H., Cuijpers, P., & Turner, E. H. (2015) · PLOS ONE · doi:10.1371/journal.pone.0137864 · Cited by 208
Studying and Constructing Concept Maps: a Meta-Analysis
Schroeder, N. L., Nesbit, J. C., Anguiano, C. J., & Adesope, O. O. (2018) · Educational Psychology Review · doi:10.1007/s10648-017-9403-9 · Cited by 266
“Publication bias is an on-going concern in meta-analysis (Rosenthal, 1979).”
BackgroundBoele, S., Van der Graaff, J., de Wied, M., Van der Valk, I. E., Crocetti, E., & Branje, S. (2019) · Journal of Youth and Adolescence · doi:10.1007/s10964-019-00993-5 · Cited by 216
“In addition to the funnel plots, the statistic Rosen-thal ’ s Fail-safe N was calculated, which indicates how many missing studies would be required in order to fi nd a non-signi fi cant ( p > 0.05) correlation (Borenstein et al. 2011; Rosenthal 1979).”
InfluentialIverson, A. L., Marín, L. E., Ennis, K. K., Gonthier, D. J., Connor‐Barrie, B. T., Remfert, J. L., Cardinale, B. J., & Perfecto, I. (2014) · Journal of Applied Ecology · doi:10.1111/1365-2664.12334 · Cited by 214
Quantitative methods for literature reviews
Green, B. F., & Hall, J. A. (1984) · Annual Review of Psychology · doi:10.1146/annurev.ps.35.020184.000345 · Cited by 213
Morning cortisol levels in schizophrenia and bipolar disorder: a meta-analysis
Girshkin, L., Matheson, S. L., Shepherd, A. M., & Green, M. J. (2014) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2014.07.013 · Cited by 215
Kim, J. S., & Quinn, D. M. (2013) · Review of Educational Research · doi:10.3102/0034654313483906 · Cited by 213
The Effect of Sugar on Behavior or Cognition in Children: A Meta-analysis
Wolraich, M. L. (1995) · JAMA · doi:10.1001/jama.1995.03530200053037 · Cited by 214
Cregg, D. R., & Cheavens, J. S. (2021) · Journal of Happiness Studies · doi:10.1007/s10902-020-00236-6 · Cited by 203
“Rosenthal (1979) suggested a fail-safe N value above 5 k + 10 reflects results that are tolerant to contradicting studies, where k is the number of studies included in the meta-analysis.”
Backgroundvan den Berk-Clark, C., Secrest, S., Walls, J., Hallberg, E., Lustman, P. J., Schneider, F. D., & Scherrer, J. F. (2018) · Health Psychology · doi:10.1037/hea0000593 · Cited by 212
A meta-analysis of sex differences in human navigation skills
Nazareth, A., Huang, X., Voyer, D., & Newcombe, N. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01633-6 · Cited by 201
“In such a case, it is often assumed that the final sample might not be representative of the entire population of studies in existence (Rosenthal, 1979).”
BackgroundEndophenotypes in schizophrenia: A selective review
Allen, A. J., Griss, M. E., Folley, B. S., Hawkins, K. A., & Pearlson, G. D. (2009) · Schizophrenia Research · doi:10.1016/j.schres.2009.01.016 · Cited by 210
Leyland, A., Rowse, G., & Emerson, L.-M. (2019) · Emotion · doi:10.1037/emo0000425 · Cited by 206
The relationship between adult attachment style and post-traumatic stress symptoms: A meta-analysis
Woodhouse, S., Ayers, S., & Field, A. P. (2015) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2015.07.002 · Cited by 210
“Publication bias refers to the decision to publish a paper based on the results of that paper rather than basing the decision on the overall standard of research (Rosenthal, 1979).”
BackgroundThe limited prosocial effects of meditation: A systematic review and meta-analysis
Kreplin, U., Farias, M., & Brazil, I. A. (2018) · Scientific Reports · doi:10.1038/s41598-018-20299-z · Cited by 211
How Robust Are Probabilistic Models of Higher-Level Cognition?
Marcus, G. F., & Davis, E. (2013) · Psychological Science · doi:10.1177/0956797613495418 · Cited by 195
Creativity and Well‐being: A Meta‐analysis
Acar, S., Tadik, H., Myers, D., van der Sman, C., & Uysal, R. (2021) · The Journal of Creative Behavior · doi:10.1002/jocb.485 · Cited by 208
Davis, M. C., Matthews, K. A., & Twamley, E. W. (1999) · Annals of Behavioral Medicine · doi:10.1007/bf02895038 · Cited by 208
Effects of UVB radiation on marine and freshwater organisms: a synthesis through meta‐analysis
Bancroft, B. A., Baker, N. J., & Blaustein, A. R. (2007) · Ecology Letters · doi:10.1111/j.1461-0248.2007.01022.x · Cited by 205
Scott-Sheldon, L. A. J., Kalichman, S. C., Carey, M. P., & Fielder, R. L. (2008) · Health Psychology · doi:10.1037/0278-6133.27.2.129 · Cited by 204
A meta-analytic review of cognitive processing therapy for adults with posttraumatic stress disorder
Asmundson, G. J. G., Thorisdottir, A. S., Roden-Foreman, J. W., Baird, S. O., Witcraft, S. M., Stein, A. T., Smits, J. A. J., & Powers, M. B. (2019) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2018.1522371 · Cited by 207
“Lastly, Rosenthal’s fail-safe N (Rosenthal, 1979; Borenstein, Hedges, Higgings, & Rothstein, 2009) indicates the number of studies with no significant effect needed to raise the p-value for the mean effect size above 0.”
BackgroundInfluentialJenkinson, R., Milne, E., & Thompson, A. (2020) · Autism · doi:10.1177/1362361320932437 · Cited by 205
“Fail-safe analysis (Rosenthal, 1979) was conducted to aid this assessment by quantifying the number of studies that would be required to invalidate the effect (Borenstein et al.”
MethodStar Wars: The Empirics Strike Back
Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. (2016) · American Economic Journal: Applied Economics · doi:10.1257/app.20150044 · Cited by 430
Moving from cognition to behavior: What the research says.
Johnson, R. E., Chang, C.-H., & Lord, R. G. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.3.381 · Cited by 193
Scott-Sheldon, L. A. J., Carey, K. B., Elliott, J. C., Garey, L., & Carey, M. P. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0035192 · Cited by 198
Lau, N., O'Daffer, A., Colt, S., Yi-Frazier, J. P., Palermo, T. M., McCauley, E., & Rosenberg, A. R. (2020) · JMIR mhealth and uhealth · doi:10.2196/17798 · Cited by 174
“However, the file-drawer problem [68] in academic research (ie, studies”
BackgroundIyer, N. S., Balsamo, L. M., Bracken, M. B., & Kadan-Lottick, N. S. (2015) · Blood · doi:10.1182/blood-2015-02-627414 · Cited by 194
Implicit cognition and depression: A meta-analysis
Phillips, W. J., Hine, D. W., & Thorsteinsson, E. B. (2010) · Clinical Psychology Review · doi:10.1016/j.cpr.2010.05.002 · Cited by 182
Meta-analysis of the effects of intranasal oxytocin on interpretation and expression of emotions
Leppanen, J., Ng, K. W., Tchanturia, K., & Treasure, J. (2017) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.04.010 · Cited by 188
“Publication bias was investigated with Begg's rank correlation test for funnel plot asymmetry (Begg and Mazumdar, 1994) and where significant effects were present Rosenthal’s file drawer analysis was also conducted (Rosenthal, 1979).”
Cai, Z., Mao, P., Wang, Z., Wang, D., He, J., & Fan, X. (2023) · Adolescent Research Review · doi:10.1007/s40894-022-00201-9 · Cited by 192
“In metaanalysis research literature, publication bias was assumed that studies reporting statistically significant results were easier to get published than those with statistically non-significant findings, and this is also called “file-drawer problem” (Rosenthal, 1979).”
BackgroundHarkness, E., Macdonald, W., Valderas, J., Coventry, P., Gask, L., & Bower, P. (2010) · Diabetes Care · doi:10.2337/dc09-1519 · Cited by 188
Boemo, T., Nieto, I., Vazquez, C., & Sanchez-Lopez, A. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104747 · Cited by 191
“Publication bias was analyzed by inspecting the funnel plot, which represents the precision of each primary study (standard error; SE) against its effect size (ES; Rosenthal, 1979).”
MethodRith-Najarian, L. R., Boustani, M. M., & Chorpita, B. F. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.06.035 · Cited by 176
Suicidality in obsessive compulsive disorder (OCD): A systematic review and meta-analysis
Angelakis, I., Gooding, P., Tarrier, N., & Panagioti, M. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.03.002 · Cited by 185
“Publication bias was examined using a test of funnel plot asymmetry (Egger’s test) (Egger, Smith, Schneider, & Minder, 1997)and the Rosenthal's fail safe N (FSN) (Rosenthal, 1979).”
MethodSong, F., Parekh-Bhurke, S., Hooper, L., Loke, Y. K., Ryder, J. J., Sutton, A. J., Hing, C. B., & Harvey, I. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-79 · Cited by 175
Effectiveness of grief therapy: A meta-analysis.
Allumbaugh, D. L., & Hoyt, W. T. (1999) · Journal of Counseling Psychology · doi:10.1037/0022-0167.46.3.370 · Cited by 185
ALE meta-analysis on facial judgments of trustworthiness and attractiveness
Bzdok, D., Langner, R., Caspers, S., Kurth, F., Habel, U., Zilles, K., Laird, A., & Eickhoff, S. B. (2011) · Brain Structure and Function · doi:10.1007/s00429-010-0287-4 · Cited by 183
A Meta-Analysis of Extensive Reading Research
Nakanishi, T. (2015) · TESOL Quarterly · doi:10.1002/tesq.157 · Cited by 184
Gallegos, A. M., Crean, H. F., Pigeon, W. R., & Heffner, K. L. (2017) · Clinical Psychology Review · doi:10.1016/j.cpr.2017.10.004 · Cited by 181
“The Duval and Tweedie (2000) trim and fill technique as well as the Rosenthal (1979) fail-safe N was applied as a form of sensitivity analysis adjusting for possible publication bias and missing studies.”
Dotson, K. B., Dunn, M. E., & Bowers, C. A. (2015) · PLOS ONE · doi:10.1371/journal.pone.0139518 · Cited by 181
“12 Publication Bias I conducted fail-safe analyses to estimate the number of missing studies with null findings that would nullify the observed summary effect (Rosenthal, 1979).”
BackgroundMethodZheng, L., Niu, J., Zhong, L., & Gyasi, J. F. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2021.2015693 · Cited by 183
“Fifth, publication bias was examined through classic fail-safe N (Rosenthal, 1979) and Orwin ’ s fail-safe N (Orwin, 1983).”
Cue-reactivity in behavioral addictions: a meta-analysis and methodological considerations
Starcke, K., Antons, S., Trotzke, P., & Brand, M. (2018) · Journal of Behavioral Addictions · doi:10.1556/2006.7.2018.39 · Cited by 180
“As a fi rst rough indicator of publication bias, the fail-safe N procedure by Rosenthal (1979) was used.”
MethodChandler, M. J., Parks, A. C., Marsiske, M., Rotblatt, L. J., & Smith, G. E. (2016) · Neuropsychology Review · doi:10.1007/s11065-016-9330-4 · Cited by 169
Joslyn, C., Hawes, D. J., Hunt, C., & Mitchell, P. B. (2016) · Bipolar Disorders · doi:10.1111/bdi.12419 · Cited by 179
Intolerance of uncertainty, anxiety, and worry in children and adolescents: A meta-analysis
Osmanağaoğlu, N., Creswell, C., & Dodd, H. F. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.07.035 · Cited by 177
Goldberg, S. B., Tucker, R. P., Greene, P. A., Davidson, R. J., Kearney, D. J., & Simpson, T. L. (2019) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2018.1556330 · Cited by 171
Toward a better understanding of behavioral intention and system usage constructs
Wu, J., & Du, H. (2012) · European Journal of Information Systems · doi:10.1057/ejis.2012.15 · Cited by 175
Cassiello-Robbins, C., Southward, M. W., Tirpak, J. W., & Sauer-Zavala, S. (2020) · Clinical Psychology Review · doi:10.1016/j.cpr.2020.101852 · Cited by 179
“That is, it is not possible to know how many studies were conducted but not reported (Rosenthal, 1979).”
BackgroundAffective divergence: Automatic responses to others' emotions depend on group membership.
Weisbuch, M., & Ambady, N. (2008) · Journal of Personality and Social Psychology · doi:10.1037/a0011993 · Cited by 176
A systematic literature review of automatic Alzheimer’s disease detection from speech and language
Petti, U., Baker, S., & Korhonen, A. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa174 · Cited by 168
“65,66 Although publication bias was undetected in the current review, it is especially common in literature reviews written in the early stages of the specific research area due to negative studies being delayed 66 and should therefore be mentioned.”
BackgroundSpence, C., & Velasco, C. (2018) · Food Quality and Preference · doi:10.1016/j.foodqual.2018.03.008 · Cited by 177
“The challenge, therefore, is to 15 And, as mentioned earlier, one should also be aware that there may be a file drawer problem here (Rosenthal, 1979), with null results of packaging colour on product experience staying in the filing draw rather than making it into the public domain. determine which…”
BackgroundInfluentialLo, C. K., & Hew, K. F. (2019) · Journal of Engineering Education · doi:10.1002/jee.20293 · Cited by 174
Relationship between work–family conflict and turnover intention in nurses: A meta‐analytic review
Yildiz, B., Yildiz, H., & Ayaz Arda, O. (2021) · Journal of Advanced Nursing · doi:10.1111/jan.14846 · Cited by 172
Hopwood, T. L., & Schutte, N. S. (2017) · Clinical Psychology Review · doi:10.1016/j.cpr.2017.08.002 · Cited by 167
Metadata
- Authors
- Rosenthal, R.
- Affiliations
- Harvard University (US)
- Corresponding author
- Rosenthal, Robert
- Venue
- Psychological Bulletin
- Published
- 1979-05
- Volume
- 86
- Issue
- 3
- Pages
- 638-641
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2006546769
- Cited by
- 9,068
- References
- 16
- Citations by year
- 2026: 421 · 2025: 521 · 2024: 509 · 2023: 526 · 2022: 525 · 2021: 536 · 2020: 612 · 2019: 607 · 2018: 492 · 2017: 436 · 2016: 417 · 2015: 392 · 2014: 337 · 2013: 289 · 2012: 252
- FWCI
- 26.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- file drawer problem, negative results, publication bias, non-significant results, type I error, selective reporting, behavioral science, evidence synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
