The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
The file drawer problem and tolerance for null results.
Abstract
For any given research area, one cannot tell how many studies have been con-ducted but never reported. The extreme view of the "file drawer problem " is that journals are filled with the 5 % of the studies that show Type I errors, while the file drawers are filled with the 95 % of the studies that show non-significant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed. Both behavioral researchers and statisti-cians have long suspected that the studies published in the behavioral sciences are a biased sample of the studies that are actually carried out (Bakan, 1967; McNemar, 1960; Smart, 1964; Sterling, 1959). The extreme view of this problem, the "file drawer prob-lem, " is that the journals are filled with the 5 % of the studies that show Type I errors, while the file drawers back at the lab are filled with the 95 % of the studies that show nonsignificant (e.g., p>.05) results. In the past there was very little one could do to assess the net effect of studies, tucked away in file drawers, that did not make the magic.05 level (Rosenthal & Gaito, 1963, 1964). Now, however, although no definitive solution to the problem is available, one can establish reasonable boundaries on the prob-lem and estimate the degree of damage to any research conclusion that could be done by the file drawer problem. This advance in our ability to cope with the file drawer is an outgrowth of the in-creasing interest of behavioral scientists in summarizing bodies of research literature sys-Preparation of this article was supported in part by the National Science Foundation. I would like to thank Judith A. Hall and Donald B. Rubin for their valuable improvements of an earlier version of this article. Requests for reprints should be sent to Robert
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We then calculated Rosenthal’s fail-safe N (Rosenthal, 1979), and determined that 2,153,789 effects evidencing no relationship between SA and psychopathology (g = 0) would be needed to reduce the significance level of the summary effect size to .05.”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“The file drawer effect is a perennial problem in the academic literature (Rosenthal, 1979)”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“First, we calculated the fail-safe N [177] to be 4,274, which is the theoretical number of unpublished studies with effect sizes averaging zero (no effect) that would be needed to render negligible the omnibus results.”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [177]
In the citing paper
“As we did not include unpublished work in our meta-analyses we tested for publication bias in the following ways.Firstly, funnel plots were examined.demonstrating no systematic publication biases (tables available from first author on request).Secondly..Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Presence of publication bias was assessed with funnel plots, 22 Rosenthal's fail-safe N , 23 and Egger's regression intercept. 24”
From Light Therapy for Patients With Bipolar Depression: Systematic Review and Meta-Analysis of Randomized Controlled Trials (2020) · cites this paper as [23]
In the citing paper
“In terms of study quality, there was some indication of publication bias whereby some associations between racism and health outcomes were stronger in published versus unpublished studies, consistent with the tendency for ‘null’ studies to remain unpublished (i.e., the ‘file drawer’ problem [418]).”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as [418]
In the citing paper
“An alternative, perhaps more plausible, explanation is that there could be a significant amount of unpublished negative or inconclusive results, a phenomenon referred to as the “file-drawer effect” or publication bias (23).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [23]
In the citing paper
“If either of the two last hypotheses is true, there should exist many unpublished studies with null results lying in laboratories' drawers (32).”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Publication bias was evaluated using Rosenthal's file drawer analysis (Rosenthal, 1979) and Egger's linear regression (Card, 2012), and applying ANOVA to compare the mean effect size of the published versus unpublished studies.”
From Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal (1979) suggested a fail-safe N value above 5; k + 10 reflects results that are tolerant to contradicting studies”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“We applied random-effects models to fit the relations between CE and the criterion variables, following the guidelines proposed by Rosenthal (1979), Hunter and Schmidt (2004) and Cumming (2013).”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal (1979)
In the citing paper
“The fail-safe value for this meta-analysis was computed to be 3001, which is more than 5(36) + 10.”
From Effects of digital game-based STEM education on students’ learning achievement: a meta-analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, we suggest our results may look like a lot of studies that ended up in the so-called “file drawer” [57].”
From Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory (2023) · cites this paper as [57]
In the citing paper
“Finally, publication bias was addressed by computing the fail-safe N for the correlation coefficients (Hunter & Schmidt, 2004;Rosenthal, 1979), which reports the number of field effect sizes or the tolerance for future null results, to check whether a publication bias affected our results.”
From Drivers of Consumer Adoption of e-Commerce: a Meta-Analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Using Rosenthal’s (1979) fail-safe N criterion that the value should be over five times greater than the number of studies included in the meta-analysis, the fail-safe N for all outcome groups were at least eight times larger than the criterion value, suggesting that the effect sizes are unlikely to be an artifact of bias.”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as Rosenthal (1979)
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, Rosenthal (1979) opined that a Fail-safe N exceeding the number of studies by fivefold would provide strong support for the robustness of the significant effects.”
From Extending the Theory of Planned Behavior in the Social Commerce Context: A Meta-Analytic SEM (MASEM) Approach (2023) · cites this paper as Rosenthal (1979)
In the citing paper
“The trim-and-fill method (Duval & Tweedie, 2000) tends to adjust too little when bias is strong (Carter et al., 2019), and Fail-Safe N (Rosenthal, 1979) neither tests nor adjusts for the presence of bias.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Rosenthal, 1979
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, a phenomenon known as ' publication bias ' (Rosenthal 1979).”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“However, a common problem is that results may not have been reported in dissertations or research reports, or published in scienti fi c (peer-reviewed) journals, due to non-signi fi cant or unfavorable fi ndings and, therefore, are dif fi cult to locate.”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [21]
In the citing paper
“The traditional fail-safe value is 1904, which is higher than the threshold value of 100, i.e., [5(18) + 10], indicating that publication bias is insignificant, and the study's results are reliable.”
From A meta-analysis to explore the role of design thinking in enhancing creativity as learning outcomes in STEM education (2026) · cites this paper as Rosenthal, 1979
In the citing paper
“The Q statistic was used to assess heterogeneity30, and publication bias was estimated using Rosenthal’s fail-safe number with α = 0.0531.”
From The limited prosocial effects of meditation: A systematic review and meta-analysis (2018) · cites this paper as [31]
In the citing paper
“Rosenthal's fail-safe number (NR) was calculated, estimating the number of unpublished studies with non-significant results needed to bring an observed result to non-significance (NR ⩾ 5k + 10 to rule out a file drawer problem) (Rosenthal, 1979).”
From Efficacy of non-invasive brain stimulation on cognitive functioning in brain disorders: a meta-analysis (2020) · cites this paper as Rosenthal, 1979
In the citing paper
“In case of statistically significant effect sizes, it was planned to examine the robustness of findings by calculating the Failsafe N (number of non‐significant studies that would bring the p‐value to non‐significant (p > 0.05)) 40.”
From Cognitive behavioral therapies for informal caregivers of patients with cancer and cancer survivors: a systematic review and meta-analysis (2017) · cites this paper as [40]
In the citing paper
“With negative results buried in file drawers (55), conclusions drawn from the published record can be misleading, and 'false facts' can become canonized in the literature (56).”
From Misinformation in and about science (2021) · cites this paper as [55]
In the citing paper
“the fail-safe N (i.e. the file drawer statistic) was used to determine how many additional studies would be necessary to turn a significant ES non-significant (32).”
From Peripheral cytokine and chemokine alterations in depression: a meta‐analysis of 82 studies (2017) · cites this paper as [32]
In the citing paper
“File drawer analysis (Rosenthal, 1979) revealed that 1919 unpublished, filed, or unretrieved studies averaging null results would be required to bring the average unweighted effect size to nonsignificance.”
From Happy to help? A systematic review and meta-analysis of the effects of performing acts of kindness on the well-being of the actor (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's (1979) fail-safe N analysis reveals 186 unpublished papers (i.e., null effect) are needed to be included in the meta-analysis in order to invalidate the population effect size of small to medium (i.e., .235)found in this review.”
From A meta-analytic review of the efficacy of physical exercise interventions on cognition in individuals with autism spectrum disorder and ADHD (2016) · cites this paper as Rosenthal's (1979)
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, the so called ' publication bias ' (Rosenthal, 1979).”
From The effects of physical activity interventions on psychosocial outcomes in adolescents: a meta-analytic review (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's fail-safe N 37 calculated 115 missing studies would be required to increase the two-tailed p-value to above .05.”
From Anxiety in Informal Dementia Carers: A Meta-Analysis of Prevalence (2020) · cites this paper as [37]
In the citing paper
“Publication bias was investigated with Begg's rank correlation test for funnel plot asymmetry (Begg and Mazumdar, 1994) and where significant effects were present Rosenthal's file drawer analysis was also conducted (Rosenthal, 1979).”
From Meta-analysis of the effects of intranasal oxytocin on interpretation and expression of emotions (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“Over time, this practice artificially inflates Type I error and may make it appear that relationships exist when in reality they do not (Rosenthal et al., 1979).”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“However, a potential problem with meta-analysis is the file drawer problem, where studies are “filed away” unpublished if they fail to find statistically significant relationships.”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“Meta-analyses are necessarily based on the available literature and hence may be affected by the potential publication-bias disfavoring null results (Rosenthal 1979).”
From Parsing the neural correlates of moral cognition: ALE meta-analysis on morality, theory of mind, and empathy (2012) · cites this paper as Rosenthal 1979
In the citing paper
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
From The Extent and Consequences of P-Hacking in Science (2015) · cites this paper as [7]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G. E., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,518
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718 · cited 1×
“Perhaps the simplest, most useful way of computing the overall p of a, set of research studies is the method of adding Zs (Cochran, 1954; Mosteller & Bush, 1954; Rosenthal, 1978).”
MethodMeta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154
Experimenter Effects in Behavioral Research
McGuigan, F. J., & Rosenthal, R. (1968) · The American Journal of Psychology · doi:10.2307/1420816 · Cited by 1,633
Self-citationArtifact in behavioral research
Rosenthal, R., & Rosnow, R. L. (1969) · Medical Entomology and Zoology · Cited by 1,186
Self-citationSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964
Combining results of independent studies.
Rosenthal, R. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.1.185 · Cited by 910
Self-citationGender effects in decoding nonverbal cues.
Hall, J. A. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.4.845 · Cited by 766
Wortman, C. B., Rosenthal, R., & Rosnow, R. L. (1976) · The American Journal of Psychology · doi:10.2307/1421418 · Cited by 428
Self-citation
Cited by
Sorted by citations
Items 501–600 of 1,374
Patterns of within‐species body size variation of birds: strong evidence for Bergmann's rule
Ashton, K. G. (2002) · Global Ecology and Biogeography · doi:10.1046/j.1466-822x.2002.00313.x · Cited by 464
The actor-observer asymmetry in attribution: A (surprising) meta-analysis.
Malle, B. F. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.6.895 · Cited by 486
Khaleque, A., & Rohner, R. P. (2002) · Journal of Marriage and the Family · doi:10.1111/j.1741-3737.2002.00054.x · Cited by 465
Customer engagement in social media: a framework and meta-analysis
de Oliveira Santini, F., Ladeira, W. J., Pinto, D. C., Herter, M. M., Sampaio, C. H., & Babin, B. J. (2020) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-020-00731-5 · Cited by 523
A Meta-Analysis of Maternal Prenatal Depression and Anxiety on Child Socioemotional Development
Madigan, S. L., Oatley, H., Racine, N. M., Fearon, P. R. M., Schumacher, L., Akbari, E. M., Cooke, J. E., & Tarabulsy, G. M. (2018) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2018.06.012 · Cited by 456
Investigating and Dealing with Publication and Other Biases
Sterne, J. A. C., Egger, M., & Smith, G. D. (2001) · Systematic Reviews in Health Care · doi:10.1002/9780470693926.ch11 · Cited by 449
Behavioral activation treatments for depression in adults: A meta-analysis and review.
Mazzucchelli, T., Kane, R., & Rees, C. (2009) · Clinical Psychology: Science and Practice · doi:10.1111/j.1468-2850.2009.01178.x · Cited by 438
Relationships of role stressors with organizational citizenship behavior: A meta-analysis.
Eatough, E. M., Chang, C.-H. D., Miloslavic, S. A., & Johnson, R. E. (2011) · Journal of Applied Psychology · doi:10.1037/a0021887 · Cited by 498
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 484
Testing a continuum structure of self-determined motivation: A meta-analysis.
Howard, J. L., Gagné, M., & Bureau, J. S. (2017) · Psychological Bulletin · doi:10.1037/bul0000125 · Cited by 447
Bayesian Analysis Reporting Guidelines
Kruschke, J. K. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01177-7 · Cited by 525
Physical activity protects from incident anxiety: A meta‐analysis of prospective cohort studies
Schuch, F. B., Stubbs, B., Meyer, J., Heissel, A., Zech, P., Vancampfort, D., Rosenbaum, S., Deenik, J., Firth, J., Ward, P. B., et al. (2019) · Depression and Anxiety · doi:10.1002/da.22915 · Cited by 427
“05) the effect size was calculated (Rosenthal, 1979).”
WANG, P. (1993) · The Lancet · doi:10.1016/0140-6736(93)90816-y · Cited by 446
Economic Shocks and Conflict: Evidence from Commodity Prices
Bazzi, S., & Blattman, C. (2013) · American Economic Journal Macroeconomics · doi:10.1257/mac.6.4.1 · Cited by 455
Talsma, K., Schüz, B., Schwarzer, R., & Norris, K. (2017) · Learning and Individual Differences · doi:10.1016/j.lindif.2017.11.015 · Cited by 544
Erickson, M. A., Ruffle, A., & Gold, J. M. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.08.025 · Cited by 426
“These three calculations are fail-safe N (28), the trim and fill method to correct for nonnormal distribution of effect size (29), and Kendall tau.”
MethodKawabata, Y., Alink, L. R. A., Tseng, W.-L., van IJzendoorn, M. H., & Crick, N. R. (2011) · Developmental Review · doi:10.1016/j.dr.2011.08.001 · Cited by 445
Ants on plants: a meta-analysis of the role of ants as plant biotic defenses
Rosumek, F. B., Silveira, F. A. O., Neves, F. S., Barbosa, N. P. U., Diniz, L., Oki, Y., Pezzini, F. F., Fernandes, G. W., & Cornelissen, T. (2009) · Oecologia · doi:10.1007/s00442-009-1309-x · Cited by 444
Vanhove, A. J., Herian, M. N., Perez, A. L. U., Harms, P. D., & Lester, P. B. (2016) · Journal of Occupational and Organizational Psychology · doi:10.1111/joop.12123 · Cited by 443
“A potential issue when conducting meta-analyses is an upward bias due to primary research showing significant effects being more likely to be published, and subsequently included in meta-analyses, than primary research showing non-significant effects (Rosenthal, 1979).”
BackgroundDuPaul, G. J., Eckert, T. L., & Vilardo, B. (2012) · School Psychology Review · doi:10.1080/02796015.2012.12087496 · Cited by 439
A meta-analytic review of age differences in theory of mind.
Henry, J. D., Phillips, L. H., Ruffman, T., & Bailey, P. E. (2012) · Psychology and Aging · doi:10.1037/a0030677 · Cited by 441
STILING, P. D., & Cornelissen, T. (2007) · Global Change Biology · doi:10.1111/j.1365-2486.2007.01392.x · Cited by 441
A reinterpretation of parental monitoring in longitudinal perspective
Kerr, M., Stattin, H., & Burk, W. J. (2010) · Journal of Research on Adolescence · doi:10.1111/j.1532-7795.2009.00623.x · Cited by 621
A Meta-Analysis of Predictors of Postpartum Depression
Beck, C. T. (1996) · Nursing Research · doi:10.1097/00006199-199609000-00008 · Cited by 440
Getting on top of the glass cliff: Reviewing a decade of evidence, explanations, and impact
Ryan, M. K., Haslam, S. A., Morgenroth, T., Rink, F., Stoker, J., & Peters, K. (2016) · The Leadership Quarterly · doi:10.1016/j.leaqua.2015.10.008 · Cited by 439
“Moreover, the “ fi le-drawer ” phenomenon suggests that there are likely to be an even greater number of studies with non-signi fi cant results that have not been published (Rosenthal, 1979).”
Philpott, S. M., Arendt, W. J., Armbrecht, I., Bichier, P., DIESTCH, T. V., Gordon, C. E., Greenberg, R., Perfecto, I., Santos, R. R., Soto‐Pinto, L., et al. (2008) · Conservation Biology · doi:10.1111/j.1523-1739.2008.01029.x · Cited by 440
A meta-analysis of online trust relationships in e-commerce.
Kim, Y., & Peterson, R. A. (2017) · Journal of Interactive Marketing · doi:10.1016/j.intmar.2017.01.001 · Cited by 541
GELLATLY, J., BOWER, P., HENNESSY, S., RICHARDS, D., GILBODY, S., & LOVELL, K. (2007) · Psychological Medicine · doi:10.1017/s0033291707000062 · Cited by 419
Differences between trait fear and trait anxiety: Implications for psychopathology
Sylvers, P. D., Lilienfeld, S. O., & LaPrairie, J. L. (2010) · Clinical Psychology Review · doi:10.1016/j.cpr.2010.08.004 · Cited by 475
The Effects of Negative Political Advertisements: A Meta-Analytic Assessment
Lau, R. R., Sigelman, L., Heldman, C., & Babbitt, P. (1999) · American Political Science Review · doi:10.2307/2586117 · Cited by 432
Putative sex differences in verbal abilities and language cortex: A critical review
Wallentin, M. (2008) · Brain and Language · doi:10.1016/j.bandl.2008.07.001 · Cited by 438
EFFECTS OF REALISTIC JOB PREVIEWS ON MULTIPLE ORGANIZATIONAL OUTCOMES: A META-ANALYSIS.
Phillips, J. M. (1998) · Academy of Management Journal · doi:10.2307/256964 · Cited by 432
Bullied Children and Psychosomatic Problems: A Meta-analysis
Gini, G., & Pozzoli, T. (2013) · Pediatrics · doi:10.1542/peds.2013-0614 · Cited by 432
Jeong, S., & Harrison, D. A. (2016) · Academy of Management Journal · doi:10.5465/amj.2014.0716 · Cited by 459
Estes, K. G., Evans, J. L., & Else-Quest, N. M. (2007) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2007/015 · Cited by 430
A meta‐analysis of positive humor in the workplace
Mesmer‐Magnus, J., Glew, D. J., & Viswesvaran, C. (2012) · Journal of Managerial Psychology · doi:10.1108/02683941211199554 · Cited by 428
Overcoming the limitations of current meta-analysis of randomised controlled trials
Pogue, J., & Yusuf, S. (1998) · The Lancet · doi:10.1016/s0140-6736(97)08461-4 · Cited by 427
Can emotional intelligence be trained? A meta-analytical investigation
Mattingly, V. P., & Kraiger, K. (2018) · Human Resource Management Review · doi:10.1016/j.hrmr.2018.03.002 · Cited by 516
“Thus, by only including studies we found within databases and through email list-serves requests of unpublished papers, we may have contributed to the “file-drawer problem” of not including null, yet valid, findings (Rosenthal, 1979).”
BackgroundA Systematic Review of Re-Identification Attacks on Health Data
El Emam, K., Jonker, E., Arbuckle, L., & Malin, B. (2011) · PLoS ONE · doi:10.1371/journal.pone.0028071 · Cited by 427
Publication and related biases.
Song, Eastwood, Gilbody, Duley, & Sutton (2000) · Health Technology Assessment · doi:10.3310/hta4100 · Cited by 427
Self-efficacy and smoking cessation: A meta-analysis.
Gwaltney, C. J., Metrik, J., Kahler, C. W., & Shiffman, S. (2009) · Psychology of Addictive Behaviors · doi:10.1037/a0013529 · Cited by 419
“N IH -PA Author M anuscript N IH -PA Author M anuscript N IH -PA Author M anuscript fail-safe N was calculated (Rosenthal, 1979; Orwin, 1983).”
MethodA meta-analytic review of the Penn Resiliency Program’s effect on depressive symptoms.
Brunwasser, S. M., Gillham, J. E., & Kim, E. S. (2009) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0017671 · Cited by 428
Pinquart, M., & Kauser, R. (2017) · Cultural Diversity & Ethnic Minority Psychology · doi:10.1037/cdp0000149 · Cited by 458
Gurevitch, J., Curtis, P. S., & Jones, M. H. (2001) · Advances in Ecological Research · doi:10.1016/s0065-2504(01)32013-5 · Cited by 421
Sockol, L. E. (2015) · Journal of Affective Disorders · doi:10.1016/j.jad.2015.01.052 · Cited by 395
“Publication bias was assessed by visual examination of funnel plots, Duval and Tweedie's (2000) trim-and-fill procedure, and classic fail-safe N values (Rosenthal, 1979).”
MethodKnowles, K. A., & Olatunji, B. O. (2020) · Clinical Psychology Review · doi:10.1016/j.cpr.2020.101928 · Cited by 533
“Next, Rosenthal’s fail-safe N was calculated to determine the number of additional studies with an effect size of zero needed to reduce the effect size of the meta-analysis to non-significance (p > 0.05; Rosenthal, 1979).”
MethodInfluentialSELF‐CONTROL AND VICTIMIZATION: A META‐ANALYSIS
Pratt, T. C., Turanovic, J. J., Fox, K. A., & Wright, K. A. (2013) · Criminology · doi:10.1111/1745-9125.12030 · Cited by 440
Fox, M. C., Ericsson, K. A., & Best, R. (2010) · Psychological Bulletin · doi:10.1037/a0021663 · Cited by 468
Open MEE : Intuitive, open‐source software for meta‐analysis in ecology and evolutionary biology
Wallace, B. C., Lajeunesse, M. J., Dietz, G., Dahabreh, I. J., Trikalinos, T. A., Schmid, C. H., & Gurevitch, J. (2017) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12708 · Cited by 421
“2), Egger’s regression-based test (Egger et al. 1997), and the ‘failsafe N’ method (Rosenthal 1979).”
MethodInfluentialWeisz, J. R., Kuppens, S., Eckshtain, D., Ugueto, A. M., Hawley, K. M., & Jensen-Doss, A. (2013) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2013.1176 · Cited by 412
McShane, B. B., Böckenholt, U., & Hansen, K. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616662243 · Cited by 398
“…& Mazumdar, 1994; Egger, Smith, Schneider, & Minder, 1997; Sterne & Egger, 2005), Rosenthal’s fail-safe N method and its many variants (Becker, 2005; Rosenthal, 1979), the trim-and-fill method (Duval, 2005; Duval & Tweedie, 2000a, 2000b), and the Precision-Effect Test– Precision-Effect Estimate…”
Moderators of nonverbal indicators of deception: A meta-analytic synthesis.
Sporer, S. L., & Schwandt, B. (2007) · Psychology Public Policy and Law · doi:10.1037/1076-8971.13.1.1 · Cited by 449
Woon, F. L., & Hedges, D. W. (2008) · Hippocampus · doi:10.1002/hipo.20437 · Cited by 416
Associations between job burnout and self-efficacy: a meta-analysis
Shoji, K., Cieślak, R., Smoktunowicz, E., Rogala, A., Benight, C. C., & Łuszczyńska, A. (2015) · Anxiety Stress & Coping · doi:10.1080/10615806.2015.1058369 · Cited by 497
“To address the file drawer problem, robustness of the calculated estimate of the average effect against the effect of unpublished null results was assessed using the fail-safe N test (Rosenthal, 1979).”
MethodDisclosure of sensitive behaviors across self-administered survey modes: a meta-analysis
Gnambs, T., & Kaspar, K. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0533-4 · Cited by 422
Sex differences in human motor activity level.
Eaton, W. O., & Enns, L. R. (1986) · Psychological Bulletin · doi:10.1037/0033-2909.100.1.19 · Cited by 418
Testing and adjusting for publication bias
Møller, A. P., & Jennions, M. D. (2001) · Trends in Ecology & Evolution · doi:10.1016/s0169-5347(01)02235-2 · Cited by 417
Sibling relationship quality and psychopathology of children and adolescents: A meta-analysis
Buist, K. L., Deković, M., & Prinzie, P. (2012) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.10.007 · Cited by 417
Exogenous (automatic) attention to emotional stimuli: a review
Carretié, L. (2014) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-014-0270-2 · Cited by 456
The effects of apolipoprotein E on non-impaired cognitive functioning: A meta-analysis
Wisdom, N. M., Callahan, J. L., & Hawkins, K. A. (2009) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2009.02.003 · Cited by 429
Appearance-related teasing, body dissatisfaction, and disordered eating: A meta-analysis
Menzel, J. E., Schaefer, L. M., Burke, N. L., Mayhew, L. L., Brannick, M. Τ., & Thompson, J. K. (2010) · Body Image · doi:10.1016/j.bodyim.2010.05.004 · Cited by 453
Macnamara, B. N., & Burgoyne, A. P. (2023) · Psychological Bulletin · doi:10.1037/bul0000352 · Cited by 415
“Publication bias is bias in effect size estimates that occurs when the likelihood of publication depends on the direction or magnitude of the study ’ s results (e.g., Dickerson, 2005); typically, null or small effects are systematically suppressed from publication (Rosenthal, 1979).”
BackgroundWoon, F. L., Sood, S., & Hedges, D. W. (2010) · Progress in Neuro-Psychopharmacology and Biological Psychiatry · doi:10.1016/j.pnpbp.2010.06.016 · Cited by 410
Doppler ultrasonography in high-risk pregnancies: Systematic review with meta-analysis
Alfirevic, Z., & Neilson, J. P. (1995) · American Journal of Obstetrics and Gynecology · doi:10.1016/0002-9378(95)90466-2 · Cited by 414
Web Versus Other Survey Modes: An Updated and Extended Meta-Analysis Comparing Response Rates
Daikeler, J., Bošnjak, M., & Manfreda, K. L. (2019) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smz008 · Cited by 422
Zeglin, L. H. (2015) · Frontiers in Microbiology · doi:10.3389/fmicb.2015.00454 · Cited by 413
Cooper, H. M. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.1.131 · Cited by 418
The psychological benefits of indoor plants: A critical review of the experimental literature
Bringslimark, T., Hartig, T., & Patil, G. G. (2009) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2009.05.001 · Cited by 424
Koch, S. C., Riege, R. F. F., Tisborn, K., Biondo, J., Martin, L., & Beelmann, A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01806 · Cited by 418
Does stereotype threat influence performance of girls in stereotyped domains? A meta-analysis
Flore, P. C., & Wicherts, J. M. (2015) · Journal of School Psychology · doi:10.1016/j.jsp.2014.10.002 · Cited by 410
A meta-analysis of the effects of teacher personality on teacher effectiveness and burnout
Kim, L. E., Jörg, V., & Klassen, R. M. (2019) · Educational Psychology Review · doi:10.1007/s10648-018-9458-2 · Cited by 406
Consumer effects of front-of-package nutrition labeling: An interdisciplinary meta-analysis
Ikonen, I., Sotgiu, F., Aydinli, A., & Verlegh, P. W. J. (2020) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-019-00663-9 · Cited by 454
“In addition, we verified howmany studies are necessary to nullify our significant results using the failsafe N (Rosenthal 1979).”
ResultTauber, N. M., O’Toole, M. S., Dinkel, A., Galica, J., Humphris, G., Lebel, S., Maheu, C., Ozakinci, G., Prins, J., Sharpe, L., et al. (2019) · Journal of Clinical Oncology · doi:10.1200/jco.19.00572 · Cited by 402
“§A failsafe number that exceeds the criterion (5 3 K + 10) indicates a robust result.(37) kK , 23, as two studies did not assess outcomes at postintervention.”
BackgroundMethodHow Bayes factors change scientific practice
Dienes, Z. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.10.003 · Cited by 365
“Butwhere the key result is non-significant, papers are less likely to be published (Rosenthal, 1979; Simonsohn, Nelson, & Simmons, 2014).”
BackgroundEmotion regulation and posttraumatic stress symptoms: a meta-analysis
Seligowski, A. V., Lee, D. J., Bardeen, J. R., & Orcutt, H. K. (2015) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2014.980753 · Cited by 408
Biringen, Z. C., Derscheid, D. J., Vliegen, N., Closson, L., & Easterbrooks, M. A. (2014) · Developmental Review · doi:10.1016/j.dr.2014.01.002 · Cited by 403
Cheng, C., Cheung, S. F., Chio, J. H.-M., & Chan, M.-P. S. (2013) · Psychological Bulletin · doi:10.1037/a0028596 · Cited by 407
Ben‐Shakhar, G., & Elaad, E. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.1.131 · Cited by 419
Robinson, E. A., Ryan, G. D., & Newman, J. A. (2012) · New Phytologist · doi:10.1111/j.1469-8137.2012.04074.x · Cited by 405
Boncoraglio, G. P., & Saino, N. (2006) · Functional Ecology · doi:10.1111/j.1365-2435.2006.01207.x · Cited by 401
Fleiss, J. L., & Gross, A. J. (1991) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(91)90261-7 · Cited by 401
Sex Differences in Aggression: A Rejoinder and Reprise
Maccoby, E. E., & Jacklin, C. N. (1980) · Child Development · doi:10.2307/1129535 · Cited by 402
EARLY EMERGENCE ENHANCES PLANT FITNESS: A PHYLOGENETICALLY CONTROLLED META-ANALYSIS
Verdú, M., & Traveset, A. (2005) · Ecology · doi:10.1890/04-1647 · Cited by 403
Is there a relation between locus of control orientation and depression?
Benassi, V. A., Sweeney, P. D., & Dufour, C. L. (1988) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.97.3.357 · Cited by 410
The own-age bias in face recognition: a meta-analytic and theoretical review
Rhodes, M. G., & Anastasi, J. S. (2012) · Psychological Bulletin · doi:10.1037/a0025750 · Cited by 346
Cleaning Up the Big Muddy: A Meta-Analytic Review of the Determinants of Escalation of Commitment
Sleesman, D. J., Conlon, D. E., McNamara, G. M., & Miles, J. (2012) · Academy of Management Journal · doi:10.5465/amj.2010.0696 · Cited by 423
The consequences of organizational commitment: Methodological investigation
Randall, D. (1990) · Journal of Organizational Behavior · doi:10.1002/job.4030110504 · Cited by 449
Stahl, G. K., & Tung, R. L. (2014) · Journal of International Business Studies · doi:10.1057/jibs.2014.68 · Cited by 399
Van den Broeck, A., Howard, J. L., Van Vaerenbergh, Y., Leroy, H., & Gagné, M. (2021) · Organizational Psychology Review · doi:10.1177/20413866211006173 · Cited by 409
“As the review process tends to be biased toward publishing significant results (Rosenthal, 1979), scholars may be tempted to include or exclude hypotheses and analyses based on whether or not they are supported, which may lead to the under-reporting of empirical evidence that does not align with…”
Costigan, S. A., Eather, N., Plotnikoff, R. C., Taaffe, D. R., & Lubans, D. R. (2015) · British Journal of Sports Medicine · doi:10.1136/bjsports-2014-094490 · Cited by 452
Vu, N. L., Jouriles, E. N., McDonald, R., & Rosenfield, D. (2016) · Clinical Psychology Review · doi:10.1016/j.cpr.2016.04.003 · Cited by 408
Takacs, Z. K., Swart, E. K., & Bus, A. G. (2015) · Review of Educational Research · doi:10.3102/0034654314566989 · Cited by 403
“A fail-safe number of 5 k + 10 is considered robust, where k is the number of studies in the meta-analysis (Rosenthal, 1979).”
BackgroundDisasters and youth: A meta-analytic examination of posttraumatic stress.
Furr, J. M., Comer, J. S., Edmunds, J. M., & Kendall, P. C. (2010) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0021482 · Cited by 401
Sex differences in visual-spatial working memory: A meta-analysis
Voyer, D., Voyer, S. D., & Saint-Aubin, J. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1085-7 · Cited by 400
“…from published studies and that our literature search strategy might have biased the retrieved material in favor of research that showed a significant sex difference, it is possible that the final sample might not be representative of the entire population of studies in existence (Rosenthal, 1979).”
ResultIdentification of and correction for publication bias
Andrews, I., & Kasy, M. (2019) · American Economic Review · doi:10.1257/aer.20180310 · Cited by 388
Cuijpers, P., Smit, F., Bohlmeijer, E., Hollon, S. D., & Andersson, G. (2010) · British Journal of Psychiatry · doi:10.1192/bjp.bp.109.066001 · Cited by 391
Contextual sensitivity in scientific reproducibility
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521897113 · Cited by 398
“Scientists have speculated that a lack of reproducibility in psychology, as well as in other fields, is the result of a wide range of questionable research practices, including a file-drawer problem (21, 22), low statistical power (23–25), researcher’s degrees of freedom (26), presenting post hoc hypotheses as a prior…”
BackgroundThe principled control of false positives in neuroimaging
Bennett, C. M., Wolford, G. L., & Miller, M. B. (2009) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsp053 · Cited by 486
Publication Bias in the Organizational Sciences
Kepes, S., Banks, G. C., McDaniel, M. A., & Whetzel, D. L. (2012) · Organizational Research Methods · doi:10.1177/1094428112452760 · Cited by 416
Is Ego Depletion Real? An Analysis of Arguments
Friese, M., Loschelder, D. D., Gieseler, K., Frankenbach, J., & Inzlicht, M. (2018) · Personality and Social Psychology Review · doi:10.1177/1088868318762183 · Cited by 438
Metadata
- Authors
- Rosenthal, R.
- Affiliations
- Harvard University (US)
- Corresponding author
- Rosenthal, Robert
- Venue
- Psychological Bulletin
- Published
- 1979-05
- Volume
- 86
- Issue
- 3
- Pages
- 638-641
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2006546769
- Cited by
- 9,068
- References
- 16
- Citations by year
- 2026: 421 · 2025: 521 · 2024: 509 · 2023: 526 · 2022: 525 · 2021: 536 · 2020: 612 · 2019: 607 · 2018: 492 · 2017: 436 · 2016: 417 · 2015: 392 · 2014: 337 · 2013: 289 · 2012: 252
- FWCI
- 26.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- file drawer problem, negative results, publication bias, non-significant results, type I error, selective reporting, behavioral science, evidence synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
