The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
The file drawer problem and tolerance for null results.
Abstract
For any given research area, one cannot tell how many studies have been con-ducted but never reported. The extreme view of the "file drawer problem " is that journals are filled with the 5 % of the studies that show Type I errors, while the file drawers are filled with the 95 % of the studies that show non-significant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed. Both behavioral researchers and statisti-cians have long suspected that the studies published in the behavioral sciences are a biased sample of the studies that are actually carried out (Bakan, 1967; McNemar, 1960; Smart, 1964; Sterling, 1959). The extreme view of this problem, the "file drawer prob-lem, " is that the journals are filled with the 5 % of the studies that show Type I errors, while the file drawers back at the lab are filled with the 95 % of the studies that show nonsignificant (e.g., p>.05) results. In the past there was very little one could do to assess the net effect of studies, tucked away in file drawers, that did not make the magic.05 level (Rosenthal & Gaito, 1963, 1964). Now, however, although no definitive solution to the problem is available, one can establish reasonable boundaries on the prob-lem and estimate the degree of damage to any research conclusion that could be done by the file drawer problem. This advance in our ability to cope with the file drawer is an outgrowth of the in-creasing interest of behavioral scientists in summarizing bodies of research literature sys-Preparation of this article was supported in part by the National Science Foundation. I would like to thank Judith A. Hall and Donald B. Rubin for their valuable improvements of an earlier version of this article. Requests for reprints should be sent to Robert
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We then calculated Rosenthal’s fail-safe N (Rosenthal, 1979), and determined that 2,153,789 effects evidencing no relationship between SA and psychopathology (g = 0) would be needed to reduce the significance level of the summary effect size to .05.”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“The file drawer effect is a perennial problem in the academic literature (Rosenthal, 1979)”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“First, we calculated the fail-safe N [177] to be 4,274, which is the theoretical number of unpublished studies with effect sizes averaging zero (no effect) that would be needed to render negligible the omnibus results.”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [177]
In the citing paper
“As we did not include unpublished work in our meta-analyses we tested for publication bias in the following ways.Firstly, funnel plots were examined.demonstrating no systematic publication biases (tables available from first author on request).Secondly..Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Presence of publication bias was assessed with funnel plots, 22 Rosenthal's fail-safe N , 23 and Egger's regression intercept. 24”
From Light Therapy for Patients With Bipolar Depression: Systematic Review and Meta-Analysis of Randomized Controlled Trials (2020) · cites this paper as [23]
In the citing paper
“In terms of study quality, there was some indication of publication bias whereby some associations between racism and health outcomes were stronger in published versus unpublished studies, consistent with the tendency for ‘null’ studies to remain unpublished (i.e., the ‘file drawer’ problem [418]).”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as [418]
In the citing paper
“An alternative, perhaps more plausible, explanation is that there could be a significant amount of unpublished negative or inconclusive results, a phenomenon referred to as the “file-drawer effect” or publication bias (23).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [23]
In the citing paper
“If either of the two last hypotheses is true, there should exist many unpublished studies with null results lying in laboratories' drawers (32).”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Publication bias was evaluated using Rosenthal's file drawer analysis (Rosenthal, 1979) and Egger's linear regression (Card, 2012), and applying ANOVA to compare the mean effect size of the published versus unpublished studies.”
From Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal (1979) suggested a fail-safe N value above 5; k + 10 reflects results that are tolerant to contradicting studies”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“We applied random-effects models to fit the relations between CE and the criterion variables, following the guidelines proposed by Rosenthal (1979), Hunter and Schmidt (2004) and Cumming (2013).”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal (1979)
In the citing paper
“The fail-safe value for this meta-analysis was computed to be 3001, which is more than 5(36) + 10.”
From Effects of digital game-based STEM education on students’ learning achievement: a meta-analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, we suggest our results may look like a lot of studies that ended up in the so-called “file drawer” [57].”
From Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory (2023) · cites this paper as [57]
In the citing paper
“Finally, publication bias was addressed by computing the fail-safe N for the correlation coefficients (Hunter & Schmidt, 2004;Rosenthal, 1979), which reports the number of field effect sizes or the tolerance for future null results, to check whether a publication bias affected our results.”
From Drivers of Consumer Adoption of e-Commerce: a Meta-Analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Using Rosenthal’s (1979) fail-safe N criterion that the value should be over five times greater than the number of studies included in the meta-analysis, the fail-safe N for all outcome groups were at least eight times larger than the criterion value, suggesting that the effect sizes are unlikely to be an artifact of bias.”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as Rosenthal (1979)
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, Rosenthal (1979) opined that a Fail-safe N exceeding the number of studies by fivefold would provide strong support for the robustness of the significant effects.”
From Extending the Theory of Planned Behavior in the Social Commerce Context: A Meta-Analytic SEM (MASEM) Approach (2023) · cites this paper as Rosenthal (1979)
In the citing paper
“The trim-and-fill method (Duval & Tweedie, 2000) tends to adjust too little when bias is strong (Carter et al., 2019), and Fail-Safe N (Rosenthal, 1979) neither tests nor adjusts for the presence of bias.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Rosenthal, 1979
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, a phenomenon known as ' publication bias ' (Rosenthal 1979).”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“However, a common problem is that results may not have been reported in dissertations or research reports, or published in scienti fi c (peer-reviewed) journals, due to non-signi fi cant or unfavorable fi ndings and, therefore, are dif fi cult to locate.”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [21]
In the citing paper
“The traditional fail-safe value is 1904, which is higher than the threshold value of 100, i.e., [5(18) + 10], indicating that publication bias is insignificant, and the study's results are reliable.”
From A meta-analysis to explore the role of design thinking in enhancing creativity as learning outcomes in STEM education (2026) · cites this paper as Rosenthal, 1979
In the citing paper
“The Q statistic was used to assess heterogeneity30, and publication bias was estimated using Rosenthal’s fail-safe number with α = 0.0531.”
From The limited prosocial effects of meditation: A systematic review and meta-analysis (2018) · cites this paper as [31]
In the citing paper
“Rosenthal's fail-safe number (NR) was calculated, estimating the number of unpublished studies with non-significant results needed to bring an observed result to non-significance (NR ⩾ 5k + 10 to rule out a file drawer problem) (Rosenthal, 1979).”
From Efficacy of non-invasive brain stimulation on cognitive functioning in brain disorders: a meta-analysis (2020) · cites this paper as Rosenthal, 1979
In the citing paper
“In case of statistically significant effect sizes, it was planned to examine the robustness of findings by calculating the Failsafe N (number of non‐significant studies that would bring the p‐value to non‐significant (p > 0.05)) 40.”
From Cognitive behavioral therapies for informal caregivers of patients with cancer and cancer survivors: a systematic review and meta-analysis (2017) · cites this paper as [40]
In the citing paper
“With negative results buried in file drawers (55), conclusions drawn from the published record can be misleading, and 'false facts' can become canonized in the literature (56).”
From Misinformation in and about science (2021) · cites this paper as [55]
In the citing paper
“the fail-safe N (i.e. the file drawer statistic) was used to determine how many additional studies would be necessary to turn a significant ES non-significant (32).”
From Peripheral cytokine and chemokine alterations in depression: a meta‐analysis of 82 studies (2017) · cites this paper as [32]
In the citing paper
“File drawer analysis (Rosenthal, 1979) revealed that 1919 unpublished, filed, or unretrieved studies averaging null results would be required to bring the average unweighted effect size to nonsignificance.”
From Happy to help? A systematic review and meta-analysis of the effects of performing acts of kindness on the well-being of the actor (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's (1979) fail-safe N analysis reveals 186 unpublished papers (i.e., null effect) are needed to be included in the meta-analysis in order to invalidate the population effect size of small to medium (i.e., .235)found in this review.”
From A meta-analytic review of the efficacy of physical exercise interventions on cognition in individuals with autism spectrum disorder and ADHD (2016) · cites this paper as Rosenthal's (1979)
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, the so called ' publication bias ' (Rosenthal, 1979).”
From The effects of physical activity interventions on psychosocial outcomes in adolescents: a meta-analytic review (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's fail-safe N 37 calculated 115 missing studies would be required to increase the two-tailed p-value to above .05.”
From Anxiety in Informal Dementia Carers: A Meta-Analysis of Prevalence (2020) · cites this paper as [37]
In the citing paper
“Publication bias was investigated with Begg's rank correlation test for funnel plot asymmetry (Begg and Mazumdar, 1994) and where significant effects were present Rosenthal's file drawer analysis was also conducted (Rosenthal, 1979).”
From Meta-analysis of the effects of intranasal oxytocin on interpretation and expression of emotions (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“Over time, this practice artificially inflates Type I error and may make it appear that relationships exist when in reality they do not (Rosenthal et al., 1979).”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“However, a potential problem with meta-analysis is the file drawer problem, where studies are “filed away” unpublished if they fail to find statistically significant relationships.”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“Meta-analyses are necessarily based on the available literature and hence may be affected by the potential publication-bias disfavoring null results (Rosenthal 1979).”
From Parsing the neural correlates of moral cognition: ALE meta-analysis on morality, theory of mind, and empathy (2012) · cites this paper as Rosenthal 1979
In the citing paper
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
From The Extent and Consequences of P-Hacking in Science (2015) · cites this paper as [7]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G. E., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,518
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718 · cited 1×
“Perhaps the simplest, most useful way of computing the overall p of a, set of research studies is the method of adding Zs (Cochran, 1954; Mosteller & Bush, 1954; Rosenthal, 1978).”
MethodMeta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154
Experimenter Effects in Behavioral Research
McGuigan, F. J., & Rosenthal, R. (1968) · The American Journal of Psychology · doi:10.2307/1420816 · Cited by 1,633
Self-citationArtifact in behavioral research
Rosenthal, R., & Rosnow, R. L. (1969) · Medical Entomology and Zoology · Cited by 1,186
Self-citationSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964
Combining results of independent studies.
Rosenthal, R. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.1.185 · Cited by 910
Self-citationGender effects in decoding nonverbal cues.
Hall, J. A. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.4.845 · Cited by 766
Wortman, C. B., Rosenthal, R., & Rosnow, R. L. (1976) · The American Journal of Psychology · doi:10.2307/1421418 · Cited by 428
Self-citation
Cited by
Sorted by citations
Items 301–400 of 1,374
The Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634
“…half century, ‘publication bias’ has been widely recognised as a potential threat to the validity of empirical science (Sterling, 1959; Tullock, 1959; Rosenthal, 1979; Glass et al., 1981; Lovell, 1983; Hedges and Olkin, 1985; Begg and Berlin, 1988; De Long and Lang, 1992; Card and Krueger, 1995;…”
BackgroundBzdok, D., Schilbach, L., Vogeley, K., Schneider, K., Laird, A. R., Langner, R., & Eickhoff, S. B. (2012) · Brain Structure and Function · doi:10.1007/s00429-012-0380-y · Cited by 653
Smith, J. J., Eather, N., Morgan, P. J., Plotnikoff, R. C., Faigenbaum, A. D., & Lubans, D. R. (2014) · Sports Medicine · doi:10.1007/s40279-014-0196-4 · Cited by 795
Delgado, P., Vargas, C., Ackerman, R., & Salmerón, L. (2018) · Educational Research Review · doi:10.1016/j.edurev.2018.09.003 · Cited by 664
“Publication bias was evaluated using Rosenthal’s file drawer analysis (Rosenthal, 1979) and Egger’s linear regression (Card, 2012), and applying ANOVA to compare the mean effect size of the published versus unpublished studies.”
BackgroundMethodAlbrecht, M., Kleijn, D., Williams, N. M., Tschumi, M., Blaauw, B. R., Bommarco, R., Campbell, A. J., Dainese, M., Drummond, F. A., Entling, M. H., et al. (2020) · Ecology Letters · doi:10.1111/ele.13576 · Cited by 648
“T o m i n i mis e p o t e nt ial public ation b ias (i.e. the fil e drawer proble m, Rosenthal 1979) and to max imise the numbe r o f relev ant d ata s ets w e also s earche d for unpublished dat a b y c o nta c t i ng p ot e n t i a l d a t a ho l d e rs t h ro u g h r e se a r c h e r ne t wo rk…”
BackgroundPerceived Racism and Mental Health Among Black American Adults: A Meta-Analytic Review
Pieterse, A. L., Todd, N. R., Neville, H. A., & Carter, R. T. (2012) · Journal of Counseling Psychology · doi:10.1037/a0026208 · Cited by 649
Cognitive function following a major depressive episode: a systematic review and meta-analysis
Semkovska, M., Quinlivan, L., O'Grady, T., Johnson, R., Collins, A., O'Connor, J., Knittle, H., Ahern, E., & Gload, T. (2019) · The Lancet Psychiatry · doi:10.1016/s2215-0366(19)30291-3 · Cited by 631
Poverty, Income Inequality, and Violent Crime: A Meta-Analysis of Recent Aggregate Data Studies
Hsieh, C.-C., & Pugh, M. D. (1993) · Criminal Justice Review · doi:10.1177/073401689301800203 · Cited by 645
Lee, E., Park, T., & Koo, B. (2015) · Psychological Bulletin · doi:10.1037/bul0000012 · Cited by 701
“Second, the issue of publication bias, defined as the selective publication of studies that have statistically significant findings, has been an important issue for psychological science for at least three decades (e.g., Ferguson & Brannick, 2012; Rosenthal, 1979, 1995).”
BackgroundA global database of soil respiration data
Bond‐Lamberty, B., & Thomson, A. M. (2010) · Biogeosciences · doi:10.5194/bg-7-1915-2010 · Cited by 642
Alien predators are more dangerous than native predators to prey populations
Salo, P., Korpimäki, E., Banks, P. B., Nordström, M., & Dickman, C. (2007) · Proceedings of the Royal Society of London. Biological Sciences · doi:10.1098/rspb.2006.0444 · Cited by 625
The effects of met expectations on newcomer attitudes and behaviors: A review and meta-analysis.
Wanous, J. P., Poland, T. D., Premack, S. L., & Davis, K. S. (1992) · Journal of Applied Psychology · doi:10.1037/0021-9010.77.3.288 · Cited by 636
Quantifying the evidence for ecological synergies
Darling, E. S., & Côté, I. M. (2008) · Ecology Letters · doi:10.1111/j.1461-0248.2008.01243.x · Cited by 723
A Meta-Analytic Review of Adult Cognitive-Behavioral Treatment Outcome Across the Anxiety Disorders
Norton, P. J., & Price, E. C. (2007) · Journal of Nervous & Mental Disease · doi:10.1097/01.nmd.0000253843.70149.9a · Cited by 612
Verbeke, W., Dietz, B., & Verwaal, E. (2010) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-010-0211-8 · Cited by 677
Soetikno, R. M., Lin, O. S., Heidenreich, P. A., Young, H. S., & Blackstone, M. O. (2002) · Gastrointestinal Endoscopy · doi:10.1067/mge.2002.125367 · Cited by 623
Earthworms increase plant production: a meta-analysis
van Groenigen, J. W., Lubbers, I. M., Vos, H. M. J., Brown, G. G., Deyn, G. B. D., & van Groenigen, K. J. (2014) · Scientific Reports · doi:10.1038/srep06365 · Cited by 629
Working memory and second language comprehension and production: A meta-analysis
Linck, J. A., Osthus, P., Koeth, J. T., & Bunting, M. F. (2013) · Psychonomic Bulletin & Review · doi:10.3758/s13423-013-0565-2 · Cited by 693
Parenting behavior and the risk of becoming a victim and a bully/victim: A meta-analysis study
Lereya, S. T., Samara, M. M., & Dieter Wolke, C. (2013) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2013.03.001 · Cited by 627
Amlung, M., Vedelago, L., Acker, J., Balodis, I., & MacKillop, J. (2017) · Addiction · doi:10.1111/add.13535 · Cited by 626
“Presence of publication bias was based on consideration of five indices: Rosenthal’s classic fail-safe N [44]; Orwin’s fail-safe N [45]; examination of the funnel plots of sample size and effect size using the two-tailed Begg–Mazumdar test [46] and the one-tailed Egger’s test [47]; and metaregression between publicati…”
Frontal midline theta reflects anxiety and cognitive control: Meta-analytic evidence
Cavanagh, J. F., & Shackman, A. J. (2014) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2014.04.003 · Cited by 684
Burke, M., Hsiang, S. M., & Miguel, E. (2015) · Annual Review of Economics · doi:10.1146/annurev-economics-080614-115430 · Cited by 655
“For example, Ranson (2014) and Rotton & Cohn (2000) test whether hot days in hotterUS locations, such asDallas, produce similar impacts on interpersonal conflict as hot days in cooler locations, such asMinneapolis, and find that impacts are in fact similar.”
BackgroundFerguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603
Gould, M. K., Dembitzer, A. D., Doyle, R. L., Hastie, T. J., & Garber, A. M. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-10-199905180-00003 · Cited by 623
Berlim, M. T., van den Eynde, F., Tovar-Perdomo, S., & Daskalakis, Z. J. (2014) · Psychological Medicine · doi:10.1017/s0033291713000512 · Cited by 620
Tradition and Innovation in Scientists’ Research Strategies
Foster, J., Rzhetsky, A., & Evans, J. A. (2013) · arXiv.org · doi:10.1177/0003122415601618 · Cited by 578
Chang, C.-H. D., Rosen, C. C., & Levy, P. E. (2009) · Academy of Management Journal · doi:10.5465/amj.2009.43670894 · Cited by 646
Blut, M., & Wang, C. (2019) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-019-00680-8 · Cited by 675
“We calculated fail-safe Ns (FSNs) to address the file-drawer problem using the formula suggested by Rosenthal (1979).”
BackgroundMethodInfluentialSynthesizing and Extending Resource Dependence Theory
Drees, J. M., & Heugens, P. P. M. A. R. (2013) · Journal of Management · doi:10.1177/0149206312471391 · Cited by 674
Facts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609
Redick, T. S., Shipstead, Z., Harrison, T. L., Hicks, K. L., Fried, D. E., Hambrick, D. Z., Kane, M. J., & Engle, R. W. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0029082 · Cited by 617
Meta-Analytic Review of Employee Turnover as a Predictor of Firm Performance
Hancock, J. I., Allen, D. G., Bosco, F. A., McDaniel, K. R., & Pierce, C. A. (2011) · Journal of Management · doi:10.1177/0149206311424943 · Cited by 618
A Meta-Analysis of Variance Accounted for and Factor Loadings in Exploratory Factor Analysis
Peterson, R. A. (2000) · Marketing Letters · doi:10.1023/a:1008191211004 · Cited by 616
Stanley, T. D. (2008) · Oxford Bulletin of Economics and Statistics · doi:10.1111/j.1468-0084.2007.00487.x · Cited by 605
Psychological impairment in the wake of disaster: The disaster-psychopathology relationship.
Rubonis, A. V., & Bickman, L. (1991) · Psychological Bulletin · doi:10.1037/0033-2909.109.3.384 · Cited by 613
A meta-analytic review of prospective memory and aging.
Henry, J., MacLeod, M., Phillips, L., & Crawford, J. (2004) · Psychology and Aging · doi:10.1037/0882-7974.19.1.27 · Cited by 616
Publication bias in reports of animal stroke studies leads to major overstatement of efficacy
Sena, E. S., van der Worp, H. B., Bath, P. M. W., Howells, D. W., & Macleod, M. R. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000344 · Cited by 600
“This is the ‘‘file drawer problem’’ [2,3]: at its most extreme, the 95% of studies that were truly neutral (that is, which reported no significant effects) remain in the files of the investigators, the 5% of experiments that were falsely positive are published, and reviewers conclude—falsely—that the literature repres…”
BackgroundVictimization Experiences of Lesbian, Gay, and Bisexual Individuals: A Meta-Analysis
Katz‐Wise, S. L., & Hyde, J. S. (2012) · The Journal of Sex Research · doi:10.1080/00224499.2011.637247 · Cited by 669
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
“Experiments that work are sent to a journal, whereas experiments that fail are sent to the file drawer (Rosenthal 1979).”
BackgroundPublication bias—a problem in interpreting medical data
Begg, C. B., & Berlin, J. A. (1988) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982993 · Cited by 814
van Mol, M. M. C., Kompanje, E. J. O., Benoît, D., Bakker, J., & Nijkamp, M. D. (2015) · PLoS ONE · doi:10.1371/journal.pone.0136955 · Cited by 702
Is Frequency of Shared Family Meals Related to the Nutritional Health of Children and Adolescents?
Hammons, A. J., & Fiese, B. H. (2011) · PEDIATRICS · doi:10.1542/peds.2010-1440 · Cited by 613
Motivation Interventions in Education: A Meta-Analytic Review
Lazowski, R. A., & Hulleman, C. S. (2016) · Review of Educational Research · doi:10.3102/0034654315617832 · Cited by 607
“The second method is the fail-safe n test, which was first developed by Rosenthal (1979) for z values, then adapted by Orwin (1983) for use with the standardized mean difference effect size.”
MethodAugé, R. M., Toler, H. D., & Saxton, A. M. (2014) · Mycorrhiza · doi:10.1007/s00572-014-0585-4 · Cited by 613
Open science challenges, benefits and tips in early career and beyond
Allen, C., & Mehler, D. M. A. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000246 · Cited by 608
“These approaches require hypotheses and analysis pipelines to be declared publicly before data collection [2,14,28,29] (although protocols can be embargoed).”
BackgroundThe Nature and Effects of Transformational School Leadership
Leithwood, K. A., & Sun, J. (2012) · Educational Administration Quarterly · doi:10.1177/0013161x11436268 · Cited by 665
Publication Bias in Meta‐Analysis
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch1 · Cited by 603
A meta-analytical integration of over 40 years of research on diversity training evaluation
Bezrukova, K., Spell, C. S., Perry, J. L., & Jehn, K. A. (2016) · Psychological Bulletin · doi:10.1037/bul0000067 · Cited by 601
Meta‐analysis of transmitter effects on avian behaviour and ecology
Barron, D. G., Brawn, J., & Weatherhead, P. J. (2010) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2010.00013.x · Cited by 575
Evaluating Six Common Stereotypes About Older Workers with Meta‐Analytical Data
Ng, T. W. H., & Feldman, D. C. (2012) · Personnel Psychology · doi:10.1111/peps.12003 · Cited by 689
Socioeconomic Status and Bullying: A Meta-Analysis
Tippett, N., & Dieter Wolke, C. (2014) · American Journal of Public Health · doi:10.2105/ajph.2014.301960 · Cited by 664
Letourneau, D. K., Jedlicka, J. A., Bothwell, S. G., & Moreno, C. R. (2009) · Annual Review of Ecology, Evolution, and Systematics · doi:10.1146/annurev.ecolsys.110308.120320 · Cited by 596
Why Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563
A meta-analysis of the relation between math anxiety and math achievement
Barroso, C., Ganley, C. M., McGraw, A. L., Geer, E. A., Hart, S. A., & Daucourt, M. C. (2021) · Psychological Bulletin · doi:10.1037/bul0000307 · Cited by 614
“…an overall effect size is from the current meta-analytic sample compared to the effect size from a sample that may more accurately account for the missing effect sizes that may not have been published and therefore are not represented on the funnel plot (i.e., file drawer problem; Rosenthal, 1979).”
InfluentialThe Relationship between Perfectionism and Psychopathology: A Meta-analysis
Limburg, K., Watson, H. J., Hagger, M. S., & Egan, S. J. (2017) · Journal of Clinical Psychology · doi:10.1002/jclp.22435 · Cited by 602
“…publication bias (Rosenthal, 1979) and funnel plots were visually inspected in order to detect asymmetry (Borenstein et al., 2009) and the fail-safe N-method was used to calculate the number of null results that would lead to a non-significant effect size if added to the analysis (Rosenthal, 1979).”
Traditional reviews, meta-analyses and pooled analyses in epidemiology
Blettner, M. (1999) · International Journal of Epidemiology · doi:10.1093/ije/28.1.1 · Cited by 596
Sex Differences in the Development of Moral Reasoning: A Critical Review
Walker, L. J. (1984) · Child Development · doi:10.2307/1130121 · Cited by 593
Corpus Use in Language Learning: A Meta‐Analysis
Boulton, A., & Cobb, T. (2017) · Language Learning · doi:10.1111/lang.12224 · Cited by 635
“Nonetheless, publication bias is still likely insofar as studies which do not produce significant differences or other desired outcomes are more likely to remain unpublished altogether (N. Ellis, 2006), the famous file-drawer problem (Rosenthal, 1979).”
BackgroundThe structure of common emotion regulation strategies: A meta-analytic examination
Naragon-Gainey, K., McMahon, T. P., & Chacko, T. P. (2017) · Psychological Bulletin · doi:10.1037/bul0000093 · Cited by 588
Environmental, social and governance (ESG) activity and firm performance: a review and consolidation
Huang, D. Z. (2019) · Accounting and Finance · doi:10.1111/acfi.12569 · Cited by 698
Effectiveness of Long-term Psychodynamic Psychotherapy: A Meta-analysis
Leichsenring, F. (2008) · JAMA · doi:10.1001/jama.300.13.1551 · Cited by 65
The psychophysiology of posttraumatic stress disorder: a meta-analysis
Pole, N. (2007) · Psychological Bulletin · doi:10.1037/0033-2909.133.5.725 · Cited by 582
What (Not) to Expect When Surveying Executives
Cycyota, C. S., & Harrison, D. A. (2006) · Organizational Research Methods · doi:10.1177/1094428105280770 · Cited by 586
Langhorne, P., Williams, B., Gilchrist, W., & Howie, K. (1993) · The Lancet · doi:10.1016/0140-6736(93)92813-9 · Cited by 585
Stanton, M., & Shadish, W. R. (1997) · Psychological Bulletin · doi:10.1037/0033-2909.122.2.170 · Cited by 585
Slotema, C. W., Blom, J. D., Hoek, H. W., & Sommer, I. E. C. (2010) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.08m04872gre · Cited by 584
Lamont, R. A., Swift, H. J., & Abrams, D. (2015) · Psychology and Aging · doi:10.1037/a0038586 · Cited by 583
Avian productivity in urban landscapes: a review and meta‐analysis
Chamberlain, D., CANNON, A. R., Toms, M. P., Leech, D. I., Hatchwell, B. J., & Gaston, K. J. (2008) · Ibis · doi:10.1111/j.1474-919x.2008.00899.x · Cited by 583
Fan, X., Jiang, X., & Deng, N. (2022) · Tourism Management · doi:10.1016/j.tourman.2022.104534 · Cited by 586
Physical Activity and Physical Self-Concept in Youth: Systematic Review and Meta-Analysis
Babic, M., Morgan, P. J., Plotnikoff, R. C., Lonsdale, C., White, R. L., & Lubans, D. R. (2014) · Sports Medicine · doi:10.1007/s40279-014-0229-z · Cited by 669
Posttraumatic stress disorder and intimate relationship problems: A meta-analysis.
Taft, C. T., Watkins, L. E., Stafford, J., Street, A. E., & Monson, C. M. (2011) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0022196 · Cited by 576
Gorgolewski, K. J., Varoquaux, G. P., Rivera, G., Schwarz, Y., Ghosh, S., Maumet, C., Sochat, V., Nichols, T. E., Poldrack, R. A., Poline, J., et al. (2015) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2015.00008 · Cited by 554
Impulse buying: a meta-analytic review
Iyer, G. R., Blut, M., Xiao, S. H., & Grewal, D. (2019) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-019-00670-w · Cited by 653
“All the significant relationships are robust to publication bias because the file-drawer N is many times greater than the tolerance levels proposed by Rosenthal (1979).”
BackgroundEndrikat, J., Guenther, E., & Hoppe, H. (2014) · European Management Journal · doi:10.1016/j.emj.2013.12.004 · Cited by 627
Future time perspective: A systematic review and meta-analysis.
Kooij, D. T. A. M., Kanfer, R., Betts, M., & Rudolph, C. W. (2018) · Journal of Applied Psychology · doi:10.1037/apl0000306 · Cited by 662
“Nevertheless, the fail-safe N, or number of studies confirming the null hypothesis that would be needed to reverse a conclusion that a significant relationship exists (Rosenthal, 1979), is above 39 for the overall relationships.”
Destination image and tourist behavioural intentions: A meta-analysis
Afshardoost, M., & Eshaghi, M. S. (2020) · Tourism Management · doi:10.1016/j.tourman.2020.104154 · Cited by 720
“Second, a “Classic Fail-safe N” test was conducted to determine the stability of meta-analysis results by computing how many missing studies would be required to retrieve and incorporate in the analysis before the p-value became non-significant (Rosenthal, 1979; Begg & Mazumdar, 1994).”
The researcher's own therapy allegiances: A "wild card" in comparisons of treatment efficacy.
Luborsky, L., Diguer, L., Seligman, D. A., Rosenthal, R., Krause, E. D., Johnson, S., Halperin, G., Bishop, M., Berman, J. S., & Schweizer, E. (1999) · Clinical Psychology: Science and Practice · doi:10.1093/clipsy.6.1.95 · Cited by 572
Self-citationRegistered Reports: A new publishing initiative at Cortex
Chambers, C. (2012) · Cortex · doi:10.1016/j.cortex.2012.12.016 · Cited by 608
Taylor, L. E., Swerdfeger, A. L., & Eslick, G. D. (2014) · Vaccine · doi:10.1016/j.vaccine.2014.04.085 · Cited by 583
Dew, M. A., DiMartini, A. F., Dabbs, A. D. V., Myaskovsky, L., Steel, J. L., Unruh, M. L., Switzer, G. E., Zomak, R. A., Kormos, R. L., & Greenhouse, J. B. (2007) · Transplantation · doi:10.1097/01.tp.0000258599.65257.a6 · Cited by 571
Test anxiety effects, predictors, and correlates: A 30-year meta-analytic review
von der Embse, N., Jester, D., Roy, D., & Post, J. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.11.048 · Cited by 691
Moral disengagement among children and youth: A meta‐analytic review of links to aggressive behavior
Gini, G., Pozzoli, T., & Hymel, S. C. (2013) · Aggressive Behavior · doi:10.1002/ab.21502 · Cited by 585
Some Statistical Methods for Combining Experimental Results
Laird, N. M., & Mosteller, F. C. (1990) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300008916 · Cited by 564
Locus of control and academic achievement: A literature review.
Findley, M. J., & Cooper, H. M. (1983) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.44.2.419 · Cited by 568
The effects of postpartum depression on child development: A meta-analysis
Beck, C. T. (1998) · Archives of Psychiatric Nursing · doi:10.1016/s0883-9417(98)80004-6 · Cited by 558
Understanding Mail Survey Response Behavior: A Meta-Analysis
Yammarino, F. J., Skinner, S. J., & Childers, T. L. (1991) · Public Opinion Quarterly · doi:10.1086/269284 · Cited by 566
Credé, M., & Niehorster, S. (2012) · Educational Psychology Review · doi:10.1007/s10648-011-9184-5 · Cited by 607
All Work and No Play? A Meta-Analytic Examination of the Correlates and Outcomes of Workaholism
Clark, M. A., Michel, J. S., Zhdanova, L. S., Pui, S. Y., & Baltes, B. B. (2014) · Journal of Management · doi:10.1177/0149206314522301 · Cited by 634
“Publication status (unpublished or published) is examined to examine the possibility that publication bias (Rosenthal, 1979) influences our results.”
BackgroundSeverity of the Effects of Invasive Rats on Seabirds: A Global Review
Jones, H. P., Tershy, B. R., Zavaleta, E. S., Croll, D. A., Keitt, B. S., Finkelstein, M. E., & Howald, G. R. (2008) · Conservation Biology · doi:10.1111/j.1523-1739.2007.00859.x · Cited by 561
Hillmayr, D., Ziernwald, L., Reinhold, F., Hofer, S. I., & Reiss, K. M. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.103897 · Cited by 574
“The limit of 5 k þ 10 studies suggested by Rosenthal (1979), which is 470 in the current study, was therefore far exceeded.”
BackgroundMethodInfluentialRethinking the Use of Tests: A Meta-Analysis of Practice Testing
Adesope, O. O., Trevisan, D. A., & Sundararajan, N. (2017) · Review of Educational Research · doi:10.3102/0034654316689306 · Cited by 544
“Studies with statistically significant results are published more often than studies with no significant results, which typically remain unpublished or at best reported in the less accessible grey literature (Lipsey & Wilson, 2001; Rosenthal, 1979).”
BackgroundResultBrønfort, G., Haas, M., Evans, R., & Bouter, L. M. (2004) · The Spine Journal · doi:10.1016/j.spinee.2003.06.002 · Cited by 565
Parental correlates in child and adolescent physical activity: a meta-analysis
Yao, C. A., & Rhodes, R. E. (2015) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-015-0163-y · Cited by 540
“To assess the extent of publication bias in our samples, Rosenthal’s classic fail-safe N [33] and Duval and Tweedie’s Trim and Fill procedures [34,35] were conducted.”
MethodA brief history of research synthesis
Chalmers, I., Hedges, L. V., & Cooper, H. (2002) · Evaluation & the Health Professions · doi:10.1177/0163278702025001003 · Cited by 552
Wu, B., Yu, X., & Gu, X. (2020) · British Journal of Educational Technology · doi:10.1111/bjet.13023 · Cited by 640
Mood-congruent recall of affectively toned stimuli: A meta-analytic review
Matt, G. E., Vázquez, C., & Campbell, W. K. (1992) · Clinical Psychology Review · doi:10.1016/0272-7358(92)90116-p · Cited by 726
Perspective-taking and altruism.
Underwood, B., & Moore, B. S. (1982) · Psychological Bulletin · doi:10.1037/0033-2909.91.1.143 · Cited by 544
A meta-analysis of age differences in job performance.
Waldman, D. A., & Avolio, B. J. (1986) · Journal of Applied Psychology · doi:10.1037/0021-9010.71.1.33 · Cited by 543
A Meta‐Analysis of Supply Chain Integration and Firm Performance
Leuschner, R., Rogers, D. S., & Charvet, F. (2013) · Journal of Supply Chain Management · doi:10.1111/jscm.12013 · Cited by 557
Curry, O. S., Rowland, L. A., Van Lissa, C. J., Zlotowitz, S., McAlaney, J., & Whitehouse, H. (2018) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2018.02.014 · Cited by 528
“File drawer analysis (Rosenthal, 1979) revealed that 1919 un-published, fi led, or unretrieved studies averaging null results would be required to bring the average unweighted e ff ect size to nonsigni fi cance.”
Metadata
- Authors
- Rosenthal, R.
- Affiliations
- Harvard University (US)
- Corresponding author
- Rosenthal, Robert
- Venue
- Psychological Bulletin
- Published
- 1979-05
- Volume
- 86
- Issue
- 3
- Pages
- 638-641
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2006546769
- Cited by
- 9,068
- References
- 16
- Citations by year
- 2026: 421 · 2025: 521 · 2024: 509 · 2023: 526 · 2022: 525 · 2021: 536 · 2020: 612 · 2019: 607 · 2018: 492 · 2017: 436 · 2016: 417 · 2015: 392 · 2014: 337 · 2013: 289 · 2012: 252
- FWCI
- 26.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- file drawer problem, negative results, publication bias, non-significant results, type I error, selective reporting, behavioral science, evidence synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
