The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
The file drawer problem and tolerance for null results.
Abstract
For any given research area, one cannot tell how many studies have been con-ducted but never reported. The extreme view of the "file drawer problem " is that journals are filled with the 5 % of the studies that show Type I errors, while the file drawers are filled with the 95 % of the studies that show non-significant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed. Both behavioral researchers and statisti-cians have long suspected that the studies published in the behavioral sciences are a biased sample of the studies that are actually carried out (Bakan, 1967; McNemar, 1960; Smart, 1964; Sterling, 1959). The extreme view of this problem, the "file drawer prob-lem, " is that the journals are filled with the 5 % of the studies that show Type I errors, while the file drawers back at the lab are filled with the 95 % of the studies that show nonsignificant (e.g., p>.05) results. In the past there was very little one could do to assess the net effect of studies, tucked away in file drawers, that did not make the magic.05 level (Rosenthal & Gaito, 1963, 1964). Now, however, although no definitive solution to the problem is available, one can establish reasonable boundaries on the prob-lem and estimate the degree of damage to any research conclusion that could be done by the file drawer problem. This advance in our ability to cope with the file drawer is an outgrowth of the in-creasing interest of behavioral scientists in summarizing bodies of research literature sys-Preparation of this article was supported in part by the National Science Foundation. I would like to thank Judith A. Hall and Donald B. Rubin for their valuable improvements of an earlier version of this article. Requests for reprints should be sent to Robert
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We then calculated Rosenthal’s fail-safe N (Rosenthal, 1979), and determined that 2,153,789 effects evidencing no relationship between SA and psychopathology (g = 0) would be needed to reduce the significance level of the summary effect size to .05.”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“The file drawer effect is a perennial problem in the academic literature (Rosenthal, 1979)”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“First, we calculated the fail-safe N [177] to be 4,274, which is the theoretical number of unpublished studies with effect sizes averaging zero (no effect) that would be needed to render negligible the omnibus results.”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [177]
In the citing paper
“As we did not include unpublished work in our meta-analyses we tested for publication bias in the following ways.Firstly, funnel plots were examined.demonstrating no systematic publication biases (tables available from first author on request).Secondly..Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Presence of publication bias was assessed with funnel plots, 22 Rosenthal's fail-safe N , 23 and Egger's regression intercept. 24”
From Light Therapy for Patients With Bipolar Depression: Systematic Review and Meta-Analysis of Randomized Controlled Trials (2020) · cites this paper as [23]
In the citing paper
“In terms of study quality, there was some indication of publication bias whereby some associations between racism and health outcomes were stronger in published versus unpublished studies, consistent with the tendency for ‘null’ studies to remain unpublished (i.e., the ‘file drawer’ problem [418]).”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as [418]
In the citing paper
“An alternative, perhaps more plausible, explanation is that there could be a significant amount of unpublished negative or inconclusive results, a phenomenon referred to as the “file-drawer effect” or publication bias (23).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [23]
In the citing paper
“If either of the two last hypotheses is true, there should exist many unpublished studies with null results lying in laboratories' drawers (32).”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Publication bias was evaluated using Rosenthal's file drawer analysis (Rosenthal, 1979) and Egger's linear regression (Card, 2012), and applying ANOVA to compare the mean effect size of the published versus unpublished studies.”
From Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal (1979) suggested a fail-safe N value above 5; k + 10 reflects results that are tolerant to contradicting studies”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“We applied random-effects models to fit the relations between CE and the criterion variables, following the guidelines proposed by Rosenthal (1979), Hunter and Schmidt (2004) and Cumming (2013).”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal (1979)
In the citing paper
“The fail-safe value for this meta-analysis was computed to be 3001, which is more than 5(36) + 10.”
From Effects of digital game-based STEM education on students’ learning achievement: a meta-analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, we suggest our results may look like a lot of studies that ended up in the so-called “file drawer” [57].”
From Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory (2023) · cites this paper as [57]
In the citing paper
“Finally, publication bias was addressed by computing the fail-safe N for the correlation coefficients (Hunter & Schmidt, 2004;Rosenthal, 1979), which reports the number of field effect sizes or the tolerance for future null results, to check whether a publication bias affected our results.”
From Drivers of Consumer Adoption of e-Commerce: a Meta-Analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Using Rosenthal’s (1979) fail-safe N criterion that the value should be over five times greater than the number of studies included in the meta-analysis, the fail-safe N for all outcome groups were at least eight times larger than the criterion value, suggesting that the effect sizes are unlikely to be an artifact of bias.”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as Rosenthal (1979)
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, Rosenthal (1979) opined that a Fail-safe N exceeding the number of studies by fivefold would provide strong support for the robustness of the significant effects.”
From Extending the Theory of Planned Behavior in the Social Commerce Context: A Meta-Analytic SEM (MASEM) Approach (2023) · cites this paper as Rosenthal (1979)
In the citing paper
“The trim-and-fill method (Duval & Tweedie, 2000) tends to adjust too little when bias is strong (Carter et al., 2019), and Fail-Safe N (Rosenthal, 1979) neither tests nor adjusts for the presence of bias.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Rosenthal, 1979
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, a phenomenon known as ' publication bias ' (Rosenthal 1979).”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“However, a common problem is that results may not have been reported in dissertations or research reports, or published in scienti fi c (peer-reviewed) journals, due to non-signi fi cant or unfavorable fi ndings and, therefore, are dif fi cult to locate.”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [21]
In the citing paper
“The traditional fail-safe value is 1904, which is higher than the threshold value of 100, i.e., [5(18) + 10], indicating that publication bias is insignificant, and the study's results are reliable.”
From A meta-analysis to explore the role of design thinking in enhancing creativity as learning outcomes in STEM education (2026) · cites this paper as Rosenthal, 1979
In the citing paper
“The Q statistic was used to assess heterogeneity30, and publication bias was estimated using Rosenthal’s fail-safe number with α = 0.0531.”
From The limited prosocial effects of meditation: A systematic review and meta-analysis (2018) · cites this paper as [31]
In the citing paper
“Rosenthal's fail-safe number (NR) was calculated, estimating the number of unpublished studies with non-significant results needed to bring an observed result to non-significance (NR ⩾ 5k + 10 to rule out a file drawer problem) (Rosenthal, 1979).”
From Efficacy of non-invasive brain stimulation on cognitive functioning in brain disorders: a meta-analysis (2020) · cites this paper as Rosenthal, 1979
In the citing paper
“In case of statistically significant effect sizes, it was planned to examine the robustness of findings by calculating the Failsafe N (number of non‐significant studies that would bring the p‐value to non‐significant (p > 0.05)) 40.”
From Cognitive behavioral therapies for informal caregivers of patients with cancer and cancer survivors: a systematic review and meta-analysis (2017) · cites this paper as [40]
In the citing paper
“With negative results buried in file drawers (55), conclusions drawn from the published record can be misleading, and 'false facts' can become canonized in the literature (56).”
From Misinformation in and about science (2021) · cites this paper as [55]
In the citing paper
“the fail-safe N (i.e. the file drawer statistic) was used to determine how many additional studies would be necessary to turn a significant ES non-significant (32).”
From Peripheral cytokine and chemokine alterations in depression: a meta‐analysis of 82 studies (2017) · cites this paper as [32]
In the citing paper
“File drawer analysis (Rosenthal, 1979) revealed that 1919 unpublished, filed, or unretrieved studies averaging null results would be required to bring the average unweighted effect size to nonsignificance.”
From Happy to help? A systematic review and meta-analysis of the effects of performing acts of kindness on the well-being of the actor (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's (1979) fail-safe N analysis reveals 186 unpublished papers (i.e., null effect) are needed to be included in the meta-analysis in order to invalidate the population effect size of small to medium (i.e., .235)found in this review.”
From A meta-analytic review of the efficacy of physical exercise interventions on cognition in individuals with autism spectrum disorder and ADHD (2016) · cites this paper as Rosenthal's (1979)
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, the so called ' publication bias ' (Rosenthal, 1979).”
From The effects of physical activity interventions on psychosocial outcomes in adolescents: a meta-analytic review (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's fail-safe N 37 calculated 115 missing studies would be required to increase the two-tailed p-value to above .05.”
From Anxiety in Informal Dementia Carers: A Meta-Analysis of Prevalence (2020) · cites this paper as [37]
In the citing paper
“Publication bias was investigated with Begg's rank correlation test for funnel plot asymmetry (Begg and Mazumdar, 1994) and where significant effects were present Rosenthal's file drawer analysis was also conducted (Rosenthal, 1979).”
From Meta-analysis of the effects of intranasal oxytocin on interpretation and expression of emotions (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“Over time, this practice artificially inflates Type I error and may make it appear that relationships exist when in reality they do not (Rosenthal et al., 1979).”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“However, a potential problem with meta-analysis is the file drawer problem, where studies are “filed away” unpublished if they fail to find statistically significant relationships.”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“Meta-analyses are necessarily based on the available literature and hence may be affected by the potential publication-bias disfavoring null results (Rosenthal 1979).”
From Parsing the neural correlates of moral cognition: ALE meta-analysis on morality, theory of mind, and empathy (2012) · cites this paper as Rosenthal 1979
In the citing paper
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
From The Extent and Consequences of P-Hacking in Science (2015) · cites this paper as [7]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G. E., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,518
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718 · cited 1×
“Perhaps the simplest, most useful way of computing the overall p of a, set of research studies is the method of adding Zs (Cochran, 1954; Mosteller & Bush, 1954; Rosenthal, 1978).”
MethodMeta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154
Experimenter Effects in Behavioral Research
McGuigan, F. J., & Rosenthal, R. (1968) · The American Journal of Psychology · doi:10.2307/1420816 · Cited by 1,633
Self-citationArtifact in behavioral research
Rosenthal, R., & Rosnow, R. L. (1969) · Medical Entomology and Zoology · Cited by 1,186
Self-citationSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964
Combining results of independent studies.
Rosenthal, R. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.1.185 · Cited by 910
Self-citationGender effects in decoding nonverbal cues.
Hall, J. A. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.4.845 · Cited by 766
Wortman, C. B., Rosenthal, R., & Rosnow, R. L. (1976) · The American Journal of Psychology · doi:10.2307/1421418 · Cited by 428
Self-citation
Cited by
Sorted by citations
Items 701–800 of 1,374
Petersen, J. L., & Hyde, J. S. (2011) · The Journal of Sex Research · doi:10.1080/00224499.2011.551851 · Cited by 345
A Meta-Analysis of the Relation between Creative Self-Efficacy and Different Creativity Measurements
Haase, J., Hoff, E. V., Hanel, P. H. P., & Innes-Ker, Å. H. (2018) · Creativity Research Journal · doi:10.1080/10400419.2018.1411436 · Cited by 388
“A statistical analysis was done to estimate the potential influence of missing studies due to the tendency that usually only significant results get published (file-drawer problem; Rosenthal, 1979).”
MethodMarlow, S. L., Lacerenza, C. N., Paoletti, J., Burke, C. S., & Salas, E. (2018) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2017.08.001 · Cited by 346
Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies
Walum, H., Waldman, I. D., & Young, L. J. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.06.016 · Cited by 347
“An alternative, perhaps more plausible, explanation is that there could be a significant amount of unpublished negative or inconclusive results, a phenomenon referred to as the “file-drawer effect” or publication bias (23).”
BackgroundImpact of ingratiation on judgments and evaluations: A meta-analytic investigation.
Gordon, R. A. (1996) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.71.1.54 · Cited by 338
Digital medicine and the curse of dimensionality
Berisha, V., Krantsevich, C., Hahn, P. R., Hahn, S., Dasarathy, G., Turaga, P., & Liss, J. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00521-5 · Cited by 344
Sex differences and the own-gender bias in face recognition: A meta-analytic review
Herlitz, A., & Lovén, J. (2013) · Visual Cognition · doi:10.1080/13506285.2013.823140 · Cited by 371
The L2 motivational self system: A meta-analysis
Al-Hoorie, A. H. (2018) · Studies in Second Language Learning and Teaching · doi:10.14746/ssllt.2018.8.4.2 · Cited by 351
“This can lead to what is known as the file drawer problem (Rosenthal, 1979).”
BackgroundFischer, S. F., Smith, G. T., & Cyders, M. A. (2008) · Clinical Psychology Review · doi:10.1016/j.cpr.2008.09.001 · Cited by 369
Tobacco and marijuana use among adolescents and young adults: A systematic review of their co-use
Ramo, D. E., Liu, H., & Prochaska, J. J. (2011) · Clinical Psychology Review · doi:10.1016/j.cpr.2011.12.002 · Cited by 339
The impact of gamification in educational settings on student learning outcomes: a meta-analysis
Huang, R., Ritzhaupt, A. D., Sommer, M., Zhu, J., Stephen, A., Valle, N., Hampton, J., & Li, J. (2020) · Educational Technology Research and Development · doi:10.1007/s11423-020-09807-z · Cited by 340
“These findings might suggest that peer reviewers have a preference for findings with positive effects from the interventions under investigation, and makes us reflect on the “file drawer problem” (Rosenthal 1979), which results from not publishing statistically non-significant findings.”
BackgroundTreatment of Fibromyalgia Syndrome With Antidepressants: A Meta-analysis
Häuser, W., Bernardy, K., Üçeyler, N., & Sommer, C. (2009) · JAMA · doi:10.1001/jama.2008.944 · Cited by 338
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496
Relaxation training for anxiety: a ten-years systematic review with meta-analysis
Manzoni, G. M., Pagnini, F., Castelnuovo, G., & Molinari, E. (2008) · BMC Psychiatry · doi:10.1186/1471-244x-8-41 · Cited by 480
Multi‐locus species delimitation in closely related animals and fungi: one marker is not enough
Dupuis, J. R., Roe, A. D., & Sperling, F. A. H. (2012) · Molecular Ecology · doi:10.1111/j.1365-294x.2012.05642.x · Cited by 339
Differences in quality of life across renal replacement therapies: A meta-analytic comparison
Cameron, J. I., Whiteside, C., Katz, J. D., & Devins, G. M. (2000) · American Journal of Kidney Diseases · doi:10.1016/s0272-6386(00)70009-6 · Cited by 339
AERA Editorial Policies regarding Statistical Significance Testing: Three Suggested Reforms
Thompson, B. (1996) · Educational Researcher · doi:10.2307/1176337 · Cited by 339
Entrepreneurial Team Composition Characteristics and New Venture Performance: A Meta–Analysis
Jin, L., Madison, K., Kraiczy, N. D., Kellermanns, F. W., Crook, T. R., & Xi, J. (2016) · Entrepreneurship Theory and Practice · doi:10.1111/etap.12232 · Cited by 369
“We followed Rosenthal (1979) and ran tests to determine the number of published studies that would be needed to negate our results (i.e., have an effect of zero).”
Bailey, A. P., Hetrick, S. E., Rosenbaum, S., Purcell, R., & Parker, A. G. (2018) · Psychological Medicine · doi:10.1017/s0033291717002653 · Cited by 338
“Publication bias was assessed by funnel plot inspection, use of the trim-and-fill method to adjust the pooled effect (Duval & Tweedie, 2000) and estimation of the failsafe N (Rosenthal, 1979).”
MethodRisk factors for persistent delinquent behavior among juveniles: A meta-analytic review
Assink, M., van der Put, C. E., Hoeve, M., de Vries, S. L. A., Stams, G. J. J. M., & Oort, F. J. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.08.002 · Cited by 338
Publication bias: the elephant in the review
Dalton, J. E., Bolen, S. D., & Mascha, E. J. (2016) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000001596 · Cited by 337
Gorrese, A., & Ruggieri, R. (2012) · Journal of Youth and Adolescence · doi:10.1007/s10964-012-9759-6 · Cited by 337
Kazdin, A. E., & Bass, D. C. (1989) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.57.1.138 · Cited by 337
The effectiveness of drug abuse treatment: a meta-analysis of comparison group studies
PRENDERGAST, M. L., Podus, D., Chang, E. Y., & Urada, D. I. (2002) · Drug and Alcohol Dependence · doi:10.1016/s0376-8716(02)00014-5 · Cited by 336
Frank, M. C., Bergelson, E., Bergmann, C., Cristià, A., Floccia, C., Gervain, J., Hamlin, J. K., Hannon, E. E., Struhl, M. K., Levelt, C. C., et al. (2017) · Infancy · doi:10.1111/infa.12182 · Cited by 353
Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0215052 · Cited by 365
“This method estimates how many effect sizes with a zero effect size have to be added to a meta-analysis for changing a statistically significant summary effect size in a meta-analysis to a nonsignificant result [38].”
MethodMorschett, D., Schramm‐Klein, H., & Swoboda, B. S. (2010) · Journal of International Management · doi:10.1016/j.intman.2009.09.002 · Cited by 334
The alliance in child and adolescent psychotherapy.
Shirk, S. R., Karver, M. S., & Brown, R. R. (2011) · Psychotherapy · doi:10.1037/a0022181 · Cited by 336
Cillessen, L., Johannsen, M., Speckens, A. E. M., & Zachariae, R. (2019) · Psycho-Oncology · doi:10.1002/pon.5214 · Cited by 326
Mann-Wrobel, M. C., Carreno, J. T., & Dickinson, D. (2011) · Bipolar Disorders · doi:10.1111/j.1399-5618.2011.00935.x · Cited by 328
Cigarette smoking and the risk of systemic lupus erythematosus: A meta‐analysis
Costenbader, K. H., Kim, D. J., Peerzada, J. M., Lockman, S., Nobles-Knight, D., Petri, M., & Karlson, E. W. (2004) · Arthritis & Rheumatism · doi:10.1002/art.20049 · Cited by 341
Davies, G. M., & Gray, A. (2015) · Ecology and Evolution · doi:10.1002/ece3.1782 · Cited by 333
“The recent reviews by Waller et al. (2013) and Ramage et al. (2013) highlight the potential dangers associated with pseudoreplicated experiments. Waller et al. (2013) found that 39% of primate communication studies were pseudoreplicated, but in 88% of these, the issue was avoidable. Ramage et al. (2013) found that 68%…”
BackgroundInfluentialZakzanis, K. K. (2001) · Archives of Clinical Neuropsychology · doi:10.1093/arclin/16.7.653 · Cited by 346
Wheeler, J. A., Gorey, K. M., & Greenblatt, B. (1998) · The International Journal of Aging and Human Development · doi:10.2190/vump-xcmf-fqyu-v0jh · Cited by 332
Educational Interventions That Improve Environmental Behaviors: A Meta-Analysis
Zelezny, L. C. (1999) · The Journal of Environmental Education · doi:10.1080/00958969909598627 · Cited by 332
Residents’ impact perceptions of and attitudes towards tourism development: a meta-analysis
Gürsoy, D., Ouyang, Z., Nunkoo, R., & Wei, W. (2018) · Journal of Hospitality Marketing & Management · doi:10.1080/19368623.2018.1516589 · Cited by 357
“Several researchers have addressed the existence of publication bias against statistically nonsignificant results in peer-reviewed journals (Orwin, 1983; Rosenthal, 1979).”
BackgroundTakacs, Z. K., & Kassai, R. (2019) · Psychological Bulletin · doi:10.1037/bul0000195 · Cited by 332
“In case of a robust effect, this fail-safe number should exceed 5k 10 where k is the number of contrasts included (Rosenthal, 1979).”
BackgroundService Recovery Paradox: A Meta-Analysis
de Matos, C. A., Henrique, J. L., & Rossi, C. A. V. (2007) · Journal of Service Research · doi:10.1177/1094670507303012 · Cited by 334
A meta-analysis of plant physiological and growth responses to temperature and elevated CO2
Wang, D., Heckathorn, S. A., Wang, X., & Philpott, S. M. (2011) · Oecologia · doi:10.1007/s00442-011-2172-0 · Cited by 331
Carlson, J. P., Vincent, L. H., Hardesty, D. M., & Bearden, W. O. (2008) · Journal of Consumer Research · doi:10.1086/593688 · Cited by 333
Global processing takes time: A meta-analysis on local–global visual processing in ASD.
der Hallen, R. V., Evers, K., Brewaeys, K., Noortgate, W. V. D., & Wagemans, J. P. (2014) · Psychological Bulletin · doi:10.1037/bul0000004 · Cited by 361
Gannon, T. A., Olver, M. E., Mallion, J. S., & James, M. (2019) · Clinical Psychology Review · doi:10.1016/j.cpr.2019.101752 · Cited by 335
“Three sets of asymmetry testing were conducted: funnel plots of precision, trim and fill (Duval & Tweedie, 2000), and fail-safe N (Rosenthal, 1979).”
MethodEffects of family and marital psychotherapies: a meta-analysis.
Shadish, W., Montgomery, L. M., Wilson, P., Wilson, M. R., Bright, I., & Okwumabua, T. (1993) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.61.6.992 · Cited by 320
Spatial Ability: Its Influence on Learning with Visualizations—a Meta-Analytic Review
Höffler, T. N. (2010) · Educational Psychology Review · doi:10.1007/s10648-010-9126-7 · Cited by 359
Does psi exist? Replicable evidence for an anomalous process of information transfer.
Bem, D. J., & Honorton, C. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.115.1.4 · Cited by 330
Salgado, J. F., Anderson, N. R., Moscoso, S., Bertua, C., Fruyt, F. D., & Rolland, J. P. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.6.1068 · Cited by 364
Toplak, M. E., Sorge, G. B., Benoit, A., West, R. F., & Stanovich, K. E. (2010) · Clinical Psychology Review · doi:10.1016/j.cpr.2010.04.002 · Cited by 326
The Importance of Quality of Primary Studies in Producing Unbiased Systematic Reviews
Khan, K. S. (1996) · Archives of Internal Medicine · doi:10.1001/archinte.1996.00440060089011 · Cited by 326
What do we know about Private Family Firms? A Meta–Analytical Review
Carney, M., van Essen, M., Gedajlovic, E. R., & Heugens, P. P. M. A. R. (2013) · Entrepreneurship Theory and Practice · doi:10.1111/etap.12054 · Cited by 328
“First, to test for the “file drawer problem” (Rosenthal, 1979), we included a dummy variable denoting whether a study was published (1) or a working paper (0) at the time we ran our analyses.”
MethodBoard Characteristics and Corporate Social Responsibility: A Meta-Analytic Investigation
Endrikat, J., de Villiers, C., Guenther, T. W., & Guenther, E. (2020) · Business & Society · doi:10.1177/0007650320930638 · Cited by 375
“The number of null effect findings that would be needed to overturn the relationships are all above the tolerance level suggested by Rosenthal (1979).”
A Bayesian Perspective on the Reproducibility Project: Psychology
Etz, A., & Vandekerckhove, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149794 · Cited by 295
“Studies that do not meet the criteria 57 are relegated to lab file drawers [9].”
Review of developments in meta-analytic method.
Bangert‐Drowns, R. L. (1986) · Psychological Bulletin · doi:10.1037/0033-2909.99.3.388 · Cited by 327
Antecedents of organizational knowledge sharing: a meta‐analysis and critique
Witherspoon, C. L., Bergner, J. M., Cockrell, C., & Stone, D. N. (2013) · Journal of Knowledge Management · doi:10.1108/13673271311315204 · Cited by 360
A Quantitative Review of Research on Charismatic Leadership
Fuller, J. B., Patterson, C. E. P., Hester, K., & Stringer, D. Y. (1996) · Psychological Reports · doi:10.2466/pr0.1996.78.1.271 · Cited by 326
Ioannidis, J., & Trikalinos, T. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.019 · Cited by 352
Social Influence and Exercise: A Meta-Analysis
Carron, A. V., Hausenblas, H. A., & Mack, D. E. (1996) · Journal of Sport and Exercise Psychology · doi:10.1123/jsep.18.1.1 · Cited by 326
Spencer, C. M., & Stith, S. (2018) · Trauma, Violence, & Abuse · doi:10.1177/1524838018781101 · Cited by 289
“The recommended number of studies for the classic fail-safe N is determined by multiplying the number of effect sizes by 5 and then adding 10 to that number (Rosenthal, 1979).”
BackgroundMethodInfluentialCram, W. A., D’Arcy, J., & Proudfoot, J. G. (2019) · MIS Quarterly · doi:10.25300/misq/2019/15117 · Cited by 367
Do Animal-Assisted Activities Effectively Treat Depression? A Meta-Analysis
Souter, M. A., & Miller, M. D. (2007) · Anthrozoös · doi:10.2752/175303707x207954 · Cited by 324
Richard-Devantoy, S., Berlim, M. T., & Jollant, F. (2014) · Psychological Medicine · doi:10.1017/s0033291713002304 · Cited by 324
Meta-Analysis of Theory of Mind and Peer Popularity in the Preschool and Early School Years
Slaughter, V., Imuta, K., Peterson, C. C., & Henry, J. D. (2015) · Child Development · doi:10.1111/cdev.12372 · Cited by 324
Multidimensional Perfectionism and Burnout: A Meta-Analysis
Hill, A. P., & Curran, T. (2016) · Personality and Social Psychology Review · doi:10.1177/1088868315596286 · Cited by 389
Measuring shared team mental models: A meta-analysis.
DeChurch, L. A., & Mesmer‐Magnus, J. (2010) · Group Dynamics Theory Research and Practice · doi:10.1037/a0017455 · Cited by 359
Twenty Years of Stereotype Threat Research: A Review of Psychological Mediators
Pennington, C. R., Heim, D., Levy, A. R., & Larkin, D. T. (2016) · PLOS ONE · doi:10.1371/journal.pone.0146487 · Cited by 324
“To control for potential publication bias [59,60,61], the lead author also enquired about any ‘in press’ articles by sending out a call for papers through the European Association for Social Psychology.”
BackgroundBailey, T. H., Álvarez‐Jiménez, M., Garcia-Sanchez, A. M., Hulbert, C. A., Barlow, E., & Bendall, S. (2017) · Schizophrenia Bulletin · doi:10.1093/schbul/sbx161 · Cited by 325
A Meta-Analytic Review of Stand-Alone Interventions to Improve Body Image
Alleva, J. M., Sheeran, P., Webb, T. L., Martijn, C., & Miles, E. (2015) · PLoS ONE · doi:10.1371/journal.pone.0139177 · Cited by 388
“We used Rosenthal’s [51] recommended tolerance level of 5k + 10 (where k is the number of independent tests): If the FSN exceeds the tolerance level, the findings are considered resistant to publication bias.”
MethodResultInfluentialStatistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.58.5.646 · Cited by 340
Effects of Exercise Interventions on Body Image
Campbell, A., & Hausenblas, H. A. (2009) · Journal of Health Psychology · doi:10.1177/1359105309338977 · Cited by 364
Statistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.58.5.646 · Cited by 323
Career-intervention outcome: A replication and extension of Oliver and Spokane (1988).
Whiston, S. C., Sexton, T. L., & Lasoff, D. L. (1998) · Journal of Counseling Psychology · doi:10.1037/0022-0167.45.2.150 · Cited by 322
Moderators of the self-congruity effect on consumer decision-making: A meta-analysis
Aguirre‐Rodriguez, A., Bošnjak, M., & Sirgy, M. J. (2011) · Journal of Business Research · doi:10.1016/j.jbusres.2011.07.031 · Cited by 353
Szászi, B., Pálinkás, A., Pálfi, B., Szollosi, A., & Aczél, B. (2017) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2035 · Cited by 336
The bases of teacher expectancies: A meta-analysis.
Dusek, J. B., & Joseph, G. E. (1983) · Journal of Educational Psychology · doi:10.1037/0022-0663.75.3.327 · Cited by 321
Incorporating uncertainty in predictive species distribution modelling
Beale, C. M., & Lennon, J. J. (2011) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2011.0178 · Cited by 336
Cross-National Predictors of Crime: A Meta-Analysis
Nivette, A. E. (2011) · Homicide Studies · doi:10.1177/1088767911406397 · Cited by 323
The Use of Structural Equation Modeling in Counseling Psychology Research
Martens, M. P. (2005) · The Counseling Psychologist · doi:10.1177/0011000004272260 · Cited by 343
Understanding the Role and Methods of Meta-Analysis in IS Research
King, W. R., & He, J. (2005) · Communications of the Association for Information Systems · doi:10.17705/1cais.01632 · Cited by 318
Behavioral activation interventions for well-being: A meta-analysis
Mazzucchelli, T. G., Kane, R. T., & Rees, C. S. (2010) · The Journal of Positive Psychology · doi:10.1080/17439760903569154 · Cited by 319
“1. k0¼ k[MESk/MESc–1], where k0 is the number of effect sizes with a value of zero needed to reduce the mean effect size to MESc, k is the number of studies in the mean effect size, MESk is the weighted mean effect size, and MESc is the criterion effect size level (Rosenthal, 1979).”
BackgroundVitamin A supplementation in infectious diseases: a meta-analysis.
Glasziou, P., & Mackerras, D. (1993) · BMJ · doi:10.1136/bmj.306.6874.366 · Cited by 319
What’s in a p? Reassessing best practices for conducting and reporting hypothesis-testing research
Meyer, K. E., van Witteloostuijn, A. C., & Beugelsdijk, S. (2017) · Journal of International Business Studies · doi:10.1057/s41267-017-0078-8 · Cited by 323
“…is known as the ‘‘file-drawer problem’’ is very common: scientific studies with negative or nil-results often Received: 15 November 2016 Revised: 10 March 2017 Accepted: 14 March 2017 Online publication date: 19 May 2017 remain unpublished (Rosenthal, 1979; Rothstein, Sutton, & Borenstein, 2005).”
BackgroundThe cognitive profile of ALS: a systematic review and meta-analysis update
Beeldman, E., Raaphorst, J., Klein Twennaar, M., de Visser, M., Schmand, B. A., & de Haan, R. J. (2016) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp-2015-310734 · Cited by 313
Personality and political orientation: Meta-analysis and test of a Threat-Constraint Model
Sibley, C. G., Osborne, D., & Duckitt, J. H. (2012) · Journal of Research in Personality · doi:10.1016/j.jrp.2012.08.002 · Cited by 335
Lincoln, T. M., Wilhelm, K. L., & Nestoriuc, Y. (2007) · Schizophrenia Research · doi:10.1016/j.schres.2007.07.022 · Cited by 317
Glynn, C. J., Hayes, A. F., & Shanahan, J. E. (1997) · Public Opinion Quarterly · doi:10.1086/297808 · Cited by 317
Zhu, X., Yin, S., Lang, M., He, R., & Li, J. (2016) · Ageing Research Reviews · doi:10.1016/j.arr.2016.07.003 · Cited by 340
The effects of consumer attitude on green purchase intention: A meta-analytic path analysis
Zaremohzzabieh, Z., Ismail, N., Ahrari, S., & Samah, A. A. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.10.053 · Cited by 360
The potential of immersive virtual reality to enhance learning: A meta-analysis
Coban, M., Bolat, Y. I., & Goksu, I. (2022) · Educational Research Review · doi:10.1016/j.edurev.2022.100452 · Cited by 354
Eden, D. (2002) · Academy of Management Journal · doi:10.5465/amj.2002.7718946 · Cited by 318
Effects of digital game-based STEM education on students’ learning achievement: a meta-analysis
Wang, L.-H., Chen, B., Hwang, G.-J., Guan, J.-Q., & Wang, Y.-Q. (2022) · International Journal of STEM Education · doi:10.1186/s40594-022-00344-0 · Cited by 316
“The classic fail-safe N test is a procedure to evaluate whether publication bias can be ignored (Rosenthal, 1979).”
BackgroundMobile Health Interventions for Improving Health Outcomes in Youth: A Meta-analysis
Fedele, D. A., Cushing, C. C., Fritz, A., Amaro, C. M., & Ortega, A. (2017) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2017.0042 · Cited by 316
“omnibus test, 634 null findings would be necessary to overturn the findings presented herein.(51)”
BackgroundSchmidt, S., Roesler, U., Kusserow, T., & Rau, R. (2012) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2012.711523 · Cited by 362
Brown, H. E., Pearson, N., Braithwaite, R. E., Brown, W. J., & Biddle, S. J. H. (2013) · Sports Medicine · doi:10.1007/s40279-012-0015-8 · Cited by 313
Koricheva, J., Nykänen, H., & Gianoli, E. (2004) · The American Naturalist · doi:10.1086/382601 · Cited by 318
A Theory-Based Meta-Analysis of Research on Instruction.
Marzano, R. J. (1998) · Cited by 309
Update on Bispectral Index monitoring
Johansen, J. W. (2006) · Best practice & research. Clinical anaesthesiology · doi:10.1016/j.bpa.2005.08.004 · Cited by 312
Obesity and renal cell cancer – a quantitative review
Bergström, A., Hsieh, C.-C., Lindblad, P., Lu, C., Cook, N. R., & Wolk, A. (2001) · British Journal of Cancer · doi:10.1054/bjoc.2001.2040 · Cited by 316
Ecological Consequences of Shoreline Hardening: A Meta-Analysis
Gittman, R. K., Scyphers, S. B., Smith, C. S., Neylan, I. P., & Grabowski, J. H. (2016) · BioScience · doi:10.1093/biosci/biw091 · Cited by 318
“In addition, when a significant effect size was observed, we calculated Rosenthal’s fail-safe number, which is the number of unpublished studies, with a mean effect of zero, required to eliminate a significant overall effect size (Rosenthal 1979, Møller and Jennions 2001).”
MethodOutcome-Wide Longitudinal Designs for Causal Inference: A New Template for Empirical Studies
VanderWeele, T. J., Mathur, M. B., & Chen, Y. (2020) · Statistical Science · doi:10.1214/19-sts728 · Cited by 338
“It has been frequently noted that one problematic aspect of current practices in scientific publishing is that it is difficult to publish null results (Rosenthal, 1979, Ziliak and McCloskey, 2008).”
BackgroundPublication Bias (The "File-Drawer Problem") in Scientific Inference
Scargle, J. D. (1999) · arXiv · doi:10.48550/arxiv.physics/9909033 · Cited by 313
Widespread evidence for incipient ecological speciation: a meta‐analysis of isolation‐by‐ecology
Shafer, A. B. A., & Wolf, J. B. W. (2013) · Ecology Letters · doi:10.1111/ele.12120 · Cited by 323
Metadata
- Authors
- Rosenthal, R.
- Affiliations
- Harvard University (US)
- Corresponding author
- Rosenthal, Robert
- Venue
- Psychological Bulletin
- Published
- 1979-05
- Volume
- 86
- Issue
- 3
- Pages
- 638-641
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2006546769
- Cited by
- 9,068
- References
- 16
- Citations by year
- 2026: 421 · 2025: 521 · 2024: 509 · 2023: 526 · 2022: 525 · 2021: 536 · 2020: 612 · 2019: 607 · 2018: 492 · 2017: 436 · 2016: 417 · 2015: 392 · 2014: 337 · 2013: 289 · 2012: 252
- FWCI
- 26.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- file drawer problem, negative results, publication bias, non-significant results, type I error, selective reporting, behavioral science, evidence synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
