The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
The file drawer problem and tolerance for null results.
Abstract
For any given research area, one cannot tell how many studies have been con-ducted but never reported. The extreme view of the "file drawer problem " is that journals are filled with the 5 % of the studies that show Type I errors, while the file drawers are filled with the 95 % of the studies that show non-significant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed. Both behavioral researchers and statisti-cians have long suspected that the studies published in the behavioral sciences are a biased sample of the studies that are actually carried out (Bakan, 1967; McNemar, 1960; Smart, 1964; Sterling, 1959). The extreme view of this problem, the "file drawer prob-lem, " is that the journals are filled with the 5 % of the studies that show Type I errors, while the file drawers back at the lab are filled with the 95 % of the studies that show nonsignificant (e.g., p>.05) results. In the past there was very little one could do to assess the net effect of studies, tucked away in file drawers, that did not make the magic.05 level (Rosenthal & Gaito, 1963, 1964). Now, however, although no definitive solution to the problem is available, one can establish reasonable boundaries on the prob-lem and estimate the degree of damage to any research conclusion that could be done by the file drawer problem. This advance in our ability to cope with the file drawer is an outgrowth of the in-creasing interest of behavioral scientists in summarizing bodies of research literature sys-Preparation of this article was supported in part by the National Science Foundation. I would like to thank Judith A. Hall and Donald B. Rubin for their valuable improvements of an earlier version of this article. Requests for reprints should be sent to Robert
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We then calculated Rosenthal’s fail-safe N (Rosenthal, 1979), and determined that 2,153,789 effects evidencing no relationship between SA and psychopathology (g = 0) would be needed to reduce the significance level of the summary effect size to .05.”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“The file drawer effect is a perennial problem in the academic literature (Rosenthal, 1979)”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“First, we calculated the fail-safe N [177] to be 4,274, which is the theoretical number of unpublished studies with effect sizes averaging zero (no effect) that would be needed to render negligible the omnibus results.”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [177]
In the citing paper
“As we did not include unpublished work in our meta-analyses we tested for publication bias in the following ways.Firstly, funnel plots were examined.demonstrating no systematic publication biases (tables available from first author on request).Secondly..Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Presence of publication bias was assessed with funnel plots, 22 Rosenthal's fail-safe N , 23 and Egger's regression intercept. 24”
From Light Therapy for Patients With Bipolar Depression: Systematic Review and Meta-Analysis of Randomized Controlled Trials (2020) · cites this paper as [23]
In the citing paper
“In terms of study quality, there was some indication of publication bias whereby some associations between racism and health outcomes were stronger in published versus unpublished studies, consistent with the tendency for ‘null’ studies to remain unpublished (i.e., the ‘file drawer’ problem [418]).”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as [418]
In the citing paper
“An alternative, perhaps more plausible, explanation is that there could be a significant amount of unpublished negative or inconclusive results, a phenomenon referred to as the “file-drawer effect” or publication bias (23).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [23]
In the citing paper
“If either of the two last hypotheses is true, there should exist many unpublished studies with null results lying in laboratories' drawers (32).”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Publication bias was evaluated using Rosenthal's file drawer analysis (Rosenthal, 1979) and Egger's linear regression (Card, 2012), and applying ANOVA to compare the mean effect size of the published versus unpublished studies.”
From Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal (1979) suggested a fail-safe N value above 5; k + 10 reflects results that are tolerant to contradicting studies”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“We applied random-effects models to fit the relations between CE and the criterion variables, following the guidelines proposed by Rosenthal (1979), Hunter and Schmidt (2004) and Cumming (2013).”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal (1979)
In the citing paper
“The fail-safe value for this meta-analysis was computed to be 3001, which is more than 5(36) + 10.”
From Effects of digital game-based STEM education on students’ learning achievement: a meta-analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, we suggest our results may look like a lot of studies that ended up in the so-called “file drawer” [57].”
From Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory (2023) · cites this paper as [57]
In the citing paper
“Finally, publication bias was addressed by computing the fail-safe N for the correlation coefficients (Hunter & Schmidt, 2004;Rosenthal, 1979), which reports the number of field effect sizes or the tolerance for future null results, to check whether a publication bias affected our results.”
From Drivers of Consumer Adoption of e-Commerce: a Meta-Analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Using Rosenthal’s (1979) fail-safe N criterion that the value should be over five times greater than the number of studies included in the meta-analysis, the fail-safe N for all outcome groups were at least eight times larger than the criterion value, suggesting that the effect sizes are unlikely to be an artifact of bias.”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as Rosenthal (1979)
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, Rosenthal (1979) opined that a Fail-safe N exceeding the number of studies by fivefold would provide strong support for the robustness of the significant effects.”
From Extending the Theory of Planned Behavior in the Social Commerce Context: A Meta-Analytic SEM (MASEM) Approach (2023) · cites this paper as Rosenthal (1979)
In the citing paper
“The trim-and-fill method (Duval & Tweedie, 2000) tends to adjust too little when bias is strong (Carter et al., 2019), and Fail-Safe N (Rosenthal, 1979) neither tests nor adjusts for the presence of bias.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Rosenthal, 1979
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, a phenomenon known as ' publication bias ' (Rosenthal 1979).”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“However, a common problem is that results may not have been reported in dissertations or research reports, or published in scienti fi c (peer-reviewed) journals, due to non-signi fi cant or unfavorable fi ndings and, therefore, are dif fi cult to locate.”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [21]
In the citing paper
“The traditional fail-safe value is 1904, which is higher than the threshold value of 100, i.e., [5(18) + 10], indicating that publication bias is insignificant, and the study's results are reliable.”
From A meta-analysis to explore the role of design thinking in enhancing creativity as learning outcomes in STEM education (2026) · cites this paper as Rosenthal, 1979
In the citing paper
“The Q statistic was used to assess heterogeneity30, and publication bias was estimated using Rosenthal’s fail-safe number with α = 0.0531.”
From The limited prosocial effects of meditation: A systematic review and meta-analysis (2018) · cites this paper as [31]
In the citing paper
“Rosenthal's fail-safe number (NR) was calculated, estimating the number of unpublished studies with non-significant results needed to bring an observed result to non-significance (NR ⩾ 5k + 10 to rule out a file drawer problem) (Rosenthal, 1979).”
From Efficacy of non-invasive brain stimulation on cognitive functioning in brain disorders: a meta-analysis (2020) · cites this paper as Rosenthal, 1979
In the citing paper
“In case of statistically significant effect sizes, it was planned to examine the robustness of findings by calculating the Failsafe N (number of non‐significant studies that would bring the p‐value to non‐significant (p > 0.05)) 40.”
From Cognitive behavioral therapies for informal caregivers of patients with cancer and cancer survivors: a systematic review and meta-analysis (2017) · cites this paper as [40]
In the citing paper
“With negative results buried in file drawers (55), conclusions drawn from the published record can be misleading, and 'false facts' can become canonized in the literature (56).”
From Misinformation in and about science (2021) · cites this paper as [55]
In the citing paper
“the fail-safe N (i.e. the file drawer statistic) was used to determine how many additional studies would be necessary to turn a significant ES non-significant (32).”
From Peripheral cytokine and chemokine alterations in depression: a meta‐analysis of 82 studies (2017) · cites this paper as [32]
In the citing paper
“File drawer analysis (Rosenthal, 1979) revealed that 1919 unpublished, filed, or unretrieved studies averaging null results would be required to bring the average unweighted effect size to nonsignificance.”
From Happy to help? A systematic review and meta-analysis of the effects of performing acts of kindness on the well-being of the actor (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's (1979) fail-safe N analysis reveals 186 unpublished papers (i.e., null effect) are needed to be included in the meta-analysis in order to invalidate the population effect size of small to medium (i.e., .235)found in this review.”
From A meta-analytic review of the efficacy of physical exercise interventions on cognition in individuals with autism spectrum disorder and ADHD (2016) · cites this paper as Rosenthal's (1979)
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, the so called ' publication bias ' (Rosenthal, 1979).”
From The effects of physical activity interventions on psychosocial outcomes in adolescents: a meta-analytic review (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's fail-safe N 37 calculated 115 missing studies would be required to increase the two-tailed p-value to above .05.”
From Anxiety in Informal Dementia Carers: A Meta-Analysis of Prevalence (2020) · cites this paper as [37]
In the citing paper
“Publication bias was investigated with Begg's rank correlation test for funnel plot asymmetry (Begg and Mazumdar, 1994) and where significant effects were present Rosenthal's file drawer analysis was also conducted (Rosenthal, 1979).”
From Meta-analysis of the effects of intranasal oxytocin on interpretation and expression of emotions (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“Over time, this practice artificially inflates Type I error and may make it appear that relationships exist when in reality they do not (Rosenthal et al., 1979).”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“However, a potential problem with meta-analysis is the file drawer problem, where studies are “filed away” unpublished if they fail to find statistically significant relationships.”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“Meta-analyses are necessarily based on the available literature and hence may be affected by the potential publication-bias disfavoring null results (Rosenthal 1979).”
From Parsing the neural correlates of moral cognition: ALE meta-analysis on morality, theory of mind, and empathy (2012) · cites this paper as Rosenthal 1979
In the citing paper
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
From The Extent and Consequences of P-Hacking in Science (2015) · cites this paper as [7]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G. E., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,518
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718 · cited 1×
“Perhaps the simplest, most useful way of computing the overall p of a, set of research studies is the method of adding Zs (Cochran, 1954; Mosteller & Bush, 1954; Rosenthal, 1978).”
MethodMeta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154
Experimenter Effects in Behavioral Research
McGuigan, F. J., & Rosenthal, R. (1968) · The American Journal of Psychology · doi:10.2307/1420816 · Cited by 1,633
Self-citationArtifact in behavioral research
Rosenthal, R., & Rosnow, R. L. (1969) · Medical Entomology and Zoology · Cited by 1,186
Self-citationSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964
Combining results of independent studies.
Rosenthal, R. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.1.185 · Cited by 910
Self-citationGender effects in decoding nonverbal cues.
Hall, J. A. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.4.845 · Cited by 766
Wortman, C. B., Rosenthal, R., & Rosnow, R. L. (1976) · The American Journal of Psychology · doi:10.2307/1421418 · Cited by 428
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Items 201–300 of 1,374
Lee, S. S., Humphreys, K. L., Flory, K., Liu, R., & Glass, K. (2011) · Clinical Psychology Review · doi:10.1016/j.cpr.2011.01.006 · Cited by 875
Depression and memory impairment: A meta-analysis of the association, its pattern, and specificity.
Burt, D. B., Zembar, M. J., & Niederehe, G. (1995) · Psychological Bulletin · doi:10.1037/0033-2909.117.2.285 · Cited by 903
Yang, Y., & Raine, A. (2009) · Psychiatry Research: Neuroimaging · doi:10.1016/j.pscychresns.2009.03.012 · Cited by 888
Hakamata, Y., Lissek, S., Bar-Haim, Y., Britton, J. C., Fox, N. A., Leibenluft, E., Ernst, M., & Pine, D. S. (2010) · Biological Psychiatry · doi:10.1016/j.biopsych.2010.07.021 · Cited by 869
Contingency management for treatment of substance use disorders: a meta‐analysis
Prendergast, M., Podus, D., Finney, J., Greenwell, L., & Roll, J. (2006) · Addiction · doi:10.1111/j.1360-0443.2006.01581.x · Cited by 877
Gender differences in temperament: A meta-analysis.
Else-Quest, N. M., Hyde, J. S., Goldsmith, H. H., & Van Hulle, C. A. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.1.33 · Cited by 1,169
Interparental Conflict and Parenting Behaviors: A Meta‐Analytic Review
Krishnakumar, A., & Buehler, C. A. (2000) · Family Relations · doi:10.1111/j.1741-3729.2000.00025.x · Cited by 876
Multimorbidity and depression: A systematic review and meta-analysis
Read, J. R., Sharpe, L., Modini, M., & Dear, B. F. (2017) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.06.009 · Cited by 869
Stalking the elusive “vividness” effect
Taylor, S. E., & Thompson, S. C. (1982) · Psychological Review · doi:10.1037/0033-295x.89.2.155 · Cited by 913
The Contact Hypothesis Revisited
Paluck, E. L., Green, S. A., & GREEN, D. P. (2017) · Behavioural Public Policy · doi:10.1017/bpp.2018.25 · Cited by 931
“This pattern is symptomatic of a ‘file drawer problem’, in which studies are more likely to be reported when they show significant results (Rosenthal, 1979).”
ResultSystematic Review and Evidence Synthesis as a Practice and Scholarship Tool
Briner, R. B., & Denyer, D. (2012) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199763986.013.0007 · Cited by 944
Shame, guilt, and depressive symptoms: A meta-analytic review.
Kim, S., Thibodeau, R., & Jorgensen, R. S. (2011) · Psychological Bulletin · doi:10.1037/a0021466 · Cited by 866
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
Mayer, A., Takimoto, M., Fritz, E., Schellander, G., Kofler, K., & Ludwig, H. (1993) · Cancer · doi:10.1002/1097-0142(19930415)71:8 · Cited by 407
Global prevalence of gaming disorder: A systematic review and meta-analysis
Stevens, M. W., Dorstyn, D., Delfabbro, P. H., & King, D. L. (2021) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867420962851 · Cited by 863
Mesmer‐Magnus, J., & Viswesvaran, C. (2005) · Journal of Business Ethics · doi:10.1007/s10551-005-0849-1 · Cited by 857
Gender differences in self-esteem: A meta-analysis.
Kling, K. C., Hyde, J. S., Showers, C. J., & Buswell, B. N. (1999) · Psychological Bulletin · doi:10.1037/0033-2909.125.4.470 · Cited by 1,438
Exercise as a treatment for depression: A meta-analysis
Kvam, S., Kleppe, C. L., Nordhus, I. H., & Hovland, A. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.03.063 · Cited by 837
“A fail-safe N of five times the number of included studies in the meta-analysis plus 10 (5kþ10) is suggested as a standard for considering the results robust (Rosenthal, 1979).”
BackgroundDeficits in facial affect recognition among antisocial populations: A meta-analysis
Marsh, A. A., & Blair, J. (2007) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2007.08.003 · Cited by 849
Fox, K. C. R., Nijeboer, S., Dixon, M. L., Floman, J. L., Ellamil, M., Rumak, S. P., Sedlmeier, P., & Christoff, K. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2014.03.016 · Cited by 804
Firm Innovativeness and Its Performance Outcomes: A Meta-Analytic Review and Theoretical Integration
Rubera, G., & Kirca, A. H. (2012) · Journal of Marketing · doi:10.1509/jm.10.0494 · Cited by 877
Association Between Bullying and Psychosomatic Problems: A Meta-analysis
Gini, G., & Pozzoli, T. (2009) · PEDIATRICS · doi:10.1542/peds.2008-1215 · Cited by 928
A social comparison theory meta-analysis 60+ years on.
Gerber, J. P., Wheeler, L., & Suls, J. (2018) · Psychological Bulletin · doi:10.1037/bul0000127 · Cited by 808
A meta-analysis of risk factors for post-traumatic stress disorder in children and adolescents
Trickey, D., Siddaway, A. P., Meiser-Stedman, R., Serpell, L., & Field, A. P. (2012) · Clinical Psychology Review · doi:10.1016/j.cpr.2011.12.001 · Cited by 828
THE RELATIONSHIPS OF AGE WITH JOB ATTITUDES: A META-ANALYSIS
Ng, T. W. H., & Feldman, D. C. (2010) · Personnel Psychology · doi:10.1111/j.1744-6570.2010.01184.x · Cited by 930
Meta-analysis of sleep disturbance and suicidal thoughts and behaviors
Pigeon, W. R., Pinquart, M., & Conner, K. (2012) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.11r07586 · Cited by 815
Hadfield, J. D., & Nakagawa, S. (2010) · Journal of Evolutionary Biology · doi:10.1111/j.1420-9101.2009.01915.x · Cited by 829
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 876
Chen, M. J., Fan, X., & Moe, S. T. (2002) · Journal of Sports Sciences · doi:10.1080/026404102320761787 · Cited by 837
Intelligent tutoring systems and learning outcomes: A meta-analysis
Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014) · Journal of Educational Psychology · doi:10.1037/a0037123 · Cited by 840
Adjusting for publication bias in the presence of heterogeneity
Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003) · Statistics in Medicine · doi:10.1002/sim.1461 · Cited by 809
Sexual assault victimization and psychopathology: A review and meta-analysis
Dworkin, E. R., Menon, S. V., Bystrynski, J., & Allen, N. E. (2017) · Clinical Psychology Review · doi:10.1016/j.cpr.2017.06.002 · Cited by 825
Hedges, L. V., Laine, R. D., & Greenwald, R. (1994) · Educational Researcher · doi:10.3102/0013189x023003005 · Cited by 837
The validity of self-report measures of proenvironmental behavior: A meta-analytic review
Kormos, C. E., & Gifford, R. (2014) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2014.09.003 · Cited by 829
Barnett, L. M., Lai, S. K., Veldman, S. L. C., Hardy, L. L., Cliff, D. P., Morgan, P. J., Zask, A., Lubans, D. R., Shultz, S. P., Ridgers, N. D., et al. (2016) · Sports Medicine · doi:10.1007/s40279-016-0495-z · Cited by 813
“Publication bias was analyzed using Rosenthal’s classic fail safe N [46].”
MethodPaap, K. R., Johnson, H. A., & Sawi, O. M. (2015) · Cortex · doi:10.1016/j.cortex.2015.04.014 · Cited by 905
“Rosenthal (1979) coined the phrase “file drawer problem” to describe the strong tendency of researchers to set aside experiments with null results rather than submit them for publication.”
BackgroundBaumgart-Getz, A., Prokopy, L. S., & Floress, K. M. (2011) · Journal of Environmental Management · doi:10.1016/j.jenvman.2011.10.006 · Cited by 812
Ten simple rules for neuroimaging meta-analysis
Müller, V. I., Cieslik, E. C., Laird, A. R., Fox, P. T., Radua, J., Mataix-Cols, D., Tench, C. R., Yarkoni, T., Nichols, T. E., Turkeltaub, P. E., et al. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.11.012 · Cited by 805
“That is, there is in general in science a bias to publish mainly significant results while experiments failing to reject the null-hypothesis are often not reported (Ioannidis et al., 2014; Rosenthal, 1979).”
Haedt-Matt, A. A., & Keel, P. K. (2011) · Psychological Bulletin · doi:10.1037/a0023660 · Cited by 799
“Publication Bias Although we sought both published and unpublished manuscripts for this review, studies with non-significant findings may have been excluded from the meta-analyses due to the file-drawer phenomenon (Rosenthal, 1979).”
MethodEmployee voice behavior: A meta‐analytic test of the conservation of resources framework
Ng, T. W. H., & Feldman, D. C. (2011) · Journal of Organizational Behavior · doi:10.1002/job.754 · Cited by 844
McClure, E. B. (2000) · Psychological Bulletin · doi:10.1037/0033-2909.126.3.424 · Cited by 1,032
The relationship between procrastination and academic performance: A meta-analysis
Kim, K. R., & Seo, E. H. (2015) · Personality and Individual Differences · doi:10.1016/j.paid.2015.02.038 · Cited by 786
Are financial incentives related to performance? A meta-analytic review of empirical research.
Jenkins, G. D., Mitra, A., Gupta, N., & Shaw, J. D. (1998) · Journal of Applied Psychology · doi:10.1037/0021-9010.83.5.777 · Cited by 791
Langellotto, G., & Denno, R. F. (2004) · Oecologia · doi:10.1007/s00442-004-1497-3 · Cited by 781
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Langner, R., & Eickhoff, S. B. (2013) · Psychological Bulletin · doi:10.1037/a0030694 · Cited by 746
The Principles of Effective Correctional Treatment Also Apply To Sexual Offenders: A Meta-Analysis
Hanson, R. K., Bourgon, G., Helmus, L., & Hodgson, S. (2009) · Criminal Justice and Behavior · doi:10.1177/0093854809338545 · Cited by 774
Perioperative Single Dose Systemic Dexamethasone for Postoperative Pain
Oliveira, G. S. D., Almeida, M., Benzon, H. T., & McCarthy, R. J. (2011) · Anesthesiology · doi:10.1097/aln.0b013e31822a24c2 · Cited by 780
The Relationship of Achievement Motivation to Entrepreneurial Behavior: A Meta-Analysis
Collins, C. J., Hanges, P. J., & Locke, E. A. (2004) · Human Performance · doi:10.1207/s15327043hup1701_5 · Cited by 769
The Scientific Status of Projective Techniques
Lilienfeld, S. O., Wood, J. M., & Garb, H. N. (2000) · Gothic.net · doi:10.1111/1529-1006.002 · Cited by 793
p -Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614553988 · Cited by 717
Publication bias: the case for an international registry of clinical trials.
Simes, R. J. (1986) · Journal of Clinical Oncology · doi:10.1200/jco.1986.4.10.1529 · Cited by 747
Parental Divorce and Adult Well-Being: A Meta-Analysis
Amato, P. R., & Keith, B. (1991) · Journal of Marriage and the Family · doi:10.2307/353132 · Cited by 756
The road to forgiveness: A meta-analytic synthesis of its situational and dispositional correlates
Fehr, R., Gelfand, M. J., & Nag, M. (2010) · Psychological Bulletin · doi:10.1037/a0019993 · Cited by 749
Riggle, R. J., Edmondson, D. R., & Hansen, J. D. (2008) · Journal of Business Research · doi:10.1016/j.jbusres.2008.05.003 · Cited by 773
Ten years of research on group size and helping.
Latané, B., & Nida, S. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.89.2.308 · Cited by 749
Domain-Specific Physical Activity and Mental Health: A Meta-analysis
White, R. L., Babic, M. J., Parker, P. D., Lubans, D. R., Astell-Burt, T., & Lonsdale, C. (2017) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.12.008 · Cited by 715
“A larger fail-safe N, relative to the number of included studies, suggests that unpublished studies not included in the review would be unlikely to overturn the effect reported.(40) To assess the risk of bias across studies, Egger’s regression asymmetry tests were conducted on the two overall meta-analyses (i.”
MethodStanley, T. (2005) · Journal of Economic Surveys · doi:10.1111/j.0950-0804.2005.00250.x · Cited by 489
Interactions between type of instruction and type of language feature: A meta-analysis
Spada, N., & Tomita, Y. (2010) · Language Learning · doi:10.1111/j.1467-9922.2010.00562.x · Cited by 746
Gender differences in moral orientation: A meta-analysis.
Jaffee, S. R., & Hyde, J. S. (2000) · Psychological Bulletin · doi:10.1037/0033-2909.126.5.703 · Cited by 742
Brinckmann, J., Grichnik, D., & Kapsa, D. (2008) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2008.10.007 · Cited by 736
Chris Fraley, R. (2002) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0602_03 · Cited by 1,066
Publication bias and clinical trials
Dickersin, K., Chan, S., Chalmersx, T. C., Sacks, H. S., & Smith, H. (1987) · Controlled Clinical Trials · doi:10.1016/0197-2456(87)90155-3 · Cited by 715
A meta-analysis of the correlates of role conflict and ambiguity.
Fisher, C. D., & Gitelson, R. (1983) · Journal of Applied Psychology · doi:10.1037/0021-9010.68.2.320 · Cited by 444
Heritability of personality: A meta-analysis of behavior genetic studies.
Vukasović, T., & Bratko, D. (2015) · Psychological Bulletin · doi:10.1037/bul0000017 · Cited by 747
Prejudice Reduction: Progress and Challenges
Paluck, E. L., Porat, R., Clark, C. S., & Green, D. P. (2021) · Annual Review of Psychology · doi:10.1146/annurev-psych-071620-030619 · Cited by 686
“The preference for significant findings creates a “file drawer problem” (Rosenthal 1979) whereby studies with small effects go unpublished and unnoticed PUBLICATION BIAS IN THE PREJUDICE REDUCTION LITERATURE Publication bias occurs when the direction or strength of a study’s outcome influences…”
BackgroundBELANGER, H. G., CURTISS, G., DEMERY, J. A., LEBOWITZ, B. K., & VANDERPLOEG, R. D. (2005) · Journal of the International Neuropsychological Society · doi:10.1017/s1355617705050277 · Cited by 715
O’Dea, R. E., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Gurevitch, J., Page, M. J., Stewart, G. B., Moher, D., et al. (2021) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.12721 · Cited by 825
Effects of self-assessment on self-regulated learning and self-efficacy: Four meta-analyses
Panadero, E., Jonsson, A., & Botella, J. (2017) · Educational Research Review · doi:10.1016/j.edurev.2017.08.004 · Cited by 712
“In studies like Zimmerman and Kitsantas (1996), students are provided a protocol or log to record their performance.”
BackgroundZachariae, R., Lyby, M. S., Ritterband, L. M., & O'Toole, M. S. (2016) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2015.10.004 · Cited by 704
Climate for Creativity: A Quantitative Review
Hunter, S. T., Bedell, K. E., & Mumford, M. D. (2007) · Creativity Research Journal · doi:10.1080/10400410709336883 · Cited by 709
Null hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090
Brown, K. H., Peerson, J. M., Rivera, J. Á., & Allen, L. H. (2002) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/75.6.1062 · Cited by 704
Individual-level interventions to reduce college student drinking: A meta-analytic review
Carey, K. B., Scott-Sheldon, L. A. J., Carey, M. P., & DeMartini, K. S. (2007) · Addictive Behaviors · doi:10.1016/j.addbeh.2007.05.004 · Cited by 699
A meta-analytic review of ethical leadership outcomes and moderators
Bedi, A., Alpaslan, C. M., & Green, S. (2016) · Journal of Business Ethics · doi:10.1007/s10551-015-2625-1 · Cited by 698
“Publication bias refers to the selective publishing of studies with significant results (Rosenthal 1979). According to Rosenthal (1979), studies producing non-significant or unexpected results are less likely to get published and are more likely to stay buried away in file drawers; this phenomenon is also known as the…”
BackgroundThe Relation of Self-Efficacy Measures to Sport Performance: A Meta-Analytic Review
Moritz, S. E., Feltz, D. L., Fahrbach, K. R., & Mack, D. E. (2000) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.2000.10608908 · Cited by 792
A large-scale meta-analysis relating the Hare measures of psychopathy to antisocial conduct.
Leistico, A.-M. R., Salekin, R. T., DeCoster, J., & Rogers, R. (2007) · Law and Human Behavior · doi:10.1007/s10979-007-9096-6 · Cited by 696
Do European agroforestry systems enhance biodiversity and ecosystem services? A meta-analysis
Torralba, M., Fagerholm, N., Burgess, P. J., Moreno, G., & Plieninger, T. (2016) · Agriculture Ecosystems & Environment · doi:10.1016/j.agee.2016.06.002 · Cited by 722
“We used the fail-safe N method (Rosenthal, 1979) and calculated a funnel plot comparing effect sizes and variance to visually explore the publication bias (Gurevitch et al., 2001).”
MethodLinde, K., Clausius, N., Ramirez, G., Melchart, D., Eitel, F., Hedges, L. V., & Jonas, W. B. (1997) · The Lancet · doi:10.1016/s0140-6736(97)02293-9 · Cited by 979
Marshall, S. J., & Biddle, S. (2001) · Annals of Behavioral Medicine · doi:10.1207/s15324796abm2304_2 · Cited by 691
Bai, S., Hew, K. F., & Huang, B. (2020) · Educational Research Review · doi:10.1016/j.edurev.2020.100322 · Cited by 994
Fox, K. C. R., Dixon, M. L., Nijeboer, S., Girn, M., Floman, J. L., Lifshitz, M., Ellamil, M., Sedlmeier, P., & Christoff, K. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.03.021 · Cited by 656
Bell, S. T., Villado, A. J., Lukasik, M. A., Belau, L., & Briggs, A. L. (2011) · Journal of Management · doi:10.1177/0149206310365001 · Cited by 907
Madigan, D. J., & Kim, L. E. (2021) · Teaching and Teacher Education · doi:10.1016/j.tate.2021.103425 · Cited by 982
Gender differences in the prevalence of nonsuicidal self-injury: A meta-analysis
Bresin, K., & Schoenleber, M. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.02.009 · Cited by 678
“All three methods of the fail-safe-N procedure (Orwin, 1983; Rosenberg, 2005; Rosenthal, 1979), indicated that thousands of unpublished studies would need to exist to reduce to overall effect size to 1 (range: 4503–26,931), which is inconsistent with publication bias.”
A survey and critique of multiagent deep reinforcement learning
Hernández-Leal, P., Kartal, B., & Taylor, M. E. (2019) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-019-09421-1 · Cited by 799
Machine learning for medical imaging: methodological failures and recommendations for the future
Varoquaux, G. P., & Cheplygina, V. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00592-y · Cited by 815
Outcomes of Meaningful Work: A Meta‐Analysis
Allan, B. A., Batz-Barbarich, C. L., Sterling, H. M., & Tay, L. (2018) · Journal of Management Studies · doi:10.1111/joms.12406 · Cited by 721
Primate handedness reconsidered
MacNeilage, P. F., Studdert‐Kennedy, M., & Lindblom, B. E. F. (1987) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00047695 · Cited by 676
The Empirical Status of Social Learning Theory: A Meta‐Analysis
Pratt, T. C., Cullen, F. T., Sellers, C. S., Winfree, L. T., Madensen, T. D., Daigle, L. E., Fearn, N. E., & Gau, J. M. (2009) · Justice Quarterly · doi:10.1080/07418820903379610 · Cited by 681
Stubbs, B., Vancampfort, D., Rosenbaum, S., Firth, J., Cosco, T., Veronese, N., Salum, G. A., & Schuch, F. B. (2017) · Psychiatry Research · doi:10.1016/j.psychres.2016.12.020 · Cited by 666
Shimokawa, K., Lambert, M. J., & Smart, D. W. (2010) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0019247 · Cited by 657
Effects of Systematic Formative Evaluation: A Meta-Analysis
Fuchs, L. S., & Fuchs, D. (1986) · Exceptional Children · doi:10.1177/001440298605300301 · Cited by 669
Sugiyama, D., Nishimura, K., Tamaki, K., Tsuji, G., Nakazawa, T., Morinobu, A., & Kumagai, S. (2009) · Annals of the Rheumatic Diseases · doi:10.1136/ard.2008.096487 · Cited by 696
Landscape-moderated biodiversity effects of agri-environmental management: a meta-analysis
Batáry, P., Báldi, A., Kleijn, D., & Tscharntke, T. (2011) · Proceedings of the Royal Society B: Biological Sciences · doi:10.1098/rspb.2010.1923 · Cited by 661
A meta‐analytic examination of the relationship between job satisfaction and subjective well‐being
Bowling, N. A., Eschleman, K. J., & Wang, Q. (2010) · Journal of Occupational and Organizational Psychology · doi:10.1348/096317909x478557 · Cited by 764
The Effects of Negative Political Campaigns: A Meta-Analytic Reassessment
Lau, R. R., Sigelman, L., & Rovner, I. B. (2007) · The Journal of Politics · doi:10.1111/j.1468-2508.2007.00618.x · Cited by 660
Psychophysiology of aggression, psychopathy, and conduct problems: A meta-analysis
Lorber, M. F. (2004) · Psychological Bulletin · doi:10.1037/0033-2909.130.4.531 · Cited by 648
Polanin, J. R., Espelage, D. L., & Pigott, T. (2012) · School Psychology Review · doi:10.1080/02796015.2012.12087375 · Cited by 696
Metadata
- Authors
- Rosenthal, R.
- Affiliations
- Harvard University (US)
- Corresponding author
- Rosenthal, Robert
- Venue
- Psychological Bulletin
- Published
- 1979-05
- Volume
- 86
- Issue
- 3
- Pages
- 638-641
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2006546769
- Cited by
- 9,068
- References
- 16
- Citations by year
- 2026: 421 · 2025: 521 · 2024: 509 · 2023: 526 · 2022: 525 · 2021: 536 · 2020: 612 · 2019: 607 · 2018: 492 · 2017: 436 · 2016: 417 · 2015: 392 · 2014: 337 · 2013: 289 · 2012: 252
- FWCI
- 26.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- file drawer problem, negative results, publication bias, non-significant results, type I error, selective reporting, behavioral science, evidence synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
