The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
The file drawer problem and tolerance for null results.
Abstract
For any given research area, one cannot tell how many studies have been con-ducted but never reported. The extreme view of the "file drawer problem " is that journals are filled with the 5 % of the studies that show Type I errors, while the file drawers are filled with the 95 % of the studies that show non-significant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed. Both behavioral researchers and statisti-cians have long suspected that the studies published in the behavioral sciences are a biased sample of the studies that are actually carried out (Bakan, 1967; McNemar, 1960; Smart, 1964; Sterling, 1959). The extreme view of this problem, the "file drawer prob-lem, " is that the journals are filled with the 5 % of the studies that show Type I errors, while the file drawers back at the lab are filled with the 95 % of the studies that show nonsignificant (e.g., p>.05) results. In the past there was very little one could do to assess the net effect of studies, tucked away in file drawers, that did not make the magic.05 level (Rosenthal & Gaito, 1963, 1964). Now, however, although no definitive solution to the problem is available, one can establish reasonable boundaries on the prob-lem and estimate the degree of damage to any research conclusion that could be done by the file drawer problem. This advance in our ability to cope with the file drawer is an outgrowth of the in-creasing interest of behavioral scientists in summarizing bodies of research literature sys-Preparation of this article was supported in part by the National Science Foundation. I would like to thank Judith A. Hall and Donald B. Rubin for their valuable improvements of an earlier version of this article. Requests for reprints should be sent to Robert
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We then calculated Rosenthal’s fail-safe N (Rosenthal, 1979), and determined that 2,153,789 effects evidencing no relationship between SA and psychopathology (g = 0) would be needed to reduce the significance level of the summary effect size to .05.”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“The file drawer effect is a perennial problem in the academic literature (Rosenthal, 1979)”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“First, we calculated the fail-safe N [177] to be 4,274, which is the theoretical number of unpublished studies with effect sizes averaging zero (no effect) that would be needed to render negligible the omnibus results.”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [177]
In the citing paper
“As we did not include unpublished work in our meta-analyses we tested for publication bias in the following ways.Firstly, funnel plots were examined.demonstrating no systematic publication biases (tables available from first author on request).Secondly..Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Presence of publication bias was assessed with funnel plots, 22 Rosenthal's fail-safe N , 23 and Egger's regression intercept. 24”
From Light Therapy for Patients With Bipolar Depression: Systematic Review and Meta-Analysis of Randomized Controlled Trials (2020) · cites this paper as [23]
In the citing paper
“In terms of study quality, there was some indication of publication bias whereby some associations between racism and health outcomes were stronger in published versus unpublished studies, consistent with the tendency for ‘null’ studies to remain unpublished (i.e., the ‘file drawer’ problem [418]).”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as [418]
In the citing paper
“An alternative, perhaps more plausible, explanation is that there could be a significant amount of unpublished negative or inconclusive results, a phenomenon referred to as the “file-drawer effect” or publication bias (23).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [23]
In the citing paper
“If either of the two last hypotheses is true, there should exist many unpublished studies with null results lying in laboratories' drawers (32).”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Publication bias was evaluated using Rosenthal's file drawer analysis (Rosenthal, 1979) and Egger's linear regression (Card, 2012), and applying ANOVA to compare the mean effect size of the published versus unpublished studies.”
From Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal (1979) suggested a fail-safe N value above 5; k + 10 reflects results that are tolerant to contradicting studies”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“We applied random-effects models to fit the relations between CE and the criterion variables, following the guidelines proposed by Rosenthal (1979), Hunter and Schmidt (2004) and Cumming (2013).”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal (1979)
In the citing paper
“The fail-safe value for this meta-analysis was computed to be 3001, which is more than 5(36) + 10.”
From Effects of digital game-based STEM education on students’ learning achievement: a meta-analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, we suggest our results may look like a lot of studies that ended up in the so-called “file drawer” [57].”
From Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory (2023) · cites this paper as [57]
In the citing paper
“Finally, publication bias was addressed by computing the fail-safe N for the correlation coefficients (Hunter & Schmidt, 2004;Rosenthal, 1979), which reports the number of field effect sizes or the tolerance for future null results, to check whether a publication bias affected our results.”
From Drivers of Consumer Adoption of e-Commerce: a Meta-Analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Using Rosenthal’s (1979) fail-safe N criterion that the value should be over five times greater than the number of studies included in the meta-analysis, the fail-safe N for all outcome groups were at least eight times larger than the criterion value, suggesting that the effect sizes are unlikely to be an artifact of bias.”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as Rosenthal (1979)
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, Rosenthal (1979) opined that a Fail-safe N exceeding the number of studies by fivefold would provide strong support for the robustness of the significant effects.”
From Extending the Theory of Planned Behavior in the Social Commerce Context: A Meta-Analytic SEM (MASEM) Approach (2023) · cites this paper as Rosenthal (1979)
In the citing paper
“The trim-and-fill method (Duval & Tweedie, 2000) tends to adjust too little when bias is strong (Carter et al., 2019), and Fail-Safe N (Rosenthal, 1979) neither tests nor adjusts for the presence of bias.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Rosenthal, 1979
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, a phenomenon known as ' publication bias ' (Rosenthal 1979).”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“However, a common problem is that results may not have been reported in dissertations or research reports, or published in scienti fi c (peer-reviewed) journals, due to non-signi fi cant or unfavorable fi ndings and, therefore, are dif fi cult to locate.”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [21]
In the citing paper
“The traditional fail-safe value is 1904, which is higher than the threshold value of 100, i.e., [5(18) + 10], indicating that publication bias is insignificant, and the study's results are reliable.”
From A meta-analysis to explore the role of design thinking in enhancing creativity as learning outcomes in STEM education (2026) · cites this paper as Rosenthal, 1979
In the citing paper
“The Q statistic was used to assess heterogeneity30, and publication bias was estimated using Rosenthal’s fail-safe number with α = 0.0531.”
From The limited prosocial effects of meditation: A systematic review and meta-analysis (2018) · cites this paper as [31]
In the citing paper
“Rosenthal's fail-safe number (NR) was calculated, estimating the number of unpublished studies with non-significant results needed to bring an observed result to non-significance (NR ⩾ 5k + 10 to rule out a file drawer problem) (Rosenthal, 1979).”
From Efficacy of non-invasive brain stimulation on cognitive functioning in brain disorders: a meta-analysis (2020) · cites this paper as Rosenthal, 1979
In the citing paper
“In case of statistically significant effect sizes, it was planned to examine the robustness of findings by calculating the Failsafe N (number of non‐significant studies that would bring the p‐value to non‐significant (p > 0.05)) 40.”
From Cognitive behavioral therapies for informal caregivers of patients with cancer and cancer survivors: a systematic review and meta-analysis (2017) · cites this paper as [40]
In the citing paper
“With negative results buried in file drawers (55), conclusions drawn from the published record can be misleading, and 'false facts' can become canonized in the literature (56).”
From Misinformation in and about science (2021) · cites this paper as [55]
In the citing paper
“the fail-safe N (i.e. the file drawer statistic) was used to determine how many additional studies would be necessary to turn a significant ES non-significant (32).”
From Peripheral cytokine and chemokine alterations in depression: a meta‐analysis of 82 studies (2017) · cites this paper as [32]
In the citing paper
“File drawer analysis (Rosenthal, 1979) revealed that 1919 unpublished, filed, or unretrieved studies averaging null results would be required to bring the average unweighted effect size to nonsignificance.”
From Happy to help? A systematic review and meta-analysis of the effects of performing acts of kindness on the well-being of the actor (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's (1979) fail-safe N analysis reveals 186 unpublished papers (i.e., null effect) are needed to be included in the meta-analysis in order to invalidate the population effect size of small to medium (i.e., .235)found in this review.”
From A meta-analytic review of the efficacy of physical exercise interventions on cognition in individuals with autism spectrum disorder and ADHD (2016) · cites this paper as Rosenthal's (1979)
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, the so called ' publication bias ' (Rosenthal, 1979).”
From The effects of physical activity interventions on psychosocial outcomes in adolescents: a meta-analytic review (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's fail-safe N 37 calculated 115 missing studies would be required to increase the two-tailed p-value to above .05.”
From Anxiety in Informal Dementia Carers: A Meta-Analysis of Prevalence (2020) · cites this paper as [37]
In the citing paper
“Publication bias was investigated with Begg's rank correlation test for funnel plot asymmetry (Begg and Mazumdar, 1994) and where significant effects were present Rosenthal's file drawer analysis was also conducted (Rosenthal, 1979).”
From Meta-analysis of the effects of intranasal oxytocin on interpretation and expression of emotions (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“Over time, this practice artificially inflates Type I error and may make it appear that relationships exist when in reality they do not (Rosenthal et al., 1979).”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“However, a potential problem with meta-analysis is the file drawer problem, where studies are “filed away” unpublished if they fail to find statistically significant relationships.”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“Meta-analyses are necessarily based on the available literature and hence may be affected by the potential publication-bias disfavoring null results (Rosenthal 1979).”
From Parsing the neural correlates of moral cognition: ALE meta-analysis on morality, theory of mind, and empathy (2012) · cites this paper as Rosenthal 1979
In the citing paper
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
From The Extent and Consequences of P-Hacking in Science (2015) · cites this paper as [7]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G. E., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,518
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718 · cited 1×
“Perhaps the simplest, most useful way of computing the overall p of a, set of research studies is the method of adding Zs (Cochran, 1954; Mosteller & Bush, 1954; Rosenthal, 1978).”
MethodMeta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154
Experimenter Effects in Behavioral Research
McGuigan, F. J., & Rosenthal, R. (1968) · The American Journal of Psychology · doi:10.2307/1420816 · Cited by 1,633
Self-citationArtifact in behavioral research
Rosenthal, R., & Rosnow, R. L. (1969) · Medical Entomology and Zoology · Cited by 1,186
Self-citationSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964
Combining results of independent studies.
Rosenthal, R. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.1.185 · Cited by 910
Self-citationGender effects in decoding nonverbal cues.
Hall, J. A. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.4.845 · Cited by 766
Wortman, C. B., Rosenthal, R., & Rosnow, R. L. (1976) · The American Journal of Psychology · doi:10.2307/1421418 · Cited by 428
Self-citation
Cited by
Sorted by citations
Items 801–900 of 1,374
VO2max Trainability and High Intensity Interval Training in Humans: A Meta-Analysis
Bacon, A. P., Carter, R. E., Ogle, E. A., & Joyner, M. J. (2013) · PLoS ONE · doi:10.1371/journal.pone.0073182 · Cited by 311
The Politics of Fear: Is There an Ideological Asymmetry in Existential Motivation?
Jost, J. T., Stern, C., Rule, N. O., & Sterling, J. L. (2017) · Social Cognition · doi:10.1521/soco.2017.35.4.324 · Cited by 328
“For each category, we also conducted two “file-drawer” analyses to estimate the degree to which the statistical significance and size of the average effect was likely influenced by publication bias (Orwin, 1983; Rosenthal, 1979).”
MethodInsomnia and cognitive performance: A systematic review and meta-analysis
Wardle‐Pinkston, S., Slavish, D. C., & Taylor, D. J. (2019) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2019.07.008 · Cited by 320
A meta-analysis of the relationship between place attachment and pro-environmental behaviour
Daryanto, A., & Song, Z. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.09.045 · Cited by 344
“Using the remaining effect sizes minus the outliers (N=124), we calculated the fail-safe N of Rosenthal (1979), which is the number of ‘missing’ studies that are non-significant and unpublished needed to nullify the meta-analysis result.”
The association between nonsuicidal self-injury and the emotional disorders: A meta-analytic review
Bentley, K. H., Cassiello-Robbins, C. F., Vittorio, L. N., Sauer-Zavala, S. E., & Barlow, D. H. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.02.006 · Cited by 343
“…SOC = social anxiety disorder. a For studies with multiple outcomes under a diagnostic subgroup, the mean of selected outcomes was used as the effect size. b Rosenthal (1979) suggests setting a tolerance level for missing studies needed to nullify observed results at 5k + 10 (k = number of…”
MethodAffective empathy deficits in aggressive children and adolescents: A critical review
Lovett, B. J., & Sheffield, R. (2006) · Clinical Psychology Review · doi:10.1016/j.cpr.2006.03.003 · Cited by 310
Does Oxytocin Increase Trust in Humans? A Critical Review of Research
Nave, G., Camerer, C., & McCullough, M. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615600138 · Cited by 316
“It is also impossible to know whether there exist unpublished data that produced null findings and got stuck in the proverbial “file drawer” (because of publication bias; e.g., Lane, Luminet, Nave, & Mikolajczak, 2015; Rosenthal, 1979).”
BackgroundThe Relationship Between Homework Compliance and Therapy Outcomes: An Updated Meta-Analysis
Mausbach, B. T., Moore, R., Roesch, S., Cardenas, V., & Patterson, T. L. (2010) · Cognitive Therapy and Research · doi:10.1007/s10608-010-9297-z · Cited by 309
Kedzior, K. K., & Laeber, L. T. (2014) · BMC Psychiatry · doi:10.1186/1471-244x-14-136 · Cited by 309
Luebbert, K., Dahme, B., & Hasenbring, M. I. (2001) · Psycho-Oncology · doi:10.1002/pon.537 · Cited by 309
The earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research
Amrhein, V., Korner‐Nievergelt, F., & Roth, T. (2017) · PeerJ · doi:10.7717/peerj.3544 · Cited by 331
DuPaul, G. J., & Eckert, T. L. (1997) · School Psychology Review · doi:10.1080/02796015.1997.12085845 · Cited by 311
Yorke, I., White, P., Weston, A., Rafla, M., Charman, T., & Simonoff, E. A. (2018) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-018-3605-y · Cited by 327
Challenges for assessing replicability in preclinical cancer biology
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.67995 · Cited by 345
“With publication bias, negative findings are much less likely to be reported than positive findings (Greenwald, 1975; Rosenthal, 1979).”
BackgroundDew, M. A., Rosenberger, E. M., Myaskovsky, L., DiMartini, A. F., DeVito Dabbs, A. J., Posluszny, D. M., Steel, J., Switzer, G. E., Shellmer, D. A., & Greenhouse, J. B. (2015) · Transplantation · doi:10.1097/tp.0000000000000901 · Cited by 304
Effect of allelopathy on plant performance: a meta‐analysis
Zhang, Z., Liu, Y., Yuan, L., Weber, E., & van Kleunen, M. (2020) · Ecology Letters · doi:10.1111/ele.13627 · Cited by 319
Comparison of reading performance on screen and on paper: A meta-analysis
Kong, Y., Seo, Y. S., & Zhai, L. (2018) · Computers & Education · doi:10.1016/j.compedu.2018.05.005 · Cited by 306
Winter, S., Bauer, T., Strauß, P., Kratschmer, S., Paredes, D., Popescu, D., Landa, B. B., Guzmán, G., Gómez, J. A., Guernion, M., et al. (2018) · Journal of Applied Ecology · doi:10.1111/1365-2664.13124 · Cited by 340
Two-sided advertising: A meta-analysis
Eisend, M. (2006) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2005.11.001 · Cited by 305
A META-ANALYSIS OF THE PREDICTORS OF CYBERBULLYING PERPETRATION AND VICTIMIZATION
Guo, S. (2016) · Psychology in the Schools · doi:10.1002/pits.21914 · Cited by 299
Developing Early Literacy Skills: A Meta‐Analysis of Alphabet Learning and Instruction
Piasta, S. B., & Wagner, R. K. (2010) · Reading Research Quarterly · doi:10.1598/rrq.45.1.2 · Cited by 305
Perioperative Systemic Magnesium to Minimize Postoperative Pain
Oliveira, G. S. D., Castro‐Alves, L. J., Khan, J. H., & McCarthy, R. J. (2013) · Anesthesiology · doi:10.1097/aln.0b013e318297630d · Cited by 305
The Stroop revisited: a meta‐analysis of interference control in AD/HD
Van Mourik, R., Oosterlaan, J., & Sergeant, J. A. (2005) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2004.00345.x · Cited by 303
Meta‐analysis of randomized, controlled treatment trials for pediatric obsessive‐compulsive disorder
Watson, H. J., & Rees, C. S. (2008) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2007.01875.x · Cited by 304
Scott, K. D., & Taylor, G. S. (1985) · Academy of Management Journal · doi:10.2307/256116 · Cited by 304
Burger, J. M. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.90.3.496 · Cited by 303
McEvoy, G. M., & Cascio, W. F. (1987) · Academy of Management Journal · doi:10.2307/256158 · Cited by 303
Cliff, D. P., Hesketh, K. D., Vella, S. A., Hinkley, T., Tsiros, M. D., Ridgers, N. D., Carver, A., Veitch, J., Parrish, A., Hardy, L. L., et al. (2016) · Obesity Reviews · doi:10.1111/obr.12371 · Cited by 327
“classic fail-safe N and Duval and Tweedie’s trim and fill procedure (40).”
BackgroundThe Relationship Between Theory and Practice: Back to the Classics
Kessels, J., & Korthagen, F. A. J. (1996) · Educational Researcher · doi:10.3102/0013189x025003017 · Cited by 302
A meta-analysis of randomized controlled trials of laparoscopic versus conventional appendectomy
Chung, R. S. K., Rowland, D. Y., Li, P., & Diaz, J. J. (1999) · The American Journal of Surgery · doi:10.1016/s0002-9610(99)00017-3 · Cited by 302
The effects of psychological distance on abstraction: Two meta-analyses.
Soderberg, C. K., Callahan, S. P., Kochersberger, A. O., Amit, E., & Ledgerwood, A. (2014) · Psychological Bulletin · doi:10.1037/bul0000005 · Cited by 350
“To address this question, construal level theory builds on the basic idea that the same object or event can be mentally represented (or construed) at varying levels of abstraction (Medin, 1989; Rosch et al., 1976; Trope & Liberman, 2003; Vallacher & Wegner, 1987).”
BackgroundAguinis, H., Dalton, D. R., Bosco, F. A., Pierce, C. A., & Dalton, C. M. (2011) · Journal of Management · doi:10.1177/0149206310377113 · Cited by 302
Janakiraman, N., Syrdal, H. A., & Freling, R. E. (2015) · Journal of Retailing · doi:10.1016/j.jretai.2015.11.002 · Cited by 310
Varotto, A., & Spagnolli, A. (2017) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2017.03.011 · Cited by 334
Replication, falsification, and the crisis of confidence in social psychology
Earp, B. D., & Trafimow, D. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00621 · Cited by 472
“This problem, known as “publication bias,” leads to a file-drawer effect whereby “negative” experimental outcomes are simply “filed away” in a researcher’s bottom drawer, rather than written up and submitted for publication (e.g., Rosenthal, 1979).”
BackgroundA meta‐analysis of the relationships of age and tenure with innovation‐related behaviour
Ng, T. W. H., & Feldman, D. C. (2013) · Journal of Occupational and Organizational Psychology · doi:10.1111/joop.12031 · Cited by 328
Depression and attributions in children and adolescents: A meta-analytic review
Gladstone, T. R. G., & Kaslow, N. J. (1995) · Research on Child and Adolescent Psychopathology · doi:10.1007/bf01447664 · Cited by 301
Rutherford, M. W., Kuratko, D. F., & Holt, D. T. (2008) · Entrepreneurship Theory and Practice · doi:10.1111/j.1540-6520.2008.00275.x · Cited by 301
Agostini, M., & van Zomeren, M. (2021) · Psychological Bulletin · doi:10.1037/bul0000256 · Cited by 301
“We used a randomeffects model analysis because of the large diversity in contexts, groups, and cultures (Lipsey & Wilson, 2001; Rosenthal, 1979).”
MethodCameron, C. D., Brown-Iannuzzi, J. L., & Payne, B. K. (2012) · Personality and Social Psychology Review · doi:10.1177/1088868312440047 · Cited by 352
Gender Differences in Responses to Moral Dilemmas
Friesdorf, R., Conway, P. J., & Gawronski, B. (2015) · Personality and Social Psychology Bulletin · doi:10.1177/0146167215575731 · Cited by 319
“Typically, there is a concern that meta-analyses may over-estimate the true size of a given effect because studies with nonsignificant results are less likely to be published (Hedges & Vevea, 1996; Rosenthal, 1979).”
ResultMultiple cues for quantification in infancy: Is number one of them?
Mix, K. S., Huttenlocher, J., & Levine, S. C. (2002) · Psychological Bulletin · doi:10.1037//0033-2909.128.2.278 · Cited by 300
López, O. L., & Hiebl, M. R. W. (2014) · Journal of Management Accounting Research · doi:10.2308/jmar-50915 · Cited by 300
Long-term psychodynamic psychotherapy in complex mental disorders: update of a meta-analysis
Leichsenring, F., & Rabung, S. (2011) · The British Journal of Psychiatry · doi:10.1192/bjp.bp.110.082776 · Cited by 312
Depressive realism: a meta-analytic review
Moore, M. T., & Fresco, D. M. (2012) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.05.004 · Cited by 291
Berlim, M. T., den Eynde, F. V., & Daskalakis, Z. J. (2012) · Neuropsychopharmacology · doi:10.1038/npp.2012.237 · Cited by 305
How Effective are Pedagogical Agents for Learning? A Meta-Analytic Review
Schroeder, N. L., Adesope, O., & Gilbert, R. B. (2013) · Journal of Educational Computing Research · doi:10.2190/ec.49.1.a · Cited by 303
Prediction of Cardiovascular Events and All-Cause Mortality With Erectile Dysfunction
Vlachopoulos, C. V., Terentes-Printzios, D. G., Ioakeimidis, N., Aznaouridis, K. A., & Stefanadis, C. (2013) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.112.966903 · Cited by 302
Kroll, A. (2015) · Public Performance & Management Review · doi:10.1080/15309576.2015.1006469 · Cited by 297
“Since published articles, the basis for a literature review, are biased toward the presentation of significant findings (Rosenthal, 1979; Sutton, 2009), reviewers will usually read more pages about the validation of hypotheses than about their rejection or other insignificant observations.”
BackgroundDonnon, T., Paolucci, E. O., & Violato, C. (2006) · Academic Medicine · doi:10.1097/01.acm.0000249878.25186.b7 · Cited by 294
Continuously Cumulating Meta-Analysis and Replicability
Braver, S. L., Thoemmes, F. J., & Rosenthal, R. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614529796 · Cited by 308
Self-citationHauk, N., Hüffmeier, J., & Krumm, S. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.01.020 · Cited by 267
A meta-analysis of treatments for perinatal depression
Sockol, L. E., Epperson, C. N., & Barber, J. P. (2011) · Clinical Psychology Review · doi:10.1016/j.cpr.2011.03.009 · Cited by 287
Abraham, L. B. (2008) · Computer Assisted Language Learning · doi:10.1080/09588220802090246 · Cited by 303
Prosocial Consequences of Interpersonal Synchrony
Rennung, M., & Göritz, A. S. (2016) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000252 · Cited by 331
“Finally, we assessed the likelihood of inclusion bias using Begg and Mazumdar’s rank correlation test (Begg & Mazumdar, 1994), Egger’s regression test (Egger, Smith, Schneider, & Minder, 1997), Rosenthal’s fail-safe N (Rosenthal, 1979), and Orwin’s fail-safe N (Orwin, 1983), as well as Duval and Tweedie’s trim and fil…”
BackgroundMethodBlut, M., Durham University Business School, U., Chong, A. Y. L., Nottingham University Business School–Ningbo, C., Tsigna, Z., Nottingham University Business School–Ningbo, C., Venkatesh, V., & Pamplin College of Business, V. T. (2022) · Journal of the Association for Information Systems · doi:10.17705/1jais.00719 · Cited by 228
Strategies for reducing employee turnover: A meta-analysis.
McEvoy, G. M., & Cascio, W. F. (1985) · Journal of Applied Psychology · doi:10.1037/0021-9010.70.2.342 · Cited by 296
Brewin, C. R., Kleiner, J. S., Vasterling, J. J., & Field, A. P. (2007) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.116.3.448 · Cited by 290
McKinney, J. S., & Kostis, W. J. (2012) · Stroke · doi:10.1161/strokeaha.112.655894 · Cited by 294
Shachar, M., & Neumann, Y. (2003) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v4i2.153 · Cited by 288
Parental socialization of child and adolescent physical activity: A meta-analysis.
Pugliese, J., & Tinsley, B. J. (2007) · Journal of Family Psychology · doi:10.1037/0893-3200.21.3.331 · Cited by 295
Dew, M. A., DiMartini, A. F., Steel, J. L., Dabbs, A. D. V., Myaskovsky, L., Unruh, M. L., & Greenhouse, J. B. (2008) · Liver Transplantation · doi:10.1002/lt.21278 · Cited by 295
Behavioral Adjustment in Children with Asthma: A Meta-Analysis
McQuaid, E. L., Kopel, S. J., & Nassau, J. H. (2001) · Journal of Developmental & Behavioral Pediatrics · doi:10.1097/00004703-200112000-00011 · Cited by 297
Attentional biases in eating disorders: A meta-analytic review of Stroop performance
Dobson, K. S., & Dozois, D. J. A. (2004) · Clinical Psychology Review · doi:10.1016/j.cpr.2003.09.004 · Cited by 311
Efficacy of biofeedback for migraine: A meta-analysis
Nestoriuc, Y., & Martin, A. (2007) · Pain · doi:10.1016/j.pain.2006.09.007 · Cited by 292
Do amphibians follow Bergmann's rule?
Ashton, K. G. (2002) · Canadian Journal of Zoology · doi:10.1139/z02-049 · Cited by 294
A cognitive approach to lie detection: A meta‐analysis
Vrij, A., Fisher, R. P., & Blank, H. (2015) · Legal and Criminological Psychology · doi:10.1111/lcrp.12088 · Cited by 326
“The fail-safeNs – the numbers of file-drawer studies with zero effects sizes needed to render the effects non-significant (Rosenberg, 2005; Rosenthal, 1979) – were comparable as well, and they indicate that publication bias is unlikely to be an issue here (certainly at the level of total accuracy).”
BackgroundHealy, S., Nacario, A., Braithwaite, R. E., & Hopper, C. A. (2018) · Autism Research · doi:10.1002/aur.1955 · Cited by 327
“Finally, a ‘Fail-Safe N’ calculation was used to determine the number of missing studies needed to nullify significant results (Duval & Tweedie, 2000a; Pearson et al., 2014; Rosenthal, 1979).”
MethodKassai, R., Futo, J., Demetrovics, Z., & Takacs, Z. K. (2019) · Psychological Bulletin · doi:10.1037/bul0000180 · Cited by 294
“In case of a robust effect, this fail-safe number should exceed 5k 10 where k is the number of contrasts included (Rosenthal, 1979).”
BackgroundSacks, H. S. (1990) · JAMA · doi:10.1001/jama.264.4.494 · Cited by 319
Medical students’ self-assessment of performance: Results from three meta-analyses
Blanch‐Hartigan, D. (2010) · Patient Education and Counseling · doi:10.1016/j.pec.2010.06.037 · Cited by 342
Casper, D. M., & Card, N. A. (2016) · Child Development · doi:10.1111/cdev.12621 · Cited by 294
Greater cognitive deterioration in women than men with Alzheimer's disease: A meta analysis
Irvine, K., Laws, K. R., Gale, T. M., & Kondel, T. K. (2012) · Journal of Clinical and Experimental Neuropsychology · doi:10.1080/13803395.2012.712676 · Cited by 290
Key factors driving attitudes towards large mammals in conflict with humans
Kansky, R., & Knight, A. T. (2014) · Biological Conservation · doi:10.1016/j.biocon.2014.09.008 · Cited by 317
Early versus late onset obsessive–compulsive disorder: Evidence for distinct subtypes
Taylor, S. J. (2011) · Clinical Psychology Review · doi:10.1016/j.cpr.2011.06.007 · Cited by 293
Contingency management in outpatient methadone treatment: a meta-analysis
Griffith, J. D., Rowan‐Szal, G. A., Roark, R. R., & Simpson, D. D. (2000) · Drug and Alcohol Dependence · doi:10.1016/s0376-8716(99)00068-x · Cited by 291
Lui, P. P. (2015) · Psychological Bulletin · doi:10.1037/a0038449 · Cited by 285
Modelling publication bias in meta-analysis: a review
Sutton, A. J., Song, F., Gilbody, S. M., & Abrams, K. R. (2000) · Statistical Methods in Medical Research · doi:10.1177/096228020000900503 · Cited by 288
Detecting Selection Bias in Meta-Analyses with Multiple Outcomes: A Simulation Study
Fernández‐Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N., Onghena, P., & Noortgate, W. V. D. (2019) · The Journal of Experimental Education · doi:10.1080/00220973.2019.1582470 · Cited by 312
“It is defined as the propensity for publishing positive, significant results to the detriment of studies that report negative or nonsignificant results (Rosenthal, 1979).”
BackgroundHalverson, T. F., Orleans-Pobee, M., Merritt, C., Sheeran, P., Fett, A.-K. J., & Penn, D. L. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.07.020 · Cited by 288
“Another potential explanation for smaller effect estimates may be the novel inclusion of unpublished findings to address the file-drawer effect (Rosenthal, 1979).”
BackgroundColl, M.-P., Hobson, H., Bird, G., & Murphy, J. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.12.012 · Cited by 262
“First, we did not consider369 publication bias (Rosenthal, 1979) and within-study biases created by the fact that most370 studies reviewed only reported significant effects from several tests.”
BackgroundThe influence of outcome desirability on optimism.
Križan, Z., & Windschitl, P. D. (2007) · Psychological Bulletin · doi:10.1037/0033-2909.133.1.95 · Cited by 297
The influence of discrete emotions on judgement and decision-making: A meta-analytic review
Angie, A. D., Connelly, S., Waples, E. P., & Kligyte, V. (2011) · Cognition & Emotion · doi:10.1080/02699931.2010.550751 · Cited by 327
Evidence of STEM Enactment Effectiveness in Asian Student Learning Outcomes
Wahono, B., Lin, P.-L., & Chang, C.-Y. (2020) · International Journal of STEM Education · doi:10.1186/s40594-020-00236-1 · Cited by 286
“Conferences, books, and dissertations were also included as data sources, namely to capture and find what is called the “file drawer” for information, which might not be published in journals (Rosenthal, 1979).”
BackgroundMcEvoy, P. M., Hyett, M. P., Shihata, S., Price, J. E., & Strachan, L. (2019) · Clinical Psychology Review · doi:10.1016/j.cpr.2019.101778 · Cited by 271
Mystery, complexity, legibility and coherence: A meta-analysis
Stamps, A. E. (2003) · Journal of Environmental Psychology · doi:10.1016/s0272-4944(03)00023-9 · Cited by 286
A global meta-analysis of the ecological impacts of nonnative crayfish
Twardochleb, L. A., Olden, J., & Larson, E. (2013) · Freshwater Science · doi:10.1899/12-203.1 · Cited by 284
A practitioner's guide to meta‐analysis
Durlak, J. A., & Lipsey, M. W. (1991) · American Journal of Community Psychology · doi:10.1007/bf00938026 · Cited by 286
Moving emotional labor beyond surface and deep acting
Mesmer‐Magnus, J., DeChurch, L. A., & Wax, A. (2011) · Organizational Psychology Review · doi:10.1177/2041386611417746 · Cited by 287
Racial Discrimination and Health Outcomes Among Racial/Ethnic Minorities: A Meta‐Analytic Review
Carter, R. T., Lau, M. Y., Johnson, V. E., & Kirkinis, K. (2017) · Journal of Multicultural Counseling and Development · doi:10.1002/jmcd.12076 · Cited by 289
Spruit, A., Goos, L., Weenink, N., Rodenburg, R., Niemeyer, H., Stams, G. J., & Colonnesi, C. (2020) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-019-00299-9 · Cited by 280
“However, a common problem is that studies may not have been published because of non-significant or unfavorable findings, and therefore are difficult to locate, the so-called “publication or file drawer bias” (Rosenthal 1979).”
BackgroundSherman, D. S., Mauser, J., Nuno, M. A., & Sherzai, D. (2017) · Neuropsychology Review · doi:10.1007/s11065-017-9363-3 · Cited by 301
“Risk of Bias - Publication Bias To examine the potential for publication bias, we performed several analyses including (i) visual inspection of funnel plot symmetry, plotting the standard difference in means to standard error (Rosenthal 1979; Sterne et al. 2000, 2011), (ii) calculation of Egger’s regression intercept…”
MethodAntecedents and dimensions of supply chain robustness: a systematic literature review
Durach, C. F., Wieland, A., & Machuca, J. A. D. (2015) · International Journal of Physical Distribution & Logistics Management · doi:10.1108/ijpdlm-05-2013-0133 · Cited by 291
DOMANSKI, M. J. (2011) · JAMA · doi:10.1001/jama.2011.99 · Cited by 283
The association between smartphone use, stress, and anxiety: A meta‐analytic review
Vahedi, Z., & Saiphoo, A. (2018) · Stress and Health · doi:10.1002/smi.2805 · Cited by 280
Efficacy of Drug Treatment in Obsessive–Compulsive Disorder: a Meta-Analytic Review
Piccinelli, M. P., Pini, S., Bellantuono, C., & Wilkinson, G. (1995) · The British Journal of Psychiatry · doi:10.1192/bjp.166.4.424 · Cited by 282
Bilingualism and working memory capacity: A comprehensive meta-analysis
Grundy, J. G., & Timmer, K. (2016) · Second language Research · doi:10.1177/0267658316678286 · Cited by 305
Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617690042 · Cited by 251
“One potential benefit of these cultural differences is that it may decrease the likelihood of publication bias stemming from the “file drawer effect” (Rosenthal, 1979).”
BackgroundHow effective are mobile devices for language learning? A meta-analysis
Sung, Y., Chang, K.-E., & Yang, J.-M. (2015) · Educational Research Review · doi:10.1016/j.edurev.2015.09.001 · Cited by 321
Henry, J. D., & Crawford, J. R. (2004) · Cognitive Neuropsychiatry · doi:10.1080/13546800344000309 · Cited by 299
Metadata
- Authors
- Rosenthal, R.
- Affiliations
- Harvard University (US)
- Corresponding author
- Rosenthal, Robert
- Venue
- Psychological Bulletin
- Published
- 1979-05
- Volume
- 86
- Issue
- 3
- Pages
- 638-641
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2006546769
- Cited by
- 9,068
- References
- 16
- Citations by year
- 2026: 421 · 2025: 521 · 2024: 509 · 2023: 526 · 2022: 525 · 2021: 536 · 2020: 612 · 2019: 607 · 2018: 492 · 2017: 436 · 2016: 417 · 2015: 392 · 2014: 337 · 2013: 289 · 2012: 252
- FWCI
- 26.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- file drawer problem, negative results, publication bias, non-significant results, type I error, selective reporting, behavioral science, evidence synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
