The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
The file drawer problem and tolerance for null results.
Abstract
For any given research area, one cannot tell how many studies have been con-ducted but never reported. The extreme view of the "file drawer problem " is that journals are filled with the 5 % of the studies that show Type I errors, while the file drawers are filled with the 95 % of the studies that show non-significant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed. Both behavioral researchers and statisti-cians have long suspected that the studies published in the behavioral sciences are a biased sample of the studies that are actually carried out (Bakan, 1967; McNemar, 1960; Smart, 1964; Sterling, 1959). The extreme view of this problem, the "file drawer prob-lem, " is that the journals are filled with the 5 % of the studies that show Type I errors, while the file drawers back at the lab are filled with the 95 % of the studies that show nonsignificant (e.g., p>.05) results. In the past there was very little one could do to assess the net effect of studies, tucked away in file drawers, that did not make the magic.05 level (Rosenthal & Gaito, 1963, 1964). Now, however, although no definitive solution to the problem is available, one can establish reasonable boundaries on the prob-lem and estimate the degree of damage to any research conclusion that could be done by the file drawer problem. This advance in our ability to cope with the file drawer is an outgrowth of the in-creasing interest of behavioral scientists in summarizing bodies of research literature sys-Preparation of this article was supported in part by the National Science Foundation. I would like to thank Judith A. Hall and Donald B. Rubin for their valuable improvements of an earlier version of this article. Requests for reprints should be sent to Robert
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We then calculated Rosenthal’s fail-safe N (Rosenthal, 1979), and determined that 2,153,789 effects evidencing no relationship between SA and psychopathology (g = 0) would be needed to reduce the significance level of the summary effect size to .05.”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“The file drawer effect is a perennial problem in the academic literature (Rosenthal, 1979)”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“First, we calculated the fail-safe N [177] to be 4,274, which is the theoretical number of unpublished studies with effect sizes averaging zero (no effect) that would be needed to render negligible the omnibus results.”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [177]
In the citing paper
“As we did not include unpublished work in our meta-analyses we tested for publication bias in the following ways.Firstly, funnel plots were examined.demonstrating no systematic publication biases (tables available from first author on request).Secondly..Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Presence of publication bias was assessed with funnel plots, 22 Rosenthal's fail-safe N , 23 and Egger's regression intercept. 24”
From Light Therapy for Patients With Bipolar Depression: Systematic Review and Meta-Analysis of Randomized Controlled Trials (2020) · cites this paper as [23]
In the citing paper
“In terms of study quality, there was some indication of publication bias whereby some associations between racism and health outcomes were stronger in published versus unpublished studies, consistent with the tendency for ‘null’ studies to remain unpublished (i.e., the ‘file drawer’ problem [418]).”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as [418]
In the citing paper
“An alternative, perhaps more plausible, explanation is that there could be a significant amount of unpublished negative or inconclusive results, a phenomenon referred to as the “file-drawer effect” or publication bias (23).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [23]
In the citing paper
“If either of the two last hypotheses is true, there should exist many unpublished studies with null results lying in laboratories' drawers (32).”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Publication bias was evaluated using Rosenthal's file drawer analysis (Rosenthal, 1979) and Egger's linear regression (Card, 2012), and applying ANOVA to compare the mean effect size of the published versus unpublished studies.”
From Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal (1979) suggested a fail-safe N value above 5; k + 10 reflects results that are tolerant to contradicting studies”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“We applied random-effects models to fit the relations between CE and the criterion variables, following the guidelines proposed by Rosenthal (1979), Hunter and Schmidt (2004) and Cumming (2013).”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal (1979)
In the citing paper
“The fail-safe value for this meta-analysis was computed to be 3001, which is more than 5(36) + 10.”
From Effects of digital game-based STEM education on students’ learning achievement: a meta-analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, we suggest our results may look like a lot of studies that ended up in the so-called “file drawer” [57].”
From Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory (2023) · cites this paper as [57]
In the citing paper
“Finally, publication bias was addressed by computing the fail-safe N for the correlation coefficients (Hunter & Schmidt, 2004;Rosenthal, 1979), which reports the number of field effect sizes or the tolerance for future null results, to check whether a publication bias affected our results.”
From Drivers of Consumer Adoption of e-Commerce: a Meta-Analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Using Rosenthal’s (1979) fail-safe N criterion that the value should be over five times greater than the number of studies included in the meta-analysis, the fail-safe N for all outcome groups were at least eight times larger than the criterion value, suggesting that the effect sizes are unlikely to be an artifact of bias.”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as Rosenthal (1979)
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, Rosenthal (1979) opined that a Fail-safe N exceeding the number of studies by fivefold would provide strong support for the robustness of the significant effects.”
From Extending the Theory of Planned Behavior in the Social Commerce Context: A Meta-Analytic SEM (MASEM) Approach (2023) · cites this paper as Rosenthal (1979)
In the citing paper
“The trim-and-fill method (Duval & Tweedie, 2000) tends to adjust too little when bias is strong (Carter et al., 2019), and Fail-Safe N (Rosenthal, 1979) neither tests nor adjusts for the presence of bias.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Rosenthal, 1979
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, a phenomenon known as ' publication bias ' (Rosenthal 1979).”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“However, a common problem is that results may not have been reported in dissertations or research reports, or published in scienti fi c (peer-reviewed) journals, due to non-signi fi cant or unfavorable fi ndings and, therefore, are dif fi cult to locate.”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [21]
In the citing paper
“The traditional fail-safe value is 1904, which is higher than the threshold value of 100, i.e., [5(18) + 10], indicating that publication bias is insignificant, and the study's results are reliable.”
From A meta-analysis to explore the role of design thinking in enhancing creativity as learning outcomes in STEM education (2026) · cites this paper as Rosenthal, 1979
In the citing paper
“The Q statistic was used to assess heterogeneity30, and publication bias was estimated using Rosenthal’s fail-safe number with α = 0.0531.”
From The limited prosocial effects of meditation: A systematic review and meta-analysis (2018) · cites this paper as [31]
In the citing paper
“Rosenthal's fail-safe number (NR) was calculated, estimating the number of unpublished studies with non-significant results needed to bring an observed result to non-significance (NR ⩾ 5k + 10 to rule out a file drawer problem) (Rosenthal, 1979).”
From Efficacy of non-invasive brain stimulation on cognitive functioning in brain disorders: a meta-analysis (2020) · cites this paper as Rosenthal, 1979
In the citing paper
“In case of statistically significant effect sizes, it was planned to examine the robustness of findings by calculating the Failsafe N (number of non‐significant studies that would bring the p‐value to non‐significant (p > 0.05)) 40.”
From Cognitive behavioral therapies for informal caregivers of patients with cancer and cancer survivors: a systematic review and meta-analysis (2017) · cites this paper as [40]
In the citing paper
“With negative results buried in file drawers (55), conclusions drawn from the published record can be misleading, and 'false facts' can become canonized in the literature (56).”
From Misinformation in and about science (2021) · cites this paper as [55]
In the citing paper
“the fail-safe N (i.e. the file drawer statistic) was used to determine how many additional studies would be necessary to turn a significant ES non-significant (32).”
From Peripheral cytokine and chemokine alterations in depression: a meta‐analysis of 82 studies (2017) · cites this paper as [32]
In the citing paper
“File drawer analysis (Rosenthal, 1979) revealed that 1919 unpublished, filed, or unretrieved studies averaging null results would be required to bring the average unweighted effect size to nonsignificance.”
From Happy to help? A systematic review and meta-analysis of the effects of performing acts of kindness on the well-being of the actor (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's (1979) fail-safe N analysis reveals 186 unpublished papers (i.e., null effect) are needed to be included in the meta-analysis in order to invalidate the population effect size of small to medium (i.e., .235)found in this review.”
From A meta-analytic review of the efficacy of physical exercise interventions on cognition in individuals with autism spectrum disorder and ADHD (2016) · cites this paper as Rosenthal's (1979)
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, the so called ' publication bias ' (Rosenthal, 1979).”
From The effects of physical activity interventions on psychosocial outcomes in adolescents: a meta-analytic review (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's fail-safe N 37 calculated 115 missing studies would be required to increase the two-tailed p-value to above .05.”
From Anxiety in Informal Dementia Carers: A Meta-Analysis of Prevalence (2020) · cites this paper as [37]
In the citing paper
“Publication bias was investigated with Begg's rank correlation test for funnel plot asymmetry (Begg and Mazumdar, 1994) and where significant effects were present Rosenthal's file drawer analysis was also conducted (Rosenthal, 1979).”
From Meta-analysis of the effects of intranasal oxytocin on interpretation and expression of emotions (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“Over time, this practice artificially inflates Type I error and may make it appear that relationships exist when in reality they do not (Rosenthal et al., 1979).”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“However, a potential problem with meta-analysis is the file drawer problem, where studies are “filed away” unpublished if they fail to find statistically significant relationships.”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“Meta-analyses are necessarily based on the available literature and hence may be affected by the potential publication-bias disfavoring null results (Rosenthal 1979).”
From Parsing the neural correlates of moral cognition: ALE meta-analysis on morality, theory of mind, and empathy (2012) · cites this paper as Rosenthal 1979
In the citing paper
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
From The Extent and Consequences of P-Hacking in Science (2015) · cites this paper as [7]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G. E., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,518
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718 · cited 1×
“Perhaps the simplest, most useful way of computing the overall p of a, set of research studies is the method of adding Zs (Cochran, 1954; Mosteller & Bush, 1954; Rosenthal, 1978).”
MethodMeta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154
Experimenter Effects in Behavioral Research
McGuigan, F. J., & Rosenthal, R. (1968) · The American Journal of Psychology · doi:10.2307/1420816 · Cited by 1,633
Self-citationArtifact in behavioral research
Rosenthal, R., & Rosnow, R. L. (1969) · Medical Entomology and Zoology · Cited by 1,186
Self-citationSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964
Combining results of independent studies.
Rosenthal, R. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.1.185 · Cited by 910
Self-citationGender effects in decoding nonverbal cues.
Hall, J. A. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.4.845 · Cited by 766
Wortman, C. B., Rosenthal, R., & Rosnow, R. L. (1976) · The American Journal of Psychology · doi:10.2307/1421418 · Cited by 428
Self-citation
Cited by
Sorted by citations
Items 601–700 of 1,374
Swedberg, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.002 · Cited by 471
Harrington, S. J. (1996) · MIS Quarterly · doi:10.2307/249656 · Cited by 393
Swanson, H. L., Zheng, X., & Jerman, O. (2009) · Journal of Learning Disabilities · doi:10.1177/0022219409331958 · Cited by 395
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
“…such as selective reporting, presenting exploratory findings as confirmatory tests, and failures of documentation, transparency, and sharing (Casadevall and Fang, 2012; Gelman and Loken, 2013; Greenwald, 1975; Kimmelman et al., 2014; Makel et al., 2012; Nosek et al., 2012; Rosenthal, 1979).”
BackgroundInfluentialMahoney, A., Pargament, K. I., Tarakeshwar, N., & Swank, A. B. (2001) · Journal of Family Psychology · doi:10.1037//0893-3200.15.4.559 · Cited by 393
Efficacy of Naltrexone and Acamprosate for Alcoholism Treatment: A Meta-Analysis
Kranzler, H. R., & Kirk, J. V. (2001) · Alcoholism Clinical and Experimental Research · doi:10.1111/j.1530-0277.2001.tb02356.x · Cited by 394
Wu, K., Zhao, Y. C., Zhu, Q., Tan, X., & Zheng, H. (2011) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2011.03.004 · Cited by 399
A meta-analytic review of the influence of pediatric cancer on parent and family functioning.
Pai, A. L. H., Greenley, R. N., Lewandowski, A. S., Drotar, D., Youngstrom, E. A., & Peterson, C. C. (2007) · Journal of Family Psychology · doi:10.1037/0893-3200.21.3.407 · Cited by 392
Misinformation in and about science
West, J. D., & Bergstrom, C. T. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912444117 · Cited by 391
Correlates of suicide attempts among self-injurers: a meta-analysis
Victor, S. E., & Klonsky, E. D. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.03.005 · Cited by 390
Ecological rules governing helminth–microparasite coinfection
Graham, A. L. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0707221105 · Cited by 407
Development of Executive Function and Attention in Preterm Children: A Systematic Review
Mulder, H., Pitchford, N. J., Hagger, M. S., & Marlow, N. (2009) · Developmental Neuropsychology · doi:10.1080/87565640902964524 · Cited by 388
Math Disabilities: A Selective Meta-Analysis of the Literature
Swanson, H. L., & Jerman, O. (2006) · Review of Educational Research · doi:10.3102/00346543076002249 · Cited by 389
Montoya, R. M., Horton, R. S., Vevea, J. L., Citkowicz, M., & Lauber, E. A. (2017) · Psychological Bulletin · doi:10.1037/bul0000085 · Cited by 389
Are Reports of Childhood Abuse Related to the Experience of Chronic Pain in Adulthood?
Davis, D. A., Luecken, L. J., & Zautra, A. (2005) · Clinical Journal of Pain · doi:10.1097/01.ajp.0000149795.08746.31 · Cited by 388
Karr, J. E., Areshenkoff, C. N., & García-Barrera, M. A. (2013) · Neuropsychology · doi:10.1037/neu0000037 · Cited by 428
Big Five personality traits and academic performance: A meta‐analysis
Mammadov, S. (2021) · Journal of Personality · doi:10.1111/jopy.12663 · Cited by 476
Effect sizes, confidence intervals, and confidence intervals for effect sizes
Thompson, B. (2007) · Psychology in the Schools · doi:10.1002/pits.20234 · Cited by 392
Evaluating the Relationship Between Boredom and Academic Outcomes: A Meta-Analysis
Tze, V. M. C., Daniels, L. M., & Klassen, R. M. (2015) · Educational Psychology Review · doi:10.1007/s10648-015-9301-y · Cited by 461
Do invasive alien plants benefit more from global environmental change than native plants?
Liu, Y., Oduor, A. M. O., Zhang, Z., Manea, A., Tooth, I. M., Leishman, M. R., Xu, X., & van Kleunen, M. (2016) · Global Change Biology · doi:10.1111/gcb.13579 · Cited by 395
Allan, N. P., Hume, L. E., Allan, D. M., Farrington, A. L., & Lonigan, C. J. (2014) · Developmental Psychology · doi:10.1037/a0037493 · Cited by 385
Improving Preventive Care by Prompting Physicians
Balas, E. A. (2000) · Archives of Internal Medicine · doi:10.1001/archinte.160.3.301 · Cited by 385
Transaction Cost Theory: Past Progress, Current Challenges, and Suggestions for the Future
Cuypers, I. R. P., Hennart, J., Silverman, B. S., & Ertug, G. (2020) · Academy of Management Annals · doi:10.5465/annals.2019.0051 · Cited by 433
“…information, and we encourage researchers to report (and journals and editors to publish) nonsignificant findings to avoid the “ file-drawer ” problem (Rosenthal, 1979), and, if applicable, to implement appropriate methods to test for the existence of such null effects (e.g., Cuypers & Martin,…”
The Relation Between Family Socioeconomic Status and Academic Achievement in China: A Meta-analysis
Liu, J., Peng, P., & Luo, L. (2019) · Educational Psychology Review · doi:10.1007/s10648-019-09494-0 · Cited by 451
“Publication bias refers to the problem of selective publication, which means that it is easier for significant findings to be accepted and published, while it is more difficult for other nonsignificant findings to be accepted and published (Rosenthal 1979).”
BackgroundKlein, C. R., DiazGranados, D., Salas, E., Le, H., Burke, C. S., Lyons, R., & Goodwin, G. F. (2009) · Small Group Research · doi:10.1177/1046496408328821 · Cited by 447
Häuser, W., Bernardy, K., Arnold, B., Offenbächer, M., & Schiltenwolf, M. (2009) · Arthritis Care & Research · doi:10.1002/art.24276 · Cited by 378
The Effectiveness of Second Language Strategy Instruction: A Meta‐analysis
Plonsky, L. (2011) · Language Learning · doi:10.1111/j.1467-9922.2011.00663.x · Cited by 417
The shared leadership of teams: A meta-analysis of proximal, distal, and moderating relationships
Nicolaides, V., LaPort, K., Chen, T. R., Tomassetti, A. J., Weis, E. J., Zaccaro, S. J., & Cortina, J. M. (2014) · The Leadership Quarterly · doi:10.1016/j.leaqua.2014.06.006 · Cited by 418
Sizing up the Effect of Portion Size on Consumption: A Meta-Analytic Review
Zlatevska, N., Dubelaar, C., & Holden, S. J. S. (2014) · Journal of Marketing · doi:10.1509/jm.12.0303 · Cited by 380
Weingarten, E., Chen, Q.-J., McAdams, M., Yi, J., Hepler, J., & Albarracín, D. (2015) · Psychological bulletin · doi:10.1037/bul0000030 · Cited by 275
“, disconfirming findings that are either opposite to predictions or null) required to reduce a statistically significant result to nonsignificance (DeCoster, 2009; Rosenthal, 1979; Rosenthal & Rosnow, 2008) using Rosenberg’s (2005) calculator.”
InfluentialMcGirr, A., Berlim, M. T., Bond, D. J., Fleck, M. P., Yatham, L. N., & Lam, R. W. (2015) · Psychological Medicine · doi:10.1017/s0033291714001603 · Cited by 373
Job Burnout and Employee Engagement
Cole, M. S., Walter, F. H., Bedeian, A. G., & O’Boyle, E. H. (2011) · Journal of Management · doi:10.1177/0149206311415252 · Cited by 392
Mediation of interpersonal expectancy effects: 31 meta-analyses
Harris, M. J., & Rosenthal, R. (1985) · Psychological Bulletin · doi:10.1037/0033-2909.97.3.363 · Cited by 542
Self-citationFrederiksen, Y., Farver-Vestergaard, I., Skovgård, N. G., Ingerslev, H. J., & Zachariae, R. (2015) · BMJ Open · doi:10.1136/bmjopen-2014-006592 · Cited by 380
Personal and Psychosocial Predictors of Doping Use in Physical Activity Settings: A Meta-Analysis
Ntoumanis, N., Ng, J. Y. Y., Barkoukis, V., & Backhouse, S. H. (2014) · Sports Medicine · doi:10.1007/s40279-014-0240-4 · Cited by 387
Murphy, R. O., & Ackermann, K. A. (2013) · Personality and Social Psychology Review · doi:10.1177/1088868313501745 · Cited by 396
Pearson, N., Braithwaite, R. E., Biddle, S., van Sluijs, E. M. F., & Atkin, A. J. (2014) · Obesity Reviews · doi:10.1111/obr.12188 · Cited by 411
Assessing Managerial Power Theory
van Essen, M., Otten, J., & Carberry, E. J. (2012) · Journal of Management · doi:10.1177/0149206311429378 · Cited by 374
Hemmi, M., Dieter Wolke, C., & Schneider, S. (2011) · Archives of Disease in Childhood · doi:10.1136/adc.2010.191312 · Cited by 391
Meta-analysis of cognitive-behavioral treatments for social phobia
Taylor, S. J. (1996) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(95)00058-5 · Cited by 365
Perceptions of Organizational Politics: A Meta-analysis of Outcomes
Miller, B. K., Rutherford, M. A., & Kolodinsky, R. W. (2008) · Journal of Business and Psychology · doi:10.1007/s10869-008-9061-5 · Cited by 373
Delay discounting and intelligence: A meta-analysis
Shamosh, N. A., & Gray, J. R. (2007) · Intelligence · doi:10.1016/j.intell.2007.09.004 · Cited by 392
Online Health Information Seeking: A Review and Meta-Analysis
Wang, X., Shi, J. J., & Kong, H. (2020) · Health Communication · doi:10.1080/10410236.2020.1748829 · Cited by 404
“We addressed that concern by examining the fail-safe N proposed by Rosenthal (1979), which predicts how many unpublished articles with nonsignificant ES would be required to reverse the standing conclusion.”
BackgroundAn Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports
Scheel, A. M., Schijen, M., & Lakens, D. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211007467 · Cited by 417
“These include editors and reviewers selectively rejecting manuscripts with negative results (reviewer bias; Greenwald, 1975; Mahoney, 1977) and researchers deciding not to submit studies with negative results for publication (file-drawering; Rosenthal, 1979).”
BackgroundIto, T. A., Miller, N., & Pollock, V. E. (1996) · Psychological Bulletin · doi:10.1037/0033-2909.120.1.60 · Cited by 364
Reporting, Appraising, and Integrating Data on Genotype Prevalence and Gene-Disease Associations
Little, J., Bradley, L., Bray, M. S., Clyne, M., Dorman, J. S., Ellsworth, D. L., Hanson, J., Khoury, M. J., Lau, J., OʼBrien, T. R., et al. (2002) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a000179 · Cited by 366
Schutter, D. J. L. G. (2008) · Psychological Medicine · doi:10.1017/s0033291708003462 · Cited by 368
Paraverbal indicators of deception: a meta‐analytic synthesis
Sporer, S. L., & Schwandt, B. (2006) · Applied Cognitive Psychology · doi:10.1002/acp.1190 · Cited by 402
Begemann, M. J., Brand, B. A., Ćurčić-Blake, B., Aleman, A., & Sommer, I. E. (2020) · Psychological Medicine · doi:10.1017/s0033291720003670 · Cited by 368
“Rosenthal’s fail-safe number (NR) was calculated, estimating the number of unpublished studies with nonsignificant results needed to bring an observed result to nonsignificance (NR⩾ 5k + 10 to rule out a file drawer problem) (Rosenthal, 1979).”
BackgroundScientific Utopia: I. Opening Scientific Communication
Nosek, B. A., & Bar‐Anan, Y. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.692215 · Cited by 387
Self-efficacy as a predictor of commitment to the teaching profession: A meta-analysis
Chesnut, S. R., & Burley, H. (2015) · Educational Research Review · doi:10.1016/j.edurev.2015.02.001 · Cited by 367
The contingent smile: A meta-analysis of sex differences in smiling.
LaFrance, M., Hecht, M. A., & Paluck, E. L. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.2.305 · Cited by 514
The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies
Raposa, E. B., Rhodes, J., Stams, G. J. J. M., Card, N., Burton, S., Schwartz, S., Sykes, L. A. Y., Kanchewa, S., Kupersmidt, J., & Hussain, S. (2019) · Journal of Youth and Adolescence · doi:10.1007/s10964-019-00982-8 · Cited by 366
“Not including these studies may lead to an overestimation of the true effect size, a phenomenon known as “publication bias” (Rosenthal 1979).”
BackgroundEffects of game‐based learning on students' mathematics achievement: A meta‐analysis
Tokac, U., Novak, E., & Thompson, C. G. (2019) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12347 · Cited by 366
Exercise as Treatment for Anxiety: Systematic Review and Analysis
Stonerock, G. L., Hoffman, B. M., Smith, P. J., & Blumenthal, J. A. (2015) · Annals of Behavioral Medicine · doi:10.1007/s12160-014-9685-9 · Cited by 363
How can we improve the pre-clinical development of drugs for stroke?
Sena, E. S., van der Worp, H. B., Howells, D. W., & Macleod, M. R. (2007) · Trends in Neurosciences · doi:10.1016/j.tins.2007.06.009 · Cited by 367
Racism and health service utilisation: A systematic review and meta-analysis
Ben, J., Cormack, D. M., Harris, R. B., & Paradies, Y. (2017) · PLoS ONE · doi:10.1371/journal.pone.0189900 · Cited by 402
Linear and generalized linear mixed models
Bolker, B. M. (2015) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199672547.003.0014 · Cited by 375
Morres, I. D., Hatzigeorgiadis, A., Stathi, A., Comoutos, N., Arpin-Cribbie, C., Krommidas, C., & Theodorakis, Y. (2019) · Depression and Anxiety · doi:10.1002/da.22842 · Cited by 350
“A fail–safe number of five times the number of reviewed comparisons plus 10 (5K + 10) is seen as the cutoff score for considering the results robust (Rosenthal, 1979).”
BackgroundMethodInfluentialEffects of psychoeducational care for adult surgical patients: A meta-analysis of 191 studies
Devine, E. C. (1992) · Patient Education and Counseling · doi:10.1016/0738-3991(92)90193-m · Cited by 360
Jensen, C. D., Cushing, C. C., Aylward, B. S., Craig, J. T., Sorell, D. M., & Steele, R. G. (2011) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0023992 · Cited by 363
Alavi, M., & Joachimsthaler, E. A. (1992) · MIS Quarterly · doi:10.2307/249703 · Cited by 371
Does animation enhance learning? A meta-analysis
Berney, S., & Bétrancourt, M. (2016) · Computers & Education · doi:10.1016/j.compedu.2016.06.005 · Cited by 421
“To account for the possibility that the present meta-analysis missed non-significant results, the fail-safe N (Rosenthal, 1979), which reflects the number of unpublished studies needed in order to lower the effect size estimate to nonsignificant, was calculated.”
BackgroundResultThe association between weight stigma and mental health: A meta‐analysis
Emmer, C., Bošnjak, M., & Mata, J. (2019) · Obesity Reviews · doi:10.1111/obr.12935 · Cited by 416
“Failsafe N indicated a robust correlation between weight stigma and mental health since at least 21 914 386 effect sizes with non‐significant findings are required to invalidate the overall association.(64)”
BackgroundHalsey, L. G. (2019) · Biology Letters · doi:10.1098/rsbl.2019.0174 · Cited by 382
Eliot, L., Ahmed, A., Khan, H., & Patel, J. Y. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.02.026 · Cited by 379
Why do people play games? A meta-analysis
Hamari, J., & Keronen, L. (2017) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2017.01.006 · Cited by 368
“There are two main approaches for such calculations (Long 2001): one method based on the sum of the Z scores (Rosenthal 1979) while the other uses effect sizes (Orwin 1983).”
MethodWinquist, E., KIRCHNER, T. S., SEGAL, R. J., Chin, J. L., Lukka, H. R., & on behalf of the GENITOURINARY CANCER DISEASE SITE GROUP OF CANCER CARE ONTARIO PROGRAM IN EVIDENCE-BASED CARE PRACTICE GUIDELINES INITIATIVE (2004) · The Journal of Urology · doi:10.1097/01.ju.0000090967.08622.33 · Cited by 359
Ingroup Bias and Self-Esteem: A Meta-Analysis
Aberson, C. L., Healy, M. R., & Romero, V. L. (2000) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0402_04 · Cited by 359
Modeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359
Self-compassion, affect, and health-promoting behaviors.
Sirois, F. M., Kitner, R., & Hirsch, J. K. (2015) · Health Psychology · doi:10.1037/hea0000158 · Cited by 357
Mahoney, A., Pargäment, K. I., Tarakeshwar, N., & Swank, A. B. (2001) · Journal of Family Psychology · doi:10.1037/0893-3200.15.4.559 · Cited by 358
Barker, F. G., & Ogilvy, C. S. (1996) · Journal of neurosurgery · doi:10.3171/jns.1996.84.3.0405 · Cited by 357
The Theories of Reasoned Action and Planned Behavior Applied to Exercise: A Meta-analytic Update
Downs, D. S., & Hausenblas, H. A. (2005) · Journal of Physical Activity and Health · doi:10.1123/jpah.2.1.76 · Cited by 357
Building, measuring, and profiting from customer loyalty
Watson, G. F., Beck, J. T., Henderson, C. M., & Palmatier, R. W. (2015) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-015-0439-4 · Cited by 375
“First, we computed and report the Q statistic (d.f.=n–1) test of homogeneity (Rosenthal 1979).”
MethodDST Studies in Psychotic Depression: A Meta-Analysis
Nelson, J. C., & Davis, J. M. (1997) · American Journal of Psychiatry · doi:10.1176/ajp.154.11.1497 · Cited by 356
McEwan, D., Ruissen, G. R., Eys, M., Zumbo, B. D., & Beauchamp, M. R. (2017) · PLoS ONE · doi:10.1371/journal.pone.0169604 · Cited by 383
“First, publication bias was examined by calculating a fail-safe N statistic, which estimates the number of unpublished studies with null findings that would have to exist to reduce the obtained effect size to zero [38].”
Beenhouwer, M. D., Aerts, R., & Honnay, O. (2013) · Agriculture Ecosystems & Environment · doi:10.1016/j.agee.2013.05.003 · Cited by 381
Gender differences in alexithymia.
Levant, R. F., Hall, R. J., Williams, C. M., & Hasan, N. T. (2009) · Psychology of Men & Masculinity · doi:10.1037/a0015652 · Cited by 356
Depression and risk of mortality in individuals with diabetes: a meta-analysis and systematic review
Park, M., Katon, W. J., & Wolf, F. M. (2013) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2013.01.006 · Cited by 355
Laparoscopic Versus Open Appendectomy: A Metaanalysis
Golub, R., Siddiqui, F., & Pohl, D. (1998) · Journal of the American College of Surgeons · doi:10.1016/s1072-7515(98)00080-5 · Cited by 353
Cognitive-behavioral therapy for anger in children and adolescents: a meta-analysis
Sukhodolsky, D. G., Kassinove, H., & Gorman, B. S. (2003) · Aggression and Violent Behavior · doi:10.1016/j.avb.2003.08.005 · Cited by 415
Statistical versus traditional procedures for summarizing research findings.
Cooper, H. M., & Rosenthal, R. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.87.3.442 · Cited by 353
Self-citationEffect of land‐use and land‐cover change on mangrove blue carbon: A systematic review
Sasmito, S. D., Taillardat, P., Clendenning, J., Cameron, C., Friess, D. A., Murdiyarso, D., & Hutley, L. B. (2019) · Global Change Biology · doi:10.1111/gcb.14774 · Cited by 370
“…by using Rosenthal's fail‐safe number where fail‐safe numbers of all datasets—except soil CH4 efflux—were reasonable, conservative and much greater than the minimum of suggested fail‐safe value threshold (5n + 10, where n is the number of data in the meta‐analysis, Rosenthal, 1979; Table S3).”
MethodReschly, A. L., Busch, T. W., Betts, J., Deno, S. L., & Long, J. D. (2009) · Journal of School Psychology · doi:10.1016/j.jsp.2009.07.001 · Cited by 376
Psychological stress and wound healing in humans: A systematic review and meta-analysis
Walburn, J., Vedhara, K., Hankins, M. C., Rixon, L., & Weinman, J. A. (2009) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2009.04.002 · Cited by 353
Sensitivity Analysis for Publication Bias in Meta-Analyses
Mathur, M. B., & VanderWeele, T. J. (2020) · Journal of the Royal Statistical Society Series C: Applied Statistics · doi:10.1111/rssc.12440 · Cited by 354
“…minimum number of unpublished studies with a mean point estimate of 0 (or another fixed value) that would need to exist for their inclusion in the meta-analysis to reduce the pooled point estimate to ‘statistical non-significance’ (Rosenthal, 1979) or to a given effect size threshold (Orwin, 1983).”
BackgroundFeeling Good, Happy, and Proud: A Meta-Analysis of Positive Ethnic–Racial Affect and Adjustment
Rivas-Drake, D., Syed, M., Umaña-Taylor, A., Markstrom, C., French, S., Schwartz, S. J., Lee, R., & Ethnic and Racial Identity in the 21st Century Study Group (2014) · Child Development · doi:10.1111/cdev.12175 · Cited by 389
STRATEGIC PLANNING AND FINANCIAL PERFORMANCE: A META‐ANALYTIC REVIEW*
Boyd, B. (1991) · Journal of Management Studies · doi:10.1111/j.1467-6486.1991.tb00286.x · Cited by 351
What makes a successful biocontrol agent? A meta-analysis of biological control agent performance
STILING, P. D., & Cornelissen, T. (2005) · Biological Control · doi:10.1016/j.biocontrol.2005.02.017 · Cited by 350
Nykänen, H., & Koricheva, J. (2004) · Oikos · doi:10.1111/j.0030-1299.2004.12768.x · Cited by 344
Sherf, E. N., Parke, M. R., & Isaakyan, S. (2021) · Academy of Management Journal · doi:10.5465/amj.2018.1428 · Cited by 349
Koch, S., Kunz, T., Lykou, S., & Cruz, R. (2014) · The Arts in Psychotherapy · doi:10.1016/j.aip.2013.10.004 · Cited by 349
Handedness, language dominance and aphasia: a genetic model
McManus, I. C. (1985) · Psychological Medicine Monograph Supplement · doi:10.1017/s0264180100001879 · Cited by 349
Gender differences in self-conscious emotional experience: A meta-analysis.
Else-Quest, N. M., Higgins, A., Allison, C., & Morton, L. C. (2012) · Psychological Bulletin · doi:10.1037/a0027930 · Cited by 464
Meta-analytic procedures for social research
Wilson, E. J. (1994) · Journal of Business Research · doi:10.1016/0148-2963(94)90020-5 · Cited by 348
The relationship between math anxiety and math performance: a meta-analytic investigation
Zhang, J., Zhao, N., & Kong, Q. P. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01613 · Cited by 347
“The second method was the fail-safe N (e.g., classic fail-safe N), which estimated the number of studies with non-significant results (unpublished data) needed to cause the mean ES to become statistically non-significant (Rosenthal, 1979).”
MethodResultInfluentialCollege Diversity Experiences and Cognitive Development: A Meta-Analysis
Bowman, N. A. (2010) · Review of Educational Research · doi:10.3102/0034654309352495 · Cited by 348
Busch, T., & Friede, G. (2018) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.1480 · Cited by 345
“Fail-Safe N states the number of (future) studies with null results, until the effect size loses its significance level (Rosenthal, 1979).”
BackgroundVocks, S., Tuschen‐Caffier, B., Pietrowsky, R., Rustenbach, S. J., Kersting, A., & Herpertz, S. (2010) · International Journal of Eating Disorders · doi:10.1002/eat.20696 · Cited by 343
Metadata
- Authors
- Rosenthal, R.
- Affiliations
- Harvard University (US)
- Corresponding author
- Rosenthal, Robert
- Venue
- Psychological Bulletin
- Published
- 1979-05
- Volume
- 86
- Issue
- 3
- Pages
- 638-641
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2006546769
- Cited by
- 9,068
- References
- 16
- Citations by year
- 2026: 421 · 2025: 521 · 2024: 509 · 2023: 526 · 2022: 525 · 2021: 536 · 2020: 612 · 2019: 607 · 2018: 492 · 2017: 436 · 2016: 417 · 2015: 392 · 2014: 337 · 2013: 289 · 2012: 252
- FWCI
- 26.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- file drawer problem, negative results, publication bias, non-significant results, type I error, selective reporting, behavioral science, evidence synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
