The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
The file drawer problem and tolerance for null results.
Abstract
For any given research area, one cannot tell how many studies have been con-ducted but never reported. The extreme view of the "file drawer problem " is that journals are filled with the 5 % of the studies that show Type I errors, while the file drawers are filled with the 95 % of the studies that show non-significant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed. Both behavioral researchers and statisti-cians have long suspected that the studies published in the behavioral sciences are a biased sample of the studies that are actually carried out (Bakan, 1967; McNemar, 1960; Smart, 1964; Sterling, 1959). The extreme view of this problem, the "file drawer prob-lem, " is that the journals are filled with the 5 % of the studies that show Type I errors, while the file drawers back at the lab are filled with the 95 % of the studies that show nonsignificant (e.g., p>.05) results. In the past there was very little one could do to assess the net effect of studies, tucked away in file drawers, that did not make the magic.05 level (Rosenthal & Gaito, 1963, 1964). Now, however, although no definitive solution to the problem is available, one can establish reasonable boundaries on the prob-lem and estimate the degree of damage to any research conclusion that could be done by the file drawer problem. This advance in our ability to cope with the file drawer is an outgrowth of the in-creasing interest of behavioral scientists in summarizing bodies of research literature sys-Preparation of this article was supported in part by the National Science Foundation. I would like to thank Judith A. Hall and Donald B. Rubin for their valuable improvements of an earlier version of this article. Requests for reprints should be sent to Robert
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We then calculated Rosenthal’s fail-safe N (Rosenthal, 1979), and determined that 2,153,789 effects evidencing no relationship between SA and psychopathology (g = 0) would be needed to reduce the significance level of the summary effect size to .05.”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“The file drawer effect is a perennial problem in the academic literature (Rosenthal, 1979)”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“First, we calculated the fail-safe N [177] to be 4,274, which is the theoretical number of unpublished studies with effect sizes averaging zero (no effect) that would be needed to render negligible the omnibus results.”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [177]
In the citing paper
“As we did not include unpublished work in our meta-analyses we tested for publication bias in the following ways.Firstly, funnel plots were examined.demonstrating no systematic publication biases (tables available from first author on request).Secondly..Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Presence of publication bias was assessed with funnel plots, 22 Rosenthal's fail-safe N , 23 and Egger's regression intercept. 24”
From Light Therapy for Patients With Bipolar Depression: Systematic Review and Meta-Analysis of Randomized Controlled Trials (2020) · cites this paper as [23]
In the citing paper
“In terms of study quality, there was some indication of publication bias whereby some associations between racism and health outcomes were stronger in published versus unpublished studies, consistent with the tendency for ‘null’ studies to remain unpublished (i.e., the ‘file drawer’ problem [418]).”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as [418]
In the citing paper
“An alternative, perhaps more plausible, explanation is that there could be a significant amount of unpublished negative or inconclusive results, a phenomenon referred to as the “file-drawer effect” or publication bias (23).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [23]
In the citing paper
“If either of the two last hypotheses is true, there should exist many unpublished studies with null results lying in laboratories' drawers (32).”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Publication bias was evaluated using Rosenthal's file drawer analysis (Rosenthal, 1979) and Egger's linear regression (Card, 2012), and applying ANOVA to compare the mean effect size of the published versus unpublished studies.”
From Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal (1979) suggested a fail-safe N value above 5; k + 10 reflects results that are tolerant to contradicting studies”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“We applied random-effects models to fit the relations between CE and the criterion variables, following the guidelines proposed by Rosenthal (1979), Hunter and Schmidt (2004) and Cumming (2013).”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal (1979)
In the citing paper
“The fail-safe value for this meta-analysis was computed to be 3001, which is more than 5(36) + 10.”
From Effects of digital game-based STEM education on students’ learning achievement: a meta-analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, we suggest our results may look like a lot of studies that ended up in the so-called “file drawer” [57].”
From Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory (2023) · cites this paper as [57]
In the citing paper
“Finally, publication bias was addressed by computing the fail-safe N for the correlation coefficients (Hunter & Schmidt, 2004;Rosenthal, 1979), which reports the number of field effect sizes or the tolerance for future null results, to check whether a publication bias affected our results.”
From Drivers of Consumer Adoption of e-Commerce: a Meta-Analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Using Rosenthal’s (1979) fail-safe N criterion that the value should be over five times greater than the number of studies included in the meta-analysis, the fail-safe N for all outcome groups were at least eight times larger than the criterion value, suggesting that the effect sizes are unlikely to be an artifact of bias.”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as Rosenthal (1979)
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, Rosenthal (1979) opined that a Fail-safe N exceeding the number of studies by fivefold would provide strong support for the robustness of the significant effects.”
From Extending the Theory of Planned Behavior in the Social Commerce Context: A Meta-Analytic SEM (MASEM) Approach (2023) · cites this paper as Rosenthal (1979)
In the citing paper
“The trim-and-fill method (Duval & Tweedie, 2000) tends to adjust too little when bias is strong (Carter et al., 2019), and Fail-Safe N (Rosenthal, 1979) neither tests nor adjusts for the presence of bias.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Rosenthal, 1979
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, a phenomenon known as ' publication bias ' (Rosenthal 1979).”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“However, a common problem is that results may not have been reported in dissertations or research reports, or published in scienti fi c (peer-reviewed) journals, due to non-signi fi cant or unfavorable fi ndings and, therefore, are dif fi cult to locate.”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [21]
In the citing paper
“The traditional fail-safe value is 1904, which is higher than the threshold value of 100, i.e., [5(18) + 10], indicating that publication bias is insignificant, and the study's results are reliable.”
From A meta-analysis to explore the role of design thinking in enhancing creativity as learning outcomes in STEM education (2026) · cites this paper as Rosenthal, 1979
In the citing paper
“The Q statistic was used to assess heterogeneity30, and publication bias was estimated using Rosenthal’s fail-safe number with α = 0.0531.”
From The limited prosocial effects of meditation: A systematic review and meta-analysis (2018) · cites this paper as [31]
In the citing paper
“Rosenthal's fail-safe number (NR) was calculated, estimating the number of unpublished studies with non-significant results needed to bring an observed result to non-significance (NR ⩾ 5k + 10 to rule out a file drawer problem) (Rosenthal, 1979).”
From Efficacy of non-invasive brain stimulation on cognitive functioning in brain disorders: a meta-analysis (2020) · cites this paper as Rosenthal, 1979
In the citing paper
“In case of statistically significant effect sizes, it was planned to examine the robustness of findings by calculating the Failsafe N (number of non‐significant studies that would bring the p‐value to non‐significant (p > 0.05)) 40.”
From Cognitive behavioral therapies for informal caregivers of patients with cancer and cancer survivors: a systematic review and meta-analysis (2017) · cites this paper as [40]
In the citing paper
“With negative results buried in file drawers (55), conclusions drawn from the published record can be misleading, and 'false facts' can become canonized in the literature (56).”
From Misinformation in and about science (2021) · cites this paper as [55]
In the citing paper
“the fail-safe N (i.e. the file drawer statistic) was used to determine how many additional studies would be necessary to turn a significant ES non-significant (32).”
From Peripheral cytokine and chemokine alterations in depression: a meta‐analysis of 82 studies (2017) · cites this paper as [32]
In the citing paper
“File drawer analysis (Rosenthal, 1979) revealed that 1919 unpublished, filed, or unretrieved studies averaging null results would be required to bring the average unweighted effect size to nonsignificance.”
From Happy to help? A systematic review and meta-analysis of the effects of performing acts of kindness on the well-being of the actor (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's (1979) fail-safe N analysis reveals 186 unpublished papers (i.e., null effect) are needed to be included in the meta-analysis in order to invalidate the population effect size of small to medium (i.e., .235)found in this review.”
From A meta-analytic review of the efficacy of physical exercise interventions on cognition in individuals with autism spectrum disorder and ADHD (2016) · cites this paper as Rosenthal's (1979)
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, the so called ' publication bias ' (Rosenthal, 1979).”
From The effects of physical activity interventions on psychosocial outcomes in adolescents: a meta-analytic review (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's fail-safe N 37 calculated 115 missing studies would be required to increase the two-tailed p-value to above .05.”
From Anxiety in Informal Dementia Carers: A Meta-Analysis of Prevalence (2020) · cites this paper as [37]
In the citing paper
“Publication bias was investigated with Begg's rank correlation test for funnel plot asymmetry (Begg and Mazumdar, 1994) and where significant effects were present Rosenthal's file drawer analysis was also conducted (Rosenthal, 1979).”
From Meta-analysis of the effects of intranasal oxytocin on interpretation and expression of emotions (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“Over time, this practice artificially inflates Type I error and may make it appear that relationships exist when in reality they do not (Rosenthal et al., 1979).”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“However, a potential problem with meta-analysis is the file drawer problem, where studies are “filed away” unpublished if they fail to find statistically significant relationships.”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“Meta-analyses are necessarily based on the available literature and hence may be affected by the potential publication-bias disfavoring null results (Rosenthal 1979).”
From Parsing the neural correlates of moral cognition: ALE meta-analysis on morality, theory of mind, and empathy (2012) · cites this paper as Rosenthal 1979
In the citing paper
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
From The Extent and Consequences of P-Hacking in Science (2015) · cites this paper as [7]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G. E., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,518
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718 · cited 1×
“Perhaps the simplest, most useful way of computing the overall p of a, set of research studies is the method of adding Zs (Cochran, 1954; Mosteller & Bush, 1954; Rosenthal, 1978).”
MethodMeta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154
Experimenter Effects in Behavioral Research
McGuigan, F. J., & Rosenthal, R. (1968) · The American Journal of Psychology · doi:10.2307/1420816 · Cited by 1,633
Self-citationArtifact in behavioral research
Rosenthal, R., & Rosnow, R. L. (1969) · Medical Entomology and Zoology · Cited by 1,186
Self-citationSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964
Combining results of independent studies.
Rosenthal, R. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.1.185 · Cited by 910
Self-citationGender effects in decoding nonverbal cues.
Hall, J. A. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.4.845 · Cited by 766
Wortman, C. B., Rosenthal, R., & Rosnow, R. L. (1976) · The American Journal of Psychology · doi:10.2307/1421418 · Cited by 428
Self-citation
Cited by
Sorted by citations
Items 901–1,000 of 1,374
Gender differences in object location memory: A meta-analysis
Voyer, D., Postma, A., Brake, B., & Imperato‐McGinley, J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194024 · Cited by 288
Publication bias and the canonization of false facts
Nissen, S. B., Magidson, T., Gross, K., & Bergstrom, C. T. (2016) · eLife · doi:10.7554/elife.21451 · Cited by 266
Connell, L., & Lynott, D. (2012) · Cognition · doi:10.1016/j.cognition.2012.07.010 · Cited by 274
Lying takes time: A meta-analysis on reaction time measures of deception.
Suchotzki, K., Verschuère, B., Bockstaele, B. V., Ben‐Shakhar, G., & Crombez, G. (2017) · Psychological Bulletin · doi:10.1037/bul0000087 · Cited by 309
“the tendency that large and significant effects are more likely to be published than small and non-significant effects, poses a problem in many research areas and may lead to overestimations of effect sizes in meta-analyses (e.g., Fanelli, 2012; Rosenthal, 1979; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
A meta‐analysis of customer engagement behaviour
Barari, M. M., Ross, M., Quach, S., & Surachartkumtonkun, J. (2020) · International Journal of Consumer Studies · doi:10.1111/ijcs.12609 · Cited by 305
The relationship between chronotype and depressive symptoms: a meta-analysis
Au, J., & Reece, J. (2017) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.04.021 · Cited by 297
Social Story™ Interventions for Students with Autism Spectrum Disorders: A Meta-Analysis
Kokina, A. A., & Kern, L. (2010) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-009-0931-0 · Cited by 308
A Meta-Analytic Investigation of Business Ethics Instruction
Waples, E. P., Antes, A. L., Murphy, S. T., Connelly, S., & Mumford, M. D. (2009) · Journal of Business Ethics · doi:10.1007/s10551-008-9875-0 · Cited by 279
A Meta-Analysis of Relationships Linking Service Failure Attributions to Customer Outcomes
Vaerenbergh, Y. V., Orsingher, C., Vermeir, I., & Larivière, B. (2014) · Journal of Service Research · doi:10.1177/1094670514538321 · Cited by 284
A Meta-Analysis of the Role of Environment-Based Voluntariness in Information Technology Acceptance1
Wu, J., & Lederer, A. L. (2009) · MIS Quarterly · doi:10.2307/20650298 · Cited by 292
The National High Blood Pressure Education Program
Jones, D. W., & Hall, J. E. (2002) · Hypertension · doi:10.1161/01.hyp.0000018303.61360.28 · Cited by 281
Palmer, A. R. (2000) · Annual Review of Ecology and Systematics · doi:10.1146/annurev.ecolsys.31.1.441 · Cited by 385
Pletzer, J. L., Nikolova, R., Kedzior, K. K., & Voelpel, S. C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0130005 · Cited by 288
Looking Toward the Future of IT–Business Strategic Alignment Through the Past: A Meta-Analysis1
Gerow, J. E., Grover, V., Thatcher, J., & Roth, P. L. (2014) · MIS Quarterly · doi:10.25300/misq/2014/38.4.10 · Cited by 277
Handbook of the Sociology of Religion
(2003) · Cambridge University Press eBooks · doi:10.1017/cbo9780511807961 · Cited by 286
Pellicane, M. J., & Ciesla, J. A. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2021.102113 · Cited by 306
Correlates of physician burnout across regions and specialties: a meta-analysis
Lee, R. T., Seo, B., Hladkyj, S., Lovell, B. L., & Schwartzmann, L. (2013) · Human Resources for Health · doi:10.1186/1478-4491-11-48 · Cited by 274
The Theory of Planned Behaviour and dietary patterns: A systematic review and meta-analysis
McDermott, M. S., Oliver, M., Simnadis, T. G., Beck, E. J., Coltman, T. R., Iverson, D., Caputi, P., & Sharma, R. (2015) · Preventive Medicine · doi:10.1016/j.ypmed.2015.08.020 · Cited by 328
Publication bias: what is it? How do we measure it? How do we avoid it?
Song, F., Hooper, L., & Loke, Y. K. (2013) · Open Access Journal of Clinical Trials · doi:10.2147/oajct.s34419 · Cited by 298
Factors Influencing the Acceptance of Self-Service Technologies
Blut, M., Wang, C., & Schoefer, K. (2016) · Journal of Service Research · doi:10.1177/1094670516662352 · Cited by 297
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.010 · Cited by 279
Hoza, B., Gerdes, A. C., Mrug, S., Hinshaw, S. P., Bukowski, W. M., Gold, J. A., Arnold, L. E., Abikoff, H. B., Conners, C. K., Elliott, G. R., et al. (2005) · Journal of Clinical Child & Adolescent Psychology · doi:10.1207/s15374424jccp3401_7 · Cited by 276
Is cross-cultural training of expatriate managers effective: A meta analysis
Deshpande, S. P., & Viswesvaran, C. (1992) · International Journal of Intercultural Relations · doi:10.1016/0147-1767(92)90054-x · Cited by 275
How family-friendly work environments affect work/family conflict: A meta-analytic examination
Mesmer‐Magnus, J., & Viswesvaran, C. (2006) · Journal of Labor Research · doi:10.1007/s12122-006-1020-1 · Cited by 277
A meta-analysis of performance on emotion recognition tasks in Parkinson’s disease.
Gray, H. M., & Tickle‐Degnen, L. (2010) · Neuropsychology · doi:10.1037/a0018104 · Cited by 293
Yin, J., Zhou, M., Lin, Z., Li, Q. Q., & Zhang, Y. (2019) · Ecology Letters · doi:10.1111/ele.13373 · Cited by 270
Depression Treatment Enhances Adherence to Antiretroviral Therapy: a Meta-Analysis
Sin, N. L., & DiMatteo, M. R. (2014) · Annals of Behavioral Medicine · doi:10.1007/s12160-013-9559-6 · Cited by 266
Hiles, S. A., Baker, A. L., de Malmanche, T., & Attia, J. (2012) · Brain, Behavior, and Immunity · doi:10.1016/j.bbi.2012.06.001 · Cited by 274
Sex-role orientation and psychological well-being: Two meta-analyses
Whitley, B. E. (1985) · Sex Roles · doi:10.1007/bf00288048 · Cited by 274
Biofeedback Treatment for Headache Disorders: A Comprehensive Efficacy Review
Nestoriuc, Y., Martin, A., Rief, W., & Andrasik, F. (2008) · Applied Psychophysiology and Biofeedback · doi:10.1007/s10484-008-9060-3 · Cited by 273
Motivational interviewing for smoking cessation: A meta-analytic review.
Hettema, J. E., & Hendricks, P. S. (2010) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0021498 · Cited by 284
Wagner, J. A., & Gooding, R. Z. (1987) · Academy of Management Journal · doi:10.2307/256012 · Cited by 273
Comprehensive longitudinal study challenges the existence of neonatal imitation in humans
Oostenbroek, J., Suddendorf, T., Nielsen, M., Redshaw, J., Kennedy-Costantini, S., Davis, J., Clark, S., & Slaughter, V. (2016) · Current Biology · doi:10.1016/j.cub.2016.03.047 · Cited by 262
“A few studies reporting null results like ours can be found in the literature [17], but many more may have been filed away due to publication bias [20, 21].”
Treatment of fibromyalgia with cyclobenzaprine: A meta‐analysis
Tofferi, J. K., Jackson, J. L., & O’Malley, P. G. (2004) · Arthritis Care & Research · doi:10.1002/art.20076 · Cited by 275
Gender effects on entrepreneurial intention: a meta‐analytical structural equation model
Haus, I., Steinmetz, H., Isidor, R., & Kabst, R. (2013) · International Journal of Gender and Entrepreneurship · doi:10.1108/17566261311328828 · Cited by 309
Forsman, A. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1317745111 · Cited by 272
On the External Validity of Social Preference Games: A Systematic Lab-Field Study
Galizzi, M. M., & Navarro‐Martinez, D. (2018) · Management Science · doi:10.1287/mnsc.2017.2908 · Cited by 291
Gooding, P., & Tarrier, N. (2009) · Behaviour Research and Therapy · doi:10.1016/j.brat.2009.04.002 · Cited by 267
Barnard, N. D., Levin, S. M., & Yokoyama, Y. (2015) · Journal of the Academy of Nutrition and Dietetics · doi:10.1016/j.jand.2014.11.016 · Cited by 266
“need to exist and be incorporated in the analysis, under the assumption of no true treatment effect, before the P value for magnitude of treatment effect became nonsignificant.(11) In addition, a “trim-and-fill” method (a means of correcting for funnel plot asymmetry) was used to adjust for publication bias.”
MethodRhoades, K. A. (2008) · Child Development · doi:10.1111/j.1467-8624.2008.01235.x · Cited by 271
The prevention of child and adolescent anxiety: a meta-analytic review
Fisak, B. J., Richard, D., & Mann, A. (2011) · Prevention Science · doi:10.1007/s11121-011-0210-0 · Cited by 271
McEwan, D., Harden, S. M., Zumbo, B. D., Sylvester, B. D., Kaulius, M., Ruissen, G. R., Dowd, A. J., & Beauchamp, M. R. (2015) · Health Psychology Review · doi:10.1080/17437199.2015.1104258 · Cited by 307
“To assess potential publication bias, we calculated the fail-safe N statistic, which estimates the number of unpublished studies with null findings that would have to exist to reduce the effect size to zero (Rosenthal, 1979).”
MethodIs the insect apocalypse upon us? How to find out
Montgomery, G. A., Dunn, R., Fox, R., Jongejans, E., Leather, S., Saunders, M. E., Shortall, C., Tingley, M., & Wagner, D. L. (2020) · Biological Conservation · doi:10.1016/j.biocon.2019.108327 · Cited by 245
“Publication bias is also no doubt at play, where statistically significant results are selected for publication more often than studies with non-significant results (Rosenthal, 1979).”
BackgroundResultReviewing the literature: A comparison of traditional methods with meta-analysis
Cook, T. D., & Leviton, L. C. (1980) · Journal of Personality · doi:10.1111/j.1467-6494.1980.tb02379.x · Cited by 272
Saeed, S., Yousafzai, S., & Engélen, A. (2014) · Entrepreneurship Theory and Practice · doi:10.1111/etap.12097 · Cited by 271
On the Magnitude of Laterality Effects and Sex Differences in Functional Lateralities
Voyer, D. (1996) · Laterality Asymmetries of Body Brain and Cognition · doi:10.1080/713754209 · Cited by 270
Meta-analysis: Neither quick nor easy
Berman, N. G., & Parker, R. A. (2002) · BMC Medical Research Methodology · doi:10.1186/1471-2288-2-10 · Cited by 270
Appelt, K. C., Milch, K. F., Handgraaf, M. J. J., & Weber, E. U. (2011) · Judgment and Decision Making · doi:10.1017/s1930297500001455 · Cited by 274
Is subjective social status a unique correlate of physical health? A meta-analysis.
Cundiff, J. M., & Matthews, K. A. (2017) · Health Psychology · doi:10.1037/hea0000534 · Cited by 271
“Rosenthal’s fail-safe N (Rosenthal, 1979) was used to estimate how many unpublished effect sizes with a mean effect of zero would be needed to bring the cumulative effect size below statistical significance.”
French, D. P., Cooper, A., & Weinman, J. (2006) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2006.07.029 · Cited by 270
Person–Environment Congruence and Holland's Theory: A Review and Reconsideration
Spokane, A. R., Meir, E. I., & Catalano, M. (2000) · Journal of Vocational Behavior · doi:10.1006/jvbe.2000.1771 · Cited by 303
Personality and smoking: individual‐participant meta‐analysis of nine cohort studies
Hakulinen, C. A., Hintsanen, M., Munafo, M. R., Virtanen, M., Mika Kivimaki, B., Batty, G. D., & Jokela, M. M. (2015) · Addiction · doi:10.1111/add.13079 · Cited by 285
“First, meta-analyses based on published data can be affected by publication bias, which is caused by selective publishing of positive findings, and can artificially inflate effect estimates [25].”
BackgroundMatthes, J., Knoll, J., & von Sikorski, C. (2018) · Communication Research · doi:10.1177/0093650217745429 · Cited by 269
Marsden, E. J., Morgan‐Short, K., Thompson‐Lee, S., & Abugaber, D. (2018) · Language Learning · doi:10.1111/lang.12286 · Cited by 286
“…to submit statistically significant findings) is a widely acknowledged problem, and null findings are confined to the “file drawer” (a term coined by Rosenthal, 1979; the phenomenon documented by, for example, Bakker, van Dijk, & Wicherts, 2012; Schmidt & Oh, 2016; Sterling, Rosenbaum, & Weinkam,…”
BackgroundHow to review the evidence: systematic identification and review of the scientific literature
Glasziou, P., Irwig, L., Bain, C., & Colditz, G. A. (2000) · Cited by 267
Roschk, H., Loureiro, S. M. C., & Breitsohl, J. (2016) · Journal of Retailing · doi:10.1016/j.jretai.2016.10.001 · Cited by 268
Låg, T., & Sæle, R. G. (2019) · AERA Open · doi:10.1177/2332858419870489 · Cited by 367
“Although we have taken steps to avoid a biased selection of studies by searching several databases indexing conference papers, doctoral dissertations, master’s theses, as well as a wide selection of institutional archives through BASE, we should nevertheless check for bias as statistically significant findings are mor…”
ResultA peculiar prevalence of p values just below .05
Masicampo, E. J., & Lalande, D. R. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.711335 · Cited by 266
Shaffer, J. A., & Postlethwaite, B. E. (2012) · Personnel Psychology · doi:10.1111/j.1744-6570.2012.01250.x · Cited by 269
Risky sexual behavior and substance use among adolescents: A meta-analysis
Ritchwood, T. D., Ford, H. L., DeCoster, J., Sutton, M., & Lochman, J. E. (2015) · Children and Youth Services Review · doi:10.1016/j.childyouth.2015.03.005 · Cited by 267
Meta-analysis using effect size distributions of only statistically significant studies.
van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015) · Psychological Methods · doi:10.1037/met0000025 · Cited by 265
Cayuela, M. L., Aguilera, E., Sanz-Cobeña, A., Adams, D. C., Ábalos, D., Barton, L., Ryals, R. A., Silver, W. L., Alfaro, M. A., Pappa, V. A., et al. (2016) · Agriculture Ecosystems & Environment · doi:10.1016/j.agee.2016.10.006 · Cited by 267
“To test the possibility of publication bias (studies showing no significant effects might not be published), the Rosenthal's fail-safe N test was used (Rosenthal, 1979).”
MethodMeta‐analysis of alexithymia in posttraumatic stress disorder
Frewen, P., Dozois, D. J. A., Neufeld, R. W. J., & Lanius, R. A. (2008) · Journal of Traumatic Stress · doi:10.1002/jts.20320 · Cited by 264
Meta-Analysis of Medical Regimen Adherence Outcomes in Pediatric Solid Organ Transplantation
Dew, M. A., Dabbs, A. D., Myaskovsky, L., Shyu, S., Shellmer, D. A., DiMartini, A. F., Steel, J., Unruh, M., Switzer, G. E., Shapiro, R., et al. (2009) · Transplantation · doi:10.1097/tp.0b013e3181b2a0e0 · Cited by 264
Palmer, A. R. (1999) · The American Naturalist · doi:10.1086/303223 · Cited by 264
Thomas, J. J., Vartanian, L. R., & Brownell, K. D. (2009) · Psychological Bulletin · doi:10.1037/a0015326 · Cited by 287
Martland, R., Mondelli, V., Gaughran, F., & Stubbs, B. (2020) · Journal of Sports Sciences · doi:10.1080/02640414.2019.1706829 · Cited by 253
“Although, it is important to note that only 42% and 36% of systematic reviews reported on adverse outcomes and adherence respectively, and even then, not all trials included in these systematic reviews discussed these measures thus there may be bias in these findings, and results may have been skewered towards a more…”
BackgroundA Meta-Analysis of Ethics Instruction Effectiveness in the Sciences
Antes, A. L., Murphy, S. T., Waples, E. P., Mumford, M. D., Brown, R. P., Connelly, S., & Devenport, L. D. (2009) · Ethics & Behavior · doi:10.1080/10508420903035380 · Cited by 270
Questions about Questions: Inquiries into the Cognitive Bases of Surveys.
Schuman, H., & Tanur, J. M. (1993) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2075046 · Cited by 262
Twenty years of bogus pipeline research: A critical review and meta-analysis.
Roese, N. J., & Jamieson, D. W. (1993) · Psychological Bulletin · doi:10.1037/0033-2909.114.2.363 · Cited by 270
Pike, K. E., Cavuoto, M. G., Li, L., Wright, B. J., & Kinsella, G. (2021) · Neuropsychology Review · doi:10.1007/s11065-021-09522-3 · Cited by 285
“The fail-safe N (Rosenthal, 1979) was used to determine the robustness of the findings by computing how many additional studies with an effect size of 0 would render significant findings non-significant.”
MethodMobile-assisted ESL/EFL vocabulary learning: a systematic review and meta-analysis
Lin, J.-J., & Lin, H. (2019) · Computer Assisted Language Learning · doi:10.1080/09588221.2018.1541359 · Cited by 262
“Thus, if the necessary data were not available in the eligible studies, we followed the computing procedures suggested by Rosenthal (1979) and Lipsey and Wilson (2001) to calculate and estimate effect sizes.”
MethodThe MASH Research Team, Scott-Sheldon, L. A. J., Carey, K. B., Cunningham, K., Johnson, B. T., & Carey, M. P. (2016) · AIDS and Behavior · doi:10.1007/s10461-015-1108-9 · Cited by 260
“Assessing Publication Bias Asymmetries in the distributions of effect sizes, indicating a possible reporting bias [61], were examined by inspecting funnel plots [62] and assessing the degree of asymmetry using Begg’s [63] and Egger’s [64] techniques.”
A Meta-Analysis of Working Memory Impairments in Autism Spectrum Disorders
Wang, Y., Zhang, Y.-B., Liu, L.-L., Cui, J.-F., Wang, J., Shum, D. H. K., van Amelsvoort, T., & Chan, R. C. K. (2017) · Neuropsychology Review · doi:10.1007/s11065-016-9336-y · Cited by 252
“The fail-safe N represents the number of additional studies with a null effect (effect size = 0) needed to increase the p value of the meta-analysis to greater than 0.05 (Rosenthal, 1979).”
BackgroundInfluentialDebunking Myths and Urban Legends About Meta-Analysis
Aguinis, H., Pierce, C. A., Bosco, F. A., Dalton, D. R., & Dalton, C. M. (2010) · Organizational Research Methods · doi:10.1177/1094428110375720 · Cited by 264
Acne vulgaris and risk of depression and anxiety: A meta-analytic review
Samuels, D. V., Rosenthal, R., Lin, R., Chaudhari, S. P., & Natsuaki, M. N. (2020) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2020.02.040 · Cited by 288
Self-citationvan Eldik, W. M., de Haan, A. D., Parry, L. Q., Davies, P. T., Luijk, M. P. C. M., Arends, L. R., & Prinzie, P. (2020) · Psychological Bulletin · doi:10.1037/bul0000233 · Cited by 301
Blanck, P., Perleth, S., Heidenreich, T., Kröger, P., Ditzen, B., Bents, H., & Mander, J. (2018) · Behaviour Research and Therapy · doi:10.1016/j.brat.2017.12.002 · Cited by 248
Cavero-Redondo, I., Peleteiro, B., Álvarez-Bueno, C., Rodriguez-Artalejo, F., & Martínez-Vizcaíno, V. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-015949 · Cited by 260
Romero, L. M., & Remage‐Healey, L. (2000) · General and Comparative Endocrinology · doi:10.1006/gcen.2000.7491 · Cited by 259
Song, Y., Yu, F., Keser, L. H., Dawson, W., Fischer, M., Dong, M., & van Kleunen, M. (2012) · Oecologia · doi:10.1007/s00442-012-2430-9 · Cited by 259
Pahlke, E. E., Hyde, J. S., & Allison, C. M. (2014) · Psychological Bulletin · doi:10.1037/a0035740 · Cited by 298
Self-Enhancement and Psychological Adjustment: A Meta-Analytic Review
Dufner, M., Gebauer, J. E., Sedikides, C., & Denissen, J. J. A. (2018) · Personality and Social Psychology Review · doi:10.1177/1088868318756467 · Cited by 279
Meta-analyses of Studies of Ventricular Enlargement and Cortical Sulcal Prominence in Mood Disorders
Elkis, H. (1995) · Archives of General Psychiatry · doi:10.1001/archpsyc.1995.03950210029008 · Cited by 259
Cuijpers, P., Hollon, S. D., van Straten, A., Bockting, C., Berking, M., & Andersson, G. (2013) · BMJ Open · doi:10.1136/bmjopen-2012-002542 · Cited by 253
Outcome of psychological treatments of pathological gambling: a review and meta-analysis
Pallesen, S., Mitsem, M., Kvale, G., Johnsen, B., & Molde, H. (2005) · Addiction · doi:10.1111/j.1360-0443.2005.01204.x · Cited by 258
Perioperative Single Dose Ketorolac to Prevent Postoperative Pain
Oliveira, G. S. D., Agarwal, D., & Benzon, H. T. (2011) · Anesthesia & Analgesia · doi:10.1213/ane.0b013e3182334d68 · Cited by 258
Invited review: Use of meta-analysis in animal health and reproduction: Methods and applications
Lean, I. J., Rabiee, A. R., Duffield, T. F., & Dohoo, I. R. (2009) · Journal of Dairy Science · doi:10.3168/jds.2009-2140 · Cited by 265
Homework and students' achievement in math and science: A 30-year meta-analysis, 1986–2015
Fan, H., Xu, J., Cai, Z., He, J., & Fan, X. (2016) · Educational Research Review · doi:10.1016/j.edurev.2016.11.003 · Cited by 272
Ecological momentary assessment methodology in chronic pain research: a systematic review
May, M., Junghaenel, D. U., Ono, M., Stone, A. A., & Schneider, S. (2018) · The Journal of Pain · doi:10.1016/j.jpain.2018.01.006 · Cited by 247
On the Perception of Social Consensus
Krueger, J. I. (1998) · Advances in experimental social psychology · doi:10.1016/s0065-2601(08)60384-6 · Cited by 254
Lauriola, M., Panno, A., Levin, I. P., & Lejuez, C. W. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1784 · Cited by 283
Blais, J., Solodukhin, E., & Forth, A. E. (2014) · Criminal Justice and Behavior · doi:10.1177/0093854813519629 · Cited by 257
Cognitive outcomes in children and adolescents born very preterm: a meta‐analysis
Brydges, C. R., Landes, J. K., Reid, C. L., Campbell, C., French, N., & Anderson, M. (2018) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.13685 · Cited by 258
What Do Unions Do to Productivity? A Meta‐Analysis
Doucouliagos, H., & Laroche, P. (2003) · Industrial Relations A Journal of Economy and Society · doi:10.1111/1468-232x.00310 · Cited by 251
Lai, C. L. E., Lau, Z., Lui, S. S. Y., Lok, E., Tam, V., Chan, Q., Cheng, K. M., Lam, S. M., & Cheung, E. F. C. (2017) · Autism Research · doi:10.1002/aur.1723 · Cited by 250
“Publication Bias Publication bias was assessed by visual examination of the funnel plots, Duval and Tweedie’s trim and fill method [Duval & Tweedie, 2000] and Rosenthal’s failsafe N test [Rosenthal, 1979].”
MethodJanicke, D. M., Steele, R. G., Gayes, L. A., Lim, C. S., Clifford, L. M., Schneider, E. M., Carmody, J., & WESTEN, S. C. (2014) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsu023 · Cited by 256
Does diversifying crop rotations suppress weeds? A meta-analysis
Weisberger, D. A., Nichols, V., & Liebman, M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0219847 · Cited by 281
“To estimate the robustness of our results against publication bias, we calculated the Rosenthal fail-safe number [36] using the metafor package [37].”
MethodComputer-delivered interventions to reduce college student drinking: A meta-analysis
Carey, K. B., Scott‐Sheldon, L. A. J., Elliott, J. C., Bolles, J. R., & Carey, M. P. (2009) · Addiction · doi:10.1111/j.1360-0443.2009.02691.x · Cited by 250
“, stronger effects reported in published studies; [11]).”
BackgroundSpruit, A., Assink, M., van Vugt, E., van der Put, C., & Stams, G. J. (2016) · Clinical Psychology Review · doi:10.1016/j.cpr.2016.03.006 · Cited by 250
“Not including these studies may lead to an overestimation of the true effect size, the so called “ publication bias ” (Rosenthal, 1979).”
Metadata
- Authors
- Rosenthal, R.
- Affiliations
- Harvard University (US)
- Corresponding author
- Rosenthal, Robert
- Venue
- Psychological Bulletin
- Published
- 1979-05
- Volume
- 86
- Issue
- 3
- Pages
- 638-641
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2006546769
- Cited by
- 9,068
- References
- 16
- Citations by year
- 2026: 421 · 2025: 521 · 2024: 509 · 2023: 526 · 2022: 525 · 2021: 536 · 2020: 612 · 2019: 607 · 2018: 492 · 2017: 436 · 2016: 417 · 2015: 392 · 2014: 337 · 2013: 289 · 2012: 252
- FWCI
- 26.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- file drawer problem, negative results, publication bias, non-significant results, type I error, selective reporting, behavioral science, evidence synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
