The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
The file drawer problem and tolerance for null results.
Abstract
For any given research area, one cannot tell how many studies have been con-ducted but never reported. The extreme view of the "file drawer problem " is that journals are filled with the 5 % of the studies that show Type I errors, while the file drawers are filled with the 95 % of the studies that show non-significant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed. Both behavioral researchers and statisti-cians have long suspected that the studies published in the behavioral sciences are a biased sample of the studies that are actually carried out (Bakan, 1967; McNemar, 1960; Smart, 1964; Sterling, 1959). The extreme view of this problem, the "file drawer prob-lem, " is that the journals are filled with the 5 % of the studies that show Type I errors, while the file drawers back at the lab are filled with the 95 % of the studies that show nonsignificant (e.g., p>.05) results. In the past there was very little one could do to assess the net effect of studies, tucked away in file drawers, that did not make the magic.05 level (Rosenthal & Gaito, 1963, 1964). Now, however, although no definitive solution to the problem is available, one can establish reasonable boundaries on the prob-lem and estimate the degree of damage to any research conclusion that could be done by the file drawer problem. This advance in our ability to cope with the file drawer is an outgrowth of the in-creasing interest of behavioral scientists in summarizing bodies of research literature sys-Preparation of this article was supported in part by the National Science Foundation. I would like to thank Judith A. Hall and Donald B. Rubin for their valuable improvements of an earlier version of this article. Requests for reprints should be sent to Robert
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We then calculated Rosenthal’s fail-safe N (Rosenthal, 1979), and determined that 2,153,789 effects evidencing no relationship between SA and psychopathology (g = 0) would be needed to reduce the significance level of the summary effect size to .05.”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“The file drawer effect is a perennial problem in the academic literature (Rosenthal, 1979)”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“First, we calculated the fail-safe N [177] to be 4,274, which is the theoretical number of unpublished studies with effect sizes averaging zero (no effect) that would be needed to render negligible the omnibus results.”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [177]
In the citing paper
“As we did not include unpublished work in our meta-analyses we tested for publication bias in the following ways.Firstly, funnel plots were examined.demonstrating no systematic publication biases (tables available from first author on request).Secondly..Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Presence of publication bias was assessed with funnel plots, 22 Rosenthal's fail-safe N , 23 and Egger's regression intercept. 24”
From Light Therapy for Patients With Bipolar Depression: Systematic Review and Meta-Analysis of Randomized Controlled Trials (2020) · cites this paper as [23]
In the citing paper
“In terms of study quality, there was some indication of publication bias whereby some associations between racism and health outcomes were stronger in published versus unpublished studies, consistent with the tendency for ‘null’ studies to remain unpublished (i.e., the ‘file drawer’ problem [418]).”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as [418]
In the citing paper
“An alternative, perhaps more plausible, explanation is that there could be a significant amount of unpublished negative or inconclusive results, a phenomenon referred to as the “file-drawer effect” or publication bias (23).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [23]
In the citing paper
“If either of the two last hypotheses is true, there should exist many unpublished studies with null results lying in laboratories' drawers (32).”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Publication bias was evaluated using Rosenthal's file drawer analysis (Rosenthal, 1979) and Egger's linear regression (Card, 2012), and applying ANOVA to compare the mean effect size of the published versus unpublished studies.”
From Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal (1979) suggested a fail-safe N value above 5; k + 10 reflects results that are tolerant to contradicting studies”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“We applied random-effects models to fit the relations between CE and the criterion variables, following the guidelines proposed by Rosenthal (1979), Hunter and Schmidt (2004) and Cumming (2013).”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal (1979)
In the citing paper
“The fail-safe value for this meta-analysis was computed to be 3001, which is more than 5(36) + 10.”
From Effects of digital game-based STEM education on students’ learning achievement: a meta-analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, we suggest our results may look like a lot of studies that ended up in the so-called “file drawer” [57].”
From Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory (2023) · cites this paper as [57]
In the citing paper
“Finally, publication bias was addressed by computing the fail-safe N for the correlation coefficients (Hunter & Schmidt, 2004;Rosenthal, 1979), which reports the number of field effect sizes or the tolerance for future null results, to check whether a publication bias affected our results.”
From Drivers of Consumer Adoption of e-Commerce: a Meta-Analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Using Rosenthal’s (1979) fail-safe N criterion that the value should be over five times greater than the number of studies included in the meta-analysis, the fail-safe N for all outcome groups were at least eight times larger than the criterion value, suggesting that the effect sizes are unlikely to be an artifact of bias.”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as Rosenthal (1979)
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, Rosenthal (1979) opined that a Fail-safe N exceeding the number of studies by fivefold would provide strong support for the robustness of the significant effects.”
From Extending the Theory of Planned Behavior in the Social Commerce Context: A Meta-Analytic SEM (MASEM) Approach (2023) · cites this paper as Rosenthal (1979)
In the citing paper
“The trim-and-fill method (Duval & Tweedie, 2000) tends to adjust too little when bias is strong (Carter et al., 2019), and Fail-Safe N (Rosenthal, 1979) neither tests nor adjusts for the presence of bias.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Rosenthal, 1979
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, a phenomenon known as ' publication bias ' (Rosenthal 1979).”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“However, a common problem is that results may not have been reported in dissertations or research reports, or published in scienti fi c (peer-reviewed) journals, due to non-signi fi cant or unfavorable fi ndings and, therefore, are dif fi cult to locate.”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [21]
In the citing paper
“The traditional fail-safe value is 1904, which is higher than the threshold value of 100, i.e., [5(18) + 10], indicating that publication bias is insignificant, and the study's results are reliable.”
From A meta-analysis to explore the role of design thinking in enhancing creativity as learning outcomes in STEM education (2026) · cites this paper as Rosenthal, 1979
In the citing paper
“The Q statistic was used to assess heterogeneity30, and publication bias was estimated using Rosenthal’s fail-safe number with α = 0.0531.”
From The limited prosocial effects of meditation: A systematic review and meta-analysis (2018) · cites this paper as [31]
In the citing paper
“Rosenthal's fail-safe number (NR) was calculated, estimating the number of unpublished studies with non-significant results needed to bring an observed result to non-significance (NR ⩾ 5k + 10 to rule out a file drawer problem) (Rosenthal, 1979).”
From Efficacy of non-invasive brain stimulation on cognitive functioning in brain disorders: a meta-analysis (2020) · cites this paper as Rosenthal, 1979
In the citing paper
“In case of statistically significant effect sizes, it was planned to examine the robustness of findings by calculating the Failsafe N (number of non‐significant studies that would bring the p‐value to non‐significant (p > 0.05)) 40.”
From Cognitive behavioral therapies for informal caregivers of patients with cancer and cancer survivors: a systematic review and meta-analysis (2017) · cites this paper as [40]
In the citing paper
“With negative results buried in file drawers (55), conclusions drawn from the published record can be misleading, and 'false facts' can become canonized in the literature (56).”
From Misinformation in and about science (2021) · cites this paper as [55]
In the citing paper
“the fail-safe N (i.e. the file drawer statistic) was used to determine how many additional studies would be necessary to turn a significant ES non-significant (32).”
From Peripheral cytokine and chemokine alterations in depression: a meta‐analysis of 82 studies (2017) · cites this paper as [32]
In the citing paper
“File drawer analysis (Rosenthal, 1979) revealed that 1919 unpublished, filed, or unretrieved studies averaging null results would be required to bring the average unweighted effect size to nonsignificance.”
From Happy to help? A systematic review and meta-analysis of the effects of performing acts of kindness on the well-being of the actor (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's (1979) fail-safe N analysis reveals 186 unpublished papers (i.e., null effect) are needed to be included in the meta-analysis in order to invalidate the population effect size of small to medium (i.e., .235)found in this review.”
From A meta-analytic review of the efficacy of physical exercise interventions on cognition in individuals with autism spectrum disorder and ADHD (2016) · cites this paper as Rosenthal's (1979)
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, the so called ' publication bias ' (Rosenthal, 1979).”
From The effects of physical activity interventions on psychosocial outcomes in adolescents: a meta-analytic review (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's fail-safe N 37 calculated 115 missing studies would be required to increase the two-tailed p-value to above .05.”
From Anxiety in Informal Dementia Carers: A Meta-Analysis of Prevalence (2020) · cites this paper as [37]
In the citing paper
“Publication bias was investigated with Begg's rank correlation test for funnel plot asymmetry (Begg and Mazumdar, 1994) and where significant effects were present Rosenthal's file drawer analysis was also conducted (Rosenthal, 1979).”
From Meta-analysis of the effects of intranasal oxytocin on interpretation and expression of emotions (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“Over time, this practice artificially inflates Type I error and may make it appear that relationships exist when in reality they do not (Rosenthal et al., 1979).”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“However, a potential problem with meta-analysis is the file drawer problem, where studies are “filed away” unpublished if they fail to find statistically significant relationships.”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“Meta-analyses are necessarily based on the available literature and hence may be affected by the potential publication-bias disfavoring null results (Rosenthal 1979).”
From Parsing the neural correlates of moral cognition: ALE meta-analysis on morality, theory of mind, and empathy (2012) · cites this paper as Rosenthal 1979
In the citing paper
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
From The Extent and Consequences of P-Hacking in Science (2015) · cites this paper as [7]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G. E., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,518
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718 · cited 1×
“Perhaps the simplest, most useful way of computing the overall p of a, set of research studies is the method of adding Zs (Cochran, 1954; Mosteller & Bush, 1954; Rosenthal, 1978).”
MethodMeta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154
Experimenter Effects in Behavioral Research
McGuigan, F. J., & Rosenthal, R. (1968) · The American Journal of Psychology · doi:10.2307/1420816 · Cited by 1,633
Self-citationArtifact in behavioral research
Rosenthal, R., & Rosnow, R. L. (1969) · Medical Entomology and Zoology · Cited by 1,186
Self-citationSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964
Combining results of independent studies.
Rosenthal, R. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.1.185 · Cited by 910
Self-citationGender effects in decoding nonverbal cues.
Hall, J. A. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.4.845 · Cited by 766
Wortman, C. B., Rosenthal, R., & Rosnow, R. L. (1976) · The American Journal of Psychology · doi:10.2307/1421418 · Cited by 428
Self-citation
Cited by
Sorted by citations
Items 101–200 of 1,374
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293
“The classic file-drawer publication bias problem (wherein nonsignificant or “uninteresting” results are not published) (29) results in bias under the assumption that multiple studies are being produced independently but a biased sample is published.”
BackgroundBrener, N. D., Billy, J. O. G., & Grady, W. R. (2003) · Journal of Adolescent Health · doi:10.1016/s1054-139x(03)00052-1 · Cited by 1,331
Fitting three-level meta-analytic models in R: A step-by-step tutorial
Assink, M., & Wibbelink, C. J. M. (2016) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.12.3.p154 · Cited by 1,330
Publication bias in meta-analysis its causes and consequences
Thornton, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00161-4 · Cited by 1,291
Sex differences in empathy and related capacities.
Eisenberg, N., & Lennon, R. (1983) · Psychological Bulletin · doi:10.1037/0033-2909.94.1.100 · Cited by 1,410
A Meta-Analytic Review of Psychosocial Interventions for Substance Use Disorders
Dutra, L., Stathopoulou, G., Basden, S. L., Leyro, T. M., Powers, M. B., & Otto, M. W. (2008) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2007.06111851 · Cited by 1,287
Chiaburu, D. S., & Harrison, D. A. (2008) · Journal of Applied Psychology · doi:10.1037/0021-9010.93.5.1082 · Cited by 1,441
The influence of human disturbance on wildlife nocturnality
Gaynor, K. M., Hojnowski, C. E., Carter, N., & Brashares, J. S. (2018) · Science · doi:10.1126/science.aar7121 · Cited by 1,307
Davidson, A. M., Jennions, M. D., & Nicotra, A. B. (2011) · Ecology Letters · doi:10.1111/j.1461-0248.2011.01596.x · Cited by 1,284
Kruschke, J. K., & Liddell, T. M. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1221-4 · Cited by 1,288
“But scientific journals (i.e., reviewers and editors) are reluctant to publish nonsignificant results (e.g., Rosenthal, 1979).”
Meta-Analysis in Clinical Research
L'ABBÉ, K. A., DETSKY, A. S., & O'ROURKE, K. (1987) · Annals of Internal Medicine · doi:10.7326/0003-4819-107-2-224 · Cited by 1,267
Blascovich, J., Loomis, J., Beall, A. C., Swinth, K. R., Hoyt, C. L., & Bailenson, J. N. (2002) · Psychological Inquiry · doi:10.1207/s15327965pli1302_01 · Cited by 1,280
Information sharing and team performance: A meta-analysis
Mesmer-Magnus, J. R., & DeChurch, L. A. (2009) · Journal of Applied Psychology · doi:10.1037/a0013773 · Cited by 1,273
Motor coordination in autism spectrum disorders: A synthesis and meta-analysis
Fournier, K. A., Hass, C. J., Naik, S. K., Lodha, N., & Cauraugh, J. H. (2010) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-010-0981-3 · Cited by 1,271
Culturally adapted mental health intervention: a meta-analytic review
Griner, D., & Smith, T. B. (2006) · Psychotherapy: Theory, Research, Practice, Training · doi:10.1037/0033-3204.43.4.531 · Cited by 1,236
Tang, B. M., Eslick, G. D., Nowson, C., Smith, C., & Bensoussan, A. (2007) · The Lancet · doi:10.1016/s0140-6736(07)61342-7 · Cited by 1,248
Meta-Analyses of Randomized Controlled Trials
Sacks, H. S., Berrier, J., Reitman, D., Ancona-Berk, V. A., & Chalmers, T. C. (1987) · New England Journal of Medicine · doi:10.1056/nejm198702193160806 · Cited by 1,257
Gender differences in emotion expression in children: A meta-analytic review.
Chaplin, T. M., & Aldao, A. (2013) · Psychological Bulletin · doi:10.1037/a0030737 · Cited by 1,247
“Boys may be more inclined than girls to express anger-related emotions with peers as part of their greater tendency to engage in rough-and-tumble play (Rose & Rudolph, 2006). Positive and internalizing emotion expressions, which are more common for girls, show the largest gender differences in the presence of an unfam…”
BackgroundPredictors of posttraumatic stress disorder and symptoms in adults: A meta-analysis.
Ozer, E. J., Best, S. R., Lipsey, T. L., & Weiss, D. S. (2008) · Psychological Trauma Theory Research Practice and Policy · doi:10.1037/1942-9681.s.1.3 · Cited by 1,252
Determinants of Entrepreneurial Intent: A Meta–Analytic Test and Integration of Competing Models
Schlaegel, C., & Koenig, M. (2013) · Entrepreneurship Theory and Practice · doi:10.1111/etap.12087 · Cited by 1,409
How bad are the effects of bad leaders? A meta-analysis of destructive leadership and its outcomes
Schyns, B., & Schilling, J. (2013) · The Leadership Quarterly · doi:10.1016/j.leaqua.2012.09.001 · Cited by 1,245
The relationship of age to ten dimensions of job performance.
Ng, T. W. H., & Feldman, D. C. (2008) · Journal of Applied Psychology · doi:10.1037/0021-9010.93.2.392 · Cited by 1,224
Crampton, S. M., & Wagner, J. A. (1994) · Journal of Applied Psychology · doi:10.1037/0021-9010.79.1.67 · Cited by 1,224
The relative efficacy of avoidant and nonavoidant coping strategies: A meta-analysis.
Suls, J. M., & Fletcher, B. (1985) · Health Psychology · doi:10.1037//0278-6133.4.3.249 · Cited by 1,224
Teachers’ self-efficacy, personality, and teaching effectiveness: A meta-analysis
Klassen, R. M., & Tze, V. M. C. (2014) · Educational Research Review · doi:10.1016/j.edurev.2014.06.001 · Cited by 1,221
Egger, M., Jüni, P., Bartlett, C., Holenstein, F., & Sterne, J. (2003) · Health Technology Assessment · doi:10.3310/hta7010 · Cited by 1,181
Rosenberg, M. S. (2005) · Evolution · doi:10.1111/j.0014-3820.2005.tb01004.x · Cited by 1,285
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196
Construct Validity: A Review of Basic Issues and Marketing Practices
Peter, J. P. (1981) · Journal of Marketing Research · doi:10.2307/3150948 · Cited by 1,189
Psychological Safety: A Meta‐Analytic Review and Extension
Frazier, M. L., Fainshmidt, S., Klinger, R. L., Pezeshkan, A., & Vracheva, V. (2017) · Personnel Psychology · doi:10.1111/peps.12183 · Cited by 1,231
Horne, R., Chapman, S. C. E., Parham, R., Freemantle, N., Forbes, A., & Cooper, V. (2013) · PLoS ONE · doi:10.1371/journal.pone.0080633 · Cited by 1,174
How numeracy influences risk comprehension and medical decision making.
Reyna, V. F., Nelson, W. L., Han, P. K. J., & Dieckmann, N. F. (2009) · Psychological Bulletin · doi:10.1037/a0017327 · Cited by 1,172
Children of divorce in the 1990s: An update of the Amato and Keith (1991) meta-analysis.
Amato, P. R. (2001) · Journal of Family Psychology · doi:10.1037/0893-3200.15.3.355 · Cited by 1,169
Field, A. P., & Gillett, R. (2010) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711010x502733 · Cited by 1,162
Sisk, V. F., Burgoyne, A. P., Sun, J., Butler, J. L., & Macnamara, B. N. (2018) · Psychological Science · doi:10.1177/0956797617739704 · Cited by 1,159
“Pub-lication bias occurs when some results are systematically less likely to be published than others (e.g., studies that find small or null effects; Rosenthal, 1979).”
BackgroundCumulative Meta-Analysis of Therapeutic Trials for Myocardial Infarction
Lau, J., Antman, E. M., Jimenez-Silva, J., Kupelnick, B., Mosteller, F., & Chalmers, T. C. (1992) · New England Journal of Medicine · doi:10.1056/nejm199207233270406 · Cited by 1,140
Smith, T. B., McCullough, M. E., & Poll, J. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.4.614 · Cited by 1,150
Health Message Framing Effects on Attitudes, Intentions, and Behavior: A Meta-analytic Review
Gallagher, K. M., & Updegraff, J. A. (2011) · Annals of Behavioral Medicine · doi:10.1007/s12160-011-9308-7 · Cited by 1,156
A systematic review and meta-analysis of the cognitive correlates of bilingualism
Adesope, O. O., Lavin, T., Thompson, T., & Ungerleider, C. (2010) · Review of Educational Research · doi:10.3102/0034654310368803 · Cited by 1,132
Interpersonal expectancy effects: the first 345 studies
Rosenthal, R., & Rubin, D. B. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00075506 · Cited by 1,158
Self-citationThe Effectiveness of Creativity Training: A Quantitative Review
Scott, G., Leritz, L. E., & Mumford, M. D. (2004) · Creativity Research Journal · doi:10.1080/10400410409534549 · Cited by 1,132
Blut, M., Wang, C., Wünderlich, N. V., & Brock, C. (2021) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-020-00762-y · Cited by 1,194
“Most FSNs (15 out of 23) exceeded the tolerance levels suggested by Rosenthal (1979), indicating results to be robust to publication bias.”
BackgroundResultPersonality and Prejudice: A Meta-Analysis and Theoretical Review
Sibley, C. G., & Duckitt, J. H. (2008) · Personality and Social Psychology Review · doi:10.1177/1088868308319226 · Cited by 1,130
The Influence of Framing on Risky Decisions: A Meta-analysis
Kühberger, A. (1998) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1998.2781 · Cited by 1,180
Inflammation and frailty in the elderly: A systematic review and meta-analysis
Soysal, P., Stubbs, B., Lucato, P., Luchini, C., Solmi, M., Peluso, R., Sergi, G., Isik, A. T., Manzato, E., Maggi, S., et al. (2016) · Ageing Research Reviews · doi:10.1016/j.arr.2016.08.006 · Cited by 1,089
“05) the effect size was calculated (Rosenthal, 1979).”
InfluentialMeta-Analysis of the Relationship Between HIV Infection and Risk for Depressive Disorders
Ciesla, J. A., & Roberts, J. E. (2001) · American Journal of Psychiatry · doi:10.1176/appi.ajp.158.5.725 · Cited by 1,106
The effect of testing versus restudy on retention: A meta-analytic review of the testing effect.
Rowland, C. A. (2014) · Psychological Bulletin · doi:10.1037/a0037559 · Cited by 1,066
The relationship between burnout, depression, and anxiety: a systematic review and meta-analysis.
Koutsimani, P., Montgomery, A., & Georganta, K. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00284 · Cited by 1,091
“…although our database search was conducted through four well–known databases, we still cannot be certain whether all the studies that examined the burnout–depression and burnout–anxiety relationships are reported; a limitation which is also known as the “file–drawer problem” (Rosenthal, 1979).”
BackgroundInfluentialReligious involvement and mortality: A meta-analytic review.
McCullough, M. E., Hoyt, W. T., Larson, D. B., Koenig, H. G., & Thoresen, C. E. (2000) · Health Psychology · doi:10.1037/0278-6133.19.3.211 · Cited by 1,115
The Role Of Context In Work Team Diversity Research: A Meta-Analytic Review
Joshi, A., & Roh, H. (2009) · Academy of Management Journal · doi:10.5465/amj.2009.41331491 · Cited by 1,362
Relative effectiveness and validity of mood induction procedures: a meta-analysis
Westermann, R., Spies, K., Stahl, G., & Hesse, F. W. (1996) · European Journal of Social Psychology · doi:10.1002/(sici)1099-0992(199607)26:4 · Cited by 1,134
Methodological issues and advances in biological meta-analysis
Nakagawa, S., & Santos, E. S. A. (2012) · Evolutionary Ecology · doi:10.1007/s10682-012-9555-5 · Cited by 1,032
STATISTICAL ISSUES IN ECOLOGICAL META-ANALYSES
Gurevitch, J., & Hedges, L. V. (1999) · Ecology · doi:10.1890/0012-9658(1999)080[1142:siiema]2.0.co;2 · Cited by 1,084
A systematic review of personality trait change through intervention.
Roberts, B. W., Luo, J., Briley, D. A., Chow, P. I., Su, R., & Hill, P. L. (2017) · Psychological Bulletin · doi:10.1037/bul0000088 · Cited by 1,047
“Techniques commonly used in psychological research, such as trim-and-fill (Duval & Tweedie, 2000) or the failsafe N (Rosenthal, 1979), have been thoroughly criticized (Moreno et al., 2009; Peters et al., 2007; Sutton, 2009).”
MethodWord-of-mouth communications in marketing: a meta-analytic review of the antecedents and moderators
de Matos, C. A., & Rossi, C. A. V. (2008) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-008-0121-1 · Cited by 1,144
A META‐ANALYSIS OF SELF‐SUPERVISOR, SELF‐PEER, AND PEER‐SUPERVISOR RATINGS
Harris, M. M., & Schaubroeck, J. M. (1988) · Personnel Psychology · doi:10.1111/j.1744-6570.1988.tb00631.x · Cited by 1,080
Aguilar, R., Ashworth, L., Galetto, L., & Aizen, M. A. (2006) · Ecology Letters · doi:10.1111/j.1461-0248.2006.00927.x · Cited by 1,078
Being Inconsistent About Consistency: When Coefficient Alpha Does and Doesn't Matter
Streiner, D. L. (2003) · Journal of Personality Assessment · doi:10.1207/s15327752jpa8003_01 · Cited by 1,073
The impacts of roads and other infrastructure on mammal and bird populations: A meta-analysis
Benítez‐López, A., Alkemade, R., & Verweij, P. A. (2010) · Biological Conservation · doi:10.1016/j.biocon.2010.02.009 · Cited by 1,123
Rudolph, C. W., Katz, I. M., Lavigne, K. N., & Zacher, H. (2017) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2017.05.008 · Cited by 951
Classical fear-conditioning in the anxiety disorders: a meta-analysis
Lissek, S., Powers, A. S., McClure, E. B., Phelps, E. A., Woldehawariat, G., Grillon, C., & Pine, D. S. (2005) · Behaviour Research and Therapy · doi:10.1016/j.brat.2004.10.007 · Cited by 1,034
Nonresident Fathers and Children's Well-Being: A Meta-Analysis
Amato, P. R., & Gilbreth, J. G. (1999) · Journal of Marriage and the Family · doi:10.2307/353560 · Cited by 1,063
A meta-analysis of bees’ responses to anthropogenic disturbance
Winfree, R., Aguilar, R., Vázquez, D. P., LeBuhn, G., & Aizen, M. A. (2009) · Ecology · doi:10.1890/08-1245.1 · Cited by 1,050
Dissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039
The Short-term Impact of Unconditional Cash Transfers to the Poor: Experimental Evidence from Kenya*
Haushofer, J., & Shapiro, J. (2016) · The Quarterly Journal of Economics · doi:10.1093/qje/qjw025 · Cited by 1,128
Relationships between psychological climate perceptions and work outcomes: a meta‐analytic review
Parker, C. P., Baltes, B. B., Young, S. A., Huff, J. W., Altmann, R. A., Lacost, H. A., & Roberts, J. E. (2003) · Journal of Organizational Behavior · doi:10.1002/job.198 · Cited by 1,079
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944
“…pervasive low-powered research plus publication bias that favors positive over negative results (Button et al., 2013; Cohen, 1962; Greenwald, 1975; Rosenthal, 1979) and because of questionable research practices, such as p -hacking, that can inflate the likelihood of obtaining false positives…”
BackgroundBiochar's role in mitigating soil nitrous oxide emissions: A review and meta-analysis
Cayuela, M. L., van Zwieten, L., Singh, B. P., Jeffery, S., Roig, A., & Sánchez-Monedero, M. A. (2014) · Agriculture, Ecosystems & Environment · doi:10.1016/j.agee.2013.10.009 · Cited by 1,053
Delayed reward discounting and addictive behavior: a meta-analysis
MacKillop, J., Amlung, M. T., Few, L. R., Ray, L. A., Sweet, L. H., & Munafò, M. R. (2011) · Psychopharmacology · doi:10.1007/s00213-011-2229-0 · Cited by 1,021
Stam, W., Arzlanian, S., & Elfring, T. (2014) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2013.01.002 · Cited by 973
The cognitive underpinnings of effective teamwork: A meta-analysis.
DeChurch, L. A., & Mesmer‐Magnus, J. (2010) · Journal of Applied Psychology · doi:10.1037/a0017328 · Cited by 1,129
Modelling neural correlates of working memory: A coordinate-based meta-analysis
Rottschy, C., Langner, R., Dogan, I., Reetz, K., Laird, A. R., Schulz, J. B., Fox, P. T., & Eickhoff, S. B. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.11.050 · Cited by 951
Hartnell, C. A., Ou, A. Y., & Kinicki, A. J. (2011) · Journal of Applied Psychology · doi:10.1037/a0021987 · Cited by 1,088
Web Surveys versus other Survey Modes: A Meta-Analysis Comparing Response Rates
Manfreda, K. L., Bošnjak, M., Berzelak, J., Haas, I., & Vehovar, V. (2008) · International Journal of Market Research · doi:10.1177/147078530805000107 · Cited by 1,003
A meta-analysis of the trait resilience and mental health
Hu, T., Zhang, D., & Wang, J. (2015) · Personality and Individual Differences · doi:10.1016/j.paid.2014.11.039 · Cited by 1,131
Nguyen, A. D., & Benet‐Martínez, V. (2012) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022111435097 · Cited by 1,155
Parsons, T. D., & Rizzo, A. A. (2008) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2007.07.007 · Cited by 994
Mini Meta‐Analysis of Your Own Studies: Some Arguments on Why and a Primer on How
Goh, J. X., Hall, J. A., & Rosenthal, R. (2016) · Social and Personality Psychology Compass · doi:10.1111/spc3.12267 · Cited by 994
“…critiques over the years (Cumming, 2014), and by limiting a paper to only significant findings, researchers will inevitably contribute to an unhealthy culture of questionable research practices such as p-hacking (Simonsohn, Nelson, & Simmons, 2014), file-drawer effects (Rosenthal, 1979), and more.”
BackgroundSelf-citationInstructional animation versus static pictures: A meta-analysis
Höffler, T. N., & Leutner, D. (2007) · Learning and Instruction · doi:10.1016/j.learninstruc.2007.09.013 · Cited by 992
Assessing Macro-Level Predictors and Theories of Crime: A Meta-Analysis
Pratt, T. C., & Cullen, F. T. (2005) · Crime and Justice · doi:10.1086/655357 · Cited by 995
The relation of empathy to aggressive and externalizing/antisocial behavior.
Miller, P. A., & Eisenberg, N. (1988) · Psychological Bulletin · doi:10.1037/0033-2909.103.3.324 · Cited by 1,211
Chapter 10: Addressing reporting biases
Egger, M., & Moher, D. (2011) · Cited by 977
Johnson, N. D., & Mislin, A. A. (2011) · Journal of Economic Psychology · doi:10.1016/j.joep.2011.05.007 · Cited by 967
Psychological Treatments That Cause Harm
Lilienfeld, S. O. (2007) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2007.00029.x · Cited by 959
Stanley, T. D., & Doucouliagos, H. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1095 · Cited by 916
Emotion regulation strategies in depressive and anxiety symptoms in youth: A meta-analytic review
Schäfer, J. Ö., Naumann, E., Holmes, E. A., Tuschen-Caffier, B., & Samson, A. C. (2017) · Journal of Youth and Adolescence · doi:10.1007/s10964-016-0585-0 · Cited by 941
Cao, Z. J., & Lumineau, F. (2014) · Journal of Operations Management · doi:10.1016/j.jom.2014.09.009 · Cited by 996
A meta-analytic review of research on gender differences in sexuality, 1993–2007.
Petersen, J. L., & Hyde, J. S. (2010) · Psychological Bulletin · doi:10.1037/a0017504 · Cited by 931
Sergerie, K., Chochol, C., & Armony, J. L. (2008) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2007.12.002 · Cited by 917
The psychological effects of meditation: a meta-analysis
Sedlmeier, P., Eberth, J., Schwarz, M., Zimmermann, D., Haarig, F., Jaeger, S., & Kunze, S. (2012) · Psychological Bulletin · doi:10.1037/a0028168 · Cited by 910
Doing More with Less: Innovation Input and Output in Family Firms
Durán, P., Kammerlander, N. H., van Essen, M., & Zellweger, T. M. (2015) · Academy of Management Journal · doi:10.5465/amj.2014.0424 · Cited by 956
“Since insignificant results are less likely to be published, we tested for the “ file drawer problem ” (Rosenthal, 1979) by including a dummy variable denoting whether a study was published or not (reference group).”
The Handbook of Research Synthesis
Ziegel, E. R., Cooper, H., & Hedges, L. V. (1995) · Technometrics · doi:10.2307/1269651 · Cited by 936
Evaluative conditioning in humans: A meta-analysis.
Hofmann, W., Houwer, J. D., Perugini, M., Baeyens, F., & Crombez, G. (2010) · Psychological Bulletin · doi:10.1037/a0018916 · Cited by 993
Role of social desirability in personality testing for personnel selection: The red herring.
Öneş, D. S., Viswesvaran, C., & Reiss, A. D. (1996) · Journal of Applied Psychology · doi:10.1037/0021-9010.81.6.660 · Cited by 924
Ruffman, T., Henry, J. D., Livingstone, V., & Phillips, L. H. (2008) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2008.01.001 · Cited by 984
The ecological importance of intraspecific variation
Roches, S. D., Post, D. M., Turley, N. E., Bailey, J. K., Hendry, A. P., Kinnison, M. T., Schweitzer, J. A., & Palkovacs, E. P. (2017) · Nature Ecology & Evolution · doi:10.1038/s41559-017-0402-5 · Cited by 957
Thoresen, C. J., Kaplan, S. A., Barsky, A. P., Warren, C. R., & de Chermont, K. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.6.914 · Cited by 617
Cross-validation failure: Small sample sizes lead to large error bars
Varoquaux, G. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.06.061 · Cited by 897
“Such selective reporting can further undermine control of the fraction of false detections in a body of literature (Rosenthal, 1979).”
BackgroundMeta-Analysis of Inquiry-Based Learning: Effects of Guidance
Lazonder, A. W., & Harmsen, R. (2016) · Review of Educational Research · doi:10.3102/0034654315627366 · Cited by 967
Weisz, J. R., Kuppens, S., Ng, M. Y., Eckshtain, D., Ugueto, A. M., Vaughn-Coaxum, R., Jensen-Doss, A., Hawley, K. M., Krumholz Marchette, L. S., Chu, B. C., et al. (2017) · American Psychologist · doi:10.1037/a0040360 · Cited by 862
“The risk that studies with null or negative findings were less likely to be published (Begg, 1994; Lipsey & Wilson, 2000; Rosenthal, 1979) was addressed in three ways.”
BackgroundMethodInfluential
Metadata
- Authors
- Rosenthal, R.
- Affiliations
- Harvard University (US)
- Corresponding author
- Rosenthal, Robert
- Venue
- Psychological Bulletin
- Published
- 1979-05
- Volume
- 86
- Issue
- 3
- Pages
- 638-641
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2006546769
- Cited by
- 9,068
- References
- 16
- Citations by year
- 2026: 421 · 2025: 521 · 2024: 509 · 2023: 526 · 2022: 525 · 2021: 536 · 2020: 612 · 2019: 607 · 2018: 492 · 2017: 436 · 2016: 417 · 2015: 392 · 2014: 337 · 2013: 289 · 2012: 252
- FWCI
- 26.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- file drawer problem, negative results, publication bias, non-significant results, type I error, selective reporting, behavioral science, evidence synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
