The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
The file drawer problem and tolerance for null results.
Abstract
For any given research area, one cannot tell how many studies have been con-ducted but never reported. The extreme view of the "file drawer problem " is that journals are filled with the 5 % of the studies that show Type I errors, while the file drawers are filled with the 95 % of the studies that show non-significant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed. Both behavioral researchers and statisti-cians have long suspected that the studies published in the behavioral sciences are a biased sample of the studies that are actually carried out (Bakan, 1967; McNemar, 1960; Smart, 1964; Sterling, 1959). The extreme view of this problem, the "file drawer prob-lem, " is that the journals are filled with the 5 % of the studies that show Type I errors, while the file drawers back at the lab are filled with the 95 % of the studies that show nonsignificant (e.g., p>.05) results. In the past there was very little one could do to assess the net effect of studies, tucked away in file drawers, that did not make the magic.05 level (Rosenthal & Gaito, 1963, 1964). Now, however, although no definitive solution to the problem is available, one can establish reasonable boundaries on the prob-lem and estimate the degree of damage to any research conclusion that could be done by the file drawer problem. This advance in our ability to cope with the file drawer is an outgrowth of the in-creasing interest of behavioral scientists in summarizing bodies of research literature sys-Preparation of this article was supported in part by the National Science Foundation. I would like to thank Judith A. Hall and Donald B. Rubin for their valuable improvements of an earlier version of this article. Requests for reprints should be sent to Robert
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We then calculated Rosenthal’s fail-safe N (Rosenthal, 1979), and determined that 2,153,789 effects evidencing no relationship between SA and psychopathology (g = 0) would be needed to reduce the significance level of the summary effect size to .05.”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“The file drawer effect is a perennial problem in the academic literature (Rosenthal, 1979)”
From Sexual assault victimization and psychopathology: A review and meta-analysis (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“First, we calculated the fail-safe N [177] to be 4,274, which is the theoretical number of unpublished studies with effect sizes averaging zero (no effect) that would be needed to render negligible the omnibus results.”
From Social Relationships and Mortality Risk: A Meta-analytic Review (2010) · cites this paper as [177]
In the citing paper
“As we did not include unpublished work in our meta-analyses we tested for publication bias in the following ways.Firstly, funnel plots were examined.demonstrating no systematic publication biases (tables available from first author on request).Secondly..Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Fail-safe N (Rosenthal, 1979) values ranged from k=2048 to k=6708 studies for the primary study aims, suggesting absence of publication bias.”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Presence of publication bias was assessed with funnel plots, 22 Rosenthal's fail-safe N , 23 and Egger's regression intercept. 24”
From Light Therapy for Patients With Bipolar Depression: Systematic Review and Meta-Analysis of Randomized Controlled Trials (2020) · cites this paper as [23]
In the citing paper
“In terms of study quality, there was some indication of publication bias whereby some associations between racism and health outcomes were stronger in published versus unpublished studies, consistent with the tendency for ‘null’ studies to remain unpublished (i.e., the ‘file drawer’ problem [418]).”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as [418]
In the citing paper
“An alternative, perhaps more plausible, explanation is that there could be a significant amount of unpublished negative or inconclusive results, a phenomenon referred to as the “file-drawer effect” or publication bias (23).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [23]
In the citing paper
“If either of the two last hypotheses is true, there should exist many unpublished studies with null results lying in laboratories' drawers (32).”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Even if the first REM episode occurs about 70 minutes after falling asleep (33), some subjects may present a REM EEG and occulomotor pattern a way farster.The REM sleep can actually occur pretty rapidly in some individuals according to the MD (sleep specialist) who co-supervised this study.Furthermore, the MSLT has specialy been built for this puporse (see32)”
From Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory (2016) · cites this paper as [32]
In the citing paper
“Publication bias was evaluated using Rosenthal's file drawer analysis (Rosenthal, 1979) and Egger's linear regression (Card, 2012), and applying ANOVA to compare the mean effect size of the published versus unpublished studies.”
From Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal (1979) suggested a fail-safe N value above 5; k + 10 reflects results that are tolerant to contradicting studies”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“We applied random-effects models to fit the relations between CE and the criterion variables, following the guidelines proposed by Rosenthal (1979), Hunter and Schmidt (2004) and Cumming (2013).”
From Emotional processes, collective behavior, and social movements: A meta-analytic review of collective effervescence outcomes during collective gatherings and demonstrations (2022) · cites this paper as Rosenthal (1979)
In the citing paper
“The fail-safe value for this meta-analysis was computed to be 3001, which is more than 5(36) + 10.”
From Effects of digital game-based STEM education on students’ learning achievement: a meta-analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, we suggest our results may look like a lot of studies that ended up in the so-called “file drawer” [57].”
From Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory (2023) · cites this paper as [57]
In the citing paper
“Finally, publication bias was addressed by computing the fail-safe N for the correlation coefficients (Hunter & Schmidt, 2004;Rosenthal, 1979), which reports the number of field effect sizes or the tolerance for future null results, to check whether a publication bias affected our results.”
From Drivers of Consumer Adoption of e-Commerce: a Meta-Analysis (2022) · cites this paper as Rosenthal, 1979
In the citing paper
“Using Rosenthal’s (1979) fail-safe N criterion that the value should be over five times greater than the number of studies included in the meta-analysis, the fail-safe N for all outcome groups were at least eight times larger than the criterion value, suggesting that the effect sizes are unlikely to be an artifact of bias.”
From Racism as a Determinant of Health: A Systematic Review and Meta-Analysis (2015) · cites this paper as Rosenthal (1979)
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Rosenthal, 1979
In the citing paper
“Moreover, Rosenthal (1979) opined that a Fail-safe N exceeding the number of studies by fivefold would provide strong support for the robustness of the significant effects.”
From Extending the Theory of Planned Behavior in the Social Commerce Context: A Meta-Analytic SEM (MASEM) Approach (2023) · cites this paper as Rosenthal (1979)
In the citing paper
“The trim-and-fill method (Duval & Tweedie, 2000) tends to adjust too little when bias is strong (Carter et al., 2019), and Fail-Safe N (Rosenthal, 1979) neither tests nor adjusts for the presence of bias.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Rosenthal, 1979
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, a phenomenon known as ' publication bias ' (Rosenthal 1979).”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“However, a common problem is that results may not have been reported in dissertations or research reports, or published in scienti fi c (peer-reviewed) journals, due to non-signi fi cant or unfavorable fi ndings and, therefore, are dif fi cult to locate.”
From The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies (2019) · cites this paper as Rosenthal 1979
In the citing paper
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [21]
In the citing paper
“The traditional fail-safe value is 1904, which is higher than the threshold value of 100, i.e., [5(18) + 10], indicating that publication bias is insignificant, and the study's results are reliable.”
From A meta-analysis to explore the role of design thinking in enhancing creativity as learning outcomes in STEM education (2026) · cites this paper as Rosenthal, 1979
In the citing paper
“The Q statistic was used to assess heterogeneity30, and publication bias was estimated using Rosenthal’s fail-safe number with α = 0.0531.”
From The limited prosocial effects of meditation: A systematic review and meta-analysis (2018) · cites this paper as [31]
In the citing paper
“Rosenthal's fail-safe number (NR) was calculated, estimating the number of unpublished studies with non-significant results needed to bring an observed result to non-significance (NR ⩾ 5k + 10 to rule out a file drawer problem) (Rosenthal, 1979).”
From Efficacy of non-invasive brain stimulation on cognitive functioning in brain disorders: a meta-analysis (2020) · cites this paper as Rosenthal, 1979
In the citing paper
“In case of statistically significant effect sizes, it was planned to examine the robustness of findings by calculating the Failsafe N (number of non‐significant studies that would bring the p‐value to non‐significant (p > 0.05)) 40.”
From Cognitive behavioral therapies for informal caregivers of patients with cancer and cancer survivors: a systematic review and meta-analysis (2017) · cites this paper as [40]
In the citing paper
“With negative results buried in file drawers (55), conclusions drawn from the published record can be misleading, and 'false facts' can become canonized in the literature (56).”
From Misinformation in and about science (2021) · cites this paper as [55]
In the citing paper
“the fail-safe N (i.e. the file drawer statistic) was used to determine how many additional studies would be necessary to turn a significant ES non-significant (32).”
From Peripheral cytokine and chemokine alterations in depression: a meta‐analysis of 82 studies (2017) · cites this paper as [32]
In the citing paper
“File drawer analysis (Rosenthal, 1979) revealed that 1919 unpublished, filed, or unretrieved studies averaging null results would be required to bring the average unweighted effect size to nonsignificance.”
From Happy to help? A systematic review and meta-analysis of the effects of performing acts of kindness on the well-being of the actor (2018) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's (1979) fail-safe N analysis reveals 186 unpublished papers (i.e., null effect) are needed to be included in the meta-analysis in order to invalidate the population effect size of small to medium (i.e., .235)found in this review.”
From A meta-analytic review of the efficacy of physical exercise interventions on cognition in individuals with autism spectrum disorder and ADHD (2016) · cites this paper as Rosenthal's (1979)
In the citing paper
“Not including these studies may lead to an overestimation of the true effect size, the so called ' publication bias ' (Rosenthal, 1979).”
From The effects of physical activity interventions on psychosocial outcomes in adolescents: a meta-analytic review (2016) · cites this paper as Rosenthal, 1979
In the citing paper
“Rosenthal's fail-safe N 37 calculated 115 missing studies would be required to increase the two-tailed p-value to above .05.”
From Anxiety in Informal Dementia Carers: A Meta-Analysis of Prevalence (2020) · cites this paper as [37]
In the citing paper
“Publication bias was investigated with Begg's rank correlation test for funnel plot asymmetry (Begg and Mazumdar, 1994) and where significant effects were present Rosenthal's file drawer analysis was also conducted (Rosenthal, 1979).”
From Meta-analysis of the effects of intranasal oxytocin on interpretation and expression of emotions (2017) · cites this paper as Rosenthal, 1979
In the citing paper
“Over time, this practice artificially inflates Type I error and may make it appear that relationships exist when in reality they do not (Rosenthal et al., 1979).”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“However, a potential problem with meta-analysis is the file drawer problem, where studies are “filed away” unpublished if they fail to find statistically significant relationships.”
From Psychosocial functioning and the cortisol awakening response: meta-analysis, P-curve analysis, and evaluation of the evidential value in existing studies (2017) · cites this paper as Rosenthal et al., 1979
In the citing paper
“Meta-analyses are necessarily based on the available literature and hence may be affected by the potential publication-bias disfavoring null results (Rosenthal 1979).”
From Parsing the neural correlates of moral cognition: ALE meta-analysis on morality, theory of mind, and empathy (2012) · cites this paper as Rosenthal 1979
In the citing paper
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
From The Extent and Consequences of P-Hacking in Science (2015) · cites this paper as [7]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G. E., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,518
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718 · cited 1×
“Perhaps the simplest, most useful way of computing the overall p of a, set of research studies is the method of adding Zs (Cochran, 1954; Mosteller & Bush, 1954; Rosenthal, 1978).”
MethodMeta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154
Experimenter Effects in Behavioral Research
McGuigan, F. J., & Rosenthal, R. (1968) · The American Journal of Psychology · doi:10.2307/1420816 · Cited by 1,633
Self-citationArtifact in behavioral research
Rosenthal, R., & Rosnow, R. L. (1969) · Medical Entomology and Zoology · Cited by 1,186
Self-citationSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964
Combining results of independent studies.
Rosenthal, R. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.1.185 · Cited by 910
Self-citationGender effects in decoding nonverbal cues.
Hall, J. A. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.4.845 · Cited by 766
Wortman, C. B., Rosenthal, R., & Rosnow, R. L. (1976) · The American Journal of Psychology · doi:10.2307/1421418 · Cited by 428
Self-citation
Cited by
Sorted by citations
Items 401–500 of 1,374
A meta-analysis of mobile commerce adoption and the moderating effect of culture
Zhang, L., Zhu, J., & Liu, Q. (2012) · Computers in Human Behavior · doi:10.1016/j.chb.2012.05.008 · Cited by 541
Coskunpinar, A., Dir, A. L., & Cyders, M. A. (2013) · Alcoholism: Clinical and Experimental Research · doi:10.1111/acer.12131 · Cited by 541
Reporting guidelines for human microbiome research: the STORMS checklist
Mirzayi, C. A., Renson, A., Massive Analysis and Quality Control Society, Furlanello, C., Sansone, S., Zohra, F., Elsafoury, S., Geistlinger, L., Kasselman, L. J., Eckenrode, K. B., et al. (2021) · Nature Medicine · doi:10.1038/s41591-021-01552-x · Cited by 587
“Journals are increasingly implementing reproducible research standards that include the publishing of data and code, and those guidelines should be followed when possible 39,40 .”
BackgroundThompson, B. (2002) · Educational Researcher · doi:10.3102/0013189x031003025 · Cited by 546
How to Estimate and Interpret Various Effect Sizes.
Vacha‐Haase, T., & Thompson, B. (2004) · Journal of Counseling Psychology · doi:10.1037/0022-0167.51.4.473 · Cited by 556
Ferguson, C. J. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615592234 · Cited by 524
“It has been known for many years (e.g., Rosenthal, 1979) that the selective publication of statistically significant reports can bias research fields and meta-analyses drawn from them.”
BackgroundA Review and Evaluation of Meta-Analysis Practices in Management Research
Geyskens, I., Krishnan, R., Steenkamp, J. E. M., & Cunha, P. V. (2008) · Journal of Management · doi:10.1177/0149206308328501 · Cited by 547
The Adult Attachment Interview and self-reports of attachment style: An empirical rapprochement.
Roisman, G. I., Holland, A. S., Fortuna, K., Fraley, R. C., Clausell, E., & Clarke, A. (2007) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.92.4.678 · Cited by 575
Estimating vertical spillovers from FDI: Why results vary and what the true effect is
Havránek, T., & Iršová, Z. (2011) · Journal of International Economics · doi:10.1016/j.jinteco.2011.07.004 · Cited by 537
Sheard, T., & Maguire, P. (1999) · British Journal of Cancer · doi:10.1038/sj.bjc.6690596 · Cited by 533
Great expectations: a meta-analytic examination of optimism and hope.
Alarcon, G. M., Bowling, N. A., & Khazon, S. (2013) · Personality and Individual Differences · doi:10.1016/j.paid.2012.12.004 · Cited by 531
Dawel, A., O’Kearney, R., McKone, E., & Palermo, R. (2012) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2012.08.006 · Cited by 525
Predicting the Big 5 personality traits from digital footprints on social media: A meta-analysis
Azucar, D., Marengo, D., & Settanni, M. (2017) · Personality and Individual Differences · doi:10.1016/j.paid.2017.12.018 · Cited by 572
“For each trait, the fail-safe N value was higher than 90 (Openness: N = 12,210; Conscientiousness: N = 7688; Extraversion: N = 11,933; Agreeableness: N = 6053; Neuroticism: N = 7197), corresponding to the recommended rule-of-thumb limit of 5 k + 10 (Rosenthal, 1979).”
Rumor psychology: Social and organizational approaches.
DiFonzo, N., & Bordia, P. (2007) · American Psychological Association eBooks · doi:10.1037/11503-000 · Cited by 535
Eagly, A. H., & Carli, L. L. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.90.1.1 · Cited by 541
Exercise and body image: A meta-analysis
Hausenblas, H. A., & Fallon, E. A. (2005) · Psychology and Health · doi:10.1080/14768320500105270 · Cited by 582
Grégoire, G., Derderian, F., & LeLorier, J. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)00098-b · Cited by 521
A Meta-Analysis of Cohesion and Performance
Gully, S. M., Devine, D. J., & Whitney, D. J. (1995) · Small Group Research · doi:10.1177/1046496495264003 · Cited by 527
Slack resources and firm performance: a meta-analysis
Daniel, F., Lohrke, F. T., Fornaciari, C. J., & Turner, R. A. (2002) · Journal of Business Research · doi:10.1016/s0148-2963(02)00439-3 · Cited by 526
Does Strategic Planning Improve Organizational Performance? A Meta‐Analysis
George, B., Walker, R. M., & Monster, J. (2019) · Public Administration Review · doi:10.1111/puar.13104 · Cited by 526
“Publication bias implies that more significant research findings are published than null results (Rosenthal 1979).”
BackgroundSocial facilitation: A meta-analysis of 241 studies.
Bond, C. F., & Titus, L. J. (1983) · Psychological Bulletin · doi:10.1037/0033-2909.94.2.265 · Cited by 684
Stability of self-esteem across the life span.
Trzesniewski, K. H., Donnellan, M. B., & Robins, R. W. (2003) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.84.1.205 · Cited by 807
Invasive mammal eradication on islands results in substantial conservation gains
Jones, H. P., Holmes, N. D., Butchart, S. H. M., Tershy, B. R., Kappes, P. J., Corkery, I., Aguirre-Muñoz, A., Armstrong, D. P., Bonnaud, E., Burbidge, A. A., et al. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521179113 · Cited by 517
“We did not find any information when searching for neutral responses by adding OR neutral OR “neutral response” OR “no response” to the above search string; that result is unsurprising because the lack of reporting nonsignificant results is well documented in the literature (43, 44).”
Medical therapy to facilitate urinary stone passage: a meta-analysis
Hollingsworth, J. M., Rogers, M. A. M., Kaufman, S. R., Bradford, T. J., Saint, S. K., Wei, J. T., & Hollenbeck, B. K. (2006) · The Lancet · doi:10.1016/s0140-6736(06)69474-9 · Cited by 517
System Usability Scale Benchmarking for Digital Health Apps: Meta-analysis
Hyzy, M., Bond, R., Mulvenna, M., Bai, L., Dix, A., Leigh, S., & Hunt, S. (2022) · JMIR mHealth and uHealth · doi:10.2196/37290 · Cited by 510
“This is related to the file drawer effect [11], where researchers withhold studies that show nonsignificant or negative results (P>.”
BackgroundThe biological impacts of artificial light at night: the research challenge
Gaston, K. J., Visser, M. E., & Hölker, F. (2015) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2014.0133 · Cited by 516
“There is almost a complete lack of published examples in which no effect was documented (but see [35]), suggesting either that biological impacts are quite pervasive or the potential for a severe ‘file drawer0 problem (see [65]) in the literature.”
BackgroundAn Assessment of Resource-Based Theorizing on Firm Growth and Suggestions for the Future
Nason, R. S., & Wiklund, J. (2015) · Journal of Management · doi:10.1177/0149206315610635 · Cited by 516
Siegel, E. H., Sands, M. K., Van den Noortgate, W., Condon, P., Chang, Y., Dy, J., Quigley, K. S., & Barrett, L. F. (2018) · Psychological Bulletin · doi:10.1037/bul0000128 · Cited by 515
“Publication bias occurs when only studies reporting results at conventional levels of statistically significant are published, while others are not (Rosenthal, 1979).”
BackgroundAlvares, G. A., Quintana, D. S., Hickie, I. B., & Guastella, A. J. (2016) · Journal of Psychiatry and Neuroscience · doi:10.1503/jpn.140217 · Cited by 503
Gómez‐Aparicio, L. (2009) · Journal of Ecology · doi:10.1111/j.1365-2745.2009.01573.x · Cited by 513
Depression, anxiety, and resting frontal EEG asymmetry: A meta-analytic review.
Thibodeau, R., Jorgensen, R. S., & Kim, S. (2006) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.115.4.715 · Cited by 620
Video Games in Health Care: Closing the Gap
Kato, P. M. (2010) · Review of General Psychology · doi:10.1037/a0019441 · Cited by 502
Zvereva, E. L., & Kozlov, M. V. (2005) · Global Change Biology · doi:10.1111/j.1365-2486.2005.01086.x · Cited by 510
A meta-analysis of the efficacy of teaching mathematics with concrete manipulatives.
Carbonneau, K. J., Marley, S. C., & Selig, J. P. (2012) · Journal of Educational Psychology · doi:10.1037/a0031084 · Cited by 508
Corcoran, R. P., Cheung, A., Kim, E. Y., & Xie, C. (2017) · Educational Research Review · doi:10.1016/j.edurev.2017.12.001 · Cited by 643
Long‐term declines and recovery of meadow area across the world’s seagrass bioregions
Dunic, J. C., Brown, C. J., Connolly, R. M., Turschwell, M. P., & Côté, I. M. (2021) · Global Change Biology · doi:10.1111/gcb.15684 · Cited by 522
“…the effects of specific disturbances or management actions; meadows could have disappeared before monitoring started; or published data can be subject to the ‘file- drawer’ problem (Csada et al., 1996; Rosenthal, 1979), meaning that sites with no change could be under- represented in our dataset.”
BackgroundDoes Burnout Affect Academic Achievement? A Meta-Analysis of over 100,000 Students
Madigan, D. J., & Curran, T. (2020) · Educational Psychology Review · doi:10.1007/s10648-020-09533-1 · Cited by 681
“To do so, the fail-safe N statistic was calculated to estimate the number of unpublished studies with null findings that would be necessary to reduce the effect size to zero (Rosenthal, 1979).”
MethodScheper, J., Holzschuh, A., Kuussaari, M., Potts, S. G., Rundlöf, M., Smith, H. G., & Kleijn, D. (2013) · Ecology Letters · doi:10.1111/ele.12128 · Cited by 502
Strauss, C., Cavanagh, K., Oliver, A., & Pettman, D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0096110 · Cited by 495
Blackhart, G. C., Nelson, B. C., Knowles, M. L., & Baumeister, R. F. (2009) · Personality and Social Psychology Review · doi:10.1177/1088868309346065 · Cited by 504
Kane, P., & Garber, J. (2004) · Clinical Psychology Review · doi:10.1016/j.cpr.2004.03.004 · Cited by 498
Chen, C.-H., & Yang, Y.-C. (2019) · Educational Research Review · doi:10.1016/j.edurev.2018.11.001 · Cited by 762
BERGMANN'S RULE IN NONAVIAN REPTILES: TURTLES FOLLOW IT, LIZARDS AND SNAKES REVERSE IT
Ashton, K. G., & Feldman, C. R. (2003) · Evolution · doi:10.1111/j.0014-3820.2003.tb00324.x · Cited by 503
Organizational Tenure and Job Performance
Ng, T. W. H., & Feldman, D. C. (2010) · Journal of Management · doi:10.1177/0149206309359809 · Cited by 536
Parsons, T. D., Rogers, S. A., Braaten, A., Woods, S. P., & Tröster, A. I. (2006) · The Lancet Neurology · doi:10.1016/s1474-4422(06)70475-6 · Cited by 500
Sharma, R., Yetton, P., & Crawford, J. (2009) · MIS Quarterly · doi:10.2307/20650305 · Cited by 496
Social cognitive theory and physical activity: a systematic review and meta‐analysis
Young, M. D., Plotnikoff, R. C., Collins, C. E., Callister, R., & Morgan, P. J. (2014) · Obesity Reviews · doi:10.1111/obr.12225 · Cited by 505
A meta-analysis of humor in advertising
Eisend, M. (2008) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-008-0096-y · Cited by 498
It's time: A meta-analysis on the self-control-deviance link
Vazsonyi, A. T., Mikuška, J., & Kelley, E. L. (2017) · Journal of Criminal Justice · doi:10.1016/j.jcrimjus.2016.10.001 · Cited by 504
Jonasson, Z. (2005) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2004.10.006 · Cited by 498
Weigel, F. K., & Hazen, B. T. (2013) · Advances in healthcare information systems and administration book series · doi:10.4018/978-1-4666-4619-3.ch008 · Cited by 497
Subjective career success: A meta-analytic review
Ng, T. W. H., & Feldman, D. C. (2014) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2014.06.001 · Cited by 509
N170 sensitivity to facial expression: A meta-analysis
Hinojosa, J. A., Mercado, F., & Carretié, L. (2015) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2015.06.002 · Cited by 494
Stewart, R. E., & Chambless, D. L. (2009) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0016032 · Cited by 483
Cheng, L., Ritzhaupt, A. D., & Antonenko, P. (2019) · Educational Technology Research and Development · doi:10.1007/s11423-018-9633-7 · Cited by 494
Ferguson, C. J., & Brannick, M. Τ. (2011) · Psychological Methods · doi:10.1037/a0024445 · Cited by 512
Lawson, G. M., Hook, C. J., & Farah, M. J. (2018) · Developmental Science · doi:10.1111/desc.12529 · Cited by 556
Effects of Interventions to Promote Self-Determination for Individuals With Disabilities
Algozzine, B., Browder, D. M., Karvonen, M., Test, D. W., & Wood, W. M. (2001) · Review of Educational Research · doi:10.3102/00346543071002219 · Cited by 489
Publication Bias in Research Synthesis: Sensitivity Analysis Using A Priori Weight Functions.
Vevea, J. L., & Woods, C. M. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.428 · Cited by 461
Rehse, B., & Pukrop, R. (2003) · Patient Education and Counseling · doi:10.1016/s0738-3991(02)00149-0 · Cited by 486
Interventions to Increase Physical Activity Among Healthy Adults: Meta-Analysis of Outcomes
Conn, V. S., Hafdahl, A. R., & Mehr, D. R. (2011) · American Journal of Public Health · doi:10.2105/ajph.2010.194381 · Cited by 486
Meta-analysis of the impact of Augmented Reality on students’ learning gains
Garzón, J., & Acevedo, J. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.04.001 · Cited by 491
Love, R., Adams, J., & van Sluijs, E. M. F. (2019) · Obesity Reviews · doi:10.1111/obr.12823 · Cited by 452
“Calculation of Rosenthal's failsafe number, representing the number of studies that would be required to refute the main effect meta‐analytic conclusion, indicates low potential for biased conclusions.(47) We estimate that at least 20 further trials, all with significant and positive intervention effects, would be nee…”
BackgroundA review of the relationships between human population density and biodiversity
Luck, G. (2007) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/j.1469-185x.2007.00028.x · Cited by 498
Harvey, B. P., Gwynn‐Jones, D., & Moore, P. J. (2013) · Ecology and Evolution · doi:10.1002/ece3.516 · Cited by 515
Three ways to make replication mainstream
Gantman, A. P., Gomila, R., Martinez, J. E., Matias, J., Paluck, E., Starck, J. G., Wu, S., & Yaffe, N. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17001972 · Cited by 302
Hispanic Mortality Paradox: A Systematic Review and Meta-Analysis of the Longitudinal Literature
Ruiz, J. M., Steffen, P. R., & Smith, T. B. (2013) · American Journal of Public Health · doi:10.2105/ajph.2012.301103 · Cited by 477
Success Factors of Product Innovation: An Updated Meta‐Analysis
Evanschitzky, H., Eisend, M., Calantone, R. J., & Jiang, Y. (2012) · Journal of Product Innovation Management · doi:10.1111/j.1540-5885.2012.00964.x · Cited by 479
Kliem, S., Kröger, C., & Kosfelder, J. (2010) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0021015 · Cited by 474
Sex differences in the ability to recognise non-verbal displays of emotion: A meta-analysis
Thompson, A. E., & Voyer, D. (2014) · Cognition and Emotion · doi:10.1080/02699931.2013.875889 · Cited by 476
A Reevaluation of Risk of In Utero Exposure to Lithium
Cohen, L. S. (1994) · JAMA · doi:10.1001/jama.1994.03510260078033 · Cited by 475
Theory of mind and prosocial behavior in childhood: A meta-analytic review.
Imuta, K., Henry, J. D., Slaughter, V., Selcuk, B., & Ruffman, T. (2016) · Developmental Psychology · doi:10.1037/dev0000140 · Cited by 474
“Particularly problematic is “the file drawer problem” (Rosenthal, 1979), which refers to the fact that significant results are more likely to be published than nonsignificant results (Easterbrook, Gopalan, Berlin, & Matthews, 1991; Ferguson & Heene, 2012; Sterling, 1959).”
BackgroundResultA Meta-Analysis of Organizational Complaint Handling and Customer Responses
Gelbrich, K., & Roschk, H. (2011) · Journal of Service Research · doi:10.1177/1094670510387914 · Cited by 481
HOW BROADLY DOES EDUCATION CONTRIBUTE TO JOB PERFORMANCE?
Ng, T. W. H., & Feldman, D. C. (2009) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.01130.x · Cited by 516
Bañares, R., Albillos, A., Rincón, D., Alonso, S., González, M., Ruíz-del-Árbol, L., Salcedo, M., & Molinero, L.-M. (2002) · Hepatology · doi:10.1053/jhep.2002.31354 · Cited by 470
Maisel, N. C., Blodgett, J. C., Wilbourne, P. L., Humphreys, K., & Finney, J. W. (2013) · Addiction · doi:10.1111/j.1360-0443.2012.04054.x · Cited by 470
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471
Prevention of NSAID-induced gastroduodenal ulcers
Rostom, A., Dube, C., Wells, G. A., Tugwell, P., Welch, V. A., Jolicoeur, E., McGowan, J., Lanas, Á., Rushton, E., & Shukla, T. (2002) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002296 · Cited by 470
Treatment of functional gastrointestinal disorders with antidepressant medications: a meta-analysis
Jackson, J. L., O’Malley, P. G., Tomkins, G., Balden, E., Santoro, J., & Kroenke, K. (2000) · The American Journal of Medicine · doi:10.1016/s0002-9343(99)00299-5 · Cited by 466
Theobald, M. (2021) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2021.101976 · Cited by 478
Presidential Address: The Scientific Outlook in Financial Economics
Harvey, C. R. (2017) · The Journal of Finance · doi:10.1111/jofi.12530 · Cited by 466
A Meta-analysis of Attachment to Parents and Delinquency
Hoeve, M., Stams, G. J. J. M., van der Put, C. E., Dubas, J. S., van der Laan, P. H., & Gerris, J. R. M. (2012) · Journal of Abnormal Child Psychology · doi:10.1007/s10802-011-9608-1 · Cited by 466
Holzman, J. B., & Bridgett, D. J. (2017) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.12.032 · Cited by 449
Quality of life in autism across the lifespan: A meta-analysis
van Heijst, B. F., & Geurts, H. M. (2015) · Autism · doi:10.1177/1362361313517053 · Cited by 460
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453
A meta-analytic test of the imagined contact hypothesis
Miles, E., & Crisp, R. J. (2013) · Group Processes & Intergroup Relations · doi:10.1177/1368430213510573 · Cited by 526
Köhler, C. A., Freitas, T. H., Stubbs, B., Maes, M., Solmi, M., Veronese, N., de Andrade, N. Q., Morris, G., Fernandes, B. S., Brunoni, A. R., et al. (2017) · Molecular Neurobiology · doi:10.1007/s12035-017-0632-1 · Cited by 449
“to turn significant ES non-significant [28].”
Cieślak, R., Shoji, K., Douglas, A., Melville, E., Łuszczyńska, A., & Benight, C. C. (2013) · Psychological Services · doi:10.1037/a0033798 · Cited by 574
Meta-analysis of Observational Studies in Epidemiology
Stroup, D. F., Berlin, J. A., Morton, S. C., Olkin, I., Williamson, G. D., Rennie, D., Moher, D., Becker, B. J., Sipe, T. A., & Thacker, S. B. (2000) · Cited by 460
Beugelsdijk, S., Kostova, T., Kunst, V. E., Spadafora, E., & van Essen, M. (2017) · Journal of Management · doi:10.1177/0149206317729027 · Cited by 469
“…subsidiary and MNC. (2) In order to test for the moderating effect of methodological artifacts, we controlled, first, for the “file drawer problem” (Meyer, van Witteloostuijn, & Beugelsdijk, 2017; Rosenthal, 1979), by including a dummy variable denoting whether a study was published (1) or not (0).”
MethodAnger, hostility, and posttraumatic stress disorder in trauma-exposed adults: A meta-analysis.
Orth, U., & Wieland, E. (2006) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.74.4.698 · Cited by 459
Insect Performance on Experimentally Stressed Woody Plants: A Meta-Analysis
Koricheva, J., Larsson, S., & Haukioja, E. (1998) · Annual Review of Entomology · doi:10.1146/annurev.ento.43.1.195 · Cited by 459
Meta-analysis of functional neuroimaging data: current and future directions
Wager, T. D., Lindquist, M., & Kaplan, L. (2007) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsm015 · Cited by 434
Monitoring Change in Vertebrate Abundance: the Living Planet Index
COLLEN, B., LOH, J., WHITMEE, S., McRAE, L., AMIN, R., & BAILLIE, J. E. M. (2009) · Conservation Biology · doi:10.1111/j.1523-1739.2008.01117.x · Cited by 441
Reshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan*
Casey, K., Glennerster, R., & Miguel, E. A. (2012) · The Quarterly Journal of Economics · doi:10.1093/qje/qje027 · Cited by 466
Methods for testing publication bias in ecological and evolutionary meta‐analyses
Nakagawa, S., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Sánchez‐Tójar, A., Yang, Y., & O'Dea, R. E. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13724 · Cited by 399
“Fail-safe N (also known as the ‘file-drawer number’) represents the number of non-302 significant unpublished results needed to exist to make the overall effect non-significant (e.g., Rosenthal, 1979; Rosenberg, 2005) or negligible in magnitude (e.g., Owrin, 1983).”
InfluentialAdriaanse, M. A., Vinkers, C. D. W., De Ridder, D. T. D., Hox, J. J., & De Wit, J. B. F. (2011) · Appetite · doi:10.1016/j.appet.2010.10.012 · Cited by 455
A meta-analysis of realistic job preview experiments.
Premack, S. L., & Wanous, J. P. (1985) · Journal of Applied Psychology · doi:10.1037/0021-9010.70.4.706 · Cited by 455
The Mediating Role of Entrepreneurial Orientation in the Task Environment–Performance Relationship
Rosenbusch, N., Rauch, A., & Bausch, A. (2011) · Journal of Management · doi:10.1177/0149206311425612 · Cited by 485
Metadata
- Authors
- Rosenthal, R.
- Affiliations
- Harvard University (US)
- Corresponding author
- Rosenthal, Robert
- Venue
- Psychological Bulletin
- Published
- 1979-05
- Volume
- 86
- Issue
- 3
- Pages
- 638-641
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2006546769
- Cited by
- 9,068
- References
- 16
- Citations by year
- 2026: 421 · 2025: 521 · 2024: 509 · 2023: 526 · 2022: 525 · 2021: 536 · 2020: 612 · 2019: 607 · 2018: 492 · 2017: 436 · 2016: 417 · 2015: 392 · 2014: 337 · 2013: 289 · 2012: 252
- FWCI
- 26.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- file drawer problem, negative results, publication bias, non-significant results, type I error, selective reporting, behavioral science, evidence synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
