A Fail-Safe N for Effect Size in Meta-Analysis
Abstract
Rosenthan's (1979) concept of fail-safe N has thus far been applied to probability levels exclusively. This note introduces a fail-safe TV for effect size. Rosenthal's (1979) fail-safe N was an ingenious response to the so-called file drawer problem in research integration, in that it provided a direct assessment of the threat posed by sampling bias in the literature search. The algorithm was derived from the Stouffer method (Mosteller & Bush, 1954), in which an overall Z-score is computed by summing individual Z-scores and dividing by the square root of the number of scores. A significant overall Z-score, Rosenthal reasoned, could be made nonsignificant by adding some knowable number of hypothetical studies that averaged null results. The exact number of such studies needed to bring a significant overall p level up to some critical level (e.g., .05) has since become known as the fail-safe N (Nfs). The formula is
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A fail-safe N calculation showed that 59 null-result studies would have to be added before the meta-analytic result became nonsignificant.
“we also calculated the classic fail-safe N test (Orwin, 1983), which revealed that it would take 59 studies reporting null results (g = 0) to be included in our analyses in order for our findings to lose statistical significance.”
From Improving fluid intelligence with training on working memory: a meta-analysis (2015) · cites this paper as Orwin, 1983
Publication bias was examined by creating a funnel plot and calculating the fail safe N.
“The file drawer problem (publication bias) was examined by creating a funnel plot and calculating the fail safe N (45).”
From A Meta-Analysis of the Corpus Callosum in Autism (2009) · cites this paper as [45]
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,406 · see citing works
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · see citing works
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695 · see citing works
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · see citing works
Smith, M. L., Glass, G. V., & Miller, T. I. (1980) · Medical Entomology and Zoology · Cited by 2,123 · see citing works
Johnson, D. W., Maruyama, G., Johnson, R., Nelson, D., & Skon, L. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.89.1.47 · Cited by 1,588 · see citing works
Interpersonal expectancy effects: the first 345 studies
Rosenthal, R., & Rubin, D. B. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00075506 · Cited by 1,143 · see citing works
Eagly, A. H., & Carli, L. L. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.90.1.1 · Cited by 525 · see citing works
Cooper, H. M. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.1.131 · Cited by 413 · see citing works
Arkin, R. M., Cooper, H., & Kolditz, T. A. (1980) · Journal of Personality · doi:10.1111/j.1467-6494.1980.tb02378.x · Cited by 87 · see citing works
Flagged first, then by citations
1 of the 11 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Emotion-regulation strategies across psychopathology: A meta-analytic review
Aldao, A., Nolen-Hoeksema, S., & Schweizer, S. (2010) · Clinical Psychology Review · doi:10.1016/j.cpr.2009.11.004 · Cited by 6,686 · see citing works
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · see citing works
Neurocognitive deficit in schizophrenia: A quantitative review of the evidence.
Heinrichs, R. W., & Zakzanis, K. K. (1998) · Neuropsychology · doi:10.1037//0894-4105.12.3.426 · Cited by 2,791 · see citing works
van Zomeren, M., Postmes, T., & Spears, R. (2008) · Psychological Bulletin · doi:10.1037/0033-2909.134.4.504 · Cited by 2,714 · see citing works
“First, the fail-safe N (Orwin, 1983; Rosenthal, 1979) for all three predictor samples suggests that many more independent samples showing null effects are needed before mean effect sizes are reduced to small effects (r .”
BackgroundEllis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461 · see citing works
Jeffery, S., Verheijen, F. G. A., van der Velde, M., & Bastos, A. C. (2011) · Agriculture, Ecosystems & Environment · doi:10.1016/j.agee.2011.08.015 · Cited by 2,422 · see citing works
A Meta-Analysis of Interventions to Reduce Loneliness
Masi, C. M., Chen, H.-Y., Hawkley, L. C., & Cacioppo, J. T. (2011) · Personality and Social Psychology Review · doi:10.1177/1088868310377394 · Cited by 1,995 · see citing works
Positive psychology interventions: a meta-analysis of randomized controlled studies
Bolier, L., Haverman, M., Westerhof, G. J., Riper, H., Smit, F., & Bohlmeijer, E. (2013) · BMC Public Health · doi:10.1186/1471-2458-13-119 · Cited by 1,996 · see citing works
“The Orwin’s fail-safe number indicates the number of nonsignificant unpublished studies needed to reduce the overall significant effect to non-significance (according to a self-stated criterium) [48].”
Factors Influencing the Effectiveness of Relationship Marketing: A Meta-AnalysisPreprint
Palmatier, R. W., Dant, R. P., Grewal, D., & Evans, K. R. (2006) · SSRN Electronic Journal · doi:10.2139/ssrn.904647 · Cited by 1,997 · see citing works
Executive Function and the Frontal Lobes: A Meta-Analytic Review
Alvarez, J. A., & Emory, E. (2006) · Neuropsychology Review · doi:10.1007/s11065-006-9002-x · Cited by 1,981 · see citing works
“The failsafe N statistic (Orwin, 1983) indicated that 78 additional studies with null results (i.”
BackgroundResultThe malleability of spatial skills: A meta-analysis of training studies.
Uttal, D. H., Meadow, N. G., Tipton, E., Hand, L. L., Alden, A. R., Warren, C. M., & Newcombe, N. S. (2012) · Psychological Bulletin · doi:10.1037/a0028446 · Cited by 1,986 · see citing works
“We therefore calculated the fail-safe N (Orwin, 1983) to determine how many unpublished studies averaging no effect of training (g 0) would need to exist to lower our mean effect size to trivial levels.”
MethodResultExploring compassion: A meta-analysis of the association between self-compassion and psychopathology
MacBeth, A., & Gumley, A. (2012) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.06.003 · Cited by 1,838 · see citing works
“Additionally, Orwin's fail-safe N (FSN; Orwin, 1983) was calculated for the total data set to indicate the number of unpublished studies of non-significant effect size required to reduce the mean observed effect size to zero.”
MethodA meta-analytic review of benefit finding and growth.
Helgeson, V. S., Reynolds, K. A., & Tomich, P. L. (2006) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.74.5.797 · Cited by 1,864 · see citing works
“We used the formula developed by Orwin (1983) to calculate the fail-safe N.”
MethodOn the Use of Beta Coefficients in Meta-Analysis.
Peterson, R. A., & Brown, S. P. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.1.175 · Cited by 1,643 · see citing works
“This is commonly referred to as the “file drawer problem” (Orwin, 1983; Rosenthal, 1979) and is typically addressed by estimating the number of studies with an effect size of zero that would be necessary to reduce a particular relationship to nonsignificance.”
BackgroundIntergroup Threat and Outgroup Attitudes: A Meta-Analytic Review
Riek, B. M., Mania, E. W., & Gaertner, S. L. (2006) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr1004_4 · Cited by 1,588 · see citing works
Sung, Y.-T., Chang, K.-E., & Liu, T.-C. (2016) · Computers & Education · doi:10.1016/j.compedu.2015.11.008 · Cited by 1,574 · see citing works
“Moreover, the present study also adopted Orwin's fail-safe N (Orwin, 1983) to estimate the number of missing null studies that would be required to bring the mean effect size to a trivial level.”
ResultP-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · see citing works
“effect size statistically (Rosenthal, 1979) or practically (Orwin, 1983) non-significant.”
Predicting Psychosis: Meta-analysis of Transition Outcomes in Individuals at High Clinical Risk
Fusar-Poli, P. (2012) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2011.1472 · Cited by 1,361 · see citing works
Paulson, J. F., & Bazemore, S. D. (2010) · JAMA · doi:10.1001/jama.2010.605 · Cited by 1,337 · see citing works
A Meta-Analytic Review of Psychosocial Interventions for Substance Use Disorders
Dutra, L., Stathopoulou, G., Basden, S. L., Leyro, T. M., Powers, M. B., & Otto, M. W. (2008) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2007.06111851 · Cited by 1,287 · see citing works
Tang, B. M., Eslick, G. D., Nowson, C., Smith, C., & Bensoussan, A. (2007) · The Lancet · doi:10.1016/s0140-6736(07)61342-7 · Cited by 1,248 · see citing works
Gender differences in emotion expression in children: A meta-analytic review.
Chaplin, T. M., & Aldao, A. (2013) · Psychological Bulletin · doi:10.1037/a0030737 · Cited by 1,247 · see citing works
A Meta-analytic Review of Components Associated with Parent Training Program Effectiveness
Wyatt Kaminski, J., Valle, L. A., Filene, J. H., & Boyle, C. L. (2008) · Journal of Abnormal Child Psychology · doi:10.1007/s10802-007-9201-9 · Cited by 1,721 · see citing works
Firth, J., Torous, J., Nicholas, J., Carney, R., Pratap, A., Rosenbaum, S., & Sarris, J. (2017) · World Psychiatry · doi:10.1002/wps.20472 · Cited by 1,097 · see citing works
Rosenberg, M. S. (2005) · Evolution · doi:10.1111/j.0014-3820.2005.tb01004.x · Cited by 1,285 · see citing works
“Because they are directly derived from the methods used in modern meta-analysis, the calculations described above are necessarily more reasonable predictors of the fail-safe number than the original approaches of Rosenthal (1979) and Orwin (1983).”
MethodTwo Decades of Terror Management Theory: A Meta-Analysis of Mortality Salience Research
Burke, B. L., Martens, A., & Faucher, E. H. (2010) · Personality and Social Psychology Review · doi:10.1177/1088868309352321 · Cited by 1,188 · see citing works
“The fail-safe N is the number of unpublished or future studies averaging null results that would be necessary to reduce our overall ES for MS to a nonsignificant value (see Orwin, 1983; Rosenthal, 1991, p. 104), which is 4,239 for our review.”
BackgroundHorne, R., Chapman, S. C. E., Parham, R., Freemantle, N., Forbes, A., & Cooper, V. (2013) · PLoS ONE · doi:10.1371/journal.pone.0080633 · Cited by 1,174 · see citing works
“Orwin’s fail-safe N [35,36] was calculated to estimate the number of unpublished studies necessary to reverse any conclusion that a significant effect exists (based on the conservative assumption that unpublished studies would have effect sizes of equal magnitude but opposite direction to the overall effect size in th…”
BackgroundField, A. P., & Gillett, R. (2010) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711010x502733 · Cited by 1,162 · see citing works
“For example, when using d as an effect size measure, Orwin (1983) suggests a variation on the fail-safe N that estimates the number of unpublished studies required to bring the average effect size down to a predetermined value.”
MethodA systematic review and meta-analysis of the cognitive correlates of bilingualism
Adesope, O. O., Lavin, T., Thompson, T., & Ungerleider, C. (2010) · Review of Educational Research · doi:10.3102/0034654310368803 · Cited by 1,132 · see citing works
“Researchers have observed that published studies are a biased sample of studies in a particular domain because research reports are more likely to be published when significant results are reported (Orwin, 1983; Rosenthal, 1979).”
BackgroundThe Effectiveness of Creativity Training: A Quantitative Review
Scott, G., Leritz, L. E., & Mumford, M. D. (2004) · Creativity Research Journal · doi:10.1080/10400410409534549 · Cited by 1,132 · see citing works
“In fact, the fail-safe N statistic indicates that 168 null studies would be required to reduce the overall effect size below .20 (Orwin, 1983).”
BackgroundMeta-Analysis of the Relationship Between HIV Infection and Risk for Depressive Disorders
Ciesla, J. A., & Roberts, J. E. (2001) · American Journal of Psychiatry · doi:10.1176/appi.ajp.158.5.725 · Cited by 1,106 · see citing works
“A procedure has been proposed by Orwin (28) to calculate the number of null results that are necessary to reduce the average effect size to a negligible level.”
MethodMemory Impairment in Schizophrenia: A Meta-Analysis
Aleman, A., Hijman, R., de Haan, E. H. F., & Kahn, R. S. (1999) · American Journal of Psychiatry · doi:10.1176/ajp.156.9.1358 · Cited by 1,093 · see citing works
The Role Of Context In Work Team Diversity Research: A Meta-Analytic Review
Joshi, A., & Roh, H. (2009) · Academy of Management Journal · doi:10.5465/amj.2009.41331491 · Cited by 1,362 · see citing works
“The failsafe k was also calculated to identify the number of “file drawer” (unknown) studies of the same relationship with a true correlation of zero needed to widen the reported confidence interval enough to include zero (Orwin, 1983; Rosenthal, 1979).”
MethodScott, A. J., Webb, T. L., Martyn-St James, M., Rowse, G., & Weich, S. (2021) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2021.101556 · Cited by 1,094 · see citing works
Self-focused attention and negative affect: A meta-analysis.
Mor, N., & Winquist, J. (2002) · Psychological Bulletin · doi:10.1037/0033-2909.128.4.638 · Cited by 1,069 · see citing works
“To address concerns of publication bias, whereby the results of the meta-analysis may be biased because of the tendency to primarily report and publish statistically significant results, we calculated a fail-safe N (Orwin, 1983).”
ResultSanders, M. R., Kirby, J. N., Tellegen, C. L., & Day, J. J. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.04.003 · Cited by 1,077 · see citing works
Depression and Diabetes Treatment Nonadherence: A Meta-Analysis
Gonzalez, J. S., Peyrot, M., McCarl, L. A., Collins, E. M., Serpa, L., Mimiaga, M. J., & Safren, S. A. (2008) · Diabetes Care · doi:10.2337/dc08-1341 · Cited by 1,060 · see citing works
“05 level, chosen as an effect size that would be trivial (23).”
BackgroundDissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039 · see citing works
Relationships between psychological climate perceptions and work outcomes: a meta‐analytic review
Parker, C. P., Baltes, B. B., Young, S. A., Huff, J. W., Altmann, R. A., Lacost, H. A., & Roberts, J. E. (2003) · Journal of Organizational Behavior · doi:10.1002/job.198 · Cited by 1,078 · see citing works
Morgan, A. B., & Lilienfeld, S. O. (2000) · Clinical Psychology Review · doi:10.1016/s0272-7358(98)00096-8 · Cited by 1,032 · see citing works
Nguyen, A. D., & Benet‐Martínez, V. (2012) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022111435097 · Cited by 1,021 · see citing works
“…identified for this meta-analysis) with a null relationship (r = 0) between biculturalism and adjustment to decrease the observed strong mean effect size (r = .51) for the biculturalism-adjustment relationship to a weak or trivial effect size (r = .10; see Orwin, 1983, for the fail-safe N formula).”
BackgroundInstructional animation versus static pictures: A meta-analysis
Höffler, T. N., & Leutner, D. (2007) · Learning and Instruction · doi:10.1016/j.learninstruc.2007.09.013 · Cited by 992 · see citing works
Prati, G., & Pietrantoni, L. (2009) · Journal of Loss and Trauma · doi:10.1080/15325020902724271 · Cited by 978 · see citing works
Chapter 10: Addressing reporting biases
Egger, M., & Moher, D. (2011) · Cited by 977 · see citing works
Meta-Analysis of the Relationships Between Social Support and Well-Being in Children and Adolescents
Chu, P. S., Saucier, D. A., & Hafner, E. (2010) · Journal of Social and Clinical Psychology · doi:10.1521/jscp.2010.29.6.624 · Cited by 962 · see citing works
“We calculated the Fail-safe N using Orwin’s (1983) formula (dc = .”
BackgroundMethodInfluentialFrontocingulate Dysfunction in Depression: Toward Biomarkers of Treatment Response
Pizzagalli, D. A. (2011) · Neuropsychopharmacology · doi:10.1038/npp.2010.166 · Cited by 945 · see citing works
“Finally, to explore the possibility of publication bias (ie, bias toward non-publication of studies with no effects), Orwin’s FailSafe Number of studies test was performed (Orwin, 1983; Rosenthal, 1991).”
BackgroundMethodInfluentialEmotion regulation strategies in depressive and anxiety symptoms in youth: A meta-analytic review
Schäfer, J. Ö., Naumann, E., Holmes, E. A., Tuschen-Caffier, B., & Samson, A. C. (2017) · Journal of Youth and Adolescence · doi:10.1007/s10964-016-0585-0 · Cited by 941 · see citing works
The Handbook of Research Synthesis
Ziegel, E. R., Cooper, H., & Hedges, L. V. (1995) · Technometrics · doi:10.2307/1269651 · Cited by 936 · see citing works
Role of social desirability in personality testing for personnel selection: The red herring.
Öneş, D. S., Viswesvaran, C., & Reiss, A. D. (1996) · Journal of Applied Psychology · doi:10.1037/0021-9010.81.6.660 · Cited by 924 · see citing works
Koolschijn, P. C. M. P., van Haren, N. E. M., Lensvelt‐Mulders, G. J. L. M., Hulshoff Pol, H. E., & Kahn, R. S. (2009) · Human Brain Mapping · doi:10.1002/hbm.20801 · Cited by 902 · see citing works
“To examine the possibility of publication bias, we computed a fail-safe number of studies [Orwin, 1983; Rosenthal, 1991].”
MethodThoresen, C. J., Kaplan, S. A., Barsky, A. P., Warren, C. R., & de Chermont, K. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.6.914 · Cited by 919 · see citing works
Lee, S. S., Humphreys, K. L., Flory, K., Liu, R., & Glass, K. (2011) · Clinical Psychology Review · doi:10.1016/j.cpr.2011.01.006 · Cited by 875 · see citing works
A meta-analysis of structural brain abnormalities in PTSD
KARL, A., SCHAEFER, M., MALTA, L., DORFEL, D., ROHLEDER, N., & WERNER, A. (2006) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2006.03.004 · Cited by 881 · see citing works
“To address the file drawer problem, Orwin’s fail safe N (Orwin, 1983) was computed.”
MethodYang, Y., & Raine, A. (2009) · Psychiatry Research: Neuroimaging · doi:10.1016/j.pscychresns.2009.03.012 · Cited by 888 · see citing works
“, 1997) and Orwin's fail-safe N (Orwin, 1983) to evaluate whether the available literature was biased toward excluding non-significant studies.”
BackgroundDickinson, D., Ramsey, M. E., & Gold, J. M. (2007) · Archives of General Psychiatry · doi:10.1001/archpsyc.64.5.532 · Cited by 863 · see citing works
Meta-analysis of psychological interventions for chronic low back pain
Hoffman, B. M., Papas, R. K., Chatkoff, D. K., & Kerns, R. D. (2007) · Health Psychology · doi:10.1037/0278-6133.26.1.1 · Cited by 844 · see citing works
Prati, G., & Mancini, A. D. (2021) · Psychological Medicine · doi:10.1017/s0033291721000015 · Cited by 842 · see citing works
“In addition, we used a fail-safe N for effect size in meta-analysis (Orwin, 1983).”
A Meta-Analysis of Compassion-Based Interventions: Current State of Knowledge and Future Directions
Kirby, J. N., Tellegen, C. L., & Steindl, S. R. (2017) · Behavior Therapy · doi:10.1016/j.beth.2017.06.003 · Cited by 827 · see citing works
“Third, Orwin ’ s (1983) failsafe N was calculated to determine the number of studies with null results (set at d = 0) needed to reduce the effect size to the smallest meaningful effect size (chosen as d = 0.10).”
InfluentialAssociation Between Bullying and Psychosomatic Problems: A Meta-analysis
Gini, G., & Pozzoli, T. (2009) · PEDIATRICS · doi:10.1542/peds.2008-1215 · Cited by 834 · see citing works
A social comparison theory meta-analysis 60+ years on.
Gerber, J. P., Wheeler, L., & Suls, J. (2018) · Psychological Bulletin · doi:10.1037/bul0000127 · Cited by 808 · see citing works
Hadfield, J. D., & Nakagawa, S. (2010) · Journal of Evolutionary Biology · doi:10.1111/j.1420-9101.2009.01915.x · Cited by 813 · see citing works
“In addition, the fail-safe N (file-drawer number; Rosenthal, 1979) and related statistics (e.g. Orwin, 1983; Rosenberg, 2005) have often been reported as a means of assessing the validity of mean effect size estimates in meta-analysis in evolution and ecology (Moller & Jennions, 2001), although…”
Intelligent tutoring systems and learning outcomes: A meta-analysis
Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014) · Journal of Educational Psychology · doi:10.1037/a0037123 · Cited by 840 · see citing works
Frazier, T. W., Demaree, H. A., & Youngstrom, E. A. (2004) · Neuropsychology · doi:10.1037/0894-4105.18.3.543 · Cited by 792 · see citing works
“…to be a result of publication bias because 140 comparisons with null results would be needed to bring the mean effect size down to d ϭ .30, a small effect, and more than 1,500 studies with null results would be needed to reduce the mean effect size to d ϭ .05, a negligible effect (Orwin, 1983).”
BackgroundRichardson, J. C., Maeda, Y., Lv, J., & Caskurlu, S. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.02.001 · Cited by 787 · see citing works
Firth, J., Torous, J., Nicholas, J., Carney, R., Rosenbaum, S., & Sarris, J. (2017) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.04.046 · Cited by 675 · see citing works
Weinhardt, L. S., Carey, M. P., Johnson, B. T., & Bickham, N. L. (1999) · American Journal of Public Health · doi:10.2105/ajph.89.9.1397 · Cited by 771 · see citing works
Osbaldiston, R., & Schott, J. P. (2012) · Environment and Behavior · doi:10.1177/0013916511402673 · Cited by 754 · see citing works
A meta-analysis of web-delivered tailored health behavior change interventions
Lustria, M. L. A., Noar, S. M., Cortese, J., Van Stee, S. K., Glueckauf, R. L., & Lee, J. (2013) · Journal of Health Communication · doi:10.1080/10810730.2013.768727 · Cited by 689 · see citing works
School-Based Adolescent Drug Prevention Programs: 1998 Meta-Analysis
Tobler, N. S., Roona, M., Ochshorn, P., Marshall, D., Streke, A., & Stackpole, K. M. (2000) · The Journal of Primary Prevention · doi:10.1023/a:1021314704811 · Cited by 757 · see citing works
Deficits in domains of social cognition in schizophrenia: a meta-analysis of the empirical evidence
Savla, G. N., Vella, L., Armstrong, C. C., Penn, D. L., & Twamley, E. W. (2013) · Schizophrenia Bulletin · doi:10.1093/schbul/sbs080 · Cited by 736 · see citing works
Interactions between type of instruction and type of language feature: A meta-analysis
Spada, N., & Tomita, Y. (2010) · Language Learning · doi:10.1111/j.1467-9922.2010.00562.x · Cited by 746 · see citing works
“…problem” or the number of unpublished studies with a null result that would be needed to reverse the results of our meta-analysis (Hsu, 2002; Orwin, 1983).7 Rosenthal (1979, 1991) developed a formula to do this that estimates the number of filedrawer studies with an average validity…”
ResultMobile Telephone Text Messaging for Medication Adherence in Chronic Disease: A Meta-analysis
Thakkar, J., Kurup, R., Laba, T.-L., Santo, K., Thiagalingam, A., Rodgers, A., Woodward, M., Redfern, J., & Chow, C. K. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2015.7667 · Cited by 685 · see citing works
“Wald et al,41 2014 CVD n = 303 (6 mo) TM vs standard care Median (range), 60 (54-68) for IG and 41 (49-69) for CG 46 Primary care clinic/London, England Patients receiving blood pressure and lipid-lowering therapy for prevention of CVD”
BackgroundA meta‐analytic investigation of the relationship between HRM bundles and firm performance
Subramony, M. V. (2009) · Human Resource Management · doi:10.1002/hrm.20315 · Cited by 745 · see citing works
Efficacy of the third wave of behavioral therapies: A systematic review and meta- analysis
Öst, L.-G. (2008) · Behaviour Research and Therapy · doi:10.1016/j.brat.2007.12.005 · Cited by 727 · see citing works
Reward, punishment, and cooperation: A meta-analysis.
Balliet, D. P., Mulder, L. B., & Lange, P. A. M. V. (2011) · Psychological Bulletin · doi:10.1037/a0023489 · Cited by 714 · see citing works
Heritability of personality: A meta-analysis of behavior genetic studies.
Vukasović, T., & Bratko, D. (2015) · Psychological Bulletin · doi:10.1037/bul0000017 · Cited by 716 · see citing works
Gender differences in rumination: A meta-analysis
Johnson, D. P., & Whisman, M. A. (2013) · Personality and Individual Differences · doi:10.1016/j.paid.2013.03.019 · Cited by 712 · see citing works
Van der Kleij, F. M., Feskens, R. C. W., & Eggen, T. J. H. M. (2015) · Review of Educational Research · doi:10.3102/0034654314564881 · Cited by 714 · see citing works
“This issue was addressed with the fail-safe N method (Orwin, 1983), which estimates how many studies reporting nonsignificant effects sizes would be need to be combined with the studies in the meta-analysis in order to arrive at a specific result.”
MethodO’Dea, R. E., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Gurevitch, J., Page, M. J., Stewart, G. B., Moher, D., et al. (2021) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.12721 · Cited by 771 · see citing works
Effects of self-assessment on self-regulated learning and self-efficacy: Four meta-analyses
Panadero, E., Jonsson, A., & Botella, J. (2017) · Educational Research Review · doi:10.1016/j.edurev.2017.08.004 · Cited by 712 · see citing works
“also calculated the Orwin’s (1983) fail-safe number. Regarding the meta-analysis of Learning SRL (k=12) the Orwin fail-safe number is Nfs = 31. Therefore, with a number of 31 estimates of effect sizes not published, and with an average zero effect size added to our meta-analysis, the combined effect would statisticall…”
BackgroundInfluentialClimate for Creativity: A Quantitative Review
Hunter, S. T., Bedell, K. E., & Mumford, M. D. (2007) · Creativity Research Journal · doi:10.1080/10400410709336883 · Cited by 709 · see citing works
Frazier, T. W., Youngstrom, E. A., Glutting, J. J., & Watkins, M. W. (2007) · Journal of Learning Disabilities · doi:10.1177/00222194070400010401 · Cited by 700 · see citing works
“…consecutive comparisons with effect size d = .00 would be needed to bring the obtained overall effect size (d = .71) down to d = .30—a small effect—and more than 158 consecutive studies with null results would be needed to reduce the mean effect size to d = .05, a negligible effect (Orwin, 1983).”
BackgroundMeta‐analysis of human connection to nature and proenvironmental behavior
Whitburn, J., Linklater, W. L., & Abrahamse, W. (2019) · Conservation Biology · doi:10.1111/cobi.13381 · Cited by 700 · see citing works
The Relation of Self-Efficacy Measures to Sport Performance: A Meta-Analytic Review
Moritz, S. E., Feltz, D. L., Fahrbach, K. R., & Mack, D. E. (2000) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.2000.10608908 · Cited by 713 · see citing works
A Meta-Analytic Review of Gender Differences in ADHD
Gershon, J., & Gershon, J. (2002) · Journal of Attention Disorders · doi:10.1177/108705470200500302 · Cited by 875 · see citing works
Linde, K., Clausius, N., Ramirez, G., Melchart, D., Eitel, F., Hedges, L. V., & Jonas, W. B. (1997) · The Lancet · doi:10.1016/s0140-6736(97)02293-9 · Cited by 979 · see citing works
Cognitive Functioning in Prodromal Psychosis: A Meta-analysis
Fusar-Poli, P., Deste, G., Smieskova, R., Barlati, S., Yung, A. R., Howes, O., Stieglitz, R.-D., Vita, A., McGuire, P., & Borgwardt, S. (2012) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2011.1592 · Cited by 691 · see citing works
Bell, S. T., Villado, A. J., Lukasik, M. A., Belau, L., & Briggs, A. L. (2011) · Journal of Management · doi:10.1177/0149206310365001 · Cited by 907 · see citing works
“Finally, we calculated a fail-safe k (n) using procedures outlined by Rosenthal (1979) and Orwin (1983).”
MethodGender differences in the prevalence of nonsuicidal self-injury: A meta-analysis
Bresin, K., & Schoenleber, M. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.02.009 · Cited by 678 · see citing works
“All three methods of the fail-safe-N procedure (Orwin, 1983; Rosenberg, 2005; Rosenthal, 1979), indicated that thousands of unpublished studies would need to exist to reduce to overall effect size to 1 (range: 4503–26,931), which is inconsistent with publication bias.”
Beyond “Does it Pay to be Green?” A Meta-Analysis of Moderators of the CEP–CFP Relationship
Dixon‐Fowler, H. R., Slater, D. J., Johnson, J. L., Ellstrand, A. E., & Romi, A. M. (2012) · Journal of Business Ethics · doi:10.1007/s10551-012-1268-8 · Cited by 681 · see citing works
“Following the method proposed by Orwin (1983), we found that the number of additional samples with a correlation of 0 that would be necessary to bring the effect size in this study of 0.062 down to 0.05 would be 17.”
MethodInfluentialIncreasing physical activity: a quantitative synthesis.
Dishman, R., & Buckworth, J. (1996) · Medicine & Science in Sports & Exercise · doi:10.1097/00005768-199606000-00010 · Cited by 693 · see citing works
Improving fluid intelligence with training on working memory: a meta-analysis
Au, J., Sheehan, E., Tsai, N., Duncan, G. J., Buschkuehl, M., & Jaeggi, S. M. (2015) · Psychonomic Bulletin & Review · doi:10.3758/s13423-014-0699-x · Cited by 671 · see citing works
“In other words, a high I 2 value implies that not all of the studies included in the meta-analysis are, in fact, measuring the same effect and that some of the variation is due to differences in study design or sampling biases across studies, rather than sampling error alone.”
Shimokawa, K., Lambert, M. J., & Smart, D. W. (2010) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0019247 · Cited by 657 · see citing works
Cabral, R. R., & Smith, T. B. (2011) · Journal of Counseling Psychology · doi:10.1037/a0025266 · Cited by 651 · see citing works
The Effects of Vocabulary Intervention on Young Children’s Word Learning
Marulis, L. M., & Neuman, S. B. (2010) · Review of Educational Research · doi:10.3102/0034654310377087 · Cited by 668 · see citing works
“We also calculated the Orwin fail-safe N (Orwin, 1983), which allowed us to use a value other than null against an effect size criterion (i.”
MethodIs Peer Victimization Associated with Academic Achievement? A Meta-analytic Review
Nakamoto, J., & Schwartz, D. (2009) · Social Development · doi:10.1111/j.1467-9507.2009.00539.x · Cited by 662 · see citing works
Relation between Emotion Regulation and Mental Health: A Meta-Analysis Review
Hu, T., Zhang, D., Wang, J., Mistry, R., Ran, G., & Wang, X. (2014) · Psychological Reports · doi:10.2466/03.20.pr0.114k22w4 · Cited by 639 · see citing works
“…bearing in mind the inherent problems associated with funnel plots ( Lau, Ioannidis, Terrin, Schmid, & Olkin, 2006 ), the authors also conducted the analysis on fail-safe N s for all studies that investigated two strategies of emotion regulation and two indicators of mental health ( Orwin, 1983 ).”
MethodYuan, P., & Raz, N. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2014.02.005 · Cited by 608 · see citing works
“The fail-safe analysis (Orwin, 1983) revealed that 17 additional studies with null findings would be needed to offset the observed effect size.”
BackgroundA Meta-Analytic Review of Adult Cognitive-Behavioral Treatment Outcome Across the Anxiety Disorders
Norton, P. J., & Price, E. C. (2007) · Journal of Nervous & Mental Disease · doi:10.1097/01.nmd.0000253843.70149.9a · Cited by 612 · see citing works
“Although definitions of a trivial value vary (0.20 in Orwin, 1983; 0.05 in Van Etten and Taylor, 1998), we selected a trivial effect size to match the average effect size of the no-treatment conditions in the studies used for this meta-analysis (pre to post d 0.25).”
MethodInfluentialWorking memory and second language comprehension and production: A meta-analysis
Linck, J. A., Osthus, P., Koeth, J. T., & Bunting, M. F. (2013) · Psychonomic Bulletin & Review · doi:10.3758/s13423-013-0565-2 · Cited by 647 · see citing works
“First, we computed a fail-safe N (Orwin, 1983; Rosenthal, 1979), which computes the number of missing, unpublished, or future studies with null effects that would be required to render the probability of a Type I error for a significance test of b ρ to increase above an acceptable level.”
Influential
- Authors
- Orwin, R. G.
- Affiliations
- United States Department of Health and Human Services (US)
- Corresponding author
- Orwin, Robert G.
- Venue
- Journal of Educational Statistics
- Publisher
- JSTOR
- Published
- 1983
- Volume
- 8
- Issue
- 2
- Pages
- 157
- ISSN
- 0362-9791 (ISSN-L)
- OpenAlex
- W2112432954
- Cited by
- 2,064
- References
- 11
- Citations by year
- 2026: 51 · 2025: 75 · 2024: 91 · 2023: 121 · 2022: 114 · 2021: 129 · 2020: 129 · 2019: 120 · 2018: 116 · 2017: 109 · 2016: 95 · 2015: 113 · 2014: 108 · 2013: 76 · 2012: 88
- FWCI
- 7.68
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Educational Statistics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- effect size, file drawer problem, sampling bias, negative results, meta-analysis, publication bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
