The DOI matches its Crossref record; 10 of the 11 references it cites were found in a registry or an index.
A Fail-Safe N for Effect Size in Meta-Analysis
Abstract
Rosenthan's (1979) concept of fail-safe N has thus far been applied to probability levels exclusively. This note introduces a fail-safe TV for effect size. Rosenthal's (1979) fail-safe N was an ingenious response to the so-called file drawer problem in research integration, in that it provided a direct assessment of the threat posed by sampling bias in the literature search. The algorithm was derived from the Stouffer method (Mosteller & Bush, 1954), in which an overall Z-score is computed by summing individual Z-scores and dividing by the square root of the number of scores. A significant overall Z-score, Rosenthal reasoned, could be made nonsignificant by adding some knowable number of hypothetical studies that averaged null results. The exact number of such studies needed to bring a significant overall p level up to some critical level (e.g., .05) has since become known as the fail-safe N (Nfs). The formula is
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“we also calculated the classic fail-safe N test (Orwin, 1983), which revealed that it would take 59 studies reporting null results (g = 0) to be included in our analyses in order for our findings to lose statistical significance.”
From Improving fluid intelligence with training on working memory: a meta-analysis (2015) · cites this paper as Orwin, 1983
In the citing paper
“The file drawer problem (publication bias) was examined by creating a funnel plot and calculating the fail safe N (45).”
From A Meta-Analysis of the Corpus Callosum in Autism (2009) · cites this paper as [45]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890
Smith, M. L., Glass, G. V., & Miller, T. I. (1980) · Medical Entomology and Zoology · Cited by 2,123
Johnson, D. W., Maruyama, G., Johnson, R., Nelson, D., & Skon, L. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.89.1.47 · Cited by 1,588
Interpersonal expectancy effects: the first 345 studies
Rosenthal, R., & Rubin, D. B. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00075506 · Cited by 1,158
Eagly, A. H., & Carli, L. L. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.90.1.1 · Cited by 541
Cooper, H. M. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.1.131 · Cited by 418
Arkin, R. M., Cooper, H., & Kolditz, T. A. (1980) · Journal of Personality · doi:10.1111/j.1467-6494.1980.tb02378.x · Cited by 88
Flagged first, then by citations
1 of the 11 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Items 601–700 of 1,020
Meta–Analysis of Mathematic Basic–Fact Fluency Interventions: A Component Analysis
Codding, R. S., Burns, M. K., & Lukito, G. (2011) · Learning Disabilities Research and Practice · doi:10.1111/j.1540-5826.2010.00323.x · Cited by 117
“Thus, a Fail-Safe N for effect sizes (Orwin, 1983) was computed to determine the potential influence of a file-drawer problem (Rosenthal, 1979).”
MethodMatthews, H., Grunfeld, E. A., & Turner, A. (2017) · Psycho-Oncology · doi:10.1002/pon.4199 · Cited by 117
“FIGURE 1 Flow diagram depicting the systematic review process N was also calculated to assess the roboustness of the overall effect.(21) This will determine the number of studies with a null effect size required to reduce the overall effect to non‐significant.”
MethodZelkowitz, R. L., & Cole, D. A. (2018) · Suicide and Life-Threatening Behavior · doi:10.1111/sltb.12436 · Cited by 116
Systematic Training of Graduate-Level Counselors
Baker, S. B., Daniels, T. G., & Greeley, A. T. (1990) · The Counseling Psychologist · doi:10.1177/0011000090183001 · Cited by 116
The effectiveness of academic admission interviews: an exploratory meta-analysis
Goho, J., & Blackman, A. (2006) · Medical Teacher · doi:10.1080/01421590600603418 · Cited by 116
“ORWIN, R.G. (1983) A fail-safe N for effect sizes in meta-analysis, Journal of Educational Statistics, 8, pp.”
BackgroundFinancial Reinforcers for Improving Medication Adherence: Findings from a Meta-analysis
Petry, N. M., Rash, C. J., Byrne, S., Ashraf, S., & White, W. B. (2012) · The American Journal of Medicine · doi:10.1016/j.amjmed.2012.01.003 · Cited by 115
Gentile, A., Boca, S., & Giammusso, I. (2018) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2018.07.013 · Cited by 142
“Finally, the stability of the meta-analysis was assessed calculating Fail-Safe N (Orwin, 1983; Rosenberg, 2005), which is the estimated number of non-significant results in the file drawers that would be required to turn a significant meta-analytic finding into one that is nonsignificant…”
MethodXu, Z., Ritzhaupt, A. D., Tian, F., & Umapathy, K. (2019) · Computer Science Education · doi:10.1080/08993408.2019.1565233 · Cited by 115
“Also, the Orwin ’ s fail-safe N test was calculated to determine the number of missing null studies required to bring the existing e ff ect size to a trivial level (Orwin, 1983).”
MethodPersonality traits and conflict resolution styles: A meta-analysis
Tehrani, H. D., & Yamini, S. (2020) · Personality and Individual Differences · doi:10.1016/j.paid.2019.109794 · Cited by 115
Meta‐analysis on anxiety and depression in adult celiac disease
Smith, D. F., & Gerdes, L. U. (2011) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.2011.01795.x · Cited by 114
“Third, bias toward publishing positive findings, known as the file-drawer problem (8, 20), must be considered.”
BackgroundDiversity in Alliance Portfolios and Performance Outcomes: A Meta-Analysis
Lee, D., Kirkpatrick-Husk, K., & Madhavan, R. (2014) · Journal of Management · doi:10.1177/0149206314556316 · Cited by 114
Differences in Pain Coping Between Black and White Americans: A Meta-Analysis
Meints, S. M., Miller, M. M., & Hirsh, A. T. (2016) · Journal of Pain · doi:10.1016/j.jpain.2015.12.017 · Cited by 116
“To examine potential publication bias, Orwin’s failsafe N [69] was calculated.”
BackgroundMethodChildhood Maltreatment and Impulsivity: A Meta-Analysis and Recommendations for Future Study
Liu, R. T. (2018) · Research on Child and Adolescent Psychopathology · doi:10.1007/s10802-018-0445-3 · Cited by 123
“To evaluate for potential publication bias, the following publication bias indices were calculated: Orwin’s fail-safe N (Orwin 1983), Duval and Tweedie’s trim-and-fill analysis (Duval and Tweedie 2000), and Egger’s regression intercept (Egger et al.”
MethodA meta-analysis of virtual reality training programs
Howard, M. C., Gutworth, M. B., & Jacobs, R. R. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106808 · Cited by 134
“According to prior guidelines (Orwin, 1983; Rothstein et al., 2005), these fail-safe k values are sufficiently large and the meta-analytic effects can be considered robust.”
MethodThe Impact of Age and Education on Response Order Effects in Attitude Measurement
Knäuper, B. (1999) · Public Opinion Quarterly · doi:10.1086/297724 · Cited by 118
DIFFERENTIAL RECALL BIAS AND SPURIOUS ASSOCIATIONS IN CASE/CONTROL STUDIES
Barry, D. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19961215)15:23 · Cited by 39
Generalizations on the effectiveness of pharmaceutical promotional expenditures
Kremer, S., Bijmolt, T. H. A., Leeflang, P. S. H., & Wieringa, J. E. (2008) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2008.08.001 · Cited by 115
Yu, B. C. L., Chio, F. H. N., Mak, W. W. S., Corrigan, P. W., & Chan, K. K. Y. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2021.102029 · Cited by 110
Spatial Abilities and Aging: A Meta-Analysis
Techentin, C., Voyer, D., & Voyer, S. D. (2014) · Experimental Aging Research · doi:10.1080/0361073x.2014.926773 · Cited by 112
Witteman, J., van IJzendoorn, M. H., van de Velde, D., van Heuven, V. J. J. P., & Schiller, N. O. (2011) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2011.09.028 · Cited by 112
Bullying Victimization at School and Headache: A Meta‐Analysis of Observational Studies
Gini, G., Pozzoli, T., Lenzi, M., & Vieno, A. (2014) · Headache The Journal of Head and Face Pain · doi:10.1111/head.12344 · Cited by 113
Generational Differences in Academic Achievement Among Immigrant Youths
Duong, M. T., Badaly, D., Liu, F. F., Schwartz, D., & McCarty, C. A. (2015) · Review of Educational Research · doi:10.3102/0034654315577680 · Cited by 115
Emotional Awareness in Depressive and Anxiety Symptoms in Youth: A Meta-Analytic Review
Sendzik, L., Schäfer, J., Samson, A. C., Naumann, E., & Tuschen‐Caffier, B. (2017) · Journal of Youth and Adolescence · doi:10.1007/s10964-017-0629-0 · Cited by 132
Multifocality as a prognostic factor in thyroid cancer: A meta-analysis
Joseph, K. R., Edirimanne, S., & Eslick, G. D. (2018) · International Journal of Surgery · doi:10.1016/j.ijsu.2017.12.035 · Cited by 112
Hendriks, T., Schotanus-Dijkstra, M., Hassankhan, A., Graafsma, T., Bohlmeijer, E. T., & de Jong, J. T. V. M. (2018) · International Journal of Wellbeing · doi:10.5502/ijw.v8i1.711 · Cited by 125
“The fail-safe N indicates the number of unpublished non-significant studies that would be required to lower the overall effect size below significance (Egger, Smith, Schneider, & Minder, 1997; Orwin, 1983).”
BackgroundComparative clinical effects of hydromorphone and morphine: a meta-analysis
Felden, L., Walter, C., Harder, S., Treede, R., Kayser, H., Drover, D. R., Geißlinger, G., & Lötsch, J. (2011) · British Journal of Anaesthesia · doi:10.1093/bja/aer232 · Cited by 112
Halasa, T., Østerås, O., Hogeveen, H., van Werven, T., & Nielen, M. (2009) · Journal of Dairy Science · doi:10.3168/jds.2008-1740 · Cited by 119
Non-suicidal self-injury and life stress: A systematic meta-analysis and theoretical elaboration
Liu, R. T., Cheek, S. M., & Nestor, B. A. (2016) · Clinical Psychology Review · doi:10.1016/j.cpr.2016.05.005 · Cited by 137
“To assess for the presence of a potential publication bias, the following publication bias indices were calculated: Orwin's fail-safe N (Orwin, 1983), Duval and Tweedie's trim-and-fill analysis (Duval & Tweedie, 2000), and Egger's regression intercept (Egger, Davey Smith, Schneider, & Minder, 1997).”
MethodWilson, K., Girdler, N. M., & Welbury, R. R. (2003) · British Journal of Anaesthesia · doi:10.1093/bja/aeg278 · Cited by 110
Schoenfeld, B. J., Ogborn, D. I., Vigotsky, A. D., Franchi, M. V., & Krieger, J. W. (2017) · Journal of Strength and Conditioning Research · doi:10.1519/jsc.0000000000001983 · Cited by 122
A meta-analysis of the association between anger and aggressive driving
Bogdan-Ganea, S. R., Măirean, C., & Havârneanu, C.-E. (2016) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2016.05.009 · Cited by 110
Gou, X., Reich, P. B., Qiu, L., Shao, M., Wei, G., Wang, J., & Wei, X. (2023) · Global Change Biology · doi:10.1111/gcb.16742 · Cited by 111
“The concentrations of NO3 − and NH4 + in the soils at the beginning and end of either laboratory or in situ incubation were analyzed to calculate the metrics for net N mineralization.”
MethodInfluentialSocial Competence in Children with Chronic Illness: A Meta-analytic Review
Martínez, W., Carter, J. S., & Legato, L. J. (2011) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsr035 · Cited by 111
Magee, L. A., Elran, E., Bull, S. B., Logan, A. G., & Koren, G. (2000) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/s0301-2115(99)00113-x · Cited by 110
Antonopoulos, C. N., Sergentanis, T. N., Daskalopoulou, S. S., & Petridou, E. T. (2011) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2010.10.002 · Cited by 108
Neuropsychological functioning after CPAP treatment in obstructive sleep apnea: A meta-analysis
Kylstra, W. A., Aaronson, J. A., Hofman, W. F., & Schmand, B. A. (2012) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2012.09.002 · Cited by 111
Biodiversity in perennial and intermittent rivers: a meta‐analysis
Soria, M., Leigh, C., Datry, T., Bini, L. M., & Bonada, N. (2017) · Oikos · doi:10.1111/oik.04118 · Cited by 121
“Second, we calculated the fail-safe number 272 according to Orwin’s equation (Orwin 1983), which gives the number of studies 273 needed to reduce the average effect size to a pre-specified value, which is 274 considered unimportant.”
MethodWalker, L. E., & Warfa, A.-R. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0186203 · Cited by 109
Kumarasamy, C., Tiwary, V., Krishnan, S., Suresh, D., Shetty, S. S., Muthukaliannan, G. K., Baxi, S., & Jayaraj, R. (2021) · Cancers · doi:10.3390/cancers13164166 · Cited by 109
“Adjustment for small and missing studies was carried out via imputation of possible small studies using the Orwin’s Fail-Safe N test [45].”
MethodWojnarowski, C., Firth, N., Finegan, M., & Delgadillo, J. (2019) · Behavioural and Cognitive Psychotherapy · doi:10.1017/s1352465819000080 · Cited by 108
Khlif, H., Hussainey, K., & Achek, I. (2015) · Meditari Accountancy Research · doi:10.1108/medar-12-2014-0064 · Cited by 116
Effects of Noninvasive Brain Stimulation on Food Cravings and Consumption: A Meta-Analytic Review
Lowe, C. J., Vincent, C. M., & Hall, P. A. (2016) · Psychosomatic Medicine · doi:10.1097/psy.0000000000000368 · Cited by 107
“Publication bias was assessed using two methods: (1) using a funnel plot and visual inspection of study ESs as a function of sample size; (2) calculating the Orwin fail-safe N for ESs (42) to determine the number of studies that would need to be added to the meta-analysis to result in a “small” ES.”
MethodHogan, S. E., L'Allier, P. L., Chetcuti, S. J., Grossman, P. M., Nallamothu, B. K., Duvernoy, C. S., Bates, E., Moscucci, M. M., & Gurm, H. S. (2008) · American Heart Journal · doi:10.1016/j.ahj.2008.05.014 · Cited by 108
“contrast-induced AKI between the 2 groups) using Rosenberg's and Orwin's method.(17,18) Because there was a suggestion of publication bias, Duval and Tweedie's(19) Trim”
MethodSmall-Study Effects in Meta-Analysis
Schwarzer, G., Carpenter, J. R., & Rücker, G. (2015) · Use R! · doi:10.1007/978-3-319-21416-0_5 · Cited by 114
Depressive symptoms and cardiovascular reactivity to laboratory behavioral stress
Kibler, J. L., & Ma, M. (2004) · International Journal of Behavioral Medicine · doi:10.1207/s15327558ijbm1102_3 · Cited by 107
Correcting for Publication Bias in a Meta-Analysis with the P-uniform* MethodPreprint
van Aert, R. C. M., & van Assen, M. A. L. M. (2018) · doi:10.31222/osf.io/zqjr9 · Cited by 108
Merkin, R. S., Taras, V., & Steel, P. (2013) · International Journal of Intercultural Relations · doi:10.1016/j.ijintrel.2013.10.004 · Cited by 108
“Table 1 also provides fail-safe N (Orwin, 1983), which is used to assess the robustness of results in light of the possibility of the “file drawer” problem.”
MethodThe impact of hypnotic suggestibility in clinical care settings
Montgomery, G. H., Schnur, J. B., & David, D. (2011) · International Journal of Clinical and Experimental Hypnosis · doi:10.1080/00207144.2011.570656 · Cited by 106
“To address this possible concern, the number of studies with an effect size of zero required to decrease the overall mean effect size to small (based on Orwin, 1983) was calculated.”
Ownership structure and voluntary disclosure: A synthesis of empirical studies
Khlif, H., Ahmed, K., & Souissi, M. (2016) · Australian Journal of Management · doi:10.1177/0312896216641475 · Cited by 108
Experimental manipulations of behavioral economic demand for addictive commodities: a meta‐analysis
Acuff, S. F., Amlung, M. T., Dennhardt, A. A., MacKillop, J. M., & Murphy, J. G. (2019) · Addiction · doi:10.1111/add.14865 · Cited by 105
“Publication bias was assessed based on four indices (Table 3): (1) the classic fail-safe n, (2) Orwin’s fail-safe n at 50% of the reported effect size [102]; the two-tailed Begg–Mazumdar test [103]; and (4) the one-tailed Egger’s test [104].”
MethodQuantitative evidence for distinct cognitive impairment in anorexia nervosa and bulimia nervosa
Zakzanis, K. K., Campbell, Z., & Polsinelli, A. J. (2009) · Journal of Neuropsychology · doi:10.1348/174866409x459674 · Cited by 105
“…error scores for each mean effect size, 95% confidence intervals, homogeneity statistics, Orwin’s fail-safe N statistic (using a conservative criterion of d ¼ 0:2; Orwin, 1983), and per cent of non-overlap of effect size based on Cohen’s (1988) idealized population distribution are also presented.”
MethodBalottin, U., Fusar‐Poli, P., Termine, C., Molteni, S., & Galli, F. (2012) · Cephalalgia · doi:10.1177/0333102412468386 · Cited by 105
“In addition, we used the fail-safe procedure (40) to generate the number of unpublished studies that would be needed to move estimates to a non-significant threshold.”
MethodViljoen, J. L., Mordell, S., & Beneteau, J. L. (2012) · Law and Human Behavior · doi:10.1037/h0093938 · Cited by 113
Gender and self-esteem: A meta-analysis.
Major, B., Barr, L., Zubek, J. M., & Babey, S. H. (2004) · American Psychological Association eBooks · doi:10.1037/10277-009 · Cited by 113
Relative Size versus Controlling for Size
Smith, R. J. (2005) · Current Anthropology · doi:10.1086/427117 · Cited by 105
“Another fail-safe number, proposed by Orwin (1983), has a value for this data set of 4.”
BackgroundWhurr, R., Lorch, M. P., & Nye, C. (1992) · International Journal of Language & Communication Disorders · doi:10.3109/13682829209012026 · Cited by 105
The impact of treatment on HPA axis activity in unipolar major depression
McKay, M. S., & Zakzanis, K. K. (2010) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2009.07.012 · Cited by 103
Effectiveness of Web-based Psychological Interventions for Depression: A Meta-analysis
Cowpertwait, L., & Clarke, D. (2013) · International Journal of Mental Health and Addiction · doi:10.1007/s11469-012-9416-z · Cited by 101
“However, a fail-safe N statistic was calculated (Lipsey and Wilson 2001; Orwin 1983).”
MethodRocchetti, M., Crescini, A., Borgwardt, S., Caverzasi, E., Politi, P., Atakan, Z., & Fusar‐Poli, P. (2013) · Psychiatry and Clinical Neurosciences · doi:10.1111/pcn.12085 · Cited by 109
Emission of CO 2 from biochar‐amended soils and implications for soil organic carbon
Sagrilo, E., Jeffery, S., Hoffland, E., & Kuyper, T. W. (2014) · GCB Bioenergy · doi:10.1111/gcbb.12234 · Cited by 110
On the sunk-cost effect in economic decision-making: a meta-analytic review
Roth, S., Robbert, T., & Straus, L. (2014) · BuR - Business Research · doi:10.1007/s40685-014-0014-8 · Cited by 118
“Because we selected the standardized mean difference effect size, we used the adaptation of Orwin (1983) and calculated the fail-safe N with k0 1⁄4 k ESK ESC 1 , where k0 is the number of effect sizes needed to reduce the effect size to ESC, k is the number of studies in the mean effect size, ESK is the weighted mean…”
MethodMiller, M. M., Meints, S. M., & Hirsh, A. T. (2018) · Pain · doi:10.1097/j.pain.0000000000001342 · Cited by 116
“Another limitation, common in meta-analyses, is the file drawer problem.(80) Although the results of fail-safe analyses instill confidence in the meta-analytic findings for all of our primary outcome variables, we cannot rule out the existence of unpublished studies that would have changed these findings had they been…”
Is Combination Sulfonylurea and Insulin Therapy Useful in NIDDM Patients?: A metaanalysis
Pugh, J. A., Wagner, M., Sawyer, J., Ramı́rez, G., Tuley, M. R., & Friedberg, S. J. (1992) · Diabetes Care · doi:10.2337/diacare.15.8.953 · Cited by 104
“To determine the number of trials that would be needed to overturn or disprove the results, we calculated a "fail-safe N" with the method of Orwin (12).”
MethodIvarsson, T., Skarphéðinsson, G., Kornør, H., Axelsdottir, B., Biedilæ, S., Heyman, I., Asbahr, F. R., Thomsen, P. H., Fineberg, N. A., & March, J. S. (2015) · Psychiatry Research · doi:10.1016/j.psychres.2015.01.015 · Cited by 103
“We tested the possiblity of a publication bias with Orwin’s FailSafe N (Orwin, 1983), which indicated that we need 24 studies with point mean difference estimate of 0 to bring the mean difference from 3.”
MethodBarrera, T. L., Mott, J. M., Hofstein, R. F., & Teng, E. J. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.09.005 · Cited by 103
Cognitive benefits of right-handedness: A meta-analysis
Somers, M., Shields, L., Boks, M. P., Kahn, R. S., & Sommer, I. E. (2015) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2015.01.003 · Cited by 103
Semkovska, M., Keane, D., Babalola, O., & McLoughlin, D. M. (2010) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2010.11.001 · Cited by 106
Effects of deficient reporting on meta-analysis: A conceptual framework and reanalysis.
Orwin, R. G., & Cordray, D. S. (1985) · Psychological Bulletin · doi:10.1037/0033-2909.97.1.134 · Cited by 102
Self-citationNie, W., Li, B., & Xiu, Q. (2014) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dku033 · Cited by 102
Hot Food and Beverage Consumption and the Risk of Esophageal Cancer
Andrici, J., & Eslick, G. D. (2015) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2015.07.023 · Cited by 102
Ibrahim, B., Aljarah, A., & Ababneh, B. (2020) · Journal of Promotion Management · doi:10.1080/10496491.2020.1719956 · Cited by 123
Hartley, M., Dorstyn, D., & Due, C. (2019) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-019-04145-3 · Cited by 99
“Finally, Orwin’s FailSafe N (Nfs) (Orwin 1983) was calculated to test for publication bias (criterion effect size = 0.”
MethodThe effectiveness of virtual reality distraction for reducing pain: A meta-analysis.
Kenney, M. P., & Milling, L. S. (2016) · Psychology of Consciousness Theory Research and Practice · doi:10.1037/cns0000084 · Cited by 101
Morrison, T. G., & Morrison, M. A. (1995) · Educational and Psychological Measurement · doi:10.1177/0013164495055002015 · Cited by 101
Antecedents and Outcomes of Hospitality Loyalty
Tanford, S. (2016) · Cornell Hospitality Quarterly · doi:10.1177/1938965516640121 · Cited by 101
“The Fail-safe N (Orwin, 1983) provides a safeguard against missing or unpublished studies.”
BackgroundChoi, K., Wykes, T., & Kurtz, M. M. (2013) · The British Journal of Psychiatry · doi:10.1192/bjp.bp.111.107359 · Cited by 99
Rieker, J. A., Avilés, J. M. R., Muiños, M., & Ballesteros, S. (2022) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2022.838968 · Cited by 117
“, the number of non-significant ES needed to change a significant into a non-significant result) following different approaches (Rosenthal, 1979; Orwin, 1983; Rosenberg, 2005).”
ResultHalasa, T., Nielen, M., Whist, A. C., & Østerås, O. (2009) · Journal of Dairy Science · doi:10.3168/jds.2008-1741 · Cited by 98
Sun, F., Chen, Y., Xiang, Y., & Zhan, S. (2008) · PubMed · Cited by 100
Borenstein, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch11 · Cited by 178
“Orwin (1983) proposed a variant on the Rosenthal formula which addresses both of these issues.”
BackgroundJung, S. B., Nagaraja, V., Kapur, A., & Eslick, G. D. (2015) · Internal Medicine Journal · doi:10.1111/imj.12697 · Cited by 99
McAlister, C. B., Schmitter‐Edgecombe, M., & Lamb, R. L. (2016) · Archives of Clinical Neuropsychology · doi:10.1093/arclin/acv089 · Cited by 99
“The failsafe N indicated that 238,830 studies would be needed to nullify the observed effect (Orwin, 1983).”
BackgroundLeech, T., Dorstyn, D., Taylor, A., & Li, W. (2021) · Children and Youth Services Review · doi:10.1016/j.childyouth.2021.106073 · Cited by 100
Penela, A. C., Sanmartín, E. R., González, P. Á., & Paillé, P. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.113984 · Cited by 122
The Usefulness of the "Fail-Safe" Statistic in Meta-Analysis
Carson, K. P., Schriesheim, C. A., & Kinicki, A. J. (1990) · Educational and Psychological Measurement · doi:10.1177/0013164490502001 · Cited by 99
Kasapis, C., Henke, P. K., Chetcuti, S. J., Koenig, G. C., Rectenwald, J. E., Krishnamurthy, V. N., Grossman, P. M., & Gurm, H. S. (2008) · European Heart Journal · doi:10.1093/eurheartj/ehn514 · Cited by 99
The effect of fish meal replacement by soyabean products on fish growth: a meta-analysis
Sales, J. (2009) · British Journal Of Nutrition · doi:10.1017/s0007114509991279 · Cited by 98
“A fail-safe number (Nfs) (51) has been calculated to indicate the number of unpublished comparisons with null effects needed to reduce the observed d to a negligible level:”
BackgroundWater productivity and seed cotton yield in response to deficit irrigation: A global meta-analysis
Cheng, M., Wang, H., Fan, J., Zhang, S., Wang, Y., Li, Y., Sun, X., Yang, L., & Zhang, F. (2021) · Agricultural Water Management · doi:10.1016/j.agwat.2021.107027 · Cited by 98
Firth, J., Stubbs, B., Sarris, J. J., Rosenbaum, S., Teasdale, S. B., Berk, M., & Yung, A. R. (2017) · Psychological Medicine · doi:10.1017/s0033291717000022 · Cited by 98
“Where a significant risk of publication bias was detected, a ‘filedrawanalysis’was conducted to calculate a ‘fail-safe N’ (Orwin, 1983); the approximate number of unpublished studies which must exist to invalidate the results of the meta-analysis (i.”
ResultMicrowave Ablation Compared to Radiofrequency Ablation for Hepatic Lesions: A Meta-Analysis
Huo, Y. R., & Eslick, G. D. (2015) · Journal of Vascular and Interventional Radiology · doi:10.1016/j.jvir.2015.04.004 · Cited by 97
Meta-analysis of targeted small-group reading interventions
Hall, M. S., & Burns, M. K. (2017) · Journal of School Psychology · doi:10.1016/j.jsp.2017.11.002 · Cited by 97
Gage, N. A., Whitford, D. K., & Katsiyannis, A. (2018) · The Journal of Special Education · doi:10.1177/0022466918767847 · Cited by 97
“The Fail-Safe N is a probability estimation of the number of nonpublished studies averaging nonsignificant results, which must be in existence before the pooled effect of the included metaanalytic interventions would no longer be statistically significant (Cooper, 1979; Rosenthal, 1979) or the effect size reduced (Orw…”
BackgroundThe production effect benefits performance in between-subject designs: A meta-analysis
Fawcett, J. M. (2012) · Acta Psychologica · doi:10.1016/j.actpsy.2012.10.001 · Cited by 97
Comparing the Effectiveness of EMDR and TF-CBT for Children and Adolescents: a Meta-Analysis
Lewey, J. H., Smith, C. L., Burcham, B., Saunders, N. L., Elfallal, D., & O’Toole, S. K. (2018) · Journal of Child & Adolescent Trauma · doi:10.1007/s40653-018-0212-1 · Cited by 114
“Second, the Classic fail-safe N was calculated to provide an estimate of how many unpublished studies it would take to raise the p value to .05 (Orwin 1983).”
MethodChronic tobacco smoking and neuropsychological impairments: A systematic review and meta-analysis
Conti, A. A., McLean, L., Tolomeo, S., Steele, J. D., & Baldacchino, A. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.11.017 · Cited by 107
“…to the number of missing studies that would allow to determine how many of these studies would bring the overall effect of the current meta-analysis to a specified level other than zero and that would be needed to change the result from significant to non-significant (Orwin, 1983; Rosenthal, 1979).”
Molineri, A. I., Camussone, C., Zbrun, M. V., Archilla, G. A. S., Cristiani, M., Neder, V. E., Calvinho, L. F., & Signorini, M. L. (2021) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2021.105261 · Cited by 96
Integrating research on the microcounseling program: A meta-analysis.
Baker, S. B., & Daniels, T. G. (1989) · Journal of Counseling Psychology · doi:10.1037//0022-0167.36.2.213 · Cited by 95
The Prediction of Recidivism with Aboriginal Offenders: A Theoretically Informed Meta-Analysis
Gutierrez, L., Wilson, H. A., Rugge, T., & Bonta, J. L. (2012) · Canadian Journal of Criminology and Criminal Justice/La Revue canadienne de criminologie et de justice pénale · doi:10.3138/cjccj.2011.e.51 · Cited by 102
Effectiveness of Computer-Assisted Instruction in Statistics
Sosa, G. W., Berger, D. E., Saw, A. T., & Mary, J. (2010) · Review of Educational Research · doi:10.3102/0034654310378174 · Cited by 111
“To address this concern, we computed a fail-safe N (Orwin, 1983; Rosenthal, 1979) to determine the number of undiscovered studies with an effect size of zero that would need to exist to reduce the random effects mean effect size from 0.33 to 0.10.”
MethodInfluentialDevlin, E. J., Denson, L. A., & Whitford, H. S. (2017) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2017.03.017 · Cited by 105
Metadata
- Authors
- Orwin, R. G.
- Affiliations
- United States Department of Health and Human Services (US)
- Corresponding author
- Orwin, Robert G.
- Venue
- Journal of Educational Statistics
- Publisher
- JSTOR
- Published
- 1983
- Volume
- 8
- Issue
- 2
- Pages
- 157
- ISSN
- 0362-9791 (ISSN-L)
- OpenAlex
- W2112432954
- Cited by
- 2,064
- References
- 11
- Citations by year
- 2026: 51 · 2025: 75 · 2024: 91 · 2023: 121 · 2022: 114 · 2021: 129 · 2020: 129 · 2019: 120 · 2018: 116 · 2017: 109 · 2016: 95 · 2015: 113 · 2014: 108 · 2013: 76 · 2012: 88
- FWCI
- 7.68
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Educational Statistics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- effect size, file drawer problem, sampling bias, negative results, meta-analysis, publication bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
