The DOI matches its Crossref record; 10 of the 11 references it cites were found in a registry or an index.
A Fail-Safe N for Effect Size in Meta-Analysis
Abstract
Rosenthan's (1979) concept of fail-safe N has thus far been applied to probability levels exclusively. This note introduces a fail-safe TV for effect size. Rosenthal's (1979) fail-safe N was an ingenious response to the so-called file drawer problem in research integration, in that it provided a direct assessment of the threat posed by sampling bias in the literature search. The algorithm was derived from the Stouffer method (Mosteller & Bush, 1954), in which an overall Z-score is computed by summing individual Z-scores and dividing by the square root of the number of scores. A significant overall Z-score, Rosenthal reasoned, could be made nonsignificant by adding some knowable number of hypothetical studies that averaged null results. The exact number of such studies needed to bring a significant overall p level up to some critical level (e.g., .05) has since become known as the fail-safe N (Nfs). The formula is
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“we also calculated the classic fail-safe N test (Orwin, 1983), which revealed that it would take 59 studies reporting null results (g = 0) to be included in our analyses in order for our findings to lose statistical significance.”
From Improving fluid intelligence with training on working memory: a meta-analysis (2015) · cites this paper as Orwin, 1983
In the citing paper
“The file drawer problem (publication bias) was examined by creating a funnel plot and calculating the fail safe N (45).”
From A Meta-Analysis of the Corpus Callosum in Autism (2009) · cites this paper as [45]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890
Smith, M. L., Glass, G. V., & Miller, T. I. (1980) · Medical Entomology and Zoology · Cited by 2,123
Johnson, D. W., Maruyama, G., Johnson, R., Nelson, D., & Skon, L. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.89.1.47 · Cited by 1,588
Interpersonal expectancy effects: the first 345 studies
Rosenthal, R., & Rubin, D. B. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00075506 · Cited by 1,158
Eagly, A. H., & Carli, L. L. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.90.1.1 · Cited by 541
Cooper, H. M. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.1.131 · Cited by 418
Arkin, R. M., Cooper, H., & Kolditz, T. A. (1980) · Journal of Personality · doi:10.1111/j.1467-6494.1980.tb02378.x · Cited by 88
Flagged first, then by citations
1 of the 11 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Items 801–900 of 1,020
Nissim, N., Moberg, P., & Hamilton, R. (2020) · Brain Science · doi:10.3390/brainsci10090597 · Cited by 75
Kittel, A. F. D., Olderbak, S. G., & Wilhelm, O. (2021) · Assessment · doi:10.1177/1073191121996469 · Cited by 89
Wang, H., Zheng, J., Fan, J., Zhang, F., & Huang, C. (2021) · Field Crops Research · doi:10.1016/j.fcr.2021.108210 · Cited by 81
Hiatal hernia and the risk of Barrett's esophagus
Andrici, J., Tio, M., Cox, M. R., & Eslick, G. D. (2012) · Journal of Gastroenterology and Hepatology · doi:10.1111/j.1440-1746.2012.07199.x · Cited by 80
Borokhovski, E., Bernard, R. M., Tamim, R. M., Schmid, R. F., & Sokolovskaya, A. (2015) · Computers & Education · doi:10.1016/j.compedu.2015.11.004 · Cited by 88
Telepsychology for Posttraumatic Stress Disorder: A systematic review
Bolton, A., & Dorstyn, D. (2015) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x15571996 · Cited by 74
“20.(46) The larger the Nfs, the more robust the findings.”
BackgroundA Meta-Analysis of Risk Markers for Intimate Partner Violence in Same-Sex Relationships
Kimmes, J. G., Mallory, A. B., Spencer, C., Beck, A. R., Cafferky, B., & Stith, S. M. (2019) · Trauma, Violence, & Abuse · doi:10.1177/1524838017708784 · Cited by 89
Kupferschmidt, D. A., & Zakzanis, K. K. (2011) · Psychiatry Research Neuroimaging · doi:10.1016/j.pscychresns.2011.02.011 · Cited by 81
Meta‐analyses and effect sizes in applied behavior analysis: A review and discussion
Dowdy, A., Peltier, C. J., Tincani, M., Schneider, W. J., Hantula, D. A., & Travers, J. C. (2021) · Journal of Applied Behavior Analysis · doi:10.1002/jaba.862 · Cited by 86
Risks, strengths, gender, and recidivism among justice-involved youth: A meta-analysis.
Scott, T., & Brown, S. L. (2018) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000343 · Cited by 96
Marshall, K. H., d’Udekem, Y., Sholler, G. F., Opotowsky, A. R., Costa, D. S. J., Sharpe, L., Celermajer, D. S., Winlaw, D. S., Newburger, J. W., & Kasparian, N. A. (2020) · Journal of the American Heart Association · doi:10.1161/jaha.119.014172 · Cited by 77
Allen, J. S., Stevenson, R. M., O’Boyle, E. H., & Seibert, S. E. (2020) · Strategic Entrepreneurship Journal · doi:10.1002/sej.1377 · Cited by 91
Systematic reviews of trials and other studies.
Sutton, A. J., Abrams, K. R., Jones, D. R. H., Sheldon, T., & Song, F. (1998) · PubMed · Cited by 79
Femoral nerve blocks for acute postoperative pain after knee replacement surgery
Chan, E., Fransen, M., Parker, D. A., Assam, P. N., & Chua, N. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009941 · Cited by 78
Database Selection Guidelines for Meta‐Analysis in Applied Linguistics
In'nami, Y., & Koizumi, R. (2010) · TESOL Quarterly · doi:10.5054/tq.2010.215253 · Cited by 62
Nagaraja, V., Cox, M. R., & Eslick, G. D. (2014) · Europe PMC (PubMed Central) · doi:10.3978/j.issn.2078-6891.2014.007 · Cited by 77
“Publication bias is generally regarded as a concern if the fail-safe number is less than 5n+10, with n being the number of studies included in the meta-analysis (18).”
BackgroundDorstyn, D., Mathias, J. L., & Denson, L. A. (2010) · Journal of Health Psychology · doi:10.1177/1359105310379063 · Cited by 78
The Association Between Anxiety and Falls: A Meta-Analysis
Hallford, D. J., Nicholson, G. C., Sanders, K. M., & McCabe, M. P. (2016) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbv160 · Cited by 79
“The fail-safe N (Orwin, 1983) for the overall effect size was 18, indicating that a minimum of 305 studies with an OR point estimate of 1.0 would have to be added to the included studies before the average effect became 1.01.”
BackgroundMethodInfluentialWilson, K. E., Welbury, R. R., & Girdler, N. M. (2002) · Anaesthesia · doi:10.1046/j.1365-2044.2002.02784.x · Cited by 77
Key Elements of mHealth Interventions to Successfully Increase Physical Activity: Meta-Regression
Eckerstorfer, L. V., Tanzer, N. K., Vogrincic-Haselbacher, C., Kedia, G., Brohmer, H., Dinslaken, I., & Corcoran, K. (2018) · JMIR mhealth and uhealth · doi:10.2196/10076 · Cited by 78
“We also performed a trim and fill analysis [94] and the Orwin Failsafe-N test to assess whether the effect might be an artifact of bias [95].”
MethodChen, J., Eslick, G. D., & Weltman, M. (2013) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.12563 · Cited by 76
“Publication bias is generally regarded as a concern if the fail-safe number is less than 5n + 10, with n being the number of studies included in the meta-analysis.(14) All analyses were performed with Comprehensive Meta-analysis (version 2.”
MethodCognitive functioning in Alzheimer’s and vascular dementia: A meta-analysis.
Mathias, J. L., & Burke, J. R. (2009) · Neuropsychology · doi:10.1037/a0015384 · Cited by 77
Effectiveness of internet-based affect induction procedures: A systematic review and meta-analysis.
Ferrer, R. A., Grenen, E. G., & Taber, J. M. (2015) · Emotion · doi:10.1037/emo0000035 · Cited by 87
User Manual for Meta-Essentials: Workbooks for Meta-AnalysisPreprint
van Rhee, H., Suurmond, R., & Hak, T. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3241355 · Cited by 76
“Orwin (1983) uses a slightly different approach by looking at effect sizes rather than at p-values.”
BackgroundMethodThe role of technology-facilitated peer assessment and supporting strategies: a meta-analysis
Zheng, L., Zhang, X., & Cui, P. (2020) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2019.1644603 · Cited by 75
“The present study adopted the classic fail-safe N proposed by Rosenthal (1979) and Orwin ’ s fail-safe N (Orwin 1983) to evaluate publication bias.”
MethodTamhane, U. U., Chetcuti, S. J., Hameed, I., Grossman, P. M., Moscucci, M. M., & Gurm, H. S. (2010) · BMC Cardiovascular Disorders · doi:10.1186/1471-2261-10-10 · Cited by 75
“We also calculated fail-safe N, using Rosenberg’s and Orwin’s method [30-32].”
MethodTamhane, U. U., Meier, P., Chetcuti, S. J., Chen, K., Rha, S., Grossman, P. M., & Gurm, H. S. (2009) · EuroIntervention · doi:10.4244/v5i3a60 · Cited by 75
Meta-Analytic Review of Sport Consumption: Factors Affecting Attendance to Sporting Events
Kim, Y., Magnusen, M., Kim, M., & Lee, H. (2019) · Sport Marketing Quarterly · doi:10.32731/smq.283.092019.01 · Cited by 76
McCary, M. A., & Schmitz, O. J. (2021) · Journal of Animal Ecology · doi:10.1111/1365-2656.13489 · Cited by 75
“Rosenthal's fail- safe number gives the number of missing case studies with non- significant results that would be necessary to nullify the combined effect size (Orwin, 1983).”
BackgroundMethodBauer, T. N., Erdoğan, B., Ellis, A. M., Truxillo, D. M., Brady, G. M., & Bodner, T. (2024) · Journal of Management · doi:10.1177/01492063241277168 · Cited by 104
Mathias, J. L., & Wheaton, P. (2015) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2015.06.001 · Cited by 75
“, 1999), the Nfs calculations were based on the number of studies that contributed to each effect size (Orwin, 1983), rather than the total number of studies that were meta-analysed (which yields higher Nfs statistics), because measures of BR/CR and specific outcomes have not been examined with equal frequency.”
MethodChan, R. C. K., Xu, T., Heinrichs, R. W., Yu, Y., & Gong, Q. (2009) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2009.11.012 · Cited by 76
“A fail-safe statistic estimates the number of insignificant, unpublished studies that would need to be added to a metaanalysis to reduce a mean observed effect to some specified and negligible level (Orwin, 1983; Rosenberg, 2005).”
Preemptive Analgesia at the Crossroad
Kissin, I. (2005) · Anesthesia & Analgesia · doi:10.1213/01.ane.0000144429.39988.9b · Cited by 76
Hypnosis and the Alleviation of Clinical Pain: A Comprehensive Meta-Analysis
Milling, L. S., Valentine, K. E., LoStimolo, L. M., Nett, A. M., & McCarley, H. S. (2021) · International Journal of Clinical and Experimental Hypnosis · doi:10.1080/00207144.2021.1920330 · Cited by 74
“To address the possibility of the file-drawer effect, a fail-safe N was calculated using the approach of Orwin (1983).”
MethodLarsen, S. E., & Pacella‐LaBarbara, M. L. (2016) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2016.01.001 · Cited by 88
Howard, M. C., & Zandt, E. C. V. (2020) · Journal of Research in Personality · doi:10.1016/j.jrp.2020.103982 · Cited by 85
“Failsafe k indicates the number of undiscovered studies with a null effect that would need to be included for an effect’s p-value to exceed 0.05 (Rosenberg, 2005; Orwin, 1983).”
BackgroundInfluentialEdens, J. F., & Campbell, J. S. (2007) · Psychological Services · doi:10.1037/1541-1559.4.1.13 · Cited by 74
Piras, F., Piras, F., Orfei, M. D., Caltagirone, C., & Spalletta, G. (2015) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2015.10.002 · Cited by 74
Kristanti, M. S., Setiyarini, S., & Effendy, C. (2017) · BMC Palliative Care · doi:10.1186/s12904-016-0178-4 · Cited by 74
“8 were considered small, moderate, and high, respectively [16].”
BackgroundMethodTurner, J. K., Hutchinson, A. D., & Wilson, C. J. (2017) · Psycho-Oncology · doi:10.1002/pon.4577 · Cited by 79
Bem, D., Tressoldi, P. E., Rabeyron, T., & Duggan, M. (2016) · F1000Research · doi:10.12688/f1000research.7177.2 · Cited by 84
“An alternative approach for estimating a Fail-Safe N focuses on the effect size rather than the p value (Orwin, 1983).”
BackgroundPerineal Talc Use and Ovarian Cancer
Penninkilampi, R., & Eslick, G. D. (2017) · Epidemiology · doi:10.1097/ede.0000000000000745 · Cited by 73
Martin, F., Sun, T., Turk, M., & Ritzhaupt, A. D. (2021) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v22i3.5263 · Cited by 92
“Finally, we used Orwin’s fail-safe N (Orwin, 1983), which assists in computing the number of missing studies needed to bring the summary effect to a level below the specified value other than zero.”
Long-term effects of prevention and treatment on youth antisocial behavior: A meta-analysis
Sawyer, A. M., Borduin, C. M., & Dopp, A. R. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.06.009 · Cited by 78
Updated Meta-Analysis of Learner Control Within Educational Technology
Karich, A. C., Burns, M. K., & Maki, K. E. (2014) · Review of Educational Research · doi:10.3102/0034654314526064 · Cited by 73
“Because only published studies were used in the meta-analysis, a failsafe N was computed (Orwin, 1983).”
MethodMeta-analysis of academic interventions derived from neuropsychological data.
Burns, M. K., Petersen‐Brown, S., Haegele, K., Rodriguez, M., Schmitt, B. A., Cooper, M., Clayton, K. L., Hutcheson, S., Conner, C., Hosp, J. L., et al. (2015) · School Psychology Quarterly · doi:10.1037/spq0000117 · Cited by 73
Family-based treatments for serious juvenile offenders: A multilevel meta-analysis.
Dopp, A. R., Borduin, C. M., White, M., & Kuppens, S. (2017) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000183 · Cited by 81
A meta-analytic study of the impact of Lean Production on business performance
Abreu-Ledón, R., Luján-García, D., Garrido-Vega, P., & Escobar-Pérez, B. (2018) · International Journal of Production Economics · doi:10.1016/j.ijpe.2018.03.015 · Cited by 78
Silva Pinto, B. G., Marques Soares, T. K., Azevedo Linhares, M., & Castilhos Ghisi, N. (2020) · Science of The Total Environment · doi:10.1016/j.scitotenv.2020.141382 · Cited by 72
Batastini, A. B., King, C. M., Morgan, R. D., & McDaniel, B. (2016) · Psychological Services · doi:10.1037/ser0000042 · Cited by 72
Xu, X., Liu, W., & Pang, W. (2019) · Sustainability · doi:10.3390/su11216123 · Cited by 90
“Then, we conducted the Orwin’s fail-safe N analysis [127] and the trim-and-fill analysis [128] with the R package metafor [129], based on the aggregated 75 effect sizes (one effect size per study).”
MethodHigher taxa are sufficient to represent biodiversity patterns
de Oliveira, S. S., Ortega, J. C. G., Ribas, L. G. D. S., Lopes, V. G., & Bini, L. M. (2020) · Ecological Indicators · doi:10.1016/j.ecolind.2019.105994 · Cited by 79
Kline, A. C., Cooper, A. A., Rytwinski, N. K., & Feeny, N. C. (2020) · Behavior Therapy · doi:10.1016/j.beth.2020.04.015 · Cited by 77
“Publication bias was probed by calculating the fail-safe N (Rosenthal, 1979) as adapted by Orwin (1983).”
MethodChan, J. Y. C., Hirai, H. W., & Tsoi, K. K. F. (2015) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2015.05.010 · Cited by 73
Gurm, H. S., Tamhane, U. U., Meier, P., Grossman, P. M., Chetcuti, S. J., & Bates, E. (2009) · Circulation Cardiovascular Interventions · doi:10.1161/circinterventions.108.847996 · Cited by 72
Claxton, S. E., Bishop, H. K. D., & van Dulmen, M. H. M. (2015) · Archives of Sexual Behavior · doi:10.1007/s10508-014-0392-1 · Cited by 74
A Meta-Analysis of Mindfulness-Based Therapies for Insomnia and Sleep Disturbance
Rash, J. A., Kavanagh, V. A. J., & Garland, S. N. (2019) · Sleep Medicine Clinics · doi:10.1016/j.jsmc.2019.01.004 · Cited by 70
Slone, M., Lavi, I., Ozer, E. J., & Pollak, A. (2017) · Children and Youth Services Review · doi:10.1016/j.childyouth.2017.01.019 · Cited by 96
Student satisfaction with courses and instructors in a flipped classroom: A meta‐analysis
Strelan, P., Osborn, A. J., & Palmer, E. (2020) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12421 · Cited by 71
A Study on Causes of Delay in Road Construction Projects across 25 Developing Countries
Rivera, L., Baguec, H., & Yeom, C. (2020) · Infrastructures · doi:10.3390/infrastructures5100084 · Cited by 71
“Furthermore, in several cases, the application of pure statistical research for peer-reviewed data has proven the existence of bias against statistically irrelevant results [40].”
BackgroundRitzhaupt, A. D., Huang, R., Sommer, M., Zhu, J., Stephen, A., Valle, N., Hampton, J. D., & Li, J. (2021) · Educational Technology Research and Development · doi:10.1007/s11423-021-10036-1 · Cited by 83
“Third, Orwin’s fail-safe N was calculated, which determines the number of missing null studies required to bring the existing effect size to a trivial level (Orwin, 1983).”
BackgroundEffectiveness of conceptual change strategies in science education: A meta‐analysis
Paçacı, Ç., Üstün, U., & Özdemir, Ö. F. (2023) · Journal of Research in Science Teaching · doi:10.1002/tea.21887 · Cited by 111
Prevalence of energy drink consumption world‐wide: A systematic review and meta‐analysis
Diego, G. A., Krotter, A., & Pérez, Á. G. (2023) · Addiction · doi:10.1111/add.16390 · Cited by 84
Gutierrez, G., Stephenson, C., Eadie, J., Asadpour, K., & Alavi, N. (2024) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2024.1356773 · Cited by 71
“These techniques show the number of hypothetical null results that would be required to make the results of the analysis insigni fi cant (56).”
BackgroundXu, T., Chan, R. C. K., & Compton, M. T. (2011) · PLoS ONE · doi:10.1371/journal.pone.0024129 · Cited by 71
“The Orwin’s fail-safe number estimates the number of non-significant, unpublished studies that would overturne an overall statistically significant observed effect to some specified and negligible level [17,18].”
BackgroundChing, C., Eslick, G. D., & Poulton, A. S. (2019) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2019.0905 · Cited by 70
Bernard, R. M., Borokhovski, E., Schmid, R. F., Waddington, D. I., & Pickup, D. I. (2019) · Campbell Systematic Reviews · doi:10.1002/cl2.1017 · Cited by 110
“Publication bias was assessed based on the examination of a Forest Plot and associated “Trim and Fill” analysis plus other tests such as Classic Fail‐safe analysis and Orwin's Fail‐safe N (Orwin, 1983).”
InfluentialGuo, S., Deng, W., Wang, H., Liu, J., Liu, X., Yang, X., He, C., Zhang, Q., Liu, B., Dong, X., et al. (2020) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2528 · Cited by 83
Luo, X., Zhang, L., Lin, Y., Wen, D., & Hou, E. (2023) · Soil Biology and Biochemistry · doi:10.1016/j.soilbio.2023.109158 · Cited by 72
Depression in Multiple Sclerosis: A Quantitative Review of the Evidence.
Dalton, E. J., & Heinrichs, R. W. (2005) · Neuropsychology · doi:10.1037/0894-4105.19.2.152 · Cited by 68
Myles, N., Newall, H. D., Nielssen, O. B., & Large, M. M. B. (2012) · Current Pharmaceutical Design · doi:10.2174/138161212802884816 · Cited by 70
Factors modifying drug and placebo responses in randomized trials for bipolar mania
Yıldız, A., Vieta, E., Tohen, M. F., & Baldessarini, R. J. (2011) · The International Journal of Neuropsychopharmacology · doi:10.1017/s1461145710001641 · Cited by 69
“10) ; larger fail-safe N indicates less likelihood of publication bias (Orwin, 1983).”
BackgroundPenninkilampi, R., Brothers, H. M., & Eslick, G. D. (2016) · Journal of Neuroimmune Pharmacology · doi:10.1007/s11481-016-9722-5 · Cited by 71
Scott, A. J., Webb, T. L., & Rowse, G. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-016873 · Cited by 72
The Trauma and Mental Health Impacts of Coercive Control: A Systematic Review and Meta-Analysis
Lohmann, S., Cowlishaw, S., Ney, L., O’Donnell, M., & Felmingham, K. (2024) · Trauma, Violence, & Abuse · doi:10.1177/15248380231162972 · Cited by 107
Emotional Dysregulation and Nonsuicidal Self-Injury: A Meta-Analytic Review
You, J., Ren, Y., Zhang, X., Wu, Z., Xu, S., & Lin, M. P. (2018) · Neuropsychiatry · doi:10.4172/neuropsychiatry.1000399 · Cited by 66
“To test for publication bias, Orwin’s fail-safe N [37] was used.”
Mindfulness and Athlete Burnout: A Systematic Review and Meta-Analysis
Li, C., Zhu, Y., Zhang, M., Gustafsson, H., & Chen, T. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16030449 · Cited by 69
“Publication bias was addressed using the analysis of “fail-safe number” for effect size [32,33].”
MethodSales, J., & Glencross, B. D. (2010) · Aquaculture Nutrition · doi:10.1111/j.1365-2095.2010.00761.x · Cited by 69
Dexter, D. D., Park, Y. J., & Hughes, C. A. (2011) · Learning Disabilities Research and Practice · doi:10.1111/j.1540-5826.2011.00341.x · Cited by 69
“To counter this potential bias, we used Rosenthal’s (1979) fail-safe N statistic (adapted by Orwin, 1983) and determined that over 40 unpublished studies with null effects would be required to bring our reported mean effect to zero.”
MethodNeelam, K., Garg, D. K., & Marshall, M. (2011) · BMC Psychiatry · doi:10.1186/1471-244x-11-139 · Cited by 69
“The Orwin fail safe N estimates the number of unpublished studies that would be required to shift the effect size towards a null result [32].”
BackgroundTressoldi, P. E. (2011) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00117 · Cited by 69
“(submitted); *one study excluded because N participants = 1; § = Darlington and Hayes’s (2000) formula; # = Orwin’s (1983) fail-safe N.”
Mahoney, A., Karatzias, T., & Hutton, P. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.09.059 · Cited by 81
Kavale, K. A., & Nye, C. (1985) · The Journal of Special Education · doi:10.1177/002246698501900408 · Cited by 69
Addressing chronic absenteeism in schools: A meta-analysis of evidence-based interventions.
Eklund, K., Burns, M. K., Oyen, K., DeMarchena, S., & McCollom, E. M. (2022) · School Psychology Review · doi:10.1080/2372966x.2020.1789436 · Cited by 79
“Fail-safe N (Orwin, 1983) was also computed to determine the potential effect of missing studies on estimates of effect for individual variables.”
MethodWhat Effects Do Didactic Interventions Have on Students’ Attitudes Towards Science? A Meta-Analysis
Aguilera, D., & Palacios, F. J. P. (2018) · Research in Science Education · doi:10.1007/s11165-018-9702-2 · Cited by 68
Garrido, V., & Morales, L. A. (2007) · Campbell Systematic Reviews · doi:10.4073/csr.2007.7 · Cited by 69
A review of meta-analytic research in accounting
Khlif, H., & Chalmers, K. (2015) · Journal of Accounting Literature · doi:10.1016/j.acclit.2015.09.001 · Cited by 68
Fraguas, D., Díaz‐Caneja, C. M., Pina‐Camacho, L., Janssen, J., & Arango, C. (2014) · Schizophrenia Research · doi:10.1016/j.schres.2014.12.022 · Cited by 70
“In addition, we used Orwin’s fail-safe N (Orwin, 1983).”
Sentinel node mapping in the prostate cancer
Tabasi, K. T., Bazaz, S. M. M., Kakhki, V. R. D., Massoom, A. F., Gholami, H., Zakavi, S. R., & Sadeghi, R. (2011) · Nuklearmedizin - NuclearMedicine · doi:10.3413/nukmed-0339-10-07 · Cited by 68
Meta-Analysis of Personality Assessments as Predictors of Military Aviation Training Success
Campbell, J. S., Castaneda, M., & Pulos, S. M. (2009) · International Journal of Aviation Psychology · doi:10.1080/10508410903415872 · Cited by 68
Absence of malice: A meta-analysis of nonresponse bias in counterproductive work behavior research.
Greco, L., O’Boyle, E. H., & Walter, S. L. (2014) · Journal of Applied Psychology · doi:10.1037/a0037495 · Cited by 73
Folate intake and the risk of upper gastrointestinal cancers: A systematic review and meta‐analysis
Tio, M., Andrici, J., Cox, M. R., & Eslick, G. D. (2013) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.12446 · Cited by 68
“However, both studies similarly reported no association between total folate intake or plasma folate levels and the risk of gastric cancer.35,52 A significant limitation of our result is that the majority of studies did not control for Helicobacter pylori, a major risk factor for the 0.1 0.2 0.5 1 2 5 10 Protective Ha…”
BackgroundInfluentialA Meta-Analysis of Hypnotic Interventions for Depression Symptoms: High Hopes for Hypnosis?
Milling, L. S., Valentine, K. E., McCarley, H. S., & LoStimolo, L. M. (2019) · American Journal of Clinical Hypnosis · doi:10.1080/00029157.2018.1489777 · Cited by 67
“To address this source of publication bias, a fail-safe N was calculated separately for our post and follow-up results using the approach of Orwin (1983). The fail-safe N is the number of studies with an effect size of 0 needed to reduce a large mean weighted effect to one that is medium or small.”
MethodEfficacy of eHealth Interventions for Adults with Diabetes: A Systematic Review and Meta-Analysis
Bassi, G., Mancinelli, E., Dell’Arciprete, G., Rizzi, S., Gabrielli, S., & Salcuni, S. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18178982 · Cited by 67
“Publication bias was assessed by visually inspecting the funnel plot and through the Egger’s regression test [28,29]; when publication bias emerged, the trim and fill procedures and the fail-safe number [30] were calculated, thereby evaluating whether results remained unchanged when accounting for publication bias.”
MethodLee, C. S., Westland, H., Faulkner, K. M., Iovino, P., Thompson, J. H., Sexton, J., Farry, E., Jaarsma, T., & Riegel, B. J. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2022.104322 · Cited by 80
“Orwin's N was calculated under different assumptions to estimate the stability of random effects estimates with the addition of additional studies (Orwin, 1986).”
MethodBack, L., Nagaraja, V., Kapur, A., & Eslick, G. D. (2015) · Internal Medicine Journal · doi:10.1111/imj.12724 · Cited by 67
Topa, G., Domínguez, J. F. M., & Depolo, M. (2008) · The Spanish Journal of Psychology · doi:10.1017/s113874160000425x · Cited by 67
“According to Orwin (1983), the “tolerance index of null results” should be calculated and there must be more than 300 unpublished studies (and not recovered by the meta-analyst) for the results to be annulled.”
BackgroundRisk Markers for Physical Teen Dating Violence Perpetration: A Meta-Analysis
Spencer, C. M., Toews, M. L., Anders, K. M., & Emanuels, S. K. (2019) · Trauma Violence & Abuse · doi:10.1177/1524838019875700 · Cited by 79
Schiller, V. F., Dorstyn, D. S., & Taylor, A. M. (2021) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-020-04601-5 · Cited by 66
Fostering scientific reasoning in education – meta-analytic evidence from intervention studies
Engelmann, K., Neuhaus, B. J., & Fischer, F. (2016) · Educational Research and Evaluation · doi:10.1080/13803611.2016.1240089 · Cited by 87
“The possibility of a publication bias was addressed by calculating the Fail-Safe N for effect size (Orwin, 1983), which resulted in the need for 75 additional studies with an average effect size of g = 0 to lower the mean effect of the meta-analysis to a small (Cohen, 1988) mean effect of g = 0.2.”
MethodInfluential
Metadata
- Authors
- Orwin, R. G.
- Affiliations
- United States Department of Health and Human Services (US)
- Corresponding author
- Orwin, Robert G.
- Venue
- Journal of Educational Statistics
- Publisher
- JSTOR
- Published
- 1983
- Volume
- 8
- Issue
- 2
- Pages
- 157
- ISSN
- 0362-9791 (ISSN-L)
- OpenAlex
- W2112432954
- Cited by
- 2,064
- References
- 11
- Citations by year
- 2026: 51 · 2025: 75 · 2024: 91 · 2023: 121 · 2022: 114 · 2021: 129 · 2020: 129 · 2019: 120 · 2018: 116 · 2017: 109 · 2016: 95 · 2015: 113 · 2014: 108 · 2013: 76 · 2012: 88
- FWCI
- 7.68
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Educational Statistics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- effect size, file drawer problem, sampling bias, negative results, meta-analysis, publication bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
