The DOI matches its Crossref record; 10 of the 11 references it cites were found in a registry or an index.
A Fail-Safe N for Effect Size in Meta-Analysis
Abstract
Rosenthan's (1979) concept of fail-safe N has thus far been applied to probability levels exclusively. This note introduces a fail-safe TV for effect size. Rosenthal's (1979) fail-safe N was an ingenious response to the so-called file drawer problem in research integration, in that it provided a direct assessment of the threat posed by sampling bias in the literature search. The algorithm was derived from the Stouffer method (Mosteller & Bush, 1954), in which an overall Z-score is computed by summing individual Z-scores and dividing by the square root of the number of scores. A significant overall Z-score, Rosenthal reasoned, could be made nonsignificant by adding some knowable number of hypothetical studies that averaged null results. The exact number of such studies needed to bring a significant overall p level up to some critical level (e.g., .05) has since become known as the fail-safe N (Nfs). The formula is
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“we also calculated the classic fail-safe N test (Orwin, 1983), which revealed that it would take 59 studies reporting null results (g = 0) to be included in our analyses in order for our findings to lose statistical significance.”
From Improving fluid intelligence with training on working memory: a meta-analysis (2015) · cites this paper as Orwin, 1983
In the citing paper
“The file drawer problem (publication bias) was examined by creating a funnel plot and calculating the fail safe N (45).”
From A Meta-Analysis of the Corpus Callosum in Autism (2009) · cites this paper as [45]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890
Smith, M. L., Glass, G. V., & Miller, T. I. (1980) · Medical Entomology and Zoology · Cited by 2,123
Johnson, D. W., Maruyama, G., Johnson, R., Nelson, D., & Skon, L. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.89.1.47 · Cited by 1,588
Interpersonal expectancy effects: the first 345 studies
Rosenthal, R., & Rubin, D. B. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00075506 · Cited by 1,158
Eagly, A. H., & Carli, L. L. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.90.1.1 · Cited by 541
Cooper, H. M. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.1.131 · Cited by 418
Arkin, R. M., Cooper, H., & Kolditz, T. A. (1980) · Journal of Personality · doi:10.1111/j.1467-6494.1980.tb02378.x · Cited by 88
Flagged first, then by citations
1 of the 11 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Items 901–1,000 of 1,020
Perpetrator Risk Markers for Intimate Terrorism and Situational Couple Violence: A Meta-Analysis
Love, H., Spencer, C. M., May, S., Méndez, M., & Stith, S. (2018) · Trauma, Violence, & Abuse · doi:10.1177/1524838018801331 · Cited by 63
“Lastly, we utilized Orwin’s fail–safe Ns to estimate the number of possible missing studies where r ¼ .00 needed to reduce the mean effect size (per correlate) down to r¼ .10 (the lower limit of small effect sizes; Cohen, 1992; Orwin, 1983).”
MethodO’Donnell, K., & delBarco‐Trillo, J. (2020) · Journal of Urban Ecology · doi:10.1093/jue/juaa014 · Cited by 65
“In this case the fail-safe N estimated the number of additional studies that would be required to find no differences in home range size between natural and urban environments (Orwin 1983).”
BackgroundTagliabue, M., Sigurjonsdottir, S. S., & Sandaker, I. (2020) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2020.1796647 · Cited by 101
Evoking stress reactivity in virtual reality: A systematic review and meta-analysis
van Dammen, L., Finseth, T., McCurdy, B. H., Barnett, N. P., Conrady, R. A., Leach, A. G., Deick, A. F., Steenis, A. L. V., Gardner, R., Smith, B. L., et al. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104709 · Cited by 78
Diehle, J., Schmitt, K., Daams, J. G., Boer, F., & Lindauer, R. (2014) · Journal of Traumatic Stress · doi:10.1002/jts.21924 · Cited by 65
“We calculated the fail-safe N with the formula as proposed by Orwin (1983).”
MethodKarl, A., & Werner, A. (2009) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2009.06.008 · Cited by 67
“Cho/Cr problem, Orwin’s fail safe N (Orwin, 1983) was computed.”
MethodMcMorris, T., Hale, B. J., Corbett, J., Robertson, K. M., & Hodgson, C. I. (2015) · Physiology & Behavior · doi:10.1016/j.physbeh.2015.01.010 · Cited by 69
Mo, J.-A., Lim, J., Sul, A.-R., Lee, M. Y., Youn, Y. C., & Kim, H.-J. (2015) · PLoS ONE · doi:10.1371/journal.pone.0116802 · Cited by 67
Heightened susceptibility to secondary traumatization: A meta-analysis of gender differences.
Baum, N., Rahav, G., & Sharon, M. (2014) · American Journal of Orthopsychiatry · doi:10.1037/h0099383 · Cited by 86
Kwak, B. O., Kim, K., Kim, S., & Lee, R. (2017) · Seizure · doi:10.1016/j.seizure.2017.09.009 · Cited by 77
Qorri, A., Gashi, S., & Krasławski, A. (2020) · Sustainable Development · doi:10.1002/sd.2140 · Cited by 65
CBT for Children with Depressive Symptoms: A Meta-Analysis
Arnberg, A., & Öst, L. (2014) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2014.947316 · Cited by 65
Guérin, E., Bales, E., Sweet, S. N., & Fortier, M. (2012) · Canadian Psychology/Psychologie canadienne · doi:10.1037/a0030215 · Cited by 74
“The fail-safe N refers to the required number of unpublished studies that would be needed to reduce the overall effect size to nonsignificant levels (Orwin, 1983).”
BackgroundHow to read and understand and use systematic reviews and meta‐analyses
Leucht, S., Kissling, W., & Davis, J. M. (2009) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.2009.01388.x · Cited by 66
“So-called fail-safe methods estimate how many unpublished studies with negative results must exist in order that a meta-analytical result be no longer significant or the effect size take on a trivial value (16).”
BackgroundDo antipsychotics prevent postoperative delirium? A systematic review and meta‐analysis
Fok, M. C., Sepehry, A. A., Frisch, L., Sztramko, R., van der Burg, B. L. S. B., Vochteloo, A. J. H., & Chan, P. (2015) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.4240 · Cited by 64
Meta‐analysis of post‐release fishing mortality in apex predatory pelagic sharks and white marlin
Musyl, M. K., & Gilman, E. L. (2019) · Fish and Fisheries · doi:10.1111/faf.12358 · Cited by 70
“…(Kendall's tau with continuity correction), Egger's regression intercept method (Egger, Davey Smith, Schneider & Minder, 1997), Orwin's fail-safe N (Orwin, 1983), Duval and Tweedie's (2000a, 2000b) “trim and fill” and funnel plots were used to test for publication bias using Comprehensive…”
A comparison of oral midazolam and nitrous oxide sedation for dental extractions in children
Wilson, K. E., Girdler, N. M., & Welbury, R. R. (2006) · Anaesthesia · doi:10.1111/j.1365-2044.2006.04835.x · Cited by 67
Leopold, R., & Backenstrass, M. (2015) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2014.12.016 · Cited by 63
Meta-analysis in vocational behavior: A systematic review and recommendations for best practices
Rudolph, C. W., Chang, C. K., Rauvola, R. S., & Zacher, H. (2020) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2020.103397 · Cited by 67
Moral injury and mental health among health-care workers during the COVID-19 pandemic: meta-analysis
Coimbra, B. M., Zylberstajn, C., van Zuiden, M., Hoeboer, C. M., Mello, A. F., Mello, M. F., & Olff, M. (2024) · European Journal of Psychotraumatology · doi:10.1080/20008066.2023.2299659 · Cited by 68
“Lastly, if the results of the meta-analyses were statistically signi fi cant, we used Rosenberg ’ s, Rosenthal ’ s, and Orwin ’ s fail-safe tests to assess the robustness of the results (Orwin, 1983; Rosenberg, 2005; Rosenthal, 1979).”
MethodLin, H. T., Chen, H., Liu, Q., Xu, J., & Li, S. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1305425 · Cited by 64
“The first is by calculating the fail-safe N to determine the number of additional non-significant studies needed for the overall effect value to become insignificant (Orwin, 1983).”
BackgroundSohl, S. J., Schnur, J. B., & Montgomery, G. H. (2009) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2009.01.008 · Cited by 67
“The fail-safe n (26,27) was calculated with Meta 5.”
MethodH ippocampal neurogenesis and pattern separation: A meta‐analysis of behavioral data
França, T. F. A., Bitencourt, A. M., Maximilla, N. R., Barros, D. M., & Monserrat, J. M. (2017) · Hippocampus · doi:10.1002/hipo.22746 · Cited by 71
The Efficacy of Hypnotherapy in the Treatment of Psychosomatic Disorders: Meta-analytical Evidence
Flammer, E., & Alladin, A. (2007) · International Journal of Clinical and Experimental Hypnosis · doi:10.1080/00207140701338696 · Cited by 63
“To address these possible sources of bias, a fail-safe N (Orwin, 1983) was determined.”
BackgroundDumb money or smart money? Meta‐analytically unpacking corporate venture capital
Huang, P., & Madhavan, R. (2020) · Strategic Entrepreneurship Journal · doi:10.1002/sej.1369 · Cited by 63
Alemdağ, E. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-11154-w · Cited by 63
God representations and aspects of psychological functioning: A meta-analysis
Stulp, H. P., Koelen, J. A., Schep‐Akkerman, A., Glas, G. G., & Eurelings‐Bontekoe, L. (2019) · Cogent Psychology · doi:10.1080/23311908.2019.1647926 · Cited by 71
Ramamoorthi, R., Gahreman, D. E., Skinner, T. C., & Moss, S. A. (2019) · PLoS ONE · doi:10.1371/journal.pone.0221067 · Cited by 66
Personality, intelligence, and counterproductive academic behaviors: A meta-analysis
Cuadrado, D., Salgado, J. F., & Moscoso, S. (2021) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000285 · Cited by 63
DOMINGUES, A. R., Pires, S. M., Halasa, T., & Hald, T. (2011) · Epidemiology and Infection · doi:10.1017/s0950268811002172 · Cited by 64
“When significant publication bias and change in the estimated pooled ORs were detected, the number of studies necessary to reverse the overall pooled effect was calculated using Orwin’s fail-safe N method [22].”
MethodRisk Markers for Physical Teen Dating Violence Victimization in the United States: A Meta-Analysis
Spencer, C. M., Anders, K. M., Toews, M. L., & Emanuels, S. K. (2020) · Journal of Youth and Adolescence · doi:10.1007/s10964-020-01194-1 · Cited by 71
“05; Orwin’s fail-safe N (Orwin 1983), which determines the number of potential studies with zero correlation between variables it would take to create a small effect where r < 0.”
BackgroundEntrepreneurial alertness: A meta-analysis and empirical review
Araújo, C. F., Karami, M., Tang, J., Roldan, L. B., & dos Santos, J. A. (2023) · Journal of Business Venturing Insights · doi:10.1016/j.jbvi.2023.e00394 · Cited by 62
Zheng, J., Zhou, M., Zhu, B., Fan, J., Lin, H., Ren, B., & Zhang, F. (2023) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2023.163804 · Cited by 62
IQ change over time in schizophrenia and healthy individuals: A meta-analysis
Hedman, A. M., van Haren, N. E. M., van Baal, C. G. C. M., Kahn, R. S., & Pol, H. E. H. (2013) · Schizophrenia Research · doi:10.1016/j.schres.2013.01.027 · Cited by 62
“To examine the possibility of publication bias, we computed a fail-safe number of studies (Orwin, 1983; Rosenthal, 1991; Lipsey and Wilson, 2001).”
MethodIll-fitting dentures and oral cancer: A meta-analysis
Manoharan, S., Nagaraja, V., & Eslick, G. D. (2014) · Oral Oncology · doi:10.1016/j.oraloncology.2014.08.002 · Cited by 62
“Publication bias is generally regarded as a concern if the fail-safe number is less than 5n + 10, with n being the number of studies included in the meta-analysis [18].”
BackgroundThe Impact of Vocal and Laryngeal Pathologies Among Professional Singers: A Meta-analysis
Kwok, M., & Eslick, G. D. (2018) · Journal of Voice · doi:10.1016/j.jvoice.2017.09.002 · Cited by 62
“Publication bias is generally regarded as a concern if the failsafe number is less than 5n + 10, with n being the number of studies included in the meta-analysis.(13) All analyses were performed with Comprehensive Meta-analysis version 3.”
MethodAllott, K., van‐der‐EL, K., Bryce, S., Parrish, E. M., McGurk, S. R., Hetrick, S. E., Bowie, C. R., Kidd, S. A., Hamilton, M. P., Killackey, E. J., et al. (2019) · Schizophrenia Bulletin · doi:10.1093/schbul/sbz134 · Cited by 62
Çetin, Ş., Gürbüz, S., & Sert, M. (2015) · Employee Responsibilities and Rights Journal · doi:10.1007/s10672-015-9266-5 · Cited by 74
Lai, L., Liu, Y., McCracken, L. M., Li, Y., & Ren, Z. (2023) · Behaviour Research and Therapy · doi:10.1016/j.brat.2023.104308 · Cited by 60
Bragdon, L. B., Gibb, B. E., & Coles, M. E. (2018) · Depression and Anxiety · doi:10.1002/da.22785 · Cited by 57
“Publication bias was assessed by calculation of Egger's regression intercept examination of the funnel plot and calculation of Orwin's fail-safeN (Orwin, 1983).”
MethodPetsch, D. K., Bertoncin, A. P. D. S., Ortega, J. C. G., & Thomaz, S. M. (2022) · Oikos · doi:10.1111/oik.08768 · Cited by 69
Pedagogical strategies for teaching literacy to ESL immigrant students: A meta‐analysis
Adesope, O., Lavin, T., Thompson, T. A., & Ungerleider, C. (2011) · British Journal of Educational Psychology · doi:10.1111/j.2044-8279.2010.02015.x · Cited by 63
A Meta-Analysis of Hypnosis in the Treatment of Depressive Symptoms:A Brief Communication
Shih, M., Yang, Y.-H., & Koo, M. (2009) · International Journal of Clinical and Experimental Hypnosis · doi:10.1080/00207140903099039 · Cited by 61
“To address this limitation, a fail-safe N, that is, the number of studies with effect sizes of zero that would be required to reduce the present effect size to zero, was calculated (Orwin, 1983).”
BackgroundImpulsivity in remitted depression: A meta-analytical review
Saddichha, S., & Schuetz, C. (2014) · Asian Journal of Psychiatry · doi:10.1016/j.ajp.2014.02.003 · Cited by 61
Youngstrom, E. A., Genzlinger, J. E., Egerton, G. A., & Meter, A. V. (2015) · Archives of Scientific Psychology · doi:10.1037/arc0000024 · Cited by 68
Yeun, Y. R., & Han, S. J. (2016) · Healthcare Informatics Research · doi:10.4258/hir.2016.22.3.217 · Cited by 61
“Because so few papers have been written on this topic, it could be concluded that the effect from the meta-analysis was reliable [16].”
BackgroundA meta-analysis of cross cultural risk markers for intimate partner violence
Mallory, A. B., Dharnidharka, P., Deitz, S. L., Barros-Gomes, P., Cafferky, B. M., Stith, S. M., & Van, K. (2016) · Aggression and Violent Behavior · doi:10.1016/j.avb.2016.08.004 · Cited by 62
Internet financial reporting determinants: a meta-analytic review
Mokhtar, E. S. (2017) · Journal of financial reporting & accounting · doi:10.1108/jfra-07-2016-0061 · Cited by 61
Ruane, A., & Carr, A. (2019) · Family Process · doi:10.1111/famp.12352 · Cited by 59
Lou, S.-Z., Zhang, B., & Zhang, D.-H. (2021) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2021.128817 · Cited by 63
Chen, J. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.1021101 · Cited by 104
“Informed by previous meta-analyses, several statistical analyses, including the funnel plot with trim and fill, Orwin’s fail safe N (Orwin, 1983), a Egger’s correlation test examining the independence of variance and effect sizes (Egger et al., 1997), adding the sample size as a moderator and…”
MethodTlili, A., Garzón, J., Salha, S. H., Huang, R., Xu, L., Burgos, D., Denden, M., Farrell, O., Farrow, R., Bozkurt, A., et al. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00424-3 · Cited by 110
“To overcome this issue, Orwin (1983) proposed a more stringent method to identify how many missing studies would bring the overall effect to a specific non-zero value.”
MethodStreptococcus bovis infection and colorectal neoplasia: a meta‐analysis
Krishnan, S., & Eslick, G. D. (2014) · Colorectal Disease · doi:10.1111/codi.12662 · Cited by 60
Risk factors of gestational diabetes mellitus recurrence: a meta-analysis
Schwartz, N., Nachum, Z., & Green, M. S. (2016) · Endocrine · doi:10.1007/s12020-016-0922-9 · Cited by 64
“In addition, the Fail Safe N was calculated for each risk factor analysis using Orwin’s method [11].”
MethodBourke, E., Barker, C., & Fornells‐Ambrojo, M. (2021) · Psychology and Psychotherapy Theory Research and Practice · doi:10.1111/papt.12330 · Cited by 73
Wilson, M. L. (2021) · Journal of Research on Technology in Education · doi:10.1080/15391523.2021.1950085 · Cited by 69
Skarphedinsson, G., Hanssen-Bauer, K., Kornør, H., Heiervang, E., Landrø, N., Axelsdóttir, B., Biedilae, S., & Ivarsson, T. (2015) · Nordic Journal of Psychiatry · doi:10.3109/08039488.2014.941395 · Cited by 60
Sex differences in dichotic listening
Voyer, D. (2011) · Brain and Cognition · doi:10.1016/j.bandc.2011.02.001 · Cited by 59
Litano, M. L., Major, D. A., Landers, R. N., Streets, V. N., & Bass, B. I. (2016) · The Leadership Quarterly · doi:10.1016/j.leaqua.2016.06.003 · Cited by 66
Drought has negative consequences on aphid fitness and plant vigor: Insights from a meta‐analysis
Leybourne, D. J., Preedy, K. F., Valentine, T. A., Bos, J. I. B., & Karley, A. J. (2021) · Ecology and Evolution · doi:10.1002/ece3.7957 · Cited by 58
“Additionally, as most null results go unpublished, failsafe analysis using Orwin's method (Orwin, 1983) was employed to estimate the number of studies reporting a null effect that would be required to reduce the observed average effect size to −0.1.”
MethodInfluentialWang, J., Zhu, L., Feng, L., & Feng, J. (2023) · Sustainable Production and Consumption · doi:10.1016/j.spc.2023.04.015 · Cited by 58
Consistency of immigrant and country‐of‐birth suicide rates: a meta‐analysis
Voracek, M., & Loibl, L. M. (2008) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.2008.01238.x · Cited by 58
“These included visual inspection of funnel plots (96), Rosenthal s fail-safe N (97), Orwin s failsafe N (98), the Begg–Mazumdar rank correlation test (99), Egger s regression test (100) and the Duval–Tweedie trim-and-fill method (101, 102).”
MethodNagaraja, V., Ooi, S., Nolan, J., Large, A., de Belder, M., Ludman, P., Bagur, R., Curzen, N. P., Matsukage, T., Yoshimachi, F., et al. (2016) · Journal of the American Heart Association · doi:10.1161/jaha.116.004598 · Cited by 58
“Publication bias was assessed using an Egger regression model(27) and the fail-safe number method.(28)”
MethodMeta-Analytic Methods to Detect Publication Bias in Behavior Science Research
Dowdy, A., Hantula, D. A., Travers, J. C., & Tincani, M. (2021) · Perspectives on Behavior Science · doi:10.1007/s40614-021-00303-0 · Cited by 61
Zgoba, K. M., & Mitchell, M. M. (2021) · Journal of Experimental Criminology · doi:10.1007/s11292-021-09480-z · Cited by 68
“Fourth, the Orwin ’ s Fail Safe N (Orwin, 1983) was used and equaled 28.”
Effectiveness of Virtual Laboratories in Science Education: A Meta-Analysis
Santos, M. L., & Prudente, M. S. (2021) · International Journal of Information and Education Technology · doi:10.18178/ijiet.2022.12.2.1598 · Cited by 57
Li, R., Mukherjee, M. B., Jin, Z., Liu, H., Lin, K., Liu, Q., Dilger, J. P., & Lin, J. (2023) · Cancers · doi:10.3390/cancers15102759 · Cited by 57
“Potential publication bias was detected by using the funnel plot and Egger’s regression (statistical significance indicates high probability of publication bias, p < 0.05 is considered significant likelihood of bias) with statistical package provided by Suurmond et al. [16,17].”
InfluentialHow online reviews affect purchase intention: A meta-analysis across contextual and cultural factors
Qiu, K., & Zhang, L. (2023) · Data and Information Management · doi:10.1016/j.dim.2023.100058 · Cited by 151
Funakoshi, N., Duny, Y., Valats, J.-C., Ségalas-Largey, F., Flori, N., Bismuth, M., Daurès, J., & Blanc, P. (2012) · Annals of Hepatology · doi:10.1016/s1665-2681(19)30934-2 · Cited by 57
“Lopez-Acosta, 2002 Mexico Abstract 56 (28/28) 26 (2-48) Gheorghe, 2002 Romania Abstract 53 (28/25) 15”
Wujeska‐Klause, A., Bossinger, G., & Tausz, M. (2013) · Tree Physiology · doi:10.1093/treephys/tpt083 · Cited by 55
“A number of approaches are recommended to address bias: First, the fail-safe number indicates how many additional studies would be needed to reduce the calculated ES to non- significant level (Orwin 1983, Møller and Jennions 2001, Rosenberg 2005).”
BackgroundConsequences of Strategic Performance Measurement Systems: A Meta-Analytic Review
Endrikat, J., Guenther, T. W., & Titus, R. (2019) · Journal of Management Accounting Research · doi:10.2308/jmar-52575 · Cited by 65
Ma, T.-Y., Sun, Y., & Ku, Y. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.01095 · Cited by 57
Harrison, J. L., O’Toole, S. K., Ammen, S., Ahlmeyer, S., Harrell, S. N., & Hernandez, J. L. (2020) · Psychiatry Psychology and Law · doi:10.1080/13218719.2018.1485526 · Cited by 57
“Finally, using the more conservative Orwin’s fail-safe N (Orwin, 1983), a total of 52 studies would raise the odds ratio to over 0.”
BackgroundMethodInfluentialTutal, Ö., & Yazar, T. (2021) · Asia Pacific Education Review · doi:10.1007/s12564-021-09706-9 · Cited by 57
“Then, we tested publication bias by using the classic fail-safe N (Rosenthal, 1979), Orwin’s fail-safe N (Orwin, 1983), the trim-and-fill method (Duval & Tweedie, 2000a, b), and the weight-function model (Vevea & Hedges, 1995), which is recommended because it gives better results in studies with large samples (Vevea &…”
MethodResultInfluentialSupporting learning from text: A meta-analysis on the timing and content of effective feedback
Swart, E. K., Nielen, T. M. J., & Sikkema - de Jong, M. T. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.100296 · Cited by 68
Lapsey, D. S., Campbell, B. A., & Plumlee, B. T. (2022) · Trauma, Violence, & Abuse · doi:10.1177/1524838021991285 · Cited by 58
“Schmidt, 2004), we relied on the classic fail-safe N test (Rosenthal, 1979) and Orwin’s fail-safe N test (Orwin, 1983) to estimate the possibility of publication biases impacting p values and mean effect size results.”
MethodInfluentialA meta-analysis of social cognitive deficits in schizophrenia: Does world region matter?
Weinreb, S., Li, F., & Kurtz, M. M. (2022) · Schizophrenia Research · doi:10.1016/j.schres.2022.04.002 · Cited by 56
Ortega, J. C. G., Thomaz, S. M., & Bini, L. M. (2018) · Oecologia · doi:10.1007/s00442-018-4150-2 · Cited by 55
Golden, H. L., Vosper, J., Kingston, J., & Ellett, L. (2021) · Mindfulness · doi:10.1007/s12671-020-01501-8 · Cited by 55
Do Manual Therapies Help Low Back Pain? A Comparative Effectiveness Meta-analysis
Menke, J. M. (2014) · Spine · doi:10.1097/brs.0000000000000230 · Cited by 56
Contemporary dental photography: selection and application.
Terry, D. A., Snow, S. R., & McLaren, E. A. (2008) · PubMed · Cited by 57
Liu, F., Cui, J., Liu, X., Chen, K. W., Chen, X., & Li, R. (2020) · BMC Complementary Medicine and Therapies · doi:10.1186/s12906-020-02967-8 · Cited by 56
Stability of Personality Across the Life Span: A Meta-Analysis
Bazana, P. G., & Stelmack, R. M. (2004) · Elsevier eBooks · doi:10.1016/b978-008044209-9/50009-9 · Cited by 57
Digital peer assessment in school teacher education and development: a systematic review
Topping, K. J. (2023) · Research Papers in Education · doi:10.1080/02671522.2021.1961301 · Cited by 54
Nicolau, A. G., Candel, O. S., Constantin, T., & Kleingeld, A. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1089797 · Cited by 70
“Secondly, by using Orwin’s fail-safe N method (Orwin, 1983), we found that to find a trivial effect size (<0.”
MethodInfluentialSales, J. (2011) · Animal Feed Science and Technology · doi:10.1016/j.anifeedsci.2011.02.008 · Cited by 54
Conducting a Meta-Analysis of Counseling Outcome Research
Erford, B. T., Savin‐Murphy, J., & Butler, C. (2010) · Counseling Outcome Research and Evaluation · doi:10.1177/2150137809356682 · Cited by 54
“The fail-safe N determines the number of unpublished studies with an effect size of zero (null) that are required to negate the obtained mean effect size, in effect lowering the obtained mean effect size to an overall effect size of .01 (Orwin, 1983).”
BackgroundSynovial Cyst as a Marker for Lumbar Instability: A Systematic Review and Meta-Analysis
Ramhmdani, S., Ishida, W., Perdomo‐Pantoja, A., Witham, T. F., Lo, S.-F. L., & Bydon, A. (2018) · World Neurosurgery · doi:10.1016/j.wneu.2018.10.228 · Cited by 54
A Meta-Analytic Review of Pretrial Research: Risk Assessment, Bond Type, and Interventions
Bechtel, K. A., Holsinger, A. M., Lowenkamp, C. T., & Warren, M. J. (2016) · American Journal of Criminal Justice · doi:10.1007/s12103-016-9367-1 · Cited by 19
“Orwin’s fail safe N requires the researcher to specify two parameters (Orwin, 1983).”
BackgroundChegeni, R., Pallesen, S., McVeigh, J., & Sagoe, D. (2021) · Psychopharmacology · doi:10.1007/s00213-021-05818-7 · Cited by 54
Nagaraja, V., & Eslick, G. D. (2013) · European Journal of Surgical Oncology · doi:10.1016/j.ejso.2013.02.022 · Cited by 53
Browne, J., Wright, A. C., Berry, K., Mueser, K. T., Cather, C., Penn, D. L., & Kurtz, M. M. (2021) · Schizophrenia Research · doi:10.1016/j.schres.2021.04.002 · Cited by 62
“…studies with negative results are less likely to be submitted or published through the peer-review process, we calculated a fail-safe N using the Orwin (1983) method which provides an estimate of the number of studies with null results that would be needed to render the obtained effect size…”
MethodInfluentialSlein, M. A., Bernhardt, J. R., O’Connor, M. I., & Fey, S. B. (2023) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2022.2225 · Cited by 53
Corrigan, J. A., & Schutte, N. S. (2023) · International Journal of Applied Positive Psychology · doi:10.1007/s41042-023-00099-1 · Cited by 53
Pinar, F. I. L., Panergayo, A. A. E., Sagcal, R. R., Acut, D. P., Roleda, L. S., & Prudente, M. S. (2025) · Disciplinary and Interdisciplinary Science Education Research · doi:10.1186/s43031-025-00137-9 · Cited by 111
“…bias assessment using multiple methods, including the Begg-Mazumdar rank correlation test, Egger’s regression test, and various failsafe tests such as Rosenthal’s, Orwin’s, and Fisher’s methods (Begg & Mazumdar, 1994; Egger et al., 1997; Lin & Chu, 2017; Rosenthal, 1979; Orwin, 1983; Fisher, 1932).”
A Meta-analysis of Weight Status and Anxiety in Children and Adolescents
Burke, N. L., & Storch, E. A. (2015) · Journal of Developmental & Behavioral Pediatrics · doi:10.1097/dbp.0000000000000143 · Cited by 53
Tan, X., Li, S., Chang, Y., Fang, C., Liu, H., Zhang, X., & Wang, Y. (2016) · Clinical and Investigative Medicine · doi:10.25011/cim.v39i4.27091 · Cited by 51
Periodontitis and Preeclampsia in Pregnancy: A Systematic Review and Meta-Analysis
Anh, L. Q., Akhter, R., Coulton, K. M., Vo, N. T. N., Duong, L. T. Y., Nong, H. V., Yaacoub, A., Condous, G. S., Eberhard, J., & Nanan, R. K. H. (2022) · Maternal and Child Health Journal · doi:10.1007/s10995-022-03556-6 · Cited by 69
“Publication bias is generally regarded as a concern if the fail-safe number is less than 5n + 10, with n being the number of studies included in the meta-analysis (Orwin, 1983) Publication bias was assessed using Stata (16.1, StataCorp LLC, College Station, TX).”
MethodInfluentialForeign language anxiety and foreign language self-efficacy: a meta-analysis
Zhou, S., Chiu, M. M., Dong, Z., & Zhou, W. (2022) · Current Psychology · doi:10.1007/s12144-022-04110-x · Cited by 74
“We evaluated the risk of publication bias through funnel charts, Egger et al.’s (1997) linear regressions, and the fail-safe number (Nfs, Orwin, 1983).”
MethodTrivasse, H., Webb, T. L., & Waller, G. (2020) · Clinical Psychology Review · doi:10.1016/j.cpr.2020.101887 · Cited by 51
Metadata
- Authors
- Orwin, R. G.
- Affiliations
- United States Department of Health and Human Services (US)
- Corresponding author
- Orwin, Robert G.
- Venue
- Journal of Educational Statistics
- Publisher
- JSTOR
- Published
- 1983
- Volume
- 8
- Issue
- 2
- Pages
- 157
- ISSN
- 0362-9791 (ISSN-L)
- OpenAlex
- W2112432954
- Cited by
- 2,064
- References
- 11
- Citations by year
- 2026: 51 · 2025: 75 · 2024: 91 · 2023: 121 · 2022: 114 · 2021: 129 · 2020: 129 · 2019: 120 · 2018: 116 · 2017: 109 · 2016: 95 · 2015: 113 · 2014: 108 · 2013: 76 · 2012: 88
- FWCI
- 7.68
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Educational Statistics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- effect size, file drawer problem, sampling bias, negative results, meta-analysis, publication bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
