The DOI matches its Crossref record; 10 of the 11 references it cites were found in a registry or an index.
A Fail-Safe N for Effect Size in Meta-Analysis
Abstract
Rosenthan's (1979) concept of fail-safe N has thus far been applied to probability levels exclusively. This note introduces a fail-safe TV for effect size. Rosenthal's (1979) fail-safe N was an ingenious response to the so-called file drawer problem in research integration, in that it provided a direct assessment of the threat posed by sampling bias in the literature search. The algorithm was derived from the Stouffer method (Mosteller & Bush, 1954), in which an overall Z-score is computed by summing individual Z-scores and dividing by the square root of the number of scores. A significant overall Z-score, Rosenthal reasoned, could be made nonsignificant by adding some knowable number of hypothetical studies that averaged null results. The exact number of such studies needed to bring a significant overall p level up to some critical level (e.g., .05) has since become known as the fail-safe N (Nfs). The formula is
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“we also calculated the classic fail-safe N test (Orwin, 1983), which revealed that it would take 59 studies reporting null results (g = 0) to be included in our analyses in order for our findings to lose statistical significance.”
From Improving fluid intelligence with training on working memory: a meta-analysis (2015) · cites this paper as Orwin, 1983
In the citing paper
“The file drawer problem (publication bias) was examined by creating a funnel plot and calculating the fail safe N (45).”
From A Meta-Analysis of the Corpus Callosum in Autism (2009) · cites this paper as [45]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890
Smith, M. L., Glass, G. V., & Miller, T. I. (1980) · Medical Entomology and Zoology · Cited by 2,123
Johnson, D. W., Maruyama, G., Johnson, R., Nelson, D., & Skon, L. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.89.1.47 · Cited by 1,588
Interpersonal expectancy effects: the first 345 studies
Rosenthal, R., & Rubin, D. B. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00075506 · Cited by 1,158
Eagly, A. H., & Carli, L. L. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.90.1.1 · Cited by 541
Cooper, H. M. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.1.131 · Cited by 418
Arkin, R. M., Cooper, H., & Kolditz, T. A. (1980) · Journal of Personality · doi:10.1111/j.1467-6494.1980.tb02378.x · Cited by 88
Flagged first, then by citations
1 of the 11 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Items 201–300 of 1,020
Why do people play games? A meta-analysis
Hamari, J., & Keronen, L. (2017) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2017.01.006 · Cited by 368
“There are two main approaches for such calculations (Long 2001): one method based on the sum of the Z scores (Rosenthal 1979) while the other uses effect sizes (Orwin 1983).”
MethodModeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359
Self-assessment in second language testing: a meta-analysis and analysis of experiential factors
Ross, S. J. (1998) · Language Testing · doi:10.1177/026553229801500101 · Cited by 397
Incidental Word Learning While Reading: A Meta-Analysis
Swanborn, M. S. L., & de Glopper, K. (1999) · Review of Educational Research · doi:10.3102/00346543069003261 · Cited by 359
A meta-analysis of group cognitive behavioral therapy for insomnia
Koffel, E. A., Koffel, J. B., & Gehrman, P. R. (2015) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2014.05.001 · Cited by 356
Cognitive-behavioral therapy for anger in children and adolescents: a meta-analysis
Sukhodolsky, D. G., Kassinove, H., & Gorman, B. S. (2003) · Aggression and Violent Behavior · doi:10.1016/j.avb.2003.08.005 · Cited by 415
Reschly, A. L., Busch, T. W., Betts, J., Deno, S. L., & Long, J. D. (2009) · Journal of School Psychology · doi:10.1016/j.jsp.2009.07.001 · Cited by 376
Organizational wellness programs: A meta-analysis.
Parks, K. M., & Steelman, L. A. (2008) · Journal of Occupational Health Psychology · doi:10.1037/1076-8998.13.1.58 · Cited by 352
“20 is met with an N of 4 for absenteeism and 7 for job satisfaction (Orwin, 1983).”
BackgroundInfluentialThe contralateral delay activity as a neural measure of visual working memory
Luria, R., Balaban, H., Awh, E., & Vogel, E. K. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.01.003 · Cited by 371
Sensitivity Analysis for Publication Bias in Meta-Analyses
Mathur, M. B., & VanderWeele, T. J. (2020) · Journal of the Royal Statistical Society Series C: Applied Statistics · doi:10.1111/rssc.12440 · Cited by 354
Anger, hostility, and male perpetrators of intimate partner violence: A meta-analytic review
Norlander, B., & Eckhardt, C. I. (2004) · Clinical Psychology Review · doi:10.1016/j.cpr.2004.10.001 · Cited by 351
Meta-Analysis of Perceived Stress and Its Association With Incident Coronary Heart Disease
Richardson, S., Shaffer, J. A., Falzon, L., Krupka, D., Davidson, K. W., & Edmondson, D. (2012) · The American Journal of Cardiology · doi:10.1016/j.amjcard.2012.08.004 · Cited by 349
“[8]”
Media use and ADHD-related behaviors in children and adolescents: A meta-analysis
Nikkelen, S. W. C., Valkenburg, P. M., Huizinga, M., & Bushman, B. J. (2014) · Developmental Psychology · doi:10.1037/a0037318 · Cited by 339
Vocks, S., Tuschen‐Caffier, B., Pietrowsky, R., Rustenbach, S. J., Kersting, A., & Herpertz, S. (2010) · International Journal of Eating Disorders · doi:10.1002/eat.20696 · Cited by 343
A meta-analytic review of social, self-concept, and behavioral outcomes of peer-assisted learning.
Ginsburg-Block, M. D., Rohrbeck, C. A., & Fantuzzo, J. W. (2006) · Journal of Educational Psychology · doi:10.1037/0022-0663.98.4.732 · Cited by 397
Brain Volumes in Relatives of Patients With Schizophrenia
Boos, H. B. M., Aleman, A., Cahn, W., Pol, H. E. H., & Kahn, R. S. (2007) · Archives of General Psychiatry · doi:10.1001/archpsyc.64.3.297 · Cited by 356
“To examine the possibility of publication bias, we computed a fail-safe number of studies.(54,58) Publication bias implies that studies with no effect may not be published, posing a threat to the stability of the obtained effect size.”
MethodPietschnig, J., Penke, L., Wicherts, J. M., Zeiler, M., & Voracek, M. (2015) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2015.09.017 · Cited by 316
Fischer, S. F., Smith, G. T., & Cyders, M. A. (2008) · Clinical Psychology Review · doi:10.1016/j.cpr.2008.09.001 · Cited by 369
The impact of gamification in educational settings on student learning outcomes: a meta-analysis
Huang, R., Ritzhaupt, A. D., Sommer, M., Zhu, J., Stephen, A., Valle, N., Hampton, J., & Li, J. (2020) · Educational Technology Research and Development · doi:10.1007/s11423-020-09807-z · Cited by 340
“Also, the Orwin’s fail-safe N test was calculated to determine the number of missing null studies required to bring the existing effect size to a trivial level (Orwin 1983).”
MethodTreatment of Fibromyalgia Syndrome With Antidepressants: A Meta-analysis
Häuser, W., Bernardy, K., Üçeyler, N., & Sommer, C. (2009) · JAMA · doi:10.1001/jama.2008.944 · Cited by 338
Relaxation training for anxiety: a ten-years systematic review with meta-analysis
Manzoni, G. M., Pagnini, F., Castelnuovo, G., & Molinari, E. (2008) · BMC Psychiatry · doi:10.1186/1471-244x-8-41 · Cited by 480
“The fail-safe N [42] for ES was also calculated.”
Methodder Oord, S. V., Prins, P. J. M., Oosterlaan, J., & Emmelkamp, P. M. G. (2007) · Clinical Psychology Review · doi:10.1016/j.cpr.2007.10.007 · Cited by 337
“To determine the likelihood of publication bias or the “file-drawer problem” (Rosenthal, 1991), the fail-safe N was computed (Orwin, 1983).”
MethodKahana, S., Drotar, D., & Frazier, T. (2008) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsm128 · Cited by 334
“The significance of the overall weightedmean effect size for adherence (mean d ¼ .34, 95 CI ¼ 0.30–0.38) is unlikely to be a result of publication bias as 522 studies with null results would be needed to reduce the mean effect size to d ¼ .05, a negligible effect (Orwin, 1983).”
BackgroundInfluentialA meta-analysis of behavior therapy for Tourette Syndrome
McGuire, J. F., Piacentini, J., Brennan, E. A., Lewin, A. B., Murphy, T. K., Small, B. J., & Storch, E. A. (2014) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2013.12.009 · Cited by 334
The alliance in child and adolescent psychotherapy.
Shirk, S. R., Karver, M. S., & Brown, R. R. (2011) · Psychotherapy · doi:10.1037/a0022181 · Cited by 336
“The FSN estimates the number of unpublished studies with effect sizes of zero that would be needed to reduce the overall effect size to a less meaningful level (Lipsey & Wilson, 2001; Orwin, 1983; Rosenthal, 1991).”
BackgroundAchievement Goals and Achievement Emotions: A Meta-analysis
Huang, C. (2011) · Educational Psychology Review · doi:10.1007/s10648-011-9155-x · Cited by 344
Zakzanis, K. K. (2001) · Archives of Clinical Neuropsychology · doi:10.1093/arclin/16.7.653 · Cited by 346
Residents’ impact perceptions of and attitudes towards tourism development: a meta-analysis
Gürsoy, D., Ouyang, Z., Nunkoo, R., & Wei, W. (2018) · Journal of Hospitality Marketing & Management · doi:10.1080/19368623.2018.1516589 · Cited by 357
“Several researchers have addressed the existence of publication bias against statistically nonsignificant results in peer-reviewed journals (Orwin, 1983; Rosenthal, 1979).”
BackgroundStress and eating behaviours in healthy adults: a systematic review and meta-analysis
Hill, D., Conner, M. T., Clancy, F., Moss, R. H., Wilding, S., Bristow, M., & O'Connor, D. B. (2021) · Health Psychology Review · doi:10.1080/17437199.2021.1923406 · Cited by 396
“, 2005) and effect sizes were calculated using Hedges’ g to account for any small sample sizes included in the review (Orwin, 1983).”
MethodCareer-intervention outcome: What contributes to client gain?
Oliver, L. W., & Spokane, A. R. (1988) · Journal of Counseling Psychology · doi:10.1037/0022-0167.35.4.447 · Cited by 330
Spatial Ability: Its Influence on Learning with Visualizations—a Meta-Analytic Review
Höffler, T. N. (2010) · Educational Psychology Review · doi:10.1007/s10648-010-9126-7 · Cited by 359
Prati, G., & Pietrantoni, L. (2010) · Journal of Community Psychology · doi:10.1002/jcop.20371 · Cited by 331
The Importance of Quality of Primary Studies in Producing Unbiased Systematic Reviews
Khan, K. S. (1996) · Archives of Internal Medicine · doi:10.1001/archinte.1996.00440060089011 · Cited by 326
Antecedents of organizational knowledge sharing: a meta‐analysis and critique
Witherspoon, C. L., Bergner, J. M., Cockrell, C., & Stone, D. N. (2013) · Journal of Knowledge Management · doi:10.1108/13673271311315204 · Cited by 360
A Quantitative Review of Research on Charismatic Leadership
Fuller, J. B., Patterson, C. E. P., Hester, K., & Stringer, D. Y. (1996) · Psychological Reports · doi:10.2466/pr0.1996.78.1.271 · Cited by 326
Spencer, C. M., & Stith, S. (2018) · Trauma, Violence, & Abuse · doi:10.1177/1524838018781101 · Cited by 289
“In order to combat the “file drawer problem” that impacts all meta-analyses (Hunter & Schmidt, 2004), Duval and Tweedie’s trim and fill test (Duval & Tweedie, 2000), the classic fail-safe N test (Rosenthal, 1979), and Orwin’s fail-safe N test (Orwin, 1983) were utilized to evaluate the possibility of publication biase…”
MethodBirkley, E. L., & Eckhardt, C. I. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.01.002 · Cited by 325
“A uthor M anuscript A uthor M anuscript A uthor M anuscript A uthor M anuscript would be needed to reduce the overall effect size across studies below the critical d value of .20 for a small effect (Lipsey & Wilson, 2001; Orwin, 1983).”
BackgroundDo Animal-Assisted Activities Effectively Treat Depression? A Meta-Analysis
Souter, M. A., & Miller, M. D. (2007) · Anthrozoös · doi:10.2752/175303707x207954 · Cited by 324
“The fail-safe N, a statistic developed by Rosenthal and later adapted to the standardized mean difference effect size by Orwin, estimates the number of unpublished studies with nonsignificant findings that would decrease the cumulated effect of the studies to nonsignificant (Rosenthal 1979; Orwin 1983; Lipsey and Wils…”
BackgroundVagelatos, N. T., & Eslick, G. D. (2013) · Epidemiologic Reviews · doi:10.1093/epirev/mxs012 · Cited by 320
“Publication bias is generally regarded as a concern if the fail-safe number is <5n + 10, with n being the number of studies included in the meta-analysis (14).”
Benbasat, I., & Lim, L. H. (1993) · Small Group Research · doi:10.1177/1046496493244002 · Cited by 322
Effects of Exercise Interventions on Body Image
Campbell, A., & Hausenblas, H. A. (2009) · Journal of Health Psychology · doi:10.1177/1359105309338977 · Cited by 364
“To assess publication bias, we undertook the following graphical and statistical methods: forest plots, funnel plots, Duval and Tweedie’s Trim and Fill, Egger’s Test, and Fail Safe N (Nfs) or file drawer analysis using both Orwin’s (1983) and Rosenthal’s (1979) procedures.”
MethodHendriks, T., Schotanus-Dijkstra, M., Hassankhan, A., de Jong, J. T. V. M., & Bohlmeijer, E. T. (2019) · Journal of Happiness Studies · doi:10.1007/s10902-019-00082-1 · Cited by 380
“This fail-safe N indicates the number of unpublished non-significant studies that would be required to lower the overall effect size below significance (Egger et al. 1997; Orwin 1983).”
BackgroundAssociation Between Parents’ PTSD Severity and Children's Psychological Distress: A Meta‐Analysis
Lambert, J. E., Holzer, J., & Hasbun, A. (2014) · Journal of Traumatic Stress · doi:10.1002/jts.21891 · Cited by 338
Insight in psychosis and neuropsychological function
Aleman, A., Agrawal, N., Morgan, K. D., & David, A. S. (2006) · The British Journal of Psychiatry · doi:10.1192/bjp.189.3.204 · Cited by 320
“To assess whether sampling bias would be large enough to render the mean effect size insignificant, we computed an additional statistic, the fail-safe n (Orwin, 1983; Rosenthal, 1991).”
MethodEffectiveness of cognitive-behavior therapy for maladapting children: A meta-analysis.
Durlak, J. A., Fuhrman, T., & Lampman, C. B. (1991) · Psychological Bulletin · doi:10.1037/0033-2909.110.2.204 · Cited by 325
Schmidt, S., Roesler, U., Kusserow, T., & Rau, R. (2012) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2012.711523 · Cited by 362
“Further, we computed Fail-Safe-Ns according to Orwin (1983) and Rosenthal (1979), in which the criterion for a trivial correlation was set to r ¼ .15, and the mean correlation in missing studies to r ¼ .10.”
MethodSocial support and posttraumatic stress disorder: A meta-analysis of longitudinal studies
Wang, Y., Chung, M. C., Wang, N., Yu, X., & Kenardy, J. A. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2021.101998 · Cited by 371
Client preferences affect treatment satisfaction, completion, and clinical outcome: a meta-analysis
Lindhiem, O., Bennett, C. B., Trentacosta, C. J., & McLear, C. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.06.002 · Cited by 315
Update on Bispectral Index monitoring
Johansen, J. W. (2006) · Best practice & research. Clinical anaesthesiology · doi:10.1016/j.bpa.2005.08.004 · Cited by 312
A Meta-Analytic Review of Social Identification and Health in Organizational Contexts
Steffens, N. K., Haslam, S. A., Schuh, S. C., Jetten, J., & van Dick, R. (2017) · Personality and Social Psychology Review · doi:10.1177/1088868316656701 · Cited by 311
The Politics of Fear: Is There an Ideological Asymmetry in Existential Motivation?
Jost, J. T., Stern, C., Rule, N. O., & Sterling, J. L. (2017) · Social Cognition · doi:10.1521/soco.2017.35.4.324 · Cited by 328
“For each category, we also conducted two “file-drawer” analyses to estimate the degree to which the statistical significance and size of the average effect was likely influenced by publication bias (Orwin, 1983; Rosenthal, 1979).”
MethodJansen, R. S., van Leeuwen, A., Janssen, J., Jak, S., & Kester, L. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.100292 · Cited by 310
“In the current study, the funnel plot with trim and fill, cumulative meta-analysis, Orwin's fail safe N (Orwin, 1983), adding the sample size as a moderator, and adding the source of the study (journal article, conference proceedings, or dissertation) were used to test for the potential presence of…”
MethodLuebbert, K., Dahme, B., & Hasenbring, M. I. (2001) · Psycho-Oncology · doi:10.1002/pon.537 · Cited by 309
“The implication of an effect size can also be shown by the fail safe number of studies necessary to reduce the a6erage effect size estimate to the 'no effect level', according to Orwin (1983), in terms of Hedges and Olkin (1985).”
InfluentialFemoral nerve blocks for acute postoperative pain after knee replacement surgery
Chan, E., Fransen, M., Parker, D. A., Assam, P. N., & Chua, N. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009941.pub2 · Cited by 308
“The trivial effect was defined by an SMD value of 0.2 (Orwin 1983).”
BackgroundMethodInfluentialThe Dyadic Adjustment Scale: A Reliability Generalization Meta‐Analysis
Graham, J. M., Liu, Y. J., & Jeziorski, J. L. (2006) · Journal of Marriage and the Family · doi:10.1111/j.1741-3737.2006.00284.x · Cited by 309
“The robustness of each of the mean reliability estimates was tested using Orwin’s (1983) fail-safe n for effect sizes.”
MethodComparison of reading performance on screen and on paper: A meta-analysis
Kong, Y., Seo, Y. S., & Zhai, L. (2018) · Computers & Education · doi:10.1016/j.compedu.2018.05.005 · Cited by 306
Meta‐analysis of randomized, controlled treatment trials for pediatric obsessive‐compulsive disorder
Watson, H. J., & Rees, C. S. (2008) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2007.01875.x · Cited by 304
“The fail-safe N estimated the number of unpublished studies producing null results that would need to exist to negate an observed pooled ES (Orwin, 1983; Rosenthal, 1979).”
BackgroundReliability of the Hamilton Rating Scale for Depression: A meta-analysis over a period of 49years
Trajković, G., Starčević, V., Latas, M., Leštarević, M., Ille, T., Bukumirić, Z., & Marinković, J. (2011) · Psychiatry Research · doi:10.1016/j.psychres.2010.12.007 · Cited by 303
The Relationship Between Theory and Practice: Back to the Classics
Kessels, J., & Korthagen, F. A. J. (1996) · Educational Researcher · doi:10.3102/0013189x025003017 · Cited by 302
A Meta-Analysis of Social Skill Interventions for Students with Emotional or Behavioral Disorders
Quinn, M. M., Kavale, K. A., Mathur, S. R., Rutherford, R. B., & Forness, S. R. (1999) · Journal of Emotional and Behavioral Disorders · doi:10.1177/106342669900700106 · Cited by 302
Antecedents and consequences of retirement planning and decision-making: A meta-analysis and model
Topa, G., Moriano, J. A., Depolo, M., de la Hera, C. M. A., & Morales, J. F. (2009) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2009.03.002 · Cited by 303
Effectiveness of anger treatments for specific anger problems: A meta-analytic review
Vecchio, T. D., & O’Leary, K. D. (2003) · Clinical Psychology Review · doi:10.1016/j.cpr.2003.09.006 · Cited by 302
A meta-analysis of randomized controlled trials of laparoscopic versus conventional appendectomy
Chung, R. S. K., Rowland, D. Y., Li, P., & Diaz, J. J. (1999) · The American Journal of Surgery · doi:10.1016/s0002-9610(99)00017-3 · Cited by 302
Olver, M., Stockdale, K. C., & Wormith, J. (2014) · Psychological Assessment · doi:10.1037/a0035080 · Cited by 264
“Finally, we applied a fail-safe N procedure (Orwin, 1983) to estimate the number of missing studies with a predictive validity correlation of .00 that would be required to bring the observed fixed effect below Cohen’s (1988) threshold for a small effect size (r .”
MethodDietary intake of people with severe mental illness: systematic review and meta-analysis
Teasdale, S. B., Ward, P. B., Samaras, K., Firth, J., Stubbs, B., Tripodi, E., & Burrows, T. L. (2019) · The British Journal of Psychiatry · doi:10.1192/bjp.2019.20 · Cited by 311
Autogenic Training: A Meta-Analysis of Clinical Outcome Studies
Stetter, F., & Kupper, S. (2002) · Applied Psychophysiology and Biofeedback · doi:10.1023/a:1014576505223 · Cited by 300
Benefits of fish oil supplementation in hyperlipidemia: a systematic review and meta-analysis
Eslick, G. D., Howe, P. R. C., Smith, C., Priest, R., & Bensoussan, A. (2009) · International Journal of Cardiology · doi:10.1016/j.ijcard.2008.03.092 · Cited by 299
“If publication bias was detected, the impact of such bias would be assessed using the ‘fail-safe number’ method [26].”
MethodImpulsivity and nonsuicidal self-injury: A review and meta-analysis
Hamza, C. A., Willoughby, T., & Heffer, T. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.02.010 · Cited by 294
“…Orwin's effect size failsafe N=Number of studies with average effect size of 0 required to reduce the observed mean effect size to Cohen's d = .20— see Orwin (1983). analyses, a meta-analysis of studies examining the correlation between NSSI frequency and impulsivity, models controlling for…”
BackgroundZimmermann, G., Favrod, J., Trieu, V. H., & Pomini, V. (2005) · Schizophrenia Research · doi:10.1016/j.schres.2005.02.018 · Cited by 371
“As the selective publication of positive findings controlled trials is an important concern in metaanalytic reviews of the literature (Lipsey and Wilson, 1993), the Fail-safe N (FSN) for the average effect sizes was computed (Orwin, 1983).”
BackgroundReview: Alleviating loneliness in young people – a meta‐analysis of interventions
Eccles, A. M., & Qualter, P. (2021) · Child and Adolescent Mental Health · doi:10.1111/camh.12389 · Cited by 285
“Visual inspection of funnel plots (see Figure S1) and an Orwin’s fail-safe N = 532 (Orwin, 1983) demonstrate no evidence of publication bias.”
MethodAbraham, L. B. (2008) · Computer Assisted Language Learning · doi:10.1080/09588220802090246 · Cited by 303
Prosocial Consequences of Interpersonal Synchrony
Rennung, M., & Göritz, A. S. (2016) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000252 · Cited by 331
“Finally, we assessed the likelihood of inclusion bias using Begg and Mazumdar’s rank correlation test (Begg & Mazumdar, 1994), Egger’s regression test (Egger, Smith, Schneider, & Minder, 1997), Rosenthal’s fail-safe N (Rosenthal, 1979), and Orwin’s fail-safe N (Orwin, 1983), as well as Duval and Tweedie’s trim and fil…”
BackgroundMethodCognitive–behavioral therapy for Internet gaming disorder: A systematic review and meta‐analysis
Stevens, M. W. R., King, D. L., Dorstyn, D., & Delfabbro, P. H. (2019) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2341 · Cited by 295
McKinney, J. S., & Kostis, W. J. (2012) · Stroke · doi:10.1161/strokeaha.112.655894 · Cited by 294
Shachar, M., & Neumann, Y. (2003) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v4i2.153 · Cited by 288
“In this study the NFS numbers were calculated based on Orwin’s (1983) formula, providing large numbers for our research questions sample sizes.”
MethodOwen, M. B., Curry, W. B., Kerner, C., Newson, L., & Fairclough, S. J. (2017) · Preventive Medicine · doi:10.1016/j.ypmed.2017.09.018 · Cited by 280
“Following these visual inspections, the trim and fill procedure (Duval & Tweedie, 2000a, 2000b), Orwin’s fail safe number (Orwin, 1983) and Egger’s regression test (Egger, Smith, Schneider, & Minder, 1997) were used to confirm the presence or absence of publication bias.”
MethodFactors affecting adoption of online banking: A meta-analytic structural equation modeling study
Montazemi, A. R., & Qahri-Saremi, H. (2015) · Information & Management · doi:10.1016/j.im.2014.11.002 · Cited by 293
“The Fail-safe K value estimates the number of missing studies with null findings that could possibly render a relationship non-significant [58, 104].”
BackgroundThe impact of board and audit committee characteristics on voluntary disclosure: A meta-analysis
Samaha, K., Khlif, H., & Hussainey, K. (2015) · Journal of International Accounting Auditing and Taxation · doi:10.1016/j.intaccaudtax.2014.11.001 · Cited by 292
“In order to test whether the publication bias influences the conclusions drawn from our meta-analysis, we apply Orwin's (1983) approach to determine the extent of the file drawer problem.”
MethodNeuroanatomical Maps of Psychosis Onset: Voxel-wise Meta-Analysis of Antipsychotic-Naive VBM Studies
Fusar‐Poli, P., Raduà, J., McGuire, P. K., & Borgwardt, S. (2011) · Schizophrenia Bulletin · doi:10.1093/schbul/sbr134 · Cited by 291
Meta-analysis of the efficacy of psychological and medical treatments for binge-eating disorder
Hilbert, A., Petroff, D., Herpertz, S., Pietrowsky, R., Tuschen-Caffier, B., Vocks, S., & Schmidt, R. (2019) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000358 · Cited by 283
Modelling publication bias in meta-analysis: a review
Sutton, A. J., Song, F., Gilbody, S. M., & Abrams, K. R. (2000) · Statistical Methods in Medical Research · doi:10.1177/096228020000900503 · Cited by 288
van Vreeswijk, M. F., & de Wilde, E. J. (2004) · Behaviour Research and Therapy · doi:10.1016/s0005-7967(03)00194-3 · Cited by 280
McEvoy, P. M., Hyett, M. P., Shihata, S., Price, J. E., & Strachan, L. (2019) · Clinical Psychology Review · doi:10.1016/j.cpr.2019.101778 · Cited by 271
A practitioner's guide to meta‐analysis
Durlak, J. A., & Lipsey, M. W. (1991) · American Journal of Community Psychology · doi:10.1007/bf00938026 · Cited by 286
Racial Discrimination and Health Outcomes Among Racial/Ethnic Minorities: A Meta‐Analytic Review
Carter, R. T., Lau, M. Y., Johnson, V. E., & Kirkinis, K. (2017) · Journal of Multicultural Counseling and Development · doi:10.1002/jmcd.12076 · Cited by 289
The Effectiveness of School-Based Anti-Bullying Programs
Ferguson, C. J., Miguel, C. S., Kilburn, J. C., & Sanchez, P. (2007) · Criminal Justice Review · doi:10.1177/0734016807311712 · Cited by 326
Telephone counseling for smoking cessation: rationales and meta-analytic review of evidence
Lichtenstein, E., Glasgow, R. E., Lando, H. A., Ossip-Klein, D. J., & Boles, S. M. (1996) · Health Education Research · doi:10.1093/her/11.2.243 · Cited by 279
How effective are mobile devices for language learning? A meta-analysis
Sung, Y., Chang, K.-E., & Yang, J.-M. (2015) · Educational Research Review · doi:10.1016/j.edurev.2015.09.001 · Cited by 321
Miller, S. W., Budy, P. E., & Schmidt, J. C. (2009) · Restoration Ecology · doi:10.1111/j.1526-100x.2009.00605.x · Cited by 281
A Meta-Analytic Investigation of Business Ethics Instruction
Waples, E. P., Antes, A. L., Murphy, S. T., Connelly, S., & Mumford, M. D. (2009) · Journal of Business Ethics · doi:10.1007/s10551-008-9875-0 · Cited by 279
The National High Blood Pressure Education Program
Jones, D. W., & Hall, J. E. (2002) · Hypertension · doi:10.1161/01.hyp.0000018303.61360.28 · Cited by 281
Inducing Self-Explanation: a Meta-Analysis
Bisra, K., Liu, Q., Nesbit, J. C., Salimi, F., & Winne, P. H. (2018) · Educational Psychology Review · doi:10.1007/s10648-018-9434-x · Cited by 286
“Orwin ’ s fail-safe N , a more stringent publication bias test, revealed 274 missing null studies would be required to bring the mean effect size found in this meta-analysis to under 0.1 (Orwin 1983).”
InfluentialPellicane, M. J., & Ciesla, J. A. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2021.102113 · Cited by 306
Social skills training with schizophrenics: A meta-analytic evaluation.
Benton, M. K., & Schrøeder, H. E. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.58.6.741 · Cited by 277
Meta-analysis of psychological assessment as a therapeutic intervention
Poston, J. M., & Hanson, W. E. (2010) · Psychological Assessment · doi:10.1037/a0018679 · Cited by 276
“that dissertations were excluded, robustness of aggregate effect sizes were also calculated through the use of fail-safe N (Orwin, 1983).”
MethodA Meta-Analysis of the Corpus Callosum in Autism
Frazier, T. W., & Hardan, A. Y. (2009) · Biological Psychiatry · doi:10.1016/j.biopsych.2009.07.022 · Cited by 267
Is cross-cultural training of expatriate managers effective: A meta analysis
Deshpande, S. P., & Viswesvaran, C. (1992) · International Journal of Intercultural Relations · doi:10.1016/0147-1767(92)90054-x · Cited by 275
The effectiveness of treatment for pediatric obsessive-compulsive disorder: A meta-analysis
Abramowitz, J. S., Whiteside, S. P. H., & Deacon, B. J. (2005) · Behavior Therapy · doi:10.1016/s0005-7894(05)80054-1 · Cited by 274
“We computed the fail safe N (Orwin, 1983) to deter3 The main difference between the Y-BOCS and CY-BOCS is the substitution of simpler language for the various probe items.”
MethodHunsu, N. J., Adesope, O., & Bayly, D. J. (2016) · Computers & Education · doi:10.1016/j.compedu.2015.11.013 · Cited by 272
“This bias typically emanates from the potential of non-significant findings being presented as conference papers while such nonsignificant results are less accepted for publication in journal outlets, thus ending up in a “file drawer” (Orwin, 1983).”
ResultRhoades, K. A. (2008) · Child Development · doi:10.1111/j.1467-8624.2008.01235.x · Cited by 271
Metadata
- Authors
- Orwin, R. G.
- Affiliations
- United States Department of Health and Human Services (US)
- Corresponding author
- Orwin, Robert G.
- Venue
- Journal of Educational Statistics
- Publisher
- JSTOR
- Published
- 1983
- Volume
- 8
- Issue
- 2
- Pages
- 157
- ISSN
- 0362-9791 (ISSN-L)
- OpenAlex
- W2112432954
- Cited by
- 2,064
- References
- 11
- Citations by year
- 2026: 51 · 2025: 75 · 2024: 91 · 2023: 121 · 2022: 114 · 2021: 129 · 2020: 129 · 2019: 120 · 2018: 116 · 2017: 109 · 2016: 95 · 2015: 113 · 2014: 108 · 2013: 76 · 2012: 88
- FWCI
- 7.68
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Educational Statistics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- effect size, file drawer problem, sampling bias, negative results, meta-analysis, publication bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
