The DOI matches its Crossref record; 10 of the 11 references it cites were found in a registry or an index.
A Fail-Safe N for Effect Size in Meta-Analysis
Abstract
Rosenthan's (1979) concept of fail-safe N has thus far been applied to probability levels exclusively. This note introduces a fail-safe TV for effect size. Rosenthal's (1979) fail-safe N was an ingenious response to the so-called file drawer problem in research integration, in that it provided a direct assessment of the threat posed by sampling bias in the literature search. The algorithm was derived from the Stouffer method (Mosteller & Bush, 1954), in which an overall Z-score is computed by summing individual Z-scores and dividing by the square root of the number of scores. A significant overall Z-score, Rosenthal reasoned, could be made nonsignificant by adding some knowable number of hypothetical studies that averaged null results. The exact number of such studies needed to bring a significant overall p level up to some critical level (e.g., .05) has since become known as the fail-safe N (Nfs). The formula is
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“we also calculated the classic fail-safe N test (Orwin, 1983), which revealed that it would take 59 studies reporting null results (g = 0) to be included in our analyses in order for our findings to lose statistical significance.”
From Improving fluid intelligence with training on working memory: a meta-analysis (2015) · cites this paper as Orwin, 1983
In the citing paper
“The file drawer problem (publication bias) was examined by creating a funnel plot and calculating the fail safe N (45).”
From A Meta-Analysis of the Corpus Callosum in Autism (2009) · cites this paper as [45]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890
Smith, M. L., Glass, G. V., & Miller, T. I. (1980) · Medical Entomology and Zoology · Cited by 2,123
Johnson, D. W., Maruyama, G., Johnson, R., Nelson, D., & Skon, L. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.89.1.47 · Cited by 1,588
Interpersonal expectancy effects: the first 345 studies
Rosenthal, R., & Rubin, D. B. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00075506 · Cited by 1,158
Eagly, A. H., & Carli, L. L. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.90.1.1 · Cited by 541
Cooper, H. M. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.1.131 · Cited by 418
Arkin, R. M., Cooper, H., & Kolditz, T. A. (1980) · Journal of Personality · doi:10.1111/j.1467-6494.1980.tb02378.x · Cited by 88
Flagged first, then by citations
1 of the 11 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Items 401–500 of 1,020
Divestiture and Firm Performance: A Meta-Analysis
Lee, D., & Madhavan, R. (2010) · Journal of Management · doi:10.1177/0149206309360931 · Cited by 208
Meta-analytic review of event-related potential studies in post-traumatic stress disorder
Karl, A., Malta, L. S., & Maercker, A. R. A. (2005) · Biological Psychology · doi:10.1016/j.biopsycho.2005.03.004 · Cited by 209
“To address the file drawer problem, Orwin’s fail safe N (Orwin, 1983) was computed.”
MethodMaki, A., Burns, R. J., Ha, L., & Rothman, A. J. (2016) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2016.07.006 · Cited by 202
Meta-analysis of the relevance of the OPRM1 118A>G genetic variant for pain treatment
Walter, C., & Lötsch, J. (2009) · Pain · doi:10.1016/j.pain.2009.07.013 · Cited by 200
“Publication bias was assessed by testing for classic fail-safe N [35].”
MethodAsymmetry of the hippocampus and amygdala in MRI volumetric measurements of normal adults
Pedraza, O., Bowers, D., & Gilmore, R. L. (2004) · Journal of the International Neuropsychological Society · doi:10.1017/s1355617704105080 · Cited by 208
“In addition, publication bias in each of the two meta-analyses was evaluated by calculating a fail-safe N, or the number of unpublished studies with no effect that must exist in order to reduce the obtained overall effect size estimate to a minimal critical level (Orwin, 1983; Rosenthal, 1991).”
MethodMeta-analysis of facial affect recognition difficulties after traumatic brain injury.
Babbage, D. R., Yim, J., Zupan, B., Neumann, D., Tomita, M. R., & Willer, B. (2011) · Neuropsychology · doi:10.1037/a0021908 · Cited by 198
Trauma-focused therapy for refugees: Meta-analytic findings.
Lambert, J. E., & Alhassoon, O. M. (2015) · Journal of Counseling Psychology · doi:10.1037/cou0000048 · Cited by 199
Impaired facial emotion perception in schizophrenia: A meta-analysis
Chan, R. C. K., Li, H., Cheung, E. F. C., & Gong, Q. (2010) · Psychiatry Research · doi:10.1016/j.psychres.2009.03.035 · Cited by 221
“A fail-safe number estimates the number of insignificant, unpublished studies that would need to be added to a meta-analysis to reduce an overall statistically significant observed effect to non-significance (Orwin, 1983; Rosenberg, 2005).”
Adult attachment representations and depressive symptoms: A meta-analysis
Dagan, O., Facompré, C. R., & Bernard, K. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.04.091 · Cited by 228
Yu, L., Zhao, X., Gao, X., & Siddique, K. H. M. (2019) · Agricultural Water Management · doi:10.1016/j.agwat.2019.105906 · Cited by 203
McGinty, H. L., Phillips, K. M., Jim, H., Cessna, J. M., Asvat, Y., Cases, M. G., Small, B. J., & Jacobsen, P. B. (2014) · Supportive Care in Cancer · doi:10.1007/s00520-014-2285-1 · Cited by 196
“Larger values for Orwin’s fail-safe N indicate more robust findings [16].”
Forness, S. R., & Kavale, K. A. (1996) · Learning Disability Quarterly · doi:10.2307/1511048 · Cited by 196
“Using ES levels, Orwin (1983) developed methods for calculating a fail-safe number”
MethodLejeune, J., Northrop, A., & Kurtz, M. M. (2021) · Schizophrenia Bulletin · doi:10.1093/schbul/sbab022 · Cited by 198
Brief report: Cognitive flexibility in autism spectrum disorders: A quantitative review
Leung, R. C., & Zakzanis, K. K. (2014) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-014-2136-4 · Cited by 192
Ruttan, L., Martin, K., Liu, A. M. M., Colella, B., & Green, R. (2008) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2008.07.007 · Cited by 191
Deng, W., Hu, D., Xu, S., Liu, X., Zhao, J., Chen, Q., Liu, J., Zhang, Z., Jiang, W., Ma, L., et al. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.07.086 · Cited by 208
Sung, Y., Yang, J.-M., & Lee, H.-Y. (2017) · Review of Educational Research · doi:10.3102/0034654317704307 · Cited by 197
Hermann, C., Kim, M., & Blanchard, E. B. (1995) · Pain · doi:10.1016/0304-3959(94)00210-6 · Cited by 190
Tellegen, C. L., & Sanders, M. R. (2013) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2013.01.022 · Cited by 200
Williams, A. J., Jones, C. A., Arcelus, J., Townsend, E., Lazaridou, A., & Michail, M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0245268 · Cited by 200
“The fail-safe N was also calculated using the Orwin algorithm [45].”
MethodBuilding child trauma theory from longitudinal studies: A meta-analysis
Alisic, E., Jongmans, M. J., van Wesel, F., & Kleber, R. J. (2011) · Clinical Psychology Review · doi:10.1016/j.cpr.2011.03.001 · Cited by 219
Tolman, A. W., & Kurtz, M. M. (2010) · Schizophrenia Bulletin · doi:10.1093/schbul/sbq077 · Cited by 189
Prevention of child sexual abuse victimization: a meta-analysis of school programs
Rispens, J., Aleman, A., & Goudena, P. P. (1997) · Child Abuse & Neglect · doi:10.1016/s0145-2134(97)00058-6 · Cited by 186
The erring brain: Error‐related negativity as an endophenotype for OCD—A review and meta‐analysis
Riesel, A. (2019) · Psychophysiology · doi:10.1111/psyp.13348 · Cited by 170
How Effective Are Family Literacy Programs?
van Steensel, R., McElvany, N., Kurvers, J., & Herppich, S. (2010) · Review of Educational Research · doi:10.3102/0034654310388819 · Cited by 199
“” One way of checking for this file-drawer effect is by computing the fail-safe N, that is, the number of unpublished studies reporting no results needed to reduce the weighted mean effect size to the point of nonsignificance (Lipsey & Wilson, 2001; Orwin, 1983).”
BackgroundEffects of Instruction in Deriving Word Meaning from Context: A Meta-Analysis
Fukkink, R. G., & de Glopper, K. (1998) · Review of Educational Research · doi:10.3102/00346543068004450 · Cited by 185
“Furthermore, Orwin's fail-safe N (Orwin, 1983) is 24.58 for a small effect size, which means that 25 (not-published) studies should report zerooutcomes to reduce the overall effect size to a nonsignificant outcome A strong effect of bias due to selection factors seems unlikely also, although…”
BackgroundInfluentialSmith‐Marek, E. N., Cafferky, B., Dharnidharka, P., Mallory, A. B., Dominguez, M., High, J., Stith, S. M., & Mendez, M. (2015) · Journal of Family Theory & Review · doi:10.1111/jftr.12113 · Cited by 188
Hartman, E. K., & Eslick, G. D. (2016) · Breast Cancer Research and Treatment · doi:10.1007/s10549-016-3989-3 · Cited by 185
“10, with n being the number of studies included in the meta-analysis [33].”
Meta-analysis of brain and cranial size in schizophrenia
Ward, K. E., Friedman, L., Wise, A. L., & Schulz, S. C. (1996) · Schizophrenia Research · doi:10.1016/s0920-9964(96)00076-x · Cited by 185
The antecedents and consequences of interpersonal forgiveness: A meta‐analytic review
RIEK, B. M., & MANIA, E. W. (2012) · Personal Relationships · doi:10.1111/j.1475-6811.2011.01363.x · Cited by 237
Assessment of School Anti-Bullying Interventions: A Meta-analysis of Randomized Clinical Trials
Fraguas, D., Díaz-Caneja, C. M., Ayora, M., Durán-Cutilla, M., Abregú-Crespo, R., Ezquiaga-Bravo, I., Martín-Babarro, J., & Arango, C. (2021) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2020.3541 · Cited by 195
Bacterial Infection and Alzheimer's Disease: A Meta-Analysis
Maheshwari, P., & Eslick, G. D. (2014) · Journal of Alzheimer's Disease · doi:10.3233/jad-140621 · Cited by 183
School Counseling Outcome: A Meta‐Analytic Examination of Interventions
Whiston, S. C., Tai, W., Rahardja, D., & Eder, K. C. (2011) · Journal of Counseling & Development · doi:10.1002/j.1556-6678.2011.tb00059.x · Cited by 184
Gallegos, A. M., Crean, H. F., Pigeon, W. R., & Heffner, K. L. (2017) · Clinical Psychology Review · doi:10.1016/j.cpr.2017.10.004 · Cited by 181
“Rosenthal's (1979) fail-safe N suggests that 170 non-significant studies would need to be added to result in a non-significant overall ES, 19 nonsignificant studies would need to be added to reduce the overall effect to −0.20 (Orwin, 1983).”
InfluentialPlatz, F., & Kopiez, R. (2012) · Music Perception An Interdisciplinary Journal · doi:10.1525/mp.2012.30.1.71 · Cited by 191
Hallford, D. J., Austin, D., Takano, K., & Raes, F. (2018) · Behaviour Research and Therapy · doi:10.1016/j.brat.2018.01.003 · Cited by 187
Anxiety sensitivity and suicidal ideation/suicide risk: A meta-analysis.
Stanley, I. H., Boffa, J. W., Rogers, M. L., Hom, M. A., Albanese, B. J., Chu, C., Capron, D. W., Schmidt, N. B., & Joiner, T. E. (2018) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000342 · Cited by 184
“First, we examined Orwin’s (1983) Fail-safe N, a metric of the number of additional studies with an effect size of 0 that would be required to bring the meta-analytic total effect down to an irrelevant value of .05 (Orwin, 1983).”
The effects of educational robotics in STEM education: a multilevel meta-analysis
Ouyang, F., & Xu, W. (2024) · International Journal of STEM Education · doi:10.1186/s40594-024-00469-4 · Cited by 300
“Funnel plot, Egger’s test (Bowden et al., 2015) and Rosenthal’s Fail-safe N test (Orwin, 1983) were used together to detect publication bias.”
MethodZheng, L., Niu, J., Zhong, L., & Gyasi, J. F. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2021.2015693 · Cited by 183
“Fifth, publication bias was examined through classic fail-safe N (Rosenthal, 1979) and Orwin ’ s fail-safe N (Orwin, 1983).”
Intolerance of uncertainty, anxiety, and worry in children and adolescents: A meta-analysis
Osmanağaoğlu, N., Creswell, C., & Dodd, H. F. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.07.035 · Cited by 177
Ruocco, A. C., Amirthavasagam, S., & Zakzanis, K. K. (2012) · Psychiatry Research Neuroimaging · doi:10.1016/j.pscychresns.2012.02.012 · Cited by 188
Cognitive behavioral therapy and graded exercise for chronic fatigue syndrome: a meta-analysis
Castell, B. D., Kazantzis, N., & Moss-Morris, R. E. (2011) · Clinical Psychology: Science and Practice · doi:10.1111/j.1468-2850.2011.01262.x · Cited by 176
A Meta-Analytic Review of Racial Discrimination: Relationships to Health and Culture
Carter, R. T., Johnson, V. E., Kirkinis, K., Roberson, K., Muchow, C., & Galgay, C. (2019) · Race and Social Problems · doi:10.1007/s12552-018-9256-y · Cited by 176
Kurtz, M. M., Gagen, E. C., Rocha, N., Machado, S. E. D. C., & Penn, D. L. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.09.003 · Cited by 176
“To partially address the “file-drawer” problem, we calculated a fail-safe N for each class of outcome variable by the method of Orwin (1983). This measure provides an estimate of the number of studies with null results that would be needed to render the effect size non-significant.”
BackgroundMethodPhillips, E. A., Gordeev, V. S., & Schreyögg, J. (2019) · Scandinavian Journal of Work, Environment & Health · doi:10.5271/sjweh.3839 · Cited by 148
“The results are considered robust if the fail-safe N is ≥5n+10, where n is the number of studies (44).”
Cushing, C. C., Jensen, C. D., Miller, M. B., & Leffingwell, T. R. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0036912 · Cited by 175
“To protect against the possibility that studies with a null or even unfavorable effect size might be unpublished, we used Orwin’s (1983) Fail-Safe N formula to determine the number of studies required to overturn the current finding.”
MethodStudent satisfaction in higher education: a meta-analytic study
Santini, F. D. O., Ladeira, W. J., Sampaio, C. H., & Costa, G. (2017) · Journal of Marketing for HIGHER EDUCATION · doi:10.1080/08841241.2017.1311980 · Cited by 175
“However, Orwin (1983) proposed a variant of Rosenthal’s formula that allows a researcher to stipulate how many missing studies would be required to raise the overall effect to a specified level other than zero (Borenstein et al., 2009).”
MethodEmpathic accuracy and relationship satisfaction: A meta-analytic review.
Sened, H., Michal Lavidor, B., Lazarus, G., Bar‐Kalifa, E., Rafaeli, E., & Ickes, W. J. (2017) · Journal of Family Psychology · doi:10.1037/fam0000320 · Cited by 175
“Using Orwin’s (1983) fail-safe N, we determined that eight additional studies with an effect size of r 0 would be required to bring the pooled effect size beneath r .”
MethodSu, D., Zhou, J., Kelley, M. S., Michaud, T. L., Siahpush, M., Kim, J., Wilson, F., Stimpson, J. P., & Pagán, J. A. (2016) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2016.04.019 · Cited by 164
Booth, J. N., Boyle, J. M. E., & Kelly, S. W. (2010) · British Journal of Developmental Psychology · doi:10.1348/026151009x485432 · Cited by 193
Gastrointestinal symptoms and obesity: a meta‐analysis
Eslick, G. D. (2011) · Obesity Reviews · doi:10.1111/j.1467-789x.2011.00969.x · Cited by 174
“If publication bias was detected, the impact of such bias would be assessed using the ‘fail-safe number’ method (34).”
MethodLarsen, B., Seitz, A., & Larsen, R. (2000) · Anesthesia & Analgesia · doi:10.1097/00000539-200001000-00035 · Cited by 172
Relapse rates after psychotherapy for depression – stable long-term effects? A meta-analysis
Steinert, C., Hofmann, M., Kruse, J., & Leichsenring, F. (2014) · Journal of Affective Disorders · doi:10.1016/j.jad.2014.06.043 · Cited by 172
“Additionally, we conducted Orwin’s failsafe N analysis (Orwin, 1983) which provides an estimate for the number of additional hypothetical studies that would overturn the obtained mean effect size to a small, typically meaningless effect size of OR1⁄40.”
MethodMen’s facial width-to-height ratio predicts aggression: a meta-analysis.
Haselhuhn, M. P., Ormiston, M. E., & Wong, E. M. (2015) · PLoS ONE · doi:10.1371/journal.pone.0122637 · Cited by 166
“To detect the risk of potential publication bias, we followed Orwin’s [29] suggested procedure to determine the number of additional studies showing null effects that would need to be included in order for the mean effect size to reduce substantially (in this case, in half).”
MethodPlasma Biomarkers for Acute Respiratory Distress Syndrome
Terpstra, M. L., Aman, J., Amerongen, G. P. V. N., & Groeneveld, A. B. J. (2013) · Critical Care Medicine · doi:10.1097/01.ccm.0000435669.60811.24 · Cited by 169
Why do people buy virtual goods: A meta-analysis
Hamari, J., & Keronen, L. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.01.042 · Cited by 176
“There are two main approaches for such calculations (Long 2001): one method based on the sum of the Z scores (Rosenthal 1979) while the other uses effect sizes (Orwin 1983).”
MethodSelf‐objectification and disordered eating: A meta‐analysis
Schaefer, L. M., & Thompson, J. K. (2018) · International Journal of Eating Disorders · doi:10.1002/eat.22854 · Cited by 168
To Flip or Not to Flip? A Meta-Analysis of the Efficacy of Flipped Learning in Higher Education
Bredow, C. A., Roehling, P. V., Knorp, A. J., & Sweet, A. M. (2021) · Review of Educational Research · doi:10.3102/00346543211019122 · Cited by 266
Olatunji, B. O., Kauffman, B. Y., Meltzer, S., Davis, M. L., Smits, J. A. J., & Powers, M. B. (2014) · Behaviour Research and Therapy · doi:10.1016/j.brat.2014.05.002 · Cited by 167
Huo, Y. R., Richards, A., Liauw, W., & Morris, D. L. (2015) · European Journal of Surgical Oncology · doi:10.1016/j.ejso.2015.08.172 · Cited by 167
Shachar, M., & Neumann, Y. (2010) · Cited by 167
Armour, M., Smith, C. A., Steel, K. A., & MacMillan, F. (2019) · BMC Complementary and Alternative Medicine · doi:10.1186/s12906-019-2433-8 · Cited by 167
“Additionally, a “fail-safe N” was used to account for the file draw problem [45] estimating the number of non-significant unpublished trials which would be needed to cause the observed p value to exceed 0.”
MethodHow environmental innovation influences firm performance: A meta‐analytic review
Hizarci‐Payne, A. K., İpek, İ., & Gümüş, G. K. (2020) · Business Strategy and the Environment · doi:10.1002/bse.2678 · Cited by 191
A historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
The Association Between Parent PTSD/Depression Symptoms and Child PTSD Symptoms: A Meta-Analysis
Morris, A. T., Gabert‐Quillen, C. A., & Delahanty, D. L. (2012) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jss091 · Cited by 176
“…fail-safe N for each mean effect size to determine the number of unpublished studies with null findings that would be necessary to reduce the mean effect size to a ‘‘trivial’’ criterion effect size of r¼ 0.10 (Borenstein, Hedges, Higgins, & Rothstein, 2005; Naragon-Gainey, 2010; Orwin, 1983).”
BackgroundInfluentialPreoperative Anemia and Outcomes in Cardiovascular Surgery: Systematic Review and Meta-Analysis
Padmanabhan, H., Siau, K., Curtis, J., Ng, A., Menon, S., Luckraz, H., & Brookes, M. (2019) · The Annals of Thoracic Surgery · doi:10.1016/j.athoracsur.2019.04.108 · Cited by 180
Two Meta-analyses Exploring the Relationship between Teacher Clarity and Student Learning
Titsworth, S., Mazer, J. P., Goodboy, A. K., Bolkan, S., & Myers, S. A. (2015) · Communication Education · doi:10.1080/03634523.2015.1041998 · Cited by 194
“, a bias in publishing nonsignificant findings which may overestimate population effect sizes; see Levine, Asada, & Carpenter, 2009), we examined (a) the funnel plots, (b) Rosenthal’s classic fail-safe N (Rosenthal, 1979), (c) Orwin’s fail-safe N (Orwin, 1983), and (d) Egger’s Test of the Intercept (Egger, Smith, Schn…”
MethodTeaching the control-of-variables strategy: A meta-analysis
Schwichow, M., Croker, S. J., Zimmerman, C., Höffler, T. N., & Härtig, H. (2015) · Developmental Review · doi:10.1016/j.dr.2015.12.001 · Cited by 164
“According to Orwin (1983), fail-safe N is computed using Nfs = N0(d0dc)/dc, where N0 is the number of studies included, d0 the estimated mean effect and dc the criterion effect size.”
MethodA meta-analysis of antidepressant outcome under "blinder" conditions.
Greenberg, R. P., Bornstein, R. F., Greenberg, M. D., & Fisher, S. (1992) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.60.5.664 · Cited by 165
The neuropsychological profile of vascular cognitive impairment not demented: A meta‐analysis
Vasquez, B. P., & Zakzanis, K. K. (2014) · Journal of Neuropsychology · doi:10.1111/jnp.12039 · Cited by 162
Subjective social status and health: A meta-analysis of community and society ladders.
Zell, E., Strickhouser, J. E., & Križan, Z. (2018) · Health Psychology · doi:10.1037/hea0000667 · Cited by 163
Keep it Coherent: A Meta-Analysis of the Seductive Details Effect
Sundararajan, N., & Adesope, O. (2020) · Educational Psychology Review · doi:10.1007/s10648-020-09522-4 · Cited by 194
“Classic fail-safe N test suggests we would need 2175 null-effect studies and Orwin’s fail-safe N test (Orwin 1983) suggests that 53 additional studies with mean effect 0.34 will be needed to achieve a critical summative effect size 0.”
BackgroundInfluentialTransdiagnostic approaches to the treatment of anxiety disorders: A quantitative review.
Norton, P. J., & Philipp, L. M. (2008) · Psychotherapy: Theory, Research, Practice, Training · doi:10.1037/0033-3204.45.2.214 · Cited by 159
“Although definitions of a trivial value vary (0.20: Orwin, 1983; 0.05: Van Etten & Taylor, 1998), we selected a “trivial” effect size to match the average effect size of the no-treatment conditions from a previous meta-analysis (Norton & Price, 2007; pre to post d 0.25).”
MethodInfluentialLouise, S., Fitzpatrick, M., Strauss, C., Rossell, S. L., & Thomas, N. (2018) · Schizophrenia Research · doi:10.1016/j.schres.2017.05.023 · Cited by 160
Hans, E., & Hiller, W. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2013.07.003 · Cited by 158
Verbal redundancy in multimedia learning environments: A meta-analysis.
Adesope, O., & Nesbit, J. C. (2011) · Journal of Educational Psychology · doi:10.1037/a0026147 · Cited by 180
Mohamadi, A., Chan, J. J., Claessen, F. M. A. P., Ring, D., & Chen, N. C. (2016) · Clinical Orthopaedics and Related Research · doi:10.1007/s11999-016-5002-1 · Cited by 162
“However, fail-safe N [48] showed 82 missing studies are needed to change the results and Orwin’s fail-safe N [42] showed at least nine missing studies are needed to bring the Hedges’s g below the trivial value.”
Kline, A. C., Cooper, A. A., Rytwinksi, N. K., & Feeny, N. C. (2017) · Clinical Psychology Review · doi:10.1016/j.cpr.2017.10.009 · Cited by 183
“Fail-safe N calculation Potential publication bias was assessed by calculating the fail-safe N (Rosenthal, 1979) adapted by Orwin (1983).”
MethodNagaraja, V., & Eslick, G. D. (2014) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.12655 · Cited by 161
“Publication bias is generally regarded as a concern if the fail-safe number is less than 5n + 10, with n being the number of studies included in the meta-analysis.(13) All analyses were performed with Comprehensive Meta-analysis (version 2.”
MethodYoon, S., Kim, S., Shim, S., Kang, S.-B., & Lee, S. (2016) · European Journal of Cancer · doi:10.1016/j.ejca.2015.12.003 · Cited by 164
Bulimia nervosa: A meta-analysis of psychosocial and pharmacological treatments
Whittal, M. L., Agras, W. S., & Gould, R. A. (1999) · Behavior Therapy · doi:10.1016/s0005-7894(99)80049-5 · Cited by 159
A meta-analysis of family accommodation and OCD symptom severity.
Wu, M. S., McGuire, J., Martino, C., Phares, V., Selles, R., & Storch, E. (2016) · Clinical Psychology Review · doi:10.1016/j.cpr.2016.03.003 · Cited by 138
“Using Orwin’s Fail-safe N (Orwin, 1983), 129 unretrieved studies with a mean correlation of 0 would be needed to bring the summary effect below .”
MethodKumarasamy, C., Sabarimurugan, S., Royam, M. M., Lakhotiya, K., Samiappan, S., Baxi, S., Nachimuthu, R., Muthukaliannan, G. K., & Jayaraj, R. (2019) · Medicine · doi:10.1097/md.0000000000014834 · Cited by 159
“Orwin Fail-SafeN test will allow us to adjust the funnel plot for small missing studies.([44]) This method allows for imputation of small studies in the plot, to more accurately assess the bias present.”
MethodA systematic review and meta‐regression analysis of prophylactic gabapentin for postoperative pain
Doleman, B., Heinink, T. P., Read, D. J., Faleiro, R. J., Lund, J. N., & Williams, J. P. (2015) · Anaesthesia · doi:10.1111/anae.13179 · Cited by 161
“We considered using Duval and Tweedie’s ‘trim and fill’ analysis and calculating Orwin’s ‘failsafe N’, but decided not to, given substantial between-trial heterogeneity [16, 28, 29].”
MethodA meta-analysis of signaling principle in multimedia learning environments
Alpizar, D., Adesope, O. O., & Wong, R. M. (2020) · Educational Technology Research and Development · doi:10.1007/s11423-020-09748-7 · Cited by 157
“Dissertations were included to minimize the potential for publication bias (Orwin 1983; Rosenthal 1979; Slavin 1995).”
BackgroundRisk markers for physical intimate partner violence victimization: A meta-analysis
Spencer, C. M., Stith, S., & Cafferky, B. M. (2019) · Aggression and Violent Behavior · doi:10.1016/j.avb.2018.10.009 · Cited by 135
The antecedents and consequences of financial literacy: a meta-analysis
Santini, F. D. O., Ladeira, W. J., Mette, F. M. B., & Ponchio, M. C. (2019) · International Journal of Bank Marketing · doi:10.1108/ijbm-10-2018-0281 · Cited by 179
Can't get it off my brain: Meta-analysis of neuroimaging studies on perseverative cognition
Makovac, E., Fagioli, S., Rae, C. L., Critchley, H., & Ottaviani, C. (2019) · Psychiatry Research Neuroimaging · doi:10.1016/j.pscychresns.2019.111020 · Cited by 165
Publication Bias in Meta-Analysis: Confidence Intervals for Rosenthal’s Fail-Safe Number
Fragkos, K. C., Tsagris, M., & Frangos, C. C. (2014) · International Scholarly Research Notices · doi:10.1155/2014/825383 · Cited by 159
“Orwin’s [38] approach is very similar to Rosenthal’s [11] approach without considering the normal variates but taking Cohen’s d [22] to compute a fail-safe number.”
MethodModerator Variables of Executive Functioning in Schizophrenia: Meta-Analytic Findings
Johnson-Selfridge, M. T., & Zalewski, C. E. (2001) · Schizophrenia Bulletin · doi:10.1093/oxfordjournals.schbul.a006876 · Cited by 156
Mishra, A. A., Shukla, A., Rana, N. P., Currie, W. L., & Dwivedi, Y. K. (2022) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2022.102571 · Cited by 196
A meta-analytic study of self-help interventions for anxiety problems.
Hirai, M., & Clum, G. (2006) · The Behavior Therapist · doi:10.1016/j.beth.2005.05.002 · Cited by 181
Effect of Obstructive Sleep Apnea Treatment on Atrial Fibrillation Recurrence
Shukla, A., Aizer, A., Holmes, D., Fowler, S., Park, D. S., Bernstein, S., Bernstein, N., & Chinitz, L. (2015) · JACC: Clinical Electrophysiology · doi:10.1016/j.jacep.2015.02.014 · Cited by 155
“Orwin R.”
MethodInfluentialDalton, D. R., Aguinis, H., Dalton, C. M., Bosco, F. A., & Pierce, C. A. (2012) · Personnel Psychology · doi:10.1111/j.1744-6570.2012.01243.x · Cited by 157
Restoration of ecosystem services in tropical forests: A global meta-analysis
Shimamoto, C. Y., Padial, A. A., da Rosa, C. M., & Marques, M. C. M. (2018) · PLoS ONE · doi:10.1371/journal.pone.0208523 · Cited by 154
Coyne, S. M., Padilla‐Walker, L. M., Holmgren, H. G., Davis, E. J., Collier, K. M., Memmott‐Elison, M. K., & Hawkins, A. J. (2017) · Developmental Psychology · doi:10.1037/dev0000412 · Cited by 152
Brunelin, J., Poulet, E., Bediou, B., Kallel, L., Dalery, J., D'amato, T., & Saoud, M. (2006) · Schizophrenia Research · doi:10.1016/j.schres.2005.10.009 · Cited by 151
When and how to update systematic reviews
Moher, D., Tsertsvadze, A., Tricco, A. C., Eccles, M., Grimshaw, J., Sampson, M., & Barrowman, N. (2008) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000023.pub3 · Cited by 157
The Cognitive Effects of Obstructive Sleep Apnea: An Updated Meta-analysis
Stranks, E. K., & Crowe, S. F. (2016) · Archives of Clinical Neuropsychology · doi:10.1093/arclin/acv087 · Cited by 151
“The formula developed by Orwin (1983) was used to calculate these values.”
MethodUPPS-P model impulsivity and marijuana use behaviors in adolescents: A meta-analysis
VanderVeen, J. D., Hershberger, A. R., & Cyders, M. A. (2016) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2016.09.016 · Cited by 163
Metadata
- Authors
- Orwin, R. G.
- Affiliations
- United States Department of Health and Human Services (US)
- Corresponding author
- Orwin, Robert G.
- Venue
- Journal of Educational Statistics
- Publisher
- JSTOR
- Published
- 1983
- Volume
- 8
- Issue
- 2
- Pages
- 157
- ISSN
- 0362-9791 (ISSN-L)
- OpenAlex
- W2112432954
- Cited by
- 2,064
- References
- 11
- Citations by year
- 2026: 51 · 2025: 75 · 2024: 91 · 2023: 121 · 2022: 114 · 2021: 129 · 2020: 129 · 2019: 120 · 2018: 116 · 2017: 109 · 2016: 95 · 2015: 113 · 2014: 108 · 2013: 76 · 2012: 88
- FWCI
- 7.68
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Educational Statistics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- effect size, file drawer problem, sampling bias, negative results, meta-analysis, publication bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
