The DOI matches its Crossref record; 10 of the 11 references it cites were found in a registry or an index.
A Fail-Safe N for Effect Size in Meta-Analysis
Abstract
Rosenthan's (1979) concept of fail-safe N has thus far been applied to probability levels exclusively. This note introduces a fail-safe TV for effect size. Rosenthal's (1979) fail-safe N was an ingenious response to the so-called file drawer problem in research integration, in that it provided a direct assessment of the threat posed by sampling bias in the literature search. The algorithm was derived from the Stouffer method (Mosteller & Bush, 1954), in which an overall Z-score is computed by summing individual Z-scores and dividing by the square root of the number of scores. A significant overall Z-score, Rosenthal reasoned, could be made nonsignificant by adding some knowable number of hypothetical studies that averaged null results. The exact number of such studies needed to bring a significant overall p level up to some critical level (e.g., .05) has since become known as the fail-safe N (Nfs). The formula is
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“we also calculated the classic fail-safe N test (Orwin, 1983), which revealed that it would take 59 studies reporting null results (g = 0) to be included in our analyses in order for our findings to lose statistical significance.”
From Improving fluid intelligence with training on working memory: a meta-analysis (2015) · cites this paper as Orwin, 1983
In the citing paper
“The file drawer problem (publication bias) was examined by creating a funnel plot and calculating the fail safe N (45).”
From A Meta-Analysis of the Corpus Callosum in Autism (2009) · cites this paper as [45]
Citations
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890
Smith, M. L., Glass, G. V., & Miller, T. I. (1980) · Medical Entomology and Zoology · Cited by 2,123
Johnson, D. W., Maruyama, G., Johnson, R., Nelson, D., & Skon, L. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.89.1.47 · Cited by 1,588
Interpersonal expectancy effects: the first 345 studies
Rosenthal, R., & Rubin, D. B. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00075506 · Cited by 1,158
Eagly, A. H., & Carli, L. L. (1981) · Psychological Bulletin · doi:10.1037/0033-2909.90.1.1 · Cited by 541
Cooper, H. M. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.1.131 · Cited by 418
Arkin, R. M., Cooper, H., & Kolditz, T. A. (1980) · Journal of Personality · doi:10.1111/j.1467-6494.1980.tb02378.x · Cited by 88
Flagged first, then by citations
1 of the 11 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Items 501–600 of 1,020
The weakest link: sensitivity to climate extremes across life stages of marine invertebrates
Pandori, L. L. M., & Sorte, C. J. B. (2018) · Oikos · doi:10.1111/oik.05886 · Cited by 162
The relation between empathy and offending: A meta-analysis
van Langen, M. A. M., Wissink, I. B., van Vugt, E. S., Van der Stouwe, T., & Stams, G. J. J. M. (2014) · Aggression and Violent Behavior · doi:10.1016/j.avb.2014.02.003 · Cited by 172
“It calculates the number of additional studies needed to reduce an observed mean effect size to a desired minimal effect size (Orwin, 1983).”
BackgroundCircular economy practices and sustainable performance: A meta-analysis
Yin, S., Jia, F., Chen, L., & Wang, Q. (2023) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2022.106838 · Cited by 150
Social Skills Interventions for Individuals with Learning Disabilities
Kavale, K. A., & Mostert, M. P. (2004) · Learning Disability Quarterly · doi:10.2307/1593630 · Cited by 149
Predictive Physiological Anticipation Preceding Seemingly Unpredictable Stimuli: A Meta-Analysis
Mossbridge, J., Tressoldi, P. E., & Utts, J. M. (2012) · Frontiers in Psychology · doi:10.3389/fpsyg.2012.00390 · Cited by 171
“A more conservative method (Orwin, 1983) gave a fail-safe”
Quality of Child Care Using the Environment Rating Scales: A Meta-Analysis of International Studies
Vermeer, H. J., van IJzendoorn, M. H., Cárcamo, R. A., & Harrison, L. J. (2016) · International Journal of Early Childhood · doi:10.1007/s13158-015-0154-9 · Cited by 150
“…2002a, b) to calculate the effect of potential publication bias on the outcomes of the meta-analysis and (2) computed the fail–safe N according to the method proposed by Orwin (1983), referring to the number of studies necessary to bring the effect size down to trivial levels (e.g., r\ .10).”
MethodYu, A., Matta, F. K., & Cornfield, B. (2017) · Academy of Management Journal · doi:10.5465/amj.2016.1212 · Cited by 163
McClelland, E. K., & Naish, K. A. (2006) · Conservation Genetics · doi:10.1007/s10592-006-9178-x · Cited by 146
Psychological Adjustment of Children and Adolescents WithChronic Arthritis: A Meta-analytic Review
LeBovidge, J. (2002) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/28.1.29 · Cited by 147
“Calculation of the fail-safe N was conducted using guidelines provided by Orwin (1983) for outcome measures that showed significant differences between children with arthritis and controls.”
MethodGonzález-Valero, G., Ortega, F. Z., Ubago-Jiménez, J. L., & Puertas-Molero, P. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16224394 · Cited by 147
“In order to examine the likely influence of publication bias on the meta-analysis the security number (Ns) was calculated [34].”
MethodMeta-Analysis of Emotional Designs in Multimedia Learning: A Replication and Extension Study
Wong, R. M., & Adesope, O. (2020) · Educational Psychology Review · doi:10.1007/s10648-020-09545-x · Cited by 164
“Further examination of publication bias for retention using Orwin’s fail-safe N (Orwin 1983) indicated that an additional 166 studies were needed to nullify the findings, suggesting that publication bias was negligent for retention.”
MethodFletcher, R., Freeman, E., & Matthey, S. (2011) · Fathering: A Journal of Theory, Research, and Practice about Men as Fathers · doi:10.3149/fth.0903.291 · Cited by 145
Hans, E., & Hiller, W. (2013) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0031080 · Cited by 211
Applying meta-analytical procedures to software engineering experiments
Miller, J. (2000) · Journal of Systems and Software · doi:10.1016/s0164-1212(00)00024-8 · Cited by 146
“Other disciplines have developed quasi-statistical methods to attempt to estimate the possible impact of publication bias (Orwin, 1983).”
BackgroundThe Proximal Effects of Acute Alcohol Consumption on Male-to-Female Aggression
Crane, C. A., Godleski, S. A., Przybyla, S., Schlauch, R. C., & Testa, M. (2015) · Trauma Violence & Abuse · doi:10.1177/1524838015584374 · Cited by 147
“Although unpublished studies were included in the current analysis, publication biases against undetectable null findings prompted us to calculate a failsafe N (Orwin, 1983; Rosenthal, 1991).”
McGuire, J. F., Ung, D., Selles, R. R., Rahman, O., Lewin, A. B., Murphy, T. K., & Storch, E. A. (2014) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2014.07.015 · Cited by 142
“Finally, sensitivity analyses included Rosenthal's Fail-safe N (Rosenthal, 1991), Orwin's Fail-safe N (Orwin, 1983), and a comparison of ES across control conditions.”
MethodRehn, S., Eslick, G. D., & Brakoulias, V. (2018) · Psychiatric Quarterly · doi:10.1007/s11126-018-9566-7 · Cited by 142
“We quantified publication bias using the Egger’s regression model [65], with the effect of bias assessed using the fail-safe number method [66].”
Lee, M., Ata, R. N., & Brannick, M. Τ. (2014) · Body Image · doi:10.1016/j.bodyim.2014.03.003 · Cited by 144
“Using Orwin’s (1983) method, fail-safe N for a small effect size f d = 0.”
The effect of prison visitation on reentry success: A meta-analysis
Mitchell, M. M., Spooner, K., Jia, D., & Yan, Z. (2016) · Journal of Criminal Justice · doi:10.1016/j.jcrimjus.2016.07.006 · Cited by 150
Neurocognitive consequences of chronic cannabis use: a systematic review and meta-analysis
Figueiredo, P. R., Tolomeo, S., Steele, J. D., & Baldacchino, A. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.10.014 · Cited by 149
“To assess publication bias a visual inspection of funnel plots was performed in addition to determination of the Fail Safe N (Orwin, 1983).”
MethodPUBLICATION BIAS: A CASE STUDY OF FOUR TEST VENDORS
McDaniel, M. A., Rothstein, H. R., & Whetzel, D. L. (2006) · Personnel Psychology · doi:10.1111/j.1744-6570.2006.00059.x · Cited by 147
Kraemer, H. C., Gardner, C., Brooks, J. O., & Yesavage, J. A. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.1.23 · Cited by 141
The effects of using mobile devices on language learning: a meta-analysis
Chen, Z., Chen, W., Jia, J., & An, H. (2020) · Educational Technology Research and Development · doi:10.1007/s11423-020-09801-5 · Cited by 143
“The third approach uses Orwin’s Fail-safe N proposed by Orwin (1983).”
MethodBarkmann, C., & Schulte‐Markwort, M. (2010) · Journal of Epidemiology & Community Health · doi:10.1136/jech.2009.102467 · Cited by 153
“To estimate the extent of a potential publication bias, the failsafe number was calculated in accordance with Orwin.(62) For the k1⁄433 included studies with a mean prevalence of 17.”
MethodInfluentialZakzanis, K. K., Graham, S. J., & Campbell, Z. (2003) · Neuropsychology Review · doi:10.1023/a:1022318921994 · Cited by 145
“Accordingly, we used Orwin’s formula (Orwin, 1983) to provide a Fail SafeN for each meand found in Table 2 at a criterion of 0.01.”
InfluentialTio, M., Cox, M. R., & Eslick, G. D. (2012) · Alimentary Pharmacology & Therapeutics · doi:10.1111/j.1365-2036.2011.04972.x · Cited by 141
“Publication bias is considered to be an issue if the fail-safe number is <5n + 10, where n is the number of studies included in the meta-analysis.(14) The trim-and-fill method simulates unpublished studies in the meta-analysis to calculate a new pooled odds ratio, which is then compared with the original pooled odds r…”
MethodSodium bicarbonate-based hydration prevents contrast-induced nephropathy: a meta-analysis
Meier, P., Ko, D. T., Tamura, A., Tamhane, U. U., & Gurm, H. S. (2009) · BMC Medicine · doi:10.1186/1741-7015-7-23 · Cited by 141
“We also calculated fail-safe N (that is, the number of studies required to nullify the significant differences in CIN between the two groups) using Rosenberg's and Orwin's method [32,33].”
Effects of the Attributes of Educational Interventions on Students’ Academic Performance
de Boer, H., Donker, A. S., & van der Werf, G. P. C. (2014) · Review of Educational Research · doi:10.3102/0034654314540006 · Cited by 161
“This result was supported by the classic fail-safe Ns of Rosenthal (1979), which was 5,196, and Orwin (1983), which was 777, indicating that 5,196 and 777 interventions, respectively, would have been needed to be added to the meta-analysis, with an effect size of 0, before the effect found would…”
ResultEffect of Sleeve Gastrectomy on Ghrelin, GLP-1, PYY, and GIP Gut Hormones
McCarty, T. R., Jirapinyo, P., & Thompson, C. C. (2019) · Annals of Surgery · doi:10.1097/sla.0000000000003614 · Cited by 153
Osborn, A. J., Mathias, J. L., & Fairweather‐Schmidt, A. K. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2014.07.007 · Cited by 140
Modeling of food intake: a meta-analytic review
Vartanian, L. R., Spanos, S., Herman, C. P., & Polivy, J. (2015) · Social Influence · doi:10.1080/15534510.2015.1008037 · Cited by 157
“The potential for publication bias was assessed by calculating Orwin’s fail-safe N (the number of studies with a correlation of r ¼ .00 that would need to be added to the meta-analysis to bring the overall effect size to a negligible level, defined as r ¼ .05 in this case; Orwin, 1983).”
MethodThe test of time: A meta-analytic review of the relation between school climate and problem behavior
Reaves, S., McMahon, S. D., Duffy, S. N., & Ruiz, L. D. (2018) · Aggression and Violent Behavior · doi:10.1016/j.avb.2018.01.006 · Cited by 149
A Meta-analytic Review of the Efficacy of Drug Treatment in Generalized Anxiety Disorder
Mitte, K., Noack, P., Steil, R., & Hautzinger, M. (2005) · Journal of Clinical Psychopharmacology · doi:10.1097/01.jcp.0000155821.74832.f9 · Cited by 137
The effects of cannabis use on neurocognition in schizophrenia: A meta-analysis
Rabin, R. A., Zakzanis, K. K., & George, T. P. (2011) · Schizophrenia Research · doi:10.1016/j.schres.2011.02.017 · Cited by 140
“However, to ascertain how robust our findings were, Orwin's fail safe N formula was also utilized so to provide an index of how many studies would be theoretically needed to overturn the obtained effect size and yield an insignificant effect (i.e., d=0.1) (Orwin, 1983).”
InfluentialYoga for Chronic Low Back Pain: A Meta‐Analysis of Randomized Controlled Trials
Holtzman, S., & Beggs, R. T. (2013) · Pain Research and Management · doi:10.1155/2013/105919 · Cited by 151
“Orwin’s (39) fail-safe n was calculated to determine the number of studies with an effect size of zero that would be required to reduce the effect size of yoga to 0.”
BackgroundWallace, E. J., Mathias, J. L., & Ward, L. (2018) · Brain Imaging and Behavior · doi:10.1007/s11682-018-9823-2 · Cited by 144
“Finally, Nfs statistics were calculated using Orwin’s method to assess the potential impact of publication bias (Orwin 1983; Rothstein et al. 2006).”
MethodEspírito‐Santo, H., & Daniel, F. (2015) · Revista Portuguesa de Investigação Comportamental e Social · doi:10.7342/ismt.rpics.2015.1.1.14 · Cited by 143
The association between risk disclosure and firm characteristics: a meta-analysis
Khlif, H., & Hussainey, K. (2014) · Journal of Risk Research · doi:10.1080/13669877.2014.961514 · Cited by 142
“’ We apply Orwin’s (1983) method to determine the extent of the file drawer problem and whether the conclusions drawn from our meta-analysis are likely to be influenced by such publication bias.”
MethodEmpirical generalizations in eco-innovation: A meta-analytic approach
Bitencourt, C. C., Santini, F. D. O., Zanandrea, G., Froehlich, C., & Ladeira, W. J. (2019) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2019.118721 · Cited by 141
Dold, M., Aigner, M., Lanzenberger, R., & Kasper, S. (2015) · International Journal of Neuropsychopharmacology · doi:10.1093/ijnp/pyv047 · Cited by 134
“Moreover, we estimated the number of negative unpublished trials (fail-safe N value) that would be necessary to dissolve statistically significant differences (Orwin, 1983).”
BackgroundVitamin D Treatment and Mortality in Chronic Kidney Disease: A Systematic Review and Meta-Analysis
Duranton, F., Rodríguez-Ortiz, M. E., Duny, Y., Rodríguez, M., Daurès, J., & Argilés, À. (2013) · American Journal of Nephrology · doi:10.1159/000346846 · Cited by 132
Simon, A. E., Borgwardt, S., Riecher-Rössler, A., Velthorst, E., de Haan, L., & Fusar-Poli, P. (2013) · Psychiatry Research · doi:10.1016/j.psychres.2013.03.004 · Cited by 133
“In addition, we used the fail-safe procedure (Orwin, 1983) to generate the number of unpublished studies that would be needed to move estimates to a non-significant difference.”
Meta-analysis of the efficacy of virtual reality exposure therapy for social anxiety
Chesham, R. K., Malouff, J. M., & Schutte, N. S. (2018) · Behaviour Change · doi:10.1017/bec.2018.15 · Cited by 135
Cao, A. M. Y., Eslick, G. D., & Cox, M. R. (2015) · Surgical Endoscopy · doi:10.1007/s00464-015-4325-4 · Cited by 133
“Publication bias is generally regarded as a concern if the fail-safe number is less than 5n ? 10, with n being the number of studies included in the meta-analysis [25].”
Ma, M., Dorstyn, D., Ward, L., & Prentice, S. (2018) · Aging & Mental Health · doi:10.1080/13607863.2017.1370689 · Cited by 133
Comparing Long-Term Placement Outcomes of Residential and Family Foster Care: A Meta-Analysis
Li, D., Chng, G. S., & Chu, C. M. (2017) · Trauma Violence & Abuse · doi:10.1177/1524838017726427 · Cited by 159
“We also examined the Egger’s regression intercept, which was the statistical test for asymmetry of the funnel plot, to assess the threat of possible bias in the sampling for publications (Orwin, 1983).”
MethodThyroid carcinoma in Graves’ disease: A meta-analysis
Staniforth, J., Erdirimanne, S., & Eslick, G. D. (2015) · International Journal of Surgery · doi:10.1016/j.ijsu.2015.11.027 · Cited by 132
“This is likely due to a higher detection rate in the surgically-resected specimens of this study, and indeed our incidence rate is approximately in concordance with that observed in developed countries [51].”
BackgroundResultInfluentialEfficacy of group treatment for posttraumatic stress disorder symptoms: A meta-analysis.
Sloan, D. M., Feinstein, B. A., Gallagher, M. W., Beck, J. G., & Keane, T. M. (2011) · Psychological Trauma Theory Research Practice and Policy · doi:10.1037/a0026291 · Cited by 143
Manning, M., Homel, R. J., & Smith, C. (2009) · Children and Youth Services Review · doi:10.1016/j.childyouth.2009.11.003 · Cited by 141
“Orwin (1983) adapted Rosenthal‟s method “…to the standardized mean effect size” (Lipsey & Wilson, 2000, p.166).”
MethodWilson, M. L., Ritzhaupt, A. D., & Cheng, L. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.103941 · Cited by 166
Digital Platforms and the Improvement of Learning Outcomes: Evidence Extracted from Meta-Analysis
Alshammary, F. M., & Alhalafawy, W. S. (2023) · Sustainability · doi:10.3390/su15021305 · Cited by 244
Bronchodilator Delivery in Acute Airflow Obstruction
TURNER, M. O. (1997) · Archives of Internal Medicine · doi:10.1001/archinte.1997.00440360162018 · Cited by 131
Papalia, N., Spivak, B., Daffern, M., & Ogloff, J. R. P. (2019) · Clinical Psychology: Science and Practice · doi:10.1111/cpsp.12282 · Cited by 130
Dissociation and emotion regulation strategies: A meta-analytic review
Cavicchioli, M., Scalabrini, A., Northoff, G., Mucci, C., Ogliari, A., & Maffei, C. (2021) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2021.09.011 · Cited by 146
The impact of pedagogical agent gesturing in multimedia learning environments: A meta-analysis
Davis, R. O. (2018) · Educational Research Review · doi:10.1016/j.edurev.2018.05.002 · Cited by 146
Leung, K. C. (2014) · Journal of Educational Psychology · doi:10.1037/a0037698 · Cited by 149
Baldacchino, A., Arbuckle, K., Petrie, D., & McCowan, C. (2014) · BMC Psychiatry · doi:10.1186/1471-244x-14-104 · Cited by 135
Koning, J., Tenback, D. E., van Os, J., Aleman, A., Kahn, R. S., & van Harten, P. N. (2008) · Schizophrenia Bulletin · doi:10.1093/schbul/sbn146 · Cited by 129
“To examine the statistical heterogeneity of the individual studies, we tested a homogeneity statistic, Q. Additionally, to examine the possibility of publication bias, a method to indicate the number of unpublished studies with null effects that must reside in file drawers to reduce the observable effect size to a neg…”
MethodInfluentialWang, S., Huang, X., Zhang, Y., Yin, C., & Richel, A. (2021) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2021.105402 · Cited by 135
Hypnosis to manage distress related to medical procedures: a meta-analysis
Schnur, J. B., Kafer, I., Marcus, C., & Montgomery, G. H. (2008) · Contemporary Hypnosis · doi:10.1002/ch.364 · Cited by 129
Schwarzer, R. (1988) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03203853 · Cited by 129
Martin, F., Wu, T., Wan, L., & Xie, K. (2022) · Online Learning · doi:10.24059/olj.v26i1.2604 · Cited by 132
“Finally, Orwin’s Fail-safe N (Orwin, 1983), which assists in computing the number of missing studies to bring the summary effect to a level below the specified value other than zero, was examined.”
BackgroundThe influence of stereotype threat on immigrants: review and meta-analysis
Appel, M., Weber, S., & Kronberger, N. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00900 · Cited by 144
“We distinguished studies that used measures of knowledge and abilities such as GRE-type tasks (verbal or maths, k = 13) from studies that used non-verbal measures of general cognitive functioning, such as (working) memory tests, or tests in the tradition of Raven’s progressive matrices (k = 6).”
BackgroundMethodHäuser, W., Bartram-Wunn, E., Bartram, C., Reinecke, H., & Tölle, T. R. (2011) · Pain · doi:10.1016/j.pain.2011.01.050 · Cited by 125
“Risk of bias across studies Publication bias was controlled for by computing ‘‘safe-n rates’’ [30] and the rank correlation test for funnel plot asymmetry.”
Cameron, S. K., Rodgers, J., & Dagnan, D. (2018) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2180 · Cited by 128
“14 Orwin’s failsafe N (Orwin, 1983) was used also to account for publication bias, which identified that 155 studies that accepted the null hypothesis would be required to reduce the correlation to a trivial value, i.”
MethodInfluentialSocial Media Marketing Activities and Brand Loyalty: A Meta-Analysis Examination
Ibrahim, B. (2021) · Journal of Promotion Management · doi:10.1080/10496491.2021.1955080 · Cited by 166
“Finally, publication bias has been addressed using Orwin’s fail-safe N (Nfs) (Orwin, 1983) methodology.”
MethodWenke, K., Zapkau, F. B., & Schwens, C. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.10.018 · Cited by 141
Random versus nonrandom assignment in controlled experiments: Do you get the same answer?
Shadish, W. R., & Ragsdale, K. (1996) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.64.6.1290 · Cited by 126
Gomez, A. F., Barthel, A. L., & Hofmann, S. G. (2018) · Expert Opinion on Pharmacotherapy · doi:10.1080/14656566.2018.1472767 · Cited by 125
“N test [33] indicated that 354 trials with an effect size of zerowould be needed to yield an overall trivial effect of treatment over”
MethodLin, N., & Roberts, K. R. (2020) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2020.102612 · Cited by 139
Jokela, M., & Hanin, Y. L. (1999) · Journal of Sports Sciences · doi:10.1080/026404199365434 · Cited by 123
The Effect of Parent Involvement in the Treatment of Anxiety Disorders in Children: A Meta-Analysis
Thulin, U., Svirsky, L., Serlachius, E., Andersson, G., & Öst, L.-G. (2014) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2014.923928 · Cited by 124
Gender-Role Differences in Spatial Ability: A Meta-Analytic Review
Reilly, D., & Neumann, D. L. (2013) · Sex Roles · doi:10.1007/s11199-013-0269-0 · Cited by 148
“The possibility of unpublished null studies (referred to as the “file drawer problem”) was also addressed by the calculation of Orwin’s Fail-Safe N, which estimates the number of null studies required to reduce mean effect sizes to a specific cutoff-point (Borenstein et al. 2009; Orwin 1983).”
BackgroundMethodProblematic Gaming and Sleep: A Systematic Review and Meta-Analysis
Kristensen, J. H., Pallesen, S., King, D. L., Hysing, M., & Erevik, E. K. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.675237 · Cited by 155
“Publication bias was assessed for each sleep parameter by visual inspection of funnel plots and statistically by the Orwin’s fail-safe N (47) and the Duval and Tweedie’s trim and fill procedure (48, 49).”
MethodIs Group Cohesiveness a Double-Edged Sword?
Langfred, C. W. (1998) · Small Group Research · doi:10.1177/1046496498291005 · Cited by 124
Fusar‐Poli, P., Howes, O. D., Bechdolf, A., & Borgwardt, S. (2012) · Journal of Psychiatry and Neuroscience · doi:10.1503/jpn.110061 · Cited by 122
“2010 MRS Offspring of BD patients SCID, KSADS/PL, YMRS, CDRS-R — None 20 (25) [15] 20 (50) [13] 20 (35) [16]”
Ethnicity and Contemporary American Culture
Vargas, J. H., & Kemmelmeier, M. (2012) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022112443733 · Cited by 123
“Still, both Oyserman et al.’s and Coon and Kemmelmeier’s (2001) studies were limited by the post hoc analysis of scale content, mainly because most scales tapped a range of different aspects of I-C that could not be captured simultaneously. Moreover, Oyserman et al. found only a few studies that utilized Triandis’s (1…”
BackgroundInfluentialThe flipped classroom in second language learning: A meta-analysis
Vitta, J. P., & Al‐Hoorie, A. H. (2020) · Language Teaching Research · doi:10.1177/1362168820981403 · Cited by 176
“We also examined the results of the classic fail-safe N test (Rosenthal, 1979), Orwin’s fail-safe N test (Orwin, 1983), and the p-curve (Simonsohn, Nelson, & Simmons, 2014) to further shed light on potential bias.”
ResultMychailyszyn, M. P., Brodman, D. M., Read, K. L., & Kendall, P. C. (2012) · Clinical Psychology Science and Practice · doi:10.1111/j.1468-2850.2012.01279.x · Cited by 141
Watkins, L., Ledbetter‐Cho, K., O’Reilly, M. F., Barnard-Brak, L. M., & Grau, P. G. (2019) · Psychological Bulletin · doi:10.1037/bul0000190 · Cited by 147
Wang, Q., Lai, Y., Xu, X., & McDowall, A. (2021) · Journal of Work-Applied Management · doi:10.1108/jwam-04-2021-0030 · Cited by 154
The Efficacy of Hypnosis as a Treatment for Anxiety: A Meta-Analysis
Valentine, K. E., Milling, L. S., Clark, L. J., & Moriarty, C. L. (2019) · International Journal of Clinical and Experimental Hypnosis · doi:10.1080/00207144.2019.1613863 · Cited by 121
“To address this concern, the fail-safe N was calculated according to the method of Orwin (1983).”
MethodFactors Influencing Sponsorship Effectiveness: A Meta-Analytic Review and Research Synthesis
Kim, Y., Lee, H., Magnusen, M., & Kim, M. (2014) · Journal of Sport Management · doi:10.1123/jsm.2014-0056 · Cited by 121
Relationship between use of antidepressants and risk of fractures: a meta-analysis
Rabenda, V., Nicolet, D., Beaudart, C., Bruyère, O., & Reginster, J.-Y. (2012) · Osteoporosis International · doi:10.1007/s00198-012-2015-9 · Cited by 125
“If publication bias was detected, the effect of such bias was assessed with the fail-safe number method [23, 24].”
MethodHow to run an experimental auction: a review of recent advances
Canavari, M., Drichoutis, A. C., Lusk, J. L., & Nayga, R. M. (2019) · European Review of Agricultural Economics · doi:10.1093/erae/jbz038 · Cited by 130
“…effects and the truth.11 This approach overcomes the limitations of previous approaches such as the ‘funnel plots’ method (Duval and Tweedie, 2000; Egger et al., 1997), the ‘fail safe’ method (Orwin, 1983; Rosenthal, 1979) and the ‘excessive-significance’ test (Ioannidis and Trikalinos, 2007)12.”
MethodIdentifying suicide risk among college students: A systematic review
Li, W., Dorstyn, D., & Jarmon, E. (2019) · Death Studies · doi:10.1080/07481187.2019.1578305 · Cited by 121
Sawamura, S., Kingery, W. S., Davies, M. F., Agashe, G. S., Clark, J. D., Kobilka, B. K., Hashimoto, T., & Maze, M. (2000) · Journal of Neuroscience · doi:10.1523/jneurosci.20-24-09242.2000 · Cited by 136
Hiller, N. J., Sin, H.-P., Ponnapalli, A. R., & Ozgen, S. (2019) · The Leadership Quarterly · doi:10.1016/j.leaqua.2018.11.003 · Cited by 144
“…significant difference in study quality and reporting between English ( M = 6.00, SD = 2.00) and non-English publications ( M = 5.88, In addition, we addressed the file-drawer concern (Rosenthal, 1979) by computing the fail-safe N (Orwin, 1983) values for the statistically significant effect sizes.”
BackgroundInfluentialCao, A. M. Y., Eslick, G. D., & Cox, M. R. (2015) · Journal of Gastrointestinal Surgery · doi:10.1007/s11605-015-2747-x · Cited by 120
“Publication bias is generally regarded as a concern if the fail-safe number is less than 5n+10, with n being the number of studies included in the meta-analysis.(30) All analyses were performed with comprehensive meta-analysis (version 2.”
MethodResultEicosapentaenoic Acid Interventions in Schizophrenia
Fusar‐Poli, P., & Berger, G. E. (2012) · Journal of Clinical Psychopharmacology · doi:10.1097/jcp.0b013e318248b7bb · Cited by 120
Amygdala Volume in Adults with Posttraumatic Stress Disorder: A Meta-Analysis
Woon, F. L., & Hedges, D. W. (2009) · The Journal of Neuropsychiatry and Clinical Neurosciences · doi:10.1176/jnp.2009.21.1.5 · Cited by 120
Nagaraja, V., Eslick, G. D., & Cox, M. R. (2014) · Europe PMC (PubMed Central) · doi:10.3978/j.issn.2078-6891.2014.016 · Cited by 120
“Publication bias is generally regarded as a concern if the fail-safe number is less than 5n+10, with n being the number of studies included in the meta-analysis (27).”
Background10 The Effects of Engagement with Entertainment
Tukachinsky, R. H., & Tokunaga, R. S. (2013) · Annals of the International Communication Association · doi:10.1080/23808985.2013.11679153 · Cited by 131
Publication Bias in Special Education Meta-Analyses
Gage, N. A., Cook, B. G., & Reichow, B. (2017) · Exceptional Children · doi:10.1177/0014402917691016 · Cited by 125
“…but are not limited to, rank correlation test (Begg & Mazumdar, 1994), Egger’s regression (Egger, Smith, Schneider, & Minder, 1997), the trim-and-fill method (Duval & Tweetie, 2000), and the fail-safe N (Orwin, 1983; Rosenthal, 1979; see Song et al., 2010, for a complete list of procedures).”
MethodEffectiveness of online and blended learning from schools: A systematic review
Topping, K. J., Douglas, W., Robertson, D. P., & Ferguson, N. (2022) · Review of Education · doi:10.1002/rev3.3353 · Cited by 198
Burlingame, G. M., Seebeck, J. D., Janis, R. A., Whitcomb, K. E., Barkowski, S., Rosendahl, J., & Strauss, B. (2016) · Psychotherapy · doi:10.1037/pst0000090 · Cited by 127
Effectiveness of psychological interventions for child maltreatment: a meta-analysis
Skowron, E. A., & Reinemann, D. H. (2005) · Psychotherapy · doi:10.1037/0033-3204.42.1.52 · Cited by 120
Fabiano, G. A., Schatz, N. K., Aloe, A. M., Chacko, A., & Chronis-Tuscano, A. M. (2015) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-015-0178-6 · Cited by 122
“Publication bias Fail-safe N (Orwin 1983)”
InfluentialHoch, J. M., Mattacola, C. G., McKeon, J. M. M., Howard, J. S., & Lattermann, C. M. (2011) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2011.09.005 · Cited by 118
Shim, S., Lee, S. J., & Kim, S. (2014) · European Journal of Cancer · doi:10.1016/j.ejca.2014.03.006 · Cited by 117
“Publication bias was evaluated using the fail-safe N test [12] and the Begg–Mazumdar rank correlation test [13].”
Method
Metadata
- Authors
- Orwin, R. G.
- Affiliations
- United States Department of Health and Human Services (US)
- Corresponding author
- Orwin, Robert G.
- Venue
- Journal of Educational Statistics
- Publisher
- JSTOR
- Published
- 1983
- Volume
- 8
- Issue
- 2
- Pages
- 157
- ISSN
- 0362-9791 (ISSN-L)
- OpenAlex
- W2112432954
- Cited by
- 2,064
- References
- 11
- Citations by year
- 2026: 51 · 2025: 75 · 2024: 91 · 2023: 121 · 2022: 114 · 2021: 129 · 2020: 129 · 2019: 120 · 2018: 116 · 2017: 109 · 2016: 95 · 2015: 113 · 2014: 108 · 2013: 76 · 2012: 88
- FWCI
- 7.68
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Educational Statistics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- effect size, file drawer problem, sampling bias, negative results, meta-analysis, publication bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
